Abstract

The increasing size and complexity of deep learning (DL) models have led to the wide adoption of distributed training methods in datacenters (DCs) and high-performance computing (HPC) systems. However, communication among distributed computing units (CUs) has emerged as a major bottleneck in the training process. In this study, we propose Flex-SiPAC, a flexible silicon photonic accelerated compute cluster designed to accelerate multi-tenant distributed DL training workloads. Flex-SiPAC takes a co-design approach that combines a silicon photonic hardware platform with a tailored collective algorithm, optimized to leverage the unique physical properties of the architecture. The hardware platform integrates a novel wavelength-reconfigurable transceiver design and a micro-resonator-based wavelength-reconfigurable switch, enabling the system to achieve flexible bandwidth steering in the wavelength domain. The collective algorithm is designed to support reconfigurable topologies, enabling efficient all-reduce communications that are commonly used in DL training. The feasibility of the Flex-SiPAC architecture is demonstrated through two testbed experiments. First, an optical testbed experiment demonstrates the flexible routing of wavelengths by shuffling an array of input wavelengths using a custom-designed spatial-wavelength selective switch. Second, a four-GPU testbed running two DL workloads shows a 23% improvement in job completion time compared to a similarly sized leaf-spine topology. We further evaluate Flex-SiPAC using large-scale simulations, which show that Flex-SiPAC is able to reduce the communication time by 26% to 29% compared to state-of-the-art compute clusters under representative collective operations.

© 2024 Optica Publishing Group

Fast and scalable all-optical network architecture for distributed deep learning

Wenzhe Li, Guojun Yuan, Zhan Wang, Guangming Tan, Peiheng Zhang, and George N. Rouskas
J. Opt. Commun. Netw. 16(3) 342-357 (2024)

SL-Hyper-FleX: a cognitive and flexible-bandwidth optical datacom network by self-supervised learning [Invited]

Che-Yu Liu, Xiaoliang Chen, Zhaohui Li, Roberto Proietti, and S. J. Ben Yoo
J. Opt. Commun. Netw. 14(2) A113-A121 (2022)

Modoru: Clos nanosecond optical switching for distributed deep training [Invited]

Cen Wang, Noboru Yoshikane, Daniel Elson, Yuta Wakayama, Daiki Soma, Shohei Beppu, and Takehiro Tsuritani
J. Opt. Commun. Netw. 16(1) A40-A52 (2024)

Accelerating model synchronization for distributed machine learning in an optical wide area network

Ling Liu, Liangjun Song, Xi Chen, Hongfang Yu, and Gang Sun
J. Opt. Commun. Netw. 14(10) 852-865 (2022)

Dynamic slicing of multidimensional resources in DCI-EON with penalty-aware deep reinforcement learning

Meng Lian, Yongli Zhao, Yajie Li, Avishek Nag, and Jie Zhang
J. Opt. Commun. Netw. 16(2) 112-126 (2024)