Efficient flow scheduling in distributed deep learning training with echelon formation
Open access
Date
2022-11Type
- Conference Paper
ETH Bibliography
yes
Altmetrics
Abstract
This paper discusses why flow scheduling does not apply to distributed deep learning training and presents EchelonFlow, the first network abstraction to bridge the gap. EchelonFlow deviates from the common belief that semantically related flows should finish at the same time. We reached the key observation, after extensive workflow analysis of diverse training paradigms, that distributed training jobs observe strict computation patterns, which may consume data at different times. We devise a generic method to model the drastically different computation patterns across training paradigms, and formulate EchelonFlow to regulate flow finish times accordingly. Case studies of mainstream training paradigms under EchelonFlow demonstrate the expressiveness of the abstraction, and our system sketch suggests the feasibility of an EchelonFlow scheduling system. Show more
Permanent link
https://doi.org/10.3929/ethz-b-000591849Publication status
publishedExternal links
Book title
HotNets '22: Proceedings of the 21st ACM Workshop on Hot Topics in NetworksPages / Article No.
Publisher
Association for Computing MachineryEvent
Subject
flow scheduling; data center networks; deep learningMore
Show all metadata
ETH Bibliography
yes
Altmetrics