Kavli Affiliate: Jia Liu
| First 5 Authors: Ziyue Luo, Jia Liu, Myungjin Lee, Ness B. Shroff,
| Summary:
The recent explosive growth of deep learning (DL) models has necessitated a
compelling need for efficient job scheduling for distributed deep learning
training with mixed parallelisms (DDLwMP) in GPU clusters. This paper proposes
an adaptive shortest-remaining-processing-time-first (A-SRPT) scheduling
algorithm, a novel prediction-assisted online scheduling approach designed to
mitigate the challenges associated with DL cluster scheduling. By modeling each
job as a graph corresponding to heterogeneous Deep Neural Network (DNN) models
and their associated distributed training configurations, A-SRPT strategically
assigns jobs to the available GPUs, thereby minimizing inter-server
communication overhead. Observing that most DDLwMP jobs recur, A-SRPT
incorporates a random forest regression model to predict training iterations.
Crucially, A-SRPT maps the complex scheduling problem into a single-machine
instance, which is addressed optimally by a preemptive
"shortest-remaining-processing-time-first" strategy. This optimized solution
serves as a guide for actual job scheduling within the GPU clusters, leading to
a theoretically provable competitive scheduling efficiency. We conduct
extensive real-world testbed and simulation experiments to verify our proposed
algorithms.
| Search Query: ArXiv Query: search_query=au:”Jia Liu”&id_list=&start=0&max_results=3