Kavli Affiliate: Jia Liu
| First 5 Authors: Menglu Yu, Ye Tian, Bo Ji, Chuan Wu, Hridesh Rajan
| Summary:
Fueled by advances in distributed deep learning (DDL), recent years have
witnessed a rapidly growing demand for resource-intensive distributed/parallel
computing to process DDL computing jobs. To resolve network communication
bottleneck and load balancing issues in distributed computing, the so-called
“ring-all-reduce” decentralized architecture has been increasingly adopted to
remove the need for dedicated parameter servers. To date, however, there
remains a lack of theoretical understanding on how to design resource
optimization algorithms for efficiently scheduling ring-all-reduce DDL jobs in
computing clusters. This motivates us to fill this gap by proposing a series of
new resource scheduling designs for ring-all-reduce DDL jobs. Our contributions
in this paper are three-fold: i) We propose a new resource scheduling
analytical model for ring-all-reduce deep learning, which covers a wide range
of objectives in DDL performance optimization (e.g., excessive training
avoidance, energy efficiency, fairness); ii) Based on the proposed performance
analytical model, we develop an efficient resource scheduling algorithm called
GADGET (greedy ring-all-reduce distributed graph embedding technique), which
enjoys a provable strong performance guarantee; iii) We conduct extensive
trace-driven experiments to demonstrate the effectiveness of the GADGET
approach and its superiority over the state of the art.
| Search Query: ArXiv Query: search_query=au:”Jia Liu”&id_list=&start=0&max_results=10