Christopher De Sa, Assistant Professor in the Department of Computer Science in the Ann S. Bowers College of Computing and Information Science, and Yucheng Lu, a Ph.D. candidate in the department specializing in distributed optimization and machine learning systems, won an Outstanding Paper Award Honorable Mention at the 2021 International Conference on Machine Learning (ICML) for "Optimal Complexity in Decentralized Training."
As the authors explain: "Decentralization is a promising method of scaling up parallel machine learning systems. In this paper, we provide a tight lower bound on the iteration complexity for such methods in a stochastic non-convex setting." In their research, De Sa and Lu show how "[o]ur lower bound reveals a theoretical gap in known convergence rates of many existing decentralized training algorithms, such as D-PSGD."
In turn, De Sa and Lu "prove by construction this lower bound is tight and achievable. Motivated by our insights, we further propose DeTAG, a practical gossip-style decentralized algorithm that achieves the lower bound with only a logarithm gap. Empirically, we compare DeTAG with other decentralized algorithms on image classification tasks, and we show DeTAG enjoys faster convergence compared to baselines, especially on unshuffled data and in sparse networks."
In related news, Christopher De Sa is among five members of the computer science department to receive an NSF Career award this year.