Toward the Design of Efficient and Robust Algorithms for Distributed Collaborative Deep Learning.

The EDIC program is happy to invite you to the public talk by our doctoral student Tao Lin.
The aim of the talk is to present his achievements to a broad audience to prepare for hiring interviews coming up soon. Be sure to join, listen to the talk and participate in the Q&A session at the end of the presentation.

Abstract:
Recent progress in deep learning has shown impressive results in many domains, which has also led to a dramatic increase in the size, complexity, and computational power of the training system. In the meantime, significant efficiency concerns and privacy risks arise as currently most learning is operated in a centralized way, while tremendous amounts of data tend to be generated on decentralized edge devices and may contain users’ personal information. All these motivate the necessity of moving to distributed deep learning.
In this talk, I will highlight my three key steps toward enabling distributed collaborative deep learning. I will show how to improve the learning efficiency of distributed deep learning in data centers, for both communication-restricted settings and the regime of poor generalization of large-batch SGD. In addition to learning in the data center, I will also discuss my attempts at improving the efficiency of learning on edge devices, where the advances come from the decentralized and sparse communication topology. Finally, I will discuss the demands of deploying robust decentralized learning algorithms, due to the hampered learning performance induced by heterogeneous learning environments. I will then conclude the talk with an outlook on future directions.

Bio:
Tao Lin is a fifth-year Ph.D. student at École Polytechnique Fédérale de Lausanne (EPFL), supervised by Prof. Martin Jaggi and Prof. Babak Falsafi. Prior to that, he received a Master of Science degree from EPFL and a Bachelor of Engineering degree from Zhejiang University (ZJU). His recent research interests include designing efficient and robust optimization algorithms for distributed collaborative deep learning.