Redesigning decentralized ML algorithms: A systems view
Event details
Date | 20.01.2022 |
Hour | 12:00 › 14:00 |
Speaker | Akash Dhasade |
Category | Conferences - Seminars |
Event Language | English |
EDIC candidacy exam
Exam president: Prof. James Larus
Thesis advisor: Prof. Anne-Marie Kermarrec
Co-examiner: Prof. Nicolas Flammarion
Abstract
Decentralized Learning and Federated Learning evolved to address the crucial need of scalability -- by leveraging compute power at edge and user data privacy -- by sharing only locally trained models instead of data. However they are not without issues that arise from the training setting -- (a) with ever growing sizes of deep models communication remains expensive for low-end edge devices that participate in training; (b) heterogeneous data on client devices significantly slows down convergence; (c) systems heterogeneity results in stragglers, dropouts or intermittently available client; etc. to cite a few. As a novel research direction, we consider systems that fundamentally offer stronger privacy guarantees like Trusted Execution Environments (TEE's) and rethink the design of learning algorithms to share raw data instead of models. By letting clients exchange raw data in decentralized settings, we aim to solve several challenges at once from expensive communication to data heterogeneity while achieving privacy and scalability. Secondly, we explore a design space of algorithms which are hard to analyze theoretically but could compensate for stringent systems constraints on the learning process in practice. This includes algorithms that guess client model updates to compensate for their unavailability or lack of computation, algorithms that share only partial models and reduce communication, etc.
Background papers
Communication-efficient learning of deep networks from decentralized data, by McMahan, H B., et al.
Tackling the objective inconsistency problem in heterogeneous federated optimisation, by Wang, J., et al.
Towards mitigating device heterogeneity in federated learning via adaptive model quantization, by Ahmed M. Abdelmoniem, Marco Canini
Exam president: Prof. James Larus
Thesis advisor: Prof. Anne-Marie Kermarrec
Co-examiner: Prof. Nicolas Flammarion
Abstract
Decentralized Learning and Federated Learning evolved to address the crucial need of scalability -- by leveraging compute power at edge and user data privacy -- by sharing only locally trained models instead of data. However they are not without issues that arise from the training setting -- (a) with ever growing sizes of deep models communication remains expensive for low-end edge devices that participate in training; (b) heterogeneous data on client devices significantly slows down convergence; (c) systems heterogeneity results in stragglers, dropouts or intermittently available client; etc. to cite a few. As a novel research direction, we consider systems that fundamentally offer stronger privacy guarantees like Trusted Execution Environments (TEE's) and rethink the design of learning algorithms to share raw data instead of models. By letting clients exchange raw data in decentralized settings, we aim to solve several challenges at once from expensive communication to data heterogeneity while achieving privacy and scalability. Secondly, we explore a design space of algorithms which are hard to analyze theoretically but could compensate for stringent systems constraints on the learning process in practice. This includes algorithms that guess client model updates to compensate for their unavailability or lack of computation, algorithms that share only partial models and reduce communication, etc.
Background papers
Communication-efficient learning of deep networks from decentralized data, by McMahan, H B., et al.
Tackling the objective inconsistency problem in heterogeneous federated optimisation, by Wang, J., et al.
Towards mitigating device heterogeneity in federated learning via adaptive model quantization, by Ahmed M. Abdelmoniem, Marco Canini
Practical information
- General public
- Free