BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Memento EPFL//
BEGIN:VEVENT
SUMMARY:Cost-Efficient Distributed Learning
DTSTART:20230517T140000
DTEND:20230517T150000
DTSTAMP:20260427T225342Z
UID:6518c15e49decbc8dd57643b77bebbaac9f0377241871a5cf42bb671
CATEGORIES:Conferences - Seminars
DESCRIPTION:Dr Rawad Bitar TUM\nWe analyze distributed gradient descent\, 
 which is an iterative algorithm that consists of evaluating the gradient o
 f a loss function at a current model and the considered data. A main node 
 sends its data to n worker devices. At each iteration\, the workers evalu
 ate the gradient at the data assigned to them and return the result to the
  main node. The main node aggregates the obtained evaluations and updates 
 the model. This setting is known to suffer from the presence of stragglers
 \, i.e.\, slow or unresponsive workers\, whose delays may outweigh the ben
 efit of distributing the computations.\n\nUnder some constraints on the re
 sponse times of the workers\, ignoring the stragglers is equivalent to run
 ning stochastic gradient descent (SGD). Hence\, the main node only aggrega
 tes the result returned from the fastest k≤n workers\, where k is fixe
 d parameter. The choice of k that can be optimized a priori presents a tr
 ade-off between the total run-time of the algorithm and the quality of the
  obtained model.\n\nIn this talk\, we design a strategy that adaptively in
 creases k throughout the run-time of the algorithm to obtain the best run
 -time possible with the best quality of the model. To that end\, we derive
  a relationship between the value of k\, the time spent per iteration\, an
 d the quality of the obtained model. Based on this relationship\, we deriv
 e the times at which the main node needs to increase k and hence reduce t
 he communication cost.\n\nWe then introduce a more cost-efficient strategy
  that only assigns tasks to k workers per iteration. To learn which k wo
 rkers are the fastest\, we rely on the theory of multi-armed bandits. We d
 esign a cost-efficient scheme that assigns computational tasks to the work
 ers while learning their speed. We show that for a fixed cost (measured by
  the number of worker employment)\, the introduce strategy outperforms the
  previous schemes. However\, this strategy has a longer run-time due to th
 e overhead of learning the speed of the workers.\n\nWe validate the result
 s and concretize the theoretical findings through numerical simulations.\n
 \nThis talk summarizes results from joint works with Antonia Wachter-Zeh\,
  Deniz Gündüz\, Maximilian Egger\, Parimal Parag\, Salim El Rouyahbe\, S
 erge Kas Hanna and Venkat Dasaari.\n\n 
LOCATION:INR 113 https://plan.epfl.ch/?room==INR%20113
STATUS:CONFIRMED
END:VEVENT
END:VCALENDAR