BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Memento EPFL//
BEGIN:VEVENT
SUMMARY:AI Center Seminar - AI Fundamentals series - Dr. Frederik Kunstner
DTSTART:20261110T140000
DTEND:20261110T150000
DTSTAMP:20261109T140501Z
UID:e03dfb6794688ea816aae72af2ca39c6bc64195430c42e693217610b
CATEGORIES:Conferences - Seminars
DESCRIPTION:Dr. Frederik Kunstner\nThe talk is jointly organized by the 
 EPFL AI Center and the MLO Lab as part of the AI fundamentals seminar ser
 ies.\n\nHosting Professor: Prof. Martin Jaggi\n\nTitle\nWhy Adam work in o
 ptimizing language models: heavy-tailed data imbalance\n\nAbstract\nAdam a
 nd related methods are the default optimizers in training language models 
 because gradient descent is so slow it is thought it simply does not work.
  Many methods have been proposed to improve on Adam\, but we have a poor u
 nderstanding of why it works better in the first place. This talk will be 
 about understanding what makes language models hard for gradient descent\,
  and how Adam fixes it. We give evidence that the problem is due to imbala
 nce in the data: in language\, some words are very frequent but the majori
 ty are rare\, following Zipf’s law\; the kh most frequent word has frequ
 ency proportional to 1/k. This heavy-tailed imbalance leads to poor perfor
 mance with gradient descent\, while Adam and other sign-like methods do no
 t suffer from the same slowdown. We show empirical evidence on language mo
 dels\, and prove a separation in the performance of gradient and a simplif
 ied form of Adam\, sign descent\, on a linear bigram model as we the vocab
 ulary size grows.\n\nBio\nFrederik Kunstner is Marie Curie postdoctoral fe
 llow at INRIA\, working with Francis Bach\, and an incoming professor at t
 he University of Waterloo. \nHis research lies at the intersection of opt
 imization and machine learning\, with a particular focus on understanding 
 the foundations of methods used in deep learning.\nHe received his PhD fro
 m the university of British Columbia under the supervision of Mark Schmidt
 \, following his undergraduate studies at EPFL.\nHis work has received a B
 est Paper Award at AISTATS 2021\, a Best Doctoral Dissertation Award from 
 the Canadian AI Association in 2025\, and an Honourable Mention from AAAI/
 ACM SIGAI in 2025.
LOCATION:ELE 117 https://plan.epfl.ch/?room==ELE%20117 https://epfl.zoom.u
 s/j/63357554422?pwd=78nYhdNWmhKeRVL7ps31u6FHBi9xiv.1
STATUS:CONFIRMED
END:VEVENT
END:VCALENDAR