BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Memento EPFL//
BEGIN:VEVENT
SUMMARY:How attention learns structure form data?
DTSTART:20261113T151500
DTEND:20261113T164500
DTSTAMP:20261109T152256Z
UID:0a9ccd2afd9df6ea2f253c75b211bc8b2c1d238ba105c30f5630799f
CATEGORIES:Conferences - Seminars
DESCRIPTION:Claire Boyer\, Université Paris-Saclay\nTransformer architec
 tures have demonstrated remarkable empirical success\, yet their ability t
 o extract statistical structure from data remains only partially understoo
 d. In this talk\, I will present recent work shedding light on attention m
 echanisms through a statistical and operator-theoretic lens.\nBased on lar
 ge-prompt asymptotics\, one can show that softmax attention converges to a
  linear operator acting on the input-token distribution under Gaussian ass
 umptions. This regime enables a precise analysis of both the outputs and t
 he training dynamics using concentration arguments\, revealing a surprisin
 g bridge between nonlinear softmax attention and tractable linear models.\
 nThese results suggest that attention can be understood as a flexible stat
 istical operator that adapts to the underlying data distribution\, providi
 ng a unifying framework to study its role in representation learning and i
 n-context inference. I will conclude by discussing the connections between
  transformer architectures and principal component analysis\, using this s
 etting to illustrate the power of concentration methods in high-dimensiona
 l statistics.\n 
LOCATION:CM 1 517 https://plan.epfl.ch/?room==CM%201%20517
STATUS:CONFIRMED
END:VEVENT
END:VCALENDAR