How attention learns structure form data?

Transformer architectures have demonstrated remarkable empirical success, yet their ability to extract statistical structure from data remains only partially understood. In this talk, I will present recent work shedding light on attention mechanisms through a statistical and operator-theoretic lens.
Based on large-prompt asymptotics, one can show that softmax attention converges to a linear operator acting on the input-token distribution under Gaussian assumptions. This regime enables a precise analysis of both the outputs and the training dynamics using concentration arguments, revealing a surprising bridge between nonlinear softmax attention and tractable linear models.
These results suggest that attention can be understood as a flexible statistical operator that adapts to the underlying data distribution, providing a unifying framework to study its role in representation learning and in-context inference. I will conclude by discussing the connections between transformer architectures and principal component analysis, using this setting to illustrate the power of concentration methods in high-dimensional statistics.