AI Center Seminar - AI Fundamentals series - Anej Svete - "Expressivity-Maxxing: What To Do When Your Transformer Is Giving Mid"

The talk is jointly organized by the EPFL AI Center and the TML Lab as part of the AI fundamentals seminar series.

Hosting Professor: Prof. Nicolas Flammarion

Title
Expressivity-Maxxing: What To Do When Your Transformer Is Giving Mid
Improving the Reasoning Abilities of Transformers with Looping, Latent Reasoning, and Diffusion

Abstract
Transformers famously compute with a fixed number of parallel steps, one per layer. That makes them fast, but it also means that there are very simple problems transformers cannot solve. The cup shuffling game where one has to follow a ball under a cup while the cups are shuffled is a classic example. Chain-of-thought reasoning makes transformers much more powerful, but it is slow: The model thinks one token at a time and has to squeeze intermediate result into words. This talk is a tour of the ways to give a transformer more computation without making it talk: continuous thoughts, looping, and diffusion. Continuous thoughts let a transformer run textbook algorithms efficiently, looping reaches progressively harder classes of parallelizable problems, and masked diffusion models turn out to be padded looped transformers in disguise. I will close with a recently proposed way of training a looped transformer to reason in parallel that comes close to explicit chain of thought while thinking 2.5 times faster.

Bio
Anej Svete is a doctoral fellow at the ETH AI Center, co-advised by Ryan Cotterell and Valentina Boeva. He studies what language models can and cannot compute, using formal language and complexity theory. He uses those results for theory-driven architecture design: Making transformers more powerful with looping or by designing hybrid models that combine the strengths of transformers and linear recurrent neural networks. He holds a bachelor's degree in computer science and mathematics from the University of Ljubljana and a master's degree in data science from ETH Zürich.