AI Center Seminar - AI Fundamentals series - Ricardo Olmedo - "Speedrunning SWE-bench, or the $1,000 coding agent"

The talk is jointly organized by the EPFL AI Center and the MLO Lab as part of the AI fundamentals seminar series.

Hosting Professor: Prof. Martin Jaggi

Title
Speedrunning SWE-bench, or the $1,000 coding agent

Abstract
Speedrunning has become a popular engineering challenge in machine learning: achieve the best performance within a fixed compute budget. We use it as a scientific tool to study how models acquire specialized skills and how far those skills transfer. We speedrun SWE-bench by training coding agents from scratch on demonstrations from stronger models. With about $1,000 of training compute, we reach a 25% solve rate, outperforming GPT-4o. Compared to models with extensive pre-training, our speedrun models solve largely the same SWE-bench problems, and generalize about as well to new bugs and codebases. What, then, does pre-training buy for SWE-bench? Our experiments suggest a one-student hypothesis: pre-training helps a student imitate its teacher, but at matched imitation loss, pre-training choices barely matter. We stress-test this hypothesis by comparing a model trained on the web against a “vintage" one trained only pre-1931 documents. Up to the SWE-bench scores we reach, pre-training is interchangeable with specialization data.  We release nanoswe as an open project for studying specialization through speedrunning.

Bio
Ricardo Olmedo is a final-year PhD student at the Max Planck Institute for Intelligent Systems in Tübingen and a Google PhD Fellow. Advised by Moritz Hardt and Bernhard Schölkopf, he works on language model benchmarking and specialization. His research has earned three oral presentations at ICLR and NeurIPS. Most recently, he spent spring and summer 2026 visiting Sanmi Koyejo at Stanford University.