AI Center Seminar - AI Fundamentals series - Simon Martin

The talk is organized by the EPFL AI Center as part of the AI fundamentals seminar series. Talk followed by a coffee session.

Hosting professor: Prof. Lenka Zdeborová

Title
High-dimensional analysis of gradient flow for extensive-width matrix sensing

Abstract
The loss landscapes of deep neural networks are known to be highly nonconvex, and gradient-based algorithms could, in principle, get trapped in spurious local minima or converge to predictors with poor generalization. Yet, empirical evidence shows that overparameterized networks often behave quite differently: gradient-based methods tend to avoid poor local minima and converge to solutions with good generalization. This raises a fundamental question: what is the actual effect of overparameterization on optimization and generalization?

In this talk, we will investigate how overparameterization impacts gradient flow dynamics for a matrix sensing problem, with generalizations to shallow neural networks with quadratic activation functions. In the extensive-width regime, where the number of neurons diverges with the dimension, we derive high-dimensional equivalent equations for the regularized gradient flow dynamics, in the spirit of dynamical mean-field theory (DMFT) equations. We then simplify these equations in the long-time limit and characterize the performance of the gradient flow predictor with a simple low-dimensional set of scalar equations. This result allows us to study the generalization ability of this predictor and analyze how overparameterization affects it. 

This work is joint with Giulio Biroli and Francis Bach. 

Bio
Simon Martin is a second-year PhD student jointly affiliated with the SIERRA project-team at INRIA Paris and the Center for Data Science at École Normale Supérieure. He is advised by Francis Bach and Giulio Biroli. Before his PhD, he studied at École Normale Supérieure (ENS Ulm), where he obtained a Master’s degree in Physics, as well as a Master’s degree in Probabilities and Statistics from Université Paris-Saclay. His research focuses on shallow neural networks and their learning dynamics. The goal of his PhD is to leverage tools from applied mathematics and statistical physics to better understand gradient flow dynamics in high-dimensional systems.