Beyond NNGP: Large Deviations and Feature Learning in Bayesian Neural Networks

Bayesian neural networks, in the overparameterized and infinite-width regime, are now well understood. Under mild assumptions, their prior converges to a Gaussian process (NNGP), and both Bayesian inference and training dynamics can be described by kernel methods. Although, these infinite-width limits provide tractable models and sharp theoretical insights, they also exhibit a fundamental rigidity: the induced feature representation becomes fixed and independent of data. As a result, feature learning disappears in the infinite-width limit, and Bayesian inference reduces to kernel regression with a predetermined kernel.

In this talk, I present a complementary large-deviation perspective on wide Bayesian neural networks. Rather than studying typical Gaussian fluctuations, we analyse exponentially rare, but statistically dominant, configurations that govern posterior concentration as width grows. At this scale, Bayesian inference becomes variational: posterior mass concentrates near minimizers of an explicit functional rate function defined directly on predictors.
Our main result shows that, in contrast to the Gaussian-process limit, the posterior large-deviation rate function involves a joint optimization over predictors and internal covariance kernels. This nested variational structure leads to data-dependent kernel selection and provides a mechanism for feature learning that persists even in the infinite-width regime. In particular, we prove that the posterior-optimal kernel generically differs from the NNGP kernel. Joint work with D. Trevisan