The interplay between data structure and neural networks: going beyond Gaussian models

Neural networks are powerful feature extractors - but what do they actually learn from their data? We discuss two recent works on this question, with a focus on the importance of non-Gaussian statistics for neural networks. We first develop a simple model for images and show that a neural network trained on these images can learn a convolution from scratch. This pattern-formation process is driven by a combination of translation-invariance of the "images" and the non-Gaussian, higher-order statistics of the inputs. Second, we conjecture a "distributional simplicity bias" whereby neural networks learn increasingly complex distributions of their inputs during training. We present analytical and experimental evidence for this conjecture, going from a simple perceptron up to deep ResNets and visual transformers.