Mean field approach to learning dynamics in neural networks
Event details
| Date | 29.08.2022 |
| Hour | 10:00 › 12:00 |
| Speaker | Emanuele Troiani |
| Category | Conferences - Seminars |
EDIC candidacy exam
Exam president: Prof. Florent Krzakala
Thesis advisor: Prof. Lenka Zdeborová
Co-examiner: Prof. Nicolas Macris
Abstract
Gradient based optimisation algorithms are invaluable
tools for training neural networks. Over the years a number
of variants and optimisations have been proposed that empirically increase performance.
While their advantage can be measured in
practice, a comprehensive theoretical description of the dynamics,
especially in non convex landscapes, is still in the making. In
the following we review an early approach to the dynamics of
linear network, as well as an effective description of mean field
dynamics in shallow neural networks, which was recently proven
to be rigorous.
We believe that this procedure can be expanded
to describe a much wider range of settings.
Background papers
Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
https://iopscience.iop.org/article/10.1088/2632-2153/ac0615/pdf
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
https://arxiv.org/pdf/1312.6120.pdf%C2%A0
The high-dimensional asymptotics of first order methods with random data
https://arxiv.org/pdf/2112.07572.pdf
Exam president: Prof. Florent Krzakala
Thesis advisor: Prof. Lenka Zdeborová
Co-examiner: Prof. Nicolas Macris
Abstract
Gradient based optimisation algorithms are invaluable
tools for training neural networks. Over the years a number
of variants and optimisations have been proposed that empirically increase performance.
While their advantage can be measured in
practice, a comprehensive theoretical description of the dynamics,
especially in non convex landscapes, is still in the making. In
the following we review an early approach to the dynamics of
linear network, as well as an effective description of mean field
dynamics in shallow neural networks, which was recently proven
to be rigorous.
We believe that this procedure can be expanded
to describe a much wider range of settings.
Background papers
Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
https://iopscience.iop.org/article/10.1088/2632-2153/ac0615/pdf
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
https://arxiv.org/pdf/1312.6120.pdf%C2%A0
The high-dimensional asymptotics of first order methods with random data
https://arxiv.org/pdf/2112.07572.pdf
Practical information
- General public
- Free