Implicit Bias of Gradient Methods
Event details
| Date | 21.07.2022 |
| Hour | 15:00 › 17:00 |
| Speaker | Aditya Varre |
| Location | |
| Category | Conferences - Seminars |
EDIC candidacy exam
Exam president: Prof. Martin Jaggi
Thesis advisor: Prof. Nicolas Flammarion
Co-examiner: Prof. Lenaic Chizat
Abstract
It is becoming increasingly clear that implicit biases introduced by the optimization algorithm play a crucial role in deep learning and in the generalization ability of the learned models. In this report, we examine the implicit bias of gradient algorithms on unregularized regression or classification problems. In the case of logistic regression, we show how gradient descent converges in the direction of the max-margin (hard margin SVM) solution. Finally, we discuss how this methodology can also aid in understanding implicit regularization in more complex models and with other optimization methods.
Background papers
a) Understanding deep learning requires rethinking generalization, https://arxiv.org/pdf/1611.03530.pdf,. Zhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. ICLR 2017.
b) Kernel and rich regimes in overparametrized models http://proceedings.mlr.press/v125/woodworth20a/woodworth20a.pdf. Woodworth, B., Gunasekar, S., Lee, J.D., Moroshko, E., Savarese, P., Golan, I., Soudry, D. and Srebro, COLT 2020.
c) Large learning rate tames homogeneity: Convergence and balancing effect. https://openreview.net/pdf?id=3tbDrs77LJ5. Wang, Y., Chen, M., Zhao, T. and Tao, M., 2021. ICLR 2022.
Exam president: Prof. Martin Jaggi
Thesis advisor: Prof. Nicolas Flammarion
Co-examiner: Prof. Lenaic Chizat
Abstract
It is becoming increasingly clear that implicit biases introduced by the optimization algorithm play a crucial role in deep learning and in the generalization ability of the learned models. In this report, we examine the implicit bias of gradient algorithms on unregularized regression or classification problems. In the case of logistic regression, we show how gradient descent converges in the direction of the max-margin (hard margin SVM) solution. Finally, we discuss how this methodology can also aid in understanding implicit regularization in more complex models and with other optimization methods.
Background papers
a) Understanding deep learning requires rethinking generalization, https://arxiv.org/pdf/1611.03530.pdf,. Zhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. ICLR 2017.
b) Kernel and rich regimes in overparametrized models http://proceedings.mlr.press/v125/woodworth20a/woodworth20a.pdf. Woodworth, B., Gunasekar, S., Lee, J.D., Moroshko, E., Savarese, P., Golan, I., Soudry, D. and Srebro, COLT 2020.
c) Large learning rate tames homogeneity: Convergence and balancing effect. https://openreview.net/pdf?id=3tbDrs77LJ5. Wang, Y., Chen, M., Zhao, T. and Tao, M., 2021. ICLR 2022.
Practical information
- General public
- Free