CIS Network Seminars: #IDIAP - Dr. James Henderson
Title: Understanding Language as Bayesian Inference of Bayesian Representations
Abstract:
The astounding recent advances in AI have been enabled by the Transformer deep learning architecture, and its incredibly effective inductive bias for Large Language Models. We argue that this architecture is fundamentally Bayesian, in that its learned representations are best thought of as probability distributions. We then propose an extension to Transformers which adds nonparametric Bayesian inference of these Bayesian representations. Our Nonparametric Variational Information Bottleneck can regularise any attention-based representation, resulting in generative models with variable-sized sparse latent representations. Existing large Transformers pretrained on text can be reinterpreted as approximating this inference, suggesting that the success of Transformers is due to their ability to approximate nonparametric Bayesian inference of Bayesian representations.
Bio:
Dr James Henderson is a Senior Researcher at Idiap Research Institute, where he heads the Natural Language Understanding group. He was previously MER at Univ Geneva and held a number of other research positions since his PhD at Univ Pennsylvania and BSc at MIT. He investigates machine learning methods for natural language processing, with a long history of work on neural network structured prediction and variational Bayesian approaches to deep learning.
The Center for Intelligent Systems at EPFL (CIS) is a collaboration among IC, ENAC, SB; SV and STI that brings together researchers working on different aspects of Intelligent Systems.
Collaboration and networking are crucial in the advancement of research and are key pillars of the CIS’ activities. We are thrilled to announce a new series of talks: the “CIS Network Seminars”, where we will invite experts from our network to share their work.
The CIS seminar will take place In hybrid mode: Room INF 328 and by Zoom https://epfl.zoom.us/meeting/register/u5wkde2rqTItE9cNYqIrPJvoc7ONUV6JFesD
Monday 24 April, 2023 from 3:15 to 4:15 pm
NB: Video recordings of the seminars will be made available on our website and published on our social media pages
Abstract:
The astounding recent advances in AI have been enabled by the Transformer deep learning architecture, and its incredibly effective inductive bias for Large Language Models. We argue that this architecture is fundamentally Bayesian, in that its learned representations are best thought of as probability distributions. We then propose an extension to Transformers which adds nonparametric Bayesian inference of these Bayesian representations. Our Nonparametric Variational Information Bottleneck can regularise any attention-based representation, resulting in generative models with variable-sized sparse latent representations. Existing large Transformers pretrained on text can be reinterpreted as approximating this inference, suggesting that the success of Transformers is due to their ability to approximate nonparametric Bayesian inference of Bayesian representations.
Bio:
Dr James Henderson is a Senior Researcher at Idiap Research Institute, where he heads the Natural Language Understanding group. He was previously MER at Univ Geneva and held a number of other research positions since his PhD at Univ Pennsylvania and BSc at MIT. He investigates machine learning methods for natural language processing, with a long history of work on neural network structured prediction and variational Bayesian approaches to deep learning.
The Center for Intelligent Systems at EPFL (CIS) is a collaboration among IC, ENAC, SB; SV and STI that brings together researchers working on different aspects of Intelligent Systems.
Collaboration and networking are crucial in the advancement of research and are key pillars of the CIS’ activities. We are thrilled to announce a new series of talks: the “CIS Network Seminars”, where we will invite experts from our network to share their work.
The CIS seminar will take place In hybrid mode: Room INF 328 and by Zoom https://epfl.zoom.us/meeting/register/u5wkde2rqTItE9cNYqIrPJvoc7ONUV6JFesD
Monday 24 April, 2023 from 3:15 to 4:15 pm
NB: Video recordings of the seminars will be made available on our website and published on our social media pages
Practical information
- General public
- Free
Organizer
- CIS