Augment Language Models with Explicit Structured Information
Event details
| Date | 26.08.2021 |
| Hour | 10:00 › 12:00 |
| Speaker | Angeliki Romanou |
| Category | Conferences - Seminars |
EDIC candidacy exam
exam president: Prof. Tanja Käser
thesis advisor: Prof. Karl Aberer
co-examiner: Prof. Antoine Bosselut
Abstract
Recent natural language approaches try to leverage
the factual power of explicit knowledge by incorporating
retrieval-based techniques into neural language models.
In this doctoral candidacy proposal, we present three papers
aimed at augmenting neural language models with explicit nonparametric
knowledge. The first paper combines Transformerlike
pre-trained language models with knowledge bases, and show
improvements in many downstream tasks. The second paper
adds a jointly trained, fully differentiable document retriever
into the pre-training phase of the language model. The third
paper extends this approach by incorporating a text retriever
in sequence-to-sequence (seq2seq) models aiming for more factual
and knowledge-intense text generation. Although retrievalaugmented
models enjoy great popularity in the NLP research
community the recent years, there is still a great amount of work
that needs to be done in order to improve the explainability,
modularity, and efficiency of these models. We build upon these
works to propose a research agenda trying to tackle the existing
limitations by creating neural language models that aim for
generalization power as well as model specificity.
Background papers
- Peters, Matthew E., et al. "Knowledge enhanced contextual word representations." arXiv preprint arXiv:1909.04164 (2019). [https://arxiv.org/abs/1909.04164]
- Guu, Kelvin, et al. "Realm: Retrieval-augmented language model pre-training." arXiv preprint arXiv:2002.08909 (2020). [https://arxiv.org/abs/2002.08909]
- Lewis, Patrick, et al. "Retrieval-augmented generation for knowledge-intensive nlp tasks." arXiv preprint arXiv:2005.11401 (2020). [https://arxiv.org/abs/2005.11401]
exam president: Prof. Tanja Käser
thesis advisor: Prof. Karl Aberer
co-examiner: Prof. Antoine Bosselut
Abstract
Recent natural language approaches try to leverage
the factual power of explicit knowledge by incorporating
retrieval-based techniques into neural language models.
In this doctoral candidacy proposal, we present three papers
aimed at augmenting neural language models with explicit nonparametric
knowledge. The first paper combines Transformerlike
pre-trained language models with knowledge bases, and show
improvements in many downstream tasks. The second paper
adds a jointly trained, fully differentiable document retriever
into the pre-training phase of the language model. The third
paper extends this approach by incorporating a text retriever
in sequence-to-sequence (seq2seq) models aiming for more factual
and knowledge-intense text generation. Although retrievalaugmented
models enjoy great popularity in the NLP research
community the recent years, there is still a great amount of work
that needs to be done in order to improve the explainability,
modularity, and efficiency of these models. We build upon these
works to propose a research agenda trying to tackle the existing
limitations by creating neural language models that aim for
generalization power as well as model specificity.
Background papers
- Peters, Matthew E., et al. "Knowledge enhanced contextual word representations." arXiv preprint arXiv:1909.04164 (2019). [https://arxiv.org/abs/1909.04164]
- Guu, Kelvin, et al. "Realm: Retrieval-augmented language model pre-training." arXiv preprint arXiv:2002.08909 (2020). [https://arxiv.org/abs/2002.08909]
- Lewis, Patrick, et al. "Retrieval-augmented generation for knowledge-intensive nlp tasks." arXiv preprint arXiv:2005.11401 (2020). [https://arxiv.org/abs/2005.11401]
Practical information
- General public
- Free
Contact
- edic@epfl.ch