Extending Language Models toward more Human-Like Language Understanding
Event details
Date | 15.06.2021 |
Hour | 13:00 › 15:00 |
Speaker | Martin Josifoski |
Category | Conferences - Seminars |
EDIC candidacy exam
exam president: Prof. Boi Faltings
thesis advisor: Prof. Robert West
co-examiner: Prof. Tanja Käser
Abstract
Transformer-based language models, pretrained on large-scale data, followed by fine-tuning on a specific task, have established state-of-the-art performance on many NLP benchmarks. Despite their success, recent studies suggest many tasks with which they struggle due to their lack of reading comprehension, common sense, memory or basic reasoning. When accompanied with a benchmark dataset, task specific solutions to address the problem have been developed, but crucially, these solutions do not generalize, and only solve the dataset without solving the general task. In contrast, humans build from previous experience, and can quickly learn to solve a new language task from only a few examples or instructions. How can we make language understanding models more human-like? First, the right level of abstraction needs to be employed -- humans reason in terms of situations and not token correlations. Situations are representation models that specify the objects/entities of interest, their properties and the relations between them. Second, models should follow a modular design that allows for learning independent mechanisms that can be flexibly reused, composed and re-purposed -- humans break down complex tasks into smaller, more fundamental sub-tasks. Building on these observations we propose a knowledge-augmented neuro-symbolic system with a modular design. More specifically, the modules will communicate via situations, and the knowledge source will be organized around entities (objects), their properties and the known relations be-tween them.
Background papers
Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
Autoregressive Entity Retrieval
Invariant Risk Minimization
exam president: Prof. Boi Faltings
thesis advisor: Prof. Robert West
co-examiner: Prof. Tanja Käser
Abstract
Transformer-based language models, pretrained on large-scale data, followed by fine-tuning on a specific task, have established state-of-the-art performance on many NLP benchmarks. Despite their success, recent studies suggest many tasks with which they struggle due to their lack of reading comprehension, common sense, memory or basic reasoning. When accompanied with a benchmark dataset, task specific solutions to address the problem have been developed, but crucially, these solutions do not generalize, and only solve the dataset without solving the general task. In contrast, humans build from previous experience, and can quickly learn to solve a new language task from only a few examples or instructions. How can we make language understanding models more human-like? First, the right level of abstraction needs to be employed -- humans reason in terms of situations and not token correlations. Situations are representation models that specify the objects/entities of interest, their properties and the relations between them. Second, models should follow a modular design that allows for learning independent mechanisms that can be flexibly reused, composed and re-purposed -- humans break down complex tasks into smaller, more fundamental sub-tasks. Building on these observations we propose a knowledge-augmented neuro-symbolic system with a modular design. More specifically, the modules will communicate via situations, and the knowledge source will be organized around entities (objects), their properties and the known relations be-tween them.
Background papers
Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
Autoregressive Entity Retrieval
Invariant Risk Minimization
Practical information
- General public
- Free
Organizer
- EDIC