Extending Language Models toward more Human-Like Language Understanding


Event details

Date and time 15.06.2021 13:0015:00  
Speaker Martin Josifoski
Category Conferences - Seminars
EDIC candidacy exam
exam president: Prof. Boi Faltings
thesis advisor: Prof. Robert West
co-examiner: Prof. Tanja Käser

Transformer-based language models, pretrained on large-scale data, followed by fine-tuning on a specific task, have established state-of-the-art performance on many NLP benchmarks. Despite their success, recent studies suggest many tasks with which they struggle due to their lack of reading comprehension, common sense, memory or basic reasoning. When accompanied with a benchmark dataset, task specific solutions to address the problem have been developed, but crucially, these solutions do not generalize, and only solve the dataset without solving the general task. In contrast, humans build from previous experience, and can quickly learn to solve a new language task from only a few examples or instructions. How can we make language understanding models more human-like? First, the right level of abstraction needs to be employed -- humans reason in terms of situations and not token correlations. Situations are representation models that specify the objects/entities of interest, their properties and the relations between them. Second, models should follow a modular design that allows for learning independent mechanisms that can be flexibly reused, composed and re-purposed -- humans break down complex tasks into smaller, more fundamental sub-tasks. Building  on  these  observations  we  propose  a  knowledge-augmented  neuro-symbolic  system  with  a  modular  design. More  specifically,  the  modules  will  communicate  via  situations, and  the  knowledge  source  will  be  organized  around entities (objects), their properties and the known relations be-tween them.

Background papers
Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
Autoregressive Entity Retrieval
Invariant Risk Minimization

Practical information

  • General public
  • Free


  • EDIC


EDIC candidacy exam