IC Colloquium: From Instructions to Interaction: Developing Steerable and Usable Open Language Models
Par : Valentina Pyatkin - Allen Institute for AI and University of Washington
IC Faculty candidate
Abstract
Making large language models usable requires post-training methods that align models to human values, robustly handle underspecified inputs, and generalize to diverse instructions. This talk addresses the challenge of developing responsible AI through post-training from the following angles.
First, I address contextual robustness. Preference data, for example, is often underspecified, and I show how underspecification in preference data can lead to diverging preferences. Standard reward models fail to properly handle these disagreements, often making decisive choices even when human annotators are split. I argue that more consequential outputs demand more context, and propose clarification question generation as one solution.
Second, I will discuss how we can train models to be better instruction followers. I will show that most models severely overfit on a small set of instruction-following constraints and are not able to generalize well to unseen output constraints. I propose to train models with reinforcement learning from verifiable rewards for verifiable instruction following, and show how this leads to improved generalization on constraint following.
Throughout the presentation, I will outline how I have applied these insights into developing open generative models, like Tülu and OLMo, and I will conclude with my research agenda for responsible post-training: critical AI evaluation, broader generalization, and expanding what models can reliably do.
Bio
Valentina Pyatkin is a postdoctoral researcher at the Allen Institute for AI and the University of Washington, advised by Prof. Hanna Hajishirzi and Prof. Yejin Choi. Additionally, she is part-time affiliated with the ETH AI Center, where she mentors students and works on post-training for the Swiss AI Initiative. She obtained her PhD in Computer Science from Bar Ilan University. Her work has been awarded an ACL Outstanding Paper Award and the ACL Best Theme Paper Award, and has been supported by a Schmidt Sciences Postdoctoral Award. During her doctoral studies, she conducted research internships at Google and the Allen Institute for AI, where she received the AI2 Outstanding Intern of the Year Award.
More information
IC Faculty candidate
Abstract
Making large language models usable requires post-training methods that align models to human values, robustly handle underspecified inputs, and generalize to diverse instructions. This talk addresses the challenge of developing responsible AI through post-training from the following angles.
First, I address contextual robustness. Preference data, for example, is often underspecified, and I show how underspecification in preference data can lead to diverging preferences. Standard reward models fail to properly handle these disagreements, often making decisive choices even when human annotators are split. I argue that more consequential outputs demand more context, and propose clarification question generation as one solution.
Second, I will discuss how we can train models to be better instruction followers. I will show that most models severely overfit on a small set of instruction-following constraints and are not able to generalize well to unseen output constraints. I propose to train models with reinforcement learning from verifiable rewards for verifiable instruction following, and show how this leads to improved generalization on constraint following.
Throughout the presentation, I will outline how I have applied these insights into developing open generative models, like Tülu and OLMo, and I will conclude with my research agenda for responsible post-training: critical AI evaluation, broader generalization, and expanding what models can reliably do.
Bio
Valentina Pyatkin is a postdoctoral researcher at the Allen Institute for AI and the University of Washington, advised by Prof. Hanna Hajishirzi and Prof. Yejin Choi. Additionally, she is part-time affiliated with the ETH AI Center, where she mentors students and works on post-training for the Swiss AI Initiative. She obtained her PhD in Computer Science from Bar Ilan University. Her work has been awarded an ACL Outstanding Paper Award and the ACL Best Theme Paper Award, and has been supported by a Schmidt Sciences Postdoctoral Award. During her doctoral studies, she conducted research internships at Google and the Allen Institute for AI, where she received the AI2 Outstanding Intern of the Year Award.
More information
Practical information
- General public
- Free
Contact
- Host: Bob West