Machine Learning Seminar: Daman Arora, "Training Language Models to Fail Gracefully."

- Sponsor
- Siebel School of Computing and Data Science
- Speaker
- Daman Arora
- Contact
- Nishant Jain
- nj27@illinois.edu
- Views
- 51
- Originating Calendar
- Siebel School Speakers Calendar
Abstract: Reinforcement learning has become a cornerstone of training increasingly capable language models. However, as these models are deployed in more complex and high-stakes environments, improving their capabilities alone is not enough. A central challenge is ensuring that models behave reliably, particularly when faced with uncertainty or situations where they are likely to make mistakes. In this talk, we will discuss a broader line of research that reconsiders the optimization landscape induced by the classical REINFORCE algorithm. By examining how conventional policy-gradient objectives shape learning dynamics, this research develops alternative optimization objectives that address fundamental limitations of standard reinforcement learning providing a unifying perspective on recent methods such as MaxRL, TailRL, and GracefulRL. The main focus of the talk will be GracefulRL, a framework for training models to recognize when they are likely to fail and choose safer alternatives without sacrificing their ability to succeed. Through theoretical insights and experiments across diverse decision-making and reasoning tasks, the talk will illustrate how RL when done right can lead to not only more capable but also more reliable AI systems.
Bio: Daman Arora is a third-year PhD student in the Machine Learning Department at Carnegie Mellon University, advised by Andrea Zanette. Previously, he was a research fellow at Microsoft Research India and earned his bachelor's and master's degrees from IIT Delhi. His research focuses on reinforcement learning for efficient and reliable language models. His work on communication among AI agents recently received an Outstanding Paper Award at COLM 2026.