Computer Science Speakers Series

Back to Listing

Daniel Russo "On the Global Convergence and Approximation Benefits of Policy Gradient Methods"

Event Type
Seminar/Symposium
Sponsor
The Department of Computer Science, University of Illinois and Illinois Institute for Data Science and Dynamical Systems
Location
https://illinois.zoom.us/j/272292042?pwd=WEFqNHpBekR6RVF1U1NFQkFyMm1CUT09
Virtual
wifi event
Date
Nov 20, 2020   2:00 - 3:00 pm  
Speaker
Daniel Russo, Assistant Professor, Columbia Business School
Contact
D'Mayza McClure
E-Mail
dmcclur2@illinois.edu
Views
64
Originating Calendar
Computer Science Speakers Calendar

Abstract: Policy gradient methods apply to complex, poorly understood, control problems by performing stochastic gradient descent over a parameterized class of polices. Unfortunately, due to the multi-period nature of the objective, they face non-convex optimization problems. The first part of the talk aims to clarify when policy gradient methods can be expected to converge to a global optimum despite this non-convexity. We show they can get stuck in suboptimal stationary points in extremely simple problems but identify structural properties – shared by several canonical control problems – that guarantee the policy gradient objective function has no suboptimal stationary points despite being non-convex. In the second part of the talk, I’ll zoom in on the special case of state aggregated policies, which induce unavoidable approximation errors as they  cannot represent an optimal policy. In this case, I'll show policy gradient provably converges to better policies than popular approximate policy iteration and value iteration methods.  

Bio: Dan's research lies at the intersection of statistical machine learning and online decision making, mostly falling under the broad umbrella of reinforcement learning. He joined the Decision, Risk, and Operations division of the Columbia Business School as an assistant professor in Summer 2017. Prior to joining Columbia, he spent one year as an assistant professor in the MEDS department at Northwestern's Kellogg School of Management and one year at Microsoft Research in New England as Postdoctoral Researcher. He received his PhD from Stanford University in 2015, where he was advised by Benjamin Van Roy. In 2011, he received his BS in Mathematics and Economics from the University of Michigan.

link for robots only