BIOE Seminar Series: Associate Professor Saurabh Gupta

Separating Visual and Physical Intelligence for Scalable Robot Learning

Abstract: The monolithic, end-to-end visuomotor policies used by most current robot learning methods require a very specific form of training data: high-fidelity, paired perception-action data. In practice, this means teleoperated demonstrations for imitation learning or visually realistic simulation for sim-to-real transfer. Both are difficult to scale, so as a field we still lack ways to quickly and cheaply build robots that can perform a broad set of tasks across diverse deployment scenarios. In this talk, I will describe how appropriately separating visual and physical intelligence significantly expands the amount of robot-relevant data available for building robot policies. Across four case studies, I will show that systems built this way generalize better than end-to-end alternatives to unseen, everyday environments: opening articulated objects, precise manipulation of small objects, open-vocabulary humanoid grasping, and contact-rich humanoid loco-manipulation. If time permits, I will also share preliminary results suggesting that a similar separation could simplify building ML models for niche image domains.

Biography: Saurabh Gupta is an Associate Professor in the ECE Department at UIUC. Before starting at UIUC in 2019, he received his Ph.D. from UC Berkeley in 2018 and spent a year as a Research Scientist at Facebook AI Research in Pittsburgh. His research interests span computer vision, robotics, and machine learning, with a focus on building agents that can intelligently interact with the physical world around them.