Computer Vision Seminar Series: Dr. Chen Sun, "Force-Conditioned Video Models as Physical Worlds Models."

Abstract: World models aim to simulate the possible physical consequences of actions to support embodied decision making. Recent advances in video generation models have sparked interest in using video to simulate such physical causes and effects. In this talk, we investigate using physical forces as a conditioning signal for video generation, enabling users and agents to interact with images and videos through both localized point forces, such as poking a plant, and global force fields, such as wind blowing on fabric. We demonstrate that these “force prompts” can enable videos to respond realistically to physical control signals by leveraging the visual and motion priors in the original pretrained model, without using any 3D asset or physics simulator at inference time. The primary challenge of force prompting is obtaining high-quality paired force-video training data: real-world force signals are difficult to acquire, while physics simulators have limited visual quality and domain diversity. Our key finding is that video generation models can generalize remarkably well when adapted to follow physical force conditioning using only synthetic videos. Building on this, we explore the role of such force-conditioned world models in robot planning and decision-making, and discuss future directions for going beyond the visual and motion priors learned by pretrained video models.

Speaker Bio.: Chen Sun (https://chensun.me/) is the Eliot Horowitz assistant professor of computer science at Brown University and a part-time researcher at Google DeepMind, studying computer vision and machine learning. His current research focuses on learning temporal dynamics from unlabeled videos and applying these video-centric world models in robotics. Chen has received an NSF CAREER Award, a Richard B. Salomon Faculty Research Award, University Research Seed Award, and a Samsung Global Research Outreach Award. Previously, Chen received his PhD from the University of Southern California in 2016 and bachelor's degree from Tsinghua University in 2011.