Abstract: Robots in human environments must reason about more than isolated objects: they need to predict how object–object and object–environment relations will change under interaction from partial, noisy perception. This talk brings together a set of recent advances in learning and planning with relational dynamics models for manipulation from partial-view point clouds. We begin with graph- and transformer-based models that infer semantic relations among a variable number of objects (and scene structure) and learn action-conditioned latent dynamics that enable multi-step planning to relational goals, with reliable sim-to-real transfer. We then tackle a core challenge in real-world settings: building rich, object-centric memory so robots can plan over what they cannot currently see—maintaining persistent representations of occluded and out-of-view objects across long interaction sequences—via DOOM and LOOM, which combine object discovery/tracking with trajectory-conditioned transformer dynamics. Next, we introduce Points2Plans, a composable, language-driven framework where a relational dynamics model bridges symbolic task plans and continuous manipulation parameters, enabling zero-shot long-horizon generalization from single-step simulated training. Finally, we present Fail2Progress, which improves robustness when the robot encounters observations outside its training distribution by generating targeted data similar to the failure case in parallel simulation and fine-tuning to reduce repeated failures. Together, these results position relational dynamics as a unifying interface for perception, planning, memory, language grounding, and adaptation in scalable real-world manipulation.

Speaker Bio: Tucker Hermans is an associate professor in the Kahlert School of Computing at the University of Utah, where he is a founding member of the University of Utah Robotics Center. He is also a senior research scientist at NVIDIA. Professor Hermans’ research focuses on autonomous planning, learning, and perception for robot manipulation. His work examines how robots can efficiently manipulate and model objects for which they have no previous knowledge or interaction. Previously, Professor Hermans was a postdoctoral researcher in the Intelligent Autonomous Systems lab at TU Darmstadt in Darmstadt, Germany working with Jan Peters on learning from tactile sensors for manipulation. He was at Georgia Tech from 2009 to 2014 in the School of Interactive Computing where he earned his Ph.D. in Robotics under the supervision of Aaron Bobick and Jim Rehg. He has an A.B. in Computer Science and German from Bowdoin College, as well as an M.Sc. in Computer Science from Georgia Tech.