Title: Learning in-the-wild 3D Modeling and Simulation
Abstract: Humans have extraordinary capabilities of comprehending and reasoning about our 3D visual world. With just a few casual glances, we can grasp the 3D structure and appearances of the surroundings and imagine all sorts of “what-if” scenarios in our minds. The long-term goal of my research is to equip computers with similar abilities. In this talk, I will present a few of our efforts toward this goal.
I will start by discussing 3D reconstruction from sparse views, where the camera poses are unknown and the images have little or no overlap. While existing approaches struggle in this setting, humans seem adept at making sense of it. Our key insight is to imitate humans and distill prior knowledge about objects into the algorithms. By leveraging those priors, we can significantly expand the applicable domains of existing 3D systems and unleash the potential of multiple downstream tasks in extreme-view setups. Then, I will talk about how to recover scene representations from a set of images with imperfect camera poses. While most existing works assume perfect calibrations, camera poses are usually noisy in the real world. To this end, we present a coarse-to-fine optimization framework that allows one to jointly estimate the underlying 3D representations and camera poses. The framework can be a plug-n-play to existing methods when transferring to real-world setups. Finally, I will showcase how we exploit “what-if” to build sensor simulation systems for self-driving.
Bio: Wei-Chiu Ma is a final-year Ph.D. candidate at MIT, working with Antonio Torralba and Raquel Urtasun. His research lies in the intersection of computer vision, robotics, and graphics, with a focus on in-the-wild 3D modeling and simulation and their applications to self-driving vehicles. Wei-Chiu is a recipient of the Siebel Scholarship. His work has been covered by media outlets such as WIRED, DeepLearning.AI, MIT News, etc. Previously, Wei-Chiu was a Sr. Research Scientist at Uber ATG Toronto. He received his M.S. in Robotics from CMU where he was advised by Kris Kitani and B.S. in EE from National Taiwan University.