Abstract

This talk introduces world foundation models and their pivotal role in enabling Physical AI. I will begin by tracing the evolution from Generative AI to Physical AI, highlighting the unique challenges that arise in this transition. Two categories of world foundation models will be presented: synthesis models, which predict future world states, and analysis models, which interpret current states to guide action planning. I will showcase how these models are applied in Physical AI systems, review the current state of the art, and conclude with a discussion on the limitations of existing approaches and promising directions for future research.

Abstract

Modern foundational models have achieved superhuman performance in many logic and mathematical reasoning tasks by learning to think step by step. However, their ability to understand videos, and, consequently, control embodied agents, lags behind. They often make mistakes in recognizing simple activities, and often hallucinate when generating videos. This raises a fundamental question: What is the equivalent of thinking step-by-step for visual recognition and prediction?

In this talk, we argue that step-by-step visual reasoning has much to do with inverting a physics simulator, that is, mapping raw video pixels back to a structured, 3D-like neural representation of the world. This involves inferring 3D representations of objects, parts, their 3D motion and appearance trajectories, estimating camera movements and 3D scene structure and physics properties. We will discuss methods to automatically extract such 3D neural representations from images and videos using generative model priors and end-to-end feed-forward models. We will present methods that inject such knowledge of camera motion and 3D scene structure in modern VLMs and show it improves their ability to ground language and control robot manipulators.

How can we scale up annotations for such simulator inversion? We will discuss methods that use generative models of language and vision to automate development of 3D simulations in physics engines. Additionally, we will discuss our efforts in developing faster and more general physics engines. The integration of physics engines with generative models aims to automate the replication of real physical environments within the physics simulator, enabling more accurate and scalable world simulation for sim-to-real learning of 3D perception and robotics. We believe such real-to-sim and sim-to-real learning as universal data engines for robotics are central for democratizing and pushing the state-of-the-art in robot learning.

Abstract

3D Computer Vision is crucial for understanding and interpreting the spatial aspects of real-world scenes. It is particularly important for the coming Embodied AI, where machines need to interact with and understand their surroundings. Sensing is crucial in this context, because it generates rich, multidimensional data that enhances AI's understanding of the world and elevates its perceptual capabilities. This talk discusses two approaches to the problem of real-world sensing, namely, learning-based and model-based approaches and advocates the synergy of these approaches. In particular, we discuss the case of photometric 3D reconstruction, where we have an access to reliable physics-based models while data-driven methods are still beneficial.