Kavli Affiliate: Cheng Peng
| First 5 Authors: Taiming Lu, Tianmin Shu, Junfei Xiao, Luoxin Ye, Jiahao Wang
| Summary:
Understanding, navigating, and exploring the 3D physical real world has long
been a central challenge in the development of artificial intelligence. In this
work, we take a step toward this goal by introducing GenEx, a system capable of
planning complex embodied world exploration, guided by its generative
imagination that forms priors (expectations) about the surrounding
environments. GenEx generates an entire 3D-consistent imaginative environment
from as little as a single RGB image, bringing it to life through panoramic
video streams. Leveraging scalable 3D world data curated from Unreal Engine,
our generative model is rounded in the physical world. It captures a continuous
360-degree environment with little effort, offering a boundless landscape for
AI agents to explore and interact with. GenEx achieves high-quality world
generation, robust loop consistency over long trajectories, and demonstrates
strong 3D capabilities such as consistency and active 3D mapping. Powered by
generative imagination of the world, GPT-assisted agents are equipped to
perform complex embodied tasks, including both goal-agnostic exploration and
goal-driven navigation. These agents utilize predictive expectation regarding
unseen parts of the physical world to refine their beliefs, simulate different
outcomes based on potential decisions, and make more informed choices. In
summary, we demonstrate that GenEx provides a transformative platform for
advancing embodied AI in imaginative spaces and brings potential for extending
these capabilities to real-world exploration.
| Search Query: ArXiv Query: search_query=au:”Cheng Peng”&id_list=&start=0&max_results=3