3D-Aware VLMs with Implicit and Explicit Geometries

Wenhao Li, Xueying Jiang, Quanhao Qian, Deli Zhao, Ran Xu, Shijian Lu, Gongjie Zhang · July 23, 2026

Vision-language models (VLMs) like CLIP are great at understanding 2D images, but they often fail at 3D tasks that require spatial reasoning—like detecting objects in 3D space or grounding language in 3D scenes. A new paper introduces VLM-IE3D, a framework that boosts VLMs' 3D awareness by learning geometry from ordinary RGB videos, without needing expensive 3D sensors.

The key idea is to combine two types of geometric tokens: Implicit Geometry Tokens (IGTs) capture high-level shape priors, while Explicit Geometry Tokens (EGTs) encode detailed 3D structures reconstructed from video. A special adapter fuses these with 2D visual features, giving the model strong 3D inductive biases.

Why it matters: This approach makes VLMs much better at tasks like 3D object detection, visual grounding, and spatial reasoning—all from standard video input. For retail investors, this could mean more capable AI for autonomous driving, robotics, and AR/VR applications, potentially reducing hardware costs since no specialized 3D sensors are needed.