SM4RT: Teaching AI to See Motion as Physics, Not Just Pixels

Shing Ho J. Lin, Wenzhao Zheng, Dong Zhuo, Yuqi Wu, Jie Zhou, Jiwen Lu · July 24, 2026

A new paper from Tsinghua University introduces SM4RT, a model that reconstructs 3D scenes and motion from a single video. Unlike previous methods that treat each pixel's movement independently, SM4RT recognizes that objects move as rigid bodies—like a car driving or a person walking—where points on the same object share a common motion. The key idea is "Structure-of-Motion": it decomposes scene motion into a small set of rigid-body transformations (6D twists in SE(3)), then assigns each pixel to one of these motions. This allows the model to infer 3D geometry, motion, and kinematic structure in one pass from a monocular RGB video. Why it matters: Current 4D reconstruction (3D + time) often produces noisy, unphysical motion. SM4RT's physics-aware approach yields cleaner, more realistic results, which could improve applications like autonomous driving, robotics, and AR/VR. Practical implications: More accurate motion understanding could help self-driving cars predict pedestrian movement or enable robots to manipulate objects. The model is efficient—processing video in a single forward pass—making it suitable for real-time systems.