Real-Time 3D Video from Multiple Streams: A Smarter Memory Approach

Baback Elmieh, Lynn Tsai, Zeman Li, Srinivas Kaza, Tiancheng Sun, Gabor Csapo, Ali Behrouz, Yuan Deng, Stephen Lombardi, Steven M. Seitz, Xuan Luo · July 16, 2026

Creating dynamic 3D scenes from multiple video streams in real time is challenging because the system must remember what it saw before (to reconstruct hidden areas) while keeping up with live video. Traditional methods update memory at every frame using heavy computations, making real-time use impossible and causing instability over time. This paper introduces a smarter approach: instead of updating memory every frame, it updates memory periodically and applies the stored memory to each frame using cross-view attention to handle motion. Two key innovations—a Memory Loss that forces the model to internalize the scene persistently, and a Memory Caching strategy that prevents forgetting—enable real-time performance with state-of-the-art quality. The method works on dynamic human motion and can remember scenes over minutes. Practical implications: this could power live VR/AR experiences, telepresence, and real-time 3D reconstruction for sports or events, reducing computational costs and enabling longer, stable sessions.