Transformers Learn Inductive Reasoning Through a Hidden Low-Dimensional Manifold

Tiberiu Musat, Tiago Pimentel, Nicholas Zucchet, Thomas Hofmann · July 13, 2026

A new paper reveals how Transformers develop inductive reasoning abilities by tracking their learning dynamics on a low-dimensional manifold. The authors unify several synthetic reasoning tasks (like in-context n-grams and multi-hop reasoning) into a general class and prove that attention models' training dynamics can be confined to a highly interpretable, low-dimensional invariant manifold. On this manifold, learning is captured by just a handful of coordinates instead of millions of parameters. This framework explains how data statistics drive the competition between in-context learning and in-weights learning, how random initializations determine which circuit wins when multiple solutions exist, and how to automatically detect learned circuits. The practical implication is a step toward a predictive theory of Transformer learning, potentially enabling better model design and interpretability. For investors, this could lead to more efficient training methods and improved performance in reasoning tasks, impacting AI companies developing next-gen language models.