Xunjian Yin / Research

ICLR 2026Representation geometry

The Geometry of Reasoning: Flowing Logics in Representation Space

Yufa Zhou1*, Yixiao Wang1*, Xunjian Yin1*, Shuyan Zhou1, Anru R. Zhang1

1 Duke University · * Equal contribution

Paper arXiv Code

When the same logical argument is expressed across topics and languages, changes along its representation trajectory reveal more shared logical structure than embedding positions alone.

Result
For Qwen3-0.6B, curvature similarity is 0.53 between flows sharing logic vs 0.13 between flows sharing language; position similarity reverses the order (0.26 vs 0.85).
New vs prior
The Linear Representation Hypothesis treats concepts as linear directions, and graph views treat chain-of-thought as discrete nodes; this work models reasoning as a continuous flow with velocity and curvature.
Limitation
The study analyzes understanding rather than generation, so the paper does not relate its geometric measures to the accuracy of model outputs.
Reasoning flows: 3-D and 2-D PCA projections of step-wise representation trajectories for one MATH500 problem with six distinct answers (left, center), and the paper's schematic of mappings among input space X, concept space C, logic space L, and representation space R (right). Original Figure 1 from the paper.
Scroll sideways to read the figure. Reasoning flows: 3-D and 2-D PCA projections of step-wise representation trajectories for one MATH500 problem with six distinct answers (left, center), and the paper's schematic of mappings among input space X, concept space C, logic space L, and representation space R (right). Original Figure 1 from the paper. Full size ↗
01 / The research question

Similar meanings can obscure similar reasoning

An argument’s representation mixes topic, language, entities and logical structure. Two English passages about weather can sit close together even when their deductions differ, while one deduction in two languages can sit far apart, so comparing isolated embeddings risks measuring surface content.

Take “If it rains, the ground is wet; it rains; therefore the ground is wet.” Replacing weather with servers (“If a server is overloaded, latency rises; …”) changes the concepts but keeps the inference. The paper applies this controlled variation to longer natural-deduction structures across topics and languages.

Closest prior work

The Linear Representation Hypothesis holds that concepts align with linear directions in embedding space, and later work finds multi-dimensional, manifold-like and compositional features. Graph-based accounts model chain-of-thought as discrete nodes (Wang et al., 2024; Minegishi et al., 2025), which the paper argues fits random walks between isolated embeddings but misses the smooth, directed dynamics under cumulative context. This work instead treats reasoning as a continuous flow and tests whether its velocity and curvature track logical structure across topics and languages.

02 / Inside the method

Measure how a trajectory changes from step to step

Each reasoning step is represented by the average of its token representations at the final transformer layer, so a trajectory follows successive text steps at one layer, not one token through successive layers.

Illustrative trajectories · paper’s formulas (Sections 3–4)

Build two reasoning flows and compare their geometry

Pick a logic, topic and language for each flow. The trajectories are illustrative 2-D paths, but position, velocity and curvature similarity are computed with the paper’s own formulas. Press Play to draw each flow step by step.

Two illustrative reasoning trajectoriesFlow 1 and Flow 2 drawn as step points joined by velocity arrows.Flow 1Flow 2
Flow 1Flow 2Velocity arrow (Δy per step)
Position similarity+0.94

mean cos(y · follows topic and language

Velocity similarity+0.82

mean cos(Δy · follows logic

Curvature similarity+1.00

Pearson of Menger κ · follows logic

Flow 1 (Logic A, Weather, English) vs Flow 2 (Logic A, Sports, Chinese): position +0.94, velocity +0.82, curvature +1.00. Same logic under different carriers keeps curvature aligned.

Measured · Table 1 (Qwen3 0.6B) On the paper’s real embeddings, mean similarity by grouping criterion is: position 0.26 / 0.30 / 0.85, velocity 0.17 / 0.07 / 0.08, curvature 0.53 / 0.11 / 0.13 for logic / topic / language. Random-shuffle of the step order drops velocity to 0.02 and curvature to 0.02 while position stays 0.30. Values from Table 1; the trajectories above are illustrative, not these embeddings.

The trajectories are illustrative 2-D paths, not the paper’s embeddings. A logic template is a fixed sequence of turns; topic and language are modelled, as in Section 4, as a translation, rotation and scaling of that path. The three numbers are the paper’s own quantities, computed in your browser: position and velocity are mean cosine similarity of the step positions and of the step differences Δyt=yt−yt−1; curvature is the Pearson correlation of the Menger curvature sequences κt=4·Area(yt−1,yt,yt+1)/(abc) (Section 3). Because a similarity transform preserves curvature up to a constant, two flows that share a logic keep curvature correlation near 1 across topics and languages, while absolute position tracks the carrier; shuffling the step order collapses velocity and curvature but not position. No model is run.

Position is where a step is embedded, velocity is the difference between neighboring steps, and curvature is the Menger curvature of three consecutive points, which measures how sharply the path turns. Position and velocity are compared by cosine similarity, curvature by Pearson correlation. Differences reduce the weight of components shared across a passage, such as its language or topic.

The theory treats the steps as samples from a continuous curve on which logical statements act as local controllers of velocity. It therefore predicts shared geometry for the same logic under different semantic carriers, which the experiment tests by grouping flows by logic, topic and language.

  1. 01

    Hold the logic fixed

    Express natural-deduction structures through different topical and linguistic contexts.

  2. 02

    Trace reasoning steps

    Pool final-layer hidden states for each step to form a representation trajectory.

  3. 03

    Compare trajectory geometry

    Measure similarity in position, stepwise velocity, and curvature across matching logic, topics, and languages.

03 / Reading the evidence

Positions group by language; velocity and curvature group by logic

Position similarity is highest for flows that share a language; velocity and curvature similarity are highest for flows that share logic. Randomly permuting the logical steps keeps the same sentences but collapses velocity and curvature similarity, while position similarity stays high: the logical signal lies in the order of the steps.

The pattern holds across six models (Qwen1.5-0.5B, Qwen2-0.5B, Qwen3 at 0.6B, 1.7B and 4B, and LLaMA3-8B) and changes little with model size or family.

Table 1 (Qwen3 0.6B row, rearranged with one row per measure). Mean cosine similarity (position, velocity) and Pearson correlation (curvature) between reasoning flows that share the same logic, topic, or language; higher means more similar. Each measure has its own scale, so compare groupings within a row, not values across rows. Source ↗

MeasureLogicTopicLang.
Position similarity0.260.300.85
Velocity similarity0.170.070.08
Curvature similarity0.530.110.13

Scope. Evidence comes from controlled reasoning sequences and representation proxies. Geometric correlations support the framework but do not establish a causal mechanism or universal theory of reasoning. Read the study ↗

Further reading in the paperRepresentation trajectories and geometric quantities · Sections 3–4 ↗Controlled semantic variation and Table 1 · Sections 5–6 ↗
Research summary

We study how large language models (LLMs) “think” through their representation space. We propose a novel geometric framework that models an LLM's reasoning as flows—embedding trajectories evolving where logic goes. We disentangle logical structure from semantics by employing the same natural deduction propositions with varied semantic carriers, allowing us to test whether LLMs internalize logic beyond surface form. This perspective connects reasoning with geometric quantities such as position, velocity, and curvature, enabling formal analysis in representation and concept spaces. Our theory establishes: (1) LLM reasoning corresponds to smooth flows in representation space, and (2) logical statements act as local controllers of these flows' velocities. Using learned representation proxies, we design controlled experiments to visualize and quantify reasoning flows, providing empirical validation of our theoretical framework. Our work serves as both a conceptual foundation and practical tools for studying reasoning phenomenon, offering a new lens for interpretability and formal analysis of LLMs' behavior.

Full paper ↗
Citation BibTeX 
@inproceedings{zhou2026the,
  title={The Geometry of Reasoning: Flowing Logics in Representation Space},
  author={Yufa Zhou and Yixiao Wang and Xunjian Yin and Shuyan Zhou and Anru Zhang},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=ixr5Pcabq7}
}