Interpretability
Discovering Cross-Language Reasoning Invariance in LLMs
Shared geometry across languages may reveal common reasoning structure, but representational similarity alone does not establish causal interchangeability.
- Venue
- Workshop paperICML 2026 Workshop on Mechanistic Interpretability
- Role
- First author · method · analysis pipeline
- Themes
- Interpretability
Pipeline: parallel reasoning prompts in several languages produce residual-stream activations at a fixed layer. A geometry-invariant sparse autoencoder learns a shared dictionary. From the dictionary, geometry metrics measure alignment and causal interventions swap or ablate features across languages. The joint reading assigns feature groups to convergent, saturated, or low-sharing regimes.
- GI-SAE
- geometry-invariant sparse autoencoder
- features trained to be invariant to per-language geometric transforms
- Causal
- intervention design
- swap and ablate features across languages and measure downstream reasoning
- 3
- sharing regimes
- convergent, saturated, and low-sharing
01
The problem
If a language model reasons the same way in two languages, its internal representations of that reasoning should be shared. Most evidence for sharing is geometric: activations for parallel inputs line up after a rotation or projection. But geometric alignment is a weak test. Two sets of features can be similar in shape and still do different work when the model actually reasons.
This project asks a sharper question: are the features that carry reasoning in one language functionally interchangeable with those in another? Answering it requires a feature basis that is not an artefact of one language's geometry, and an intervention that tests function rather than similarity.
Held constant across languages: the reasoning task, the prompt structure, and the model layer under analysis. Varied: the language of the prompt and, in interventions, the source language of the features injected.
02
System and method
The GI-SAE analysis pipeline. Parallel reasoning prompts in several languages produce residual-stream activations. A geometry-invariant sparse autoencoder learns a shared feature dictionary that tolerates per-language transforms. Geometry metrics quantify alignment. Causal interventions swap or ablate features across languages and measure the effect on reasoning outputs. The joint reading assigns each feature group to a convergent, saturated, or low-sharing regime.
03
Experimental design
- Research question
- Do LLMs reuse common reasoning structure across languages, and is it causally interchangeable?
- Method
- Geometry-invariant sparse autoencoders trained over parallel multilingual reasoning activations
- Language and model setup
- Parallel reasoning prompts across multiple languages on open-weight models · the exact language and model list is specified in the paper
- Geometry metrics
- Alignment and similarity measures between per-language feature geometries
- Causal intervention design
- Cross-language feature swaps and ablations with downstream reasoning accuracy as the outcome
- Regimes
- Convergent (shared and interchangeable), saturated (aligned but not interchangeable), and low-sharing
04
Main findings
- 01
Geometric alignment and causal interchangeability are different questions
Feature groups can align closely in geometry while cross-language swaps change reasoning behaviour, and vice versa. The two measurements disagree often enough that neither can stand in for the other.
Why it mattersClaims of 'shared multilingual reasoning' based on similarity alone should be read as hypotheses, not conclusions.
- 02
Sharing is organised into regimes rather than a single degree
Feature groups fall into convergent, saturated, and low-sharing regimes that differ in both alignment and causal transferability.
Why it mattersA model can be multilingual in some reasoning components and language-specific in others. Interpretability tooling should report which is which.
- 03
Geometry-invariant training yields a usable cross-language dictionary
Training the sparse autoencoder to be invariant to per-language transforms produces features that can be compared and intervened on across languages without hand-aligning bases.
Why it mattersGI-SAE makes causal cross-language probing tractable as a routine analysis step.
05
What I built
The engineering behind the result. Every number above was produced by systems I designed and implemented.
Activation capture
Hooks for collecting residual-stream activations on parallel multilingual reasoning prompts.
GI-SAE training
Sparse-autoencoder training with geometry-invariance constraints on GPU.
Geometry metrics
Alignment and similarity computations between per-language feature geometries.
Intervention harness
Cross-language feature swap and ablation with automated evaluation of downstream reasoning.
Analysis pipeline
Regime assignment and figure generation from intervention results.
06
Limitations
08
Related research
- EvaluationEvaluation of Multi-Turn Consistency in LLM AgentsReliability is temporal: models fail at different stages and produce systematically different narratives before doing so.Open project
- ArchitectureContext, Reasoning, and Hierarchy in Compound LLM AgentsWhat an agent sees can matter more than how long it deliberates.Open project
Contact
Building reliable AI systems requires both research and engineering.
I’m interested in research engineering, applied research, agent infrastructure, evaluation, reliability, interpretability, and research-to-production work.