PI logo

Can a model reason in a language of its own?

Seven designs, and what the tests showed

PI Project · R&D notes/ Machine-native reasoning · overview· Note 1 · method & result· Note 2 · the failures· Note 3 · the molecular test

§01 The question

Meaning does not need words to exist

The same thought in several human languages, then in a machine-native code.

You may have heard that prompting a generative model in Mandarin sometimes costs fewer tokens. The reason is instructive: an ideogram is not a letter. It is a graphical unit of meaning, sometimes elementary, sometimes composed, so the same idea can ride a more compact structure without losing its descriptive power.

The point is not to compare human languages. It is that the observation raises a more general question: at what level of abstraction should a thought be represented? When a model reasons "step by step", it writes those steps in human sentences, as if narrating its thoughts aloud. That narration is for us. Nothing says a machine has to reason in English, or in French, to reason well.

Natural language The king fought the dragon and won. 8 tokens
Ideograms 王斗龙胜 4 glyphs
Learned discrete codes 0412008709110733 4 codes

An intuition, not a result: meaning can ride units denser than words, and the units need not be designed by anyone. Whether a model can genuinely reason in such units, and how you would ever know, is what this series tests.

§02 The distinction

Reasoning and explaining the reasoning are two different functions

Two extremes bracket how a model can reason today. A chain of thought in natural language is readable, but verbose: it consumes context, inference time and cost, and it is not always a faithful record of what the model actually computed. The model's raw internal activations are efficient, but completely opaque. Between the two there may be a more useful space: an internal language that is discrete, compressed and learned, whose units stand for concepts, relations or intermediate states rather than words.

One temptation is worth resisting from the start: designing that language ourselves. Nothing guarantees that a human-designed code is optimal for a model, any more than human grammar is. The goal is to create the conditions for an internal language to emerge during training, then to inspect what emerged.

And that is where the real difficulty lives. A machine-native code is, by construction, harder to audit than a sentence. So any system of this kind owes you a proof: perturb the internal code, and the answers must degrade. If they do not, the model is not reasoning through the code. It is bypassing it, and the symbols are decoration. That verification requirement, more than raw performance, is what this research line came to be about.

§03 The design space

Seven ways to wire a discrete reasoning channel

Between October 2025 and January 2026 we explored the design space below. The status column is the honest part: some families were run to a verdict, some were explored and archived, one never left paper. Only the entries marked with a run status carry evidence. The notes hold the records.

VQ bottleneck retrofitted into a pretrained modelOct 2025 · Pythia 410M / 1.4B Forward hooks quantise hidden states on GSM8K. The codebook was used. Accuracy was zero at both scales. Note 2, run 01 → run · negative
Two-pass architecture with a structured IR bufferNov 2025 · Pythia-70M + LoRA The model must emit, then consume, a discrete buffer. Every dashboard gate passed. The content changed nothing. Note 2, run 02 → run · negative
Air-gapped encoder and decoder, trained from scratchDec 2025 · Jan 2026 · 26.5M params The discrete code is the decoder's only input, so necessity becomes measurable. Under intervention, the channel proved to be the sole causal path. Note 1, the method and the run → run · mechanism proof
The same air gap, scaled roughly seven-foldDec 2025 · ~190M params · H100 Identical recipe, larger model. Phase 2 collapsed on an aggressive setup, three days into the H100 run. Gentler hyperparameters were the obvious next attempt, and the run was never relaunched. Note 2, run 03 → run · collapsed
Modular language pivot: translate, reason, translate backseparate translator and reasoner modules A disciplined experiment ledger, but the translators were deterministic parsers on synthetic arithmetic: informative engineering, not evidence about the thesis. Archived. explored · archived
Parallel tracks: readable chain of thought plus dense codecoupled by cross-attention Two coexisting reasoning tracks, one for humans and one for the machine. Designed in detail. Never built. designed · never run
Continuous reasoning with discrete checkpointsthe one forward direction Keep the reasoning continuous, where the field's momentum is, but force it through discrete, auditable checkpoints between segments, so the same causal test still applies. Stated as a direction, not a result. parked direction

§04 Where it stands

What the test settles

The summary fits in four sentences. At small scale, under a test strict enough to fail, a model demonstrably routed its computation through a learned discrete code: corrupt the code and the answers fall apart. The same test refused both designs that produced a model to test, which is what a test that can fail is for, while the seven-fold scale-up collapsed in optimisation before there was a usable checkpoint to test. Larger teams, at Meta and IBM among others, have taken similar mechanisms to scale, so the question these notes answer applies to systems considerably larger than ours. What is documented here is that question and the method that settles it: whether a given internal code carries the reasoning, or only accompanies it.

The three notes below hold the full records: the architecture and the causal-necessity protocol with the run that passed it, the three dead ends with the exact numbers that exposed them, and the molecular design task where the same standard, applied in a second domain, returned a controlled null.

Every factual claim on this page is documented in the three notes and their run records. This overview adds none of its own.