PI logo

Can a model reason in a language of its own?

The question behind our machine-native reasoning notes: why we asked it, the designs we tried, what survived a strict causal test, and what remains open.

PI Project · R&D notes/ Machine-native reasoning · overview· Note 1 · method & result· Note 2 · the failures

§01 The question

Meaning does not need words to exist

You may have heard that prompting a generative model in Mandarin sometimes costs fewer tokens. The reason is instructive: an ideogram is not a letter. It is a graphical unit of meaning, sometimes elementary, sometimes composed, so the same idea can ride a more compact structure without losing its descriptive power.

The point is not to compare human languages. It is that the observation raises a more general question: at what level of abstraction should a thought be represented? When a model reasons "step by step", it writes those steps in human sentences, as if narrating its thoughts aloud. That narration is for us. Nothing says a machine has to reason in English, or in French, to reason well.

Natural language The king fought the dragon and won. 8 tokens
Ideograms 王斗龙胜 4 glyphs
Learned discrete codes 0412008709110733 4 codes

An intuition, not a result: meaning can ride units denser than words, and the units need not be designed by anyone. Whether a model can genuinely reason in such units, and how you would ever know, is what this series tests.

§02 The distinction

Reasoning and explaining the reasoning are two different functions

Two extremes bracket how a model can reason today. A chain of thought in natural language is readable, but verbose: it consumes context, inference time and cost, and it is not always a faithful record of what the model actually computed. The model's raw internal activations are efficient, but completely opaque. Between the two there may be a more useful space: an internal language that is discrete, compressed and learned, whose units stand for concepts, relations or intermediate states rather than words.

One temptation is worth resisting from the start: designing that language ourselves. Nothing guarantees that a human-designed code is optimal for a model, any more than human grammar is. The goal is to create the conditions for an internal language to emerge during training, then to inspect what emerged.

And that is where the real difficulty lives. A machine-native code is, by construction, harder to audit than a sentence. So any system of this kind owes you a proof: perturb the internal code, and the answers must degrade. If they do not, the model is not reasoning through the code; it is bypassing it, and the symbols are decoration. That verification requirement, more than raw performance, is what this research line came to be about.

§03 The design space

Seven ways to wire a discrete reasoning channel

Between October 2025 and January 2026 we explored the design space below. The status column is the honest part: some families were run to a verdict, some were explored and archived, one never left paper. Only the entries marked with a run status carry evidence; the notes hold the records.

VQ bottleneck retrofitted into a pretrained modelOct 2025 · Pythia 410M / 1.4B Forward hooks quantise hidden states on GSM8K. The codebook was used; accuracy was zero at both scales. Note 2, run 01 → run · negative
Two-pass architecture with a structured IR bufferNov 2025 · Pythia-70M + LoRA The model must emit, then consume, a discrete buffer. Every dashboard gate passed; the content changed nothing. Note 2, run 02 → run · negative
Air-gapped encoder and decoder, trained from scratchDec 2025 · Jan 2026 · 26.5M params The discrete code is the decoder's only input, so necessity becomes measurable. Under intervention, the channel proved to be the sole causal path. Note 1, the method and the run → run · mechanism proof
The same air gap, scaled roughly seven-foldDec 2025 · ~190M params · H100 Identical recipe, larger model. Training collapsed in phase 2 and was never relaunched. Note 2, run 03 → run · collapsed
Modular language pivot: translate, reason, translate backseparate translator and reasoner modules A disciplined experiment ledger, but the translators were deterministic parsers on synthetic arithmetic: informative engineering, not evidence about the thesis. Archived. explored · archived
Parallel tracks: readable chain of thought plus dense codecoupled by cross-attention Two coexisting reasoning tracks, one for humans and one for the machine. Designed in detail; never built. designed · never run
Continuous reasoning with discrete checkpointsthe one forward direction Keep the reasoning continuous, where the field's momentum is, but force it through discrete, auditable checkpoints between segments, so the same causal test still applies. Stated as a direction, not a result. parked direction

§04 Where it stands

One clean proof, three instructive failures, one open question

The honest summary fits in three sentences. At small scale, under a test strict enough to fail, a model demonstrably routed its computation through a learned discrete code: corrupt the code and the answers fall apart. Every attempt to scale that mechanism, or to graft it onto a pretrained model, failed, and the failures taught us more than the success. Meanwhile larger teams, at Meta and IBM among others, took the same mechanism to real scale and real usefulness; their work is the reference point, and what we keep is the verification standard the field mostly does not report.

The two notes below hold the full records: the architecture and the causal-necessity protocol with the run that passed it, and the three dead ends with the exact numbers that exposed them.

§05 The open question

The question in the title is still open

This line began, candidly, as an attempt to run ahead: be early on an original mechanism and build something of our own around it. On that ambition, it failed. The mechanism proved to be public territory, better-resourced labs carried it to scale, and our own scale-up collapsed. The arc is published anyway, because what survived is more transferable than what was attempted: a way of knowing, for any system of this kind, whether its internal code actually carries the reasoning.

So the question in the title remains open, for everyone. Such a channel can exist and be made provably load-bearing; it fails in ways dashboards cannot see; and one test separates the two. If the parked direction comes off the shelf one day, it will run under that same test, and be published either way.

Every factual claim on this page is documented in the two notes and their run records; this overview adds none of its own.