§01 Why publish failures
A channel can be used, accurate, and carry none of the reasoning
When a latent-reasoning idea fails, it usually fails quietly. The codebook fills up, the loss goes down, accuracy clears its gate, and nothing in the training curve tells you the channel is hollow. You only find out if you go looking with a counterfactual: corrupting the channel's content and watching whether the answers notice.
We went looking three times in late 2025, and the work broke three different ways: one loud zero, one outright collapse, and one failure so quiet that every dashboard metric called it a success. These are the full records, kept because negative results produced under a strict test are evidence, and because the quiet failure mode is now independently documented in the literature. Each run's lesson fed the next design; the companion note records where the same test finally came back positive.
GSM8K · both scales · axis 0–100%
The cheap route: retrofit a bottleneck
First attempt, October 2025: install a hard VQ bottleneck into pretrained Pythia models (410M and 1.4B parameters) via forward hooks, and train on GSM8K, real grade-school maths word problems. If reasoning could be routed through a discrete code this cheaply, everything downstream would be easier.
The dashboard said the mechanism worked
And mechanically, it did. The quantisation was live, codes flowed: 61% of the codebook in active use at both scales, around 180 codes emitted per problem at 410M. Watching codebook utilisation, which is what this kind of work usually reports, the experiment looked healthy.
Accuracy: zero. At both scales.
Task accuracy was exactly 0%, at 410M and at 1.4B. The scale-invariance is the diagnostic: if the problem were capacity, a 3.4× parameter jump would move the number. It did not. The model compressed information into the codes and then routed around them; the codes were on the forward path but not on the causal path.
Lesson: utilisation is evidence of quantisation, not of reasoning
Building a discrete bottleneck is easy; making it causal is hard, and adding parameters to a non-causal bottleneck buys nothing. Here the competence bar itself failed loudly. The next run passed the easy bars, which is what made it dangerous.
accuracy by IR condition · 150 examples · axis 0–8%
→ identical to two decimals under content corruption; zero only when the channel is absent
Force the model through the channel, structurally
November 2025, applying run 01's lesson: a two-pass architecture on a Pythia-70M backbone adapted with LoRA. Pass one emits a structured IR buffer: scaffolding tags around 512 learnable discrete codes; pass two must consume that buffer through cross-attention to answer a small arithmetic-reasoning task. This time the channel is architecturally in the loop.
Every gate we had defined came back green
Validation accuracy 6.27% against a >5% gate. IR structural integrity 100%. Codebook utilisation 26.95% in evaluation (138 of 512 codes) against a >20% gate. Loss falling, structure valid, codes diverse. By every metric a training curve can show, this channel was alive.
The counterfactual: content changes nothing
Then the intervention table. Remove the IR entirely and accuracy goes to zero: the answer pass cannot even start without it, so the channel is structurally necessary. But randomise it, or shuffle it, and accuracy does not move: 6.00% intact, 6.00% random, 6.00% shuffled, to two decimals. The model needs a token stream in that slot and is completely indifferent to what it says.
Our pre-registered causality gate wanted an average relative drop of ≥70% under content corruption; this scored 33.3%, carried entirely by the tautological drop case.
Lesson: a positional placeholder can pass every dashboard gate
The IR acted as a "start thinking here" delimiter while the real computation bypassed its semantics: a retrofit onto a backbone whose answer path was never sealed off from the input, and gradient descent took the cheap route. This is the dangerous failure: the only instrument that saw it was the counterfactual. It is why the air gap in note 1 exists, and it is the failure mode "Do Latent Tokens Think?" later documented independently in other latent-reasoning systems.
phase-2 output stream · epochs 3–8
stopped at epoch 8 of 15 · flat across evals · no recovery
The real test: scale the design that worked
December 2025. The air-gapped architecture had passed the causal test cleanly at 26.5M parameters (note 1), so we scaled the same recipe roughly seven-fold: two 12-layer transformers at d=768, a 4,096-entry codebook, trained on an H100 with the identical two-phase schedule.
Phase 1 completed. Phase 2 fell apart.
Partway into the task fine-tune the outputs degenerated into the unknown-token filler and never recovered: arithmetic 0%, story F1 around 0.06, flat across evaluations. This is a different pathology from runs 01 and 02: not a hollow channel but an optimisation collapse, the known brittleness of hard discrete bottlenecks under an aggressive setup, where quantiser and decoder drift out of step until the decoder emits the safest token it knows.
What the record does and does not support
We stopped at epoch 8 of 15, after ~3 days and ~€300 of compute. An evaluation bug in that run (only the last item per batch was scored) means the fine-grained numbers are not trustworthy; the collapse itself, the 0% arithmetic and the unusable phase-2 checkpoint are solid. The phase-1 checkpoint exists but is unvalidated. Gentler hyperparameters were the obvious next attempt; the run was never relaunched, and it stands in the repository as an archive, not a result.
Lesson: the gap between mechanism proof and system is where this breaks
A discrete bottleneck that trains cleanly when small can collapse outright when larger under the same recipe. Between the 26.5M proof and anything deployable sits an unsolved optimisation problem, and we say so, rather than presenting the small-scale result as if it generalised.
§05 Reading them together
The dangerous failure is the one only a counterfactual catches
| Run | Backbone | Channel used? | Channel causal? | Failure mode |
|---|---|---|---|---|
| 01 · vq_bottleneck | Pythia 410M / 1.4B | yes · 61% utilisation | no · 0% accuracy, scale-invariant | used, not causal |
| 02 · seed_emergent | Pythia-70M + LoRA | yes · 27% utilisation, 100% integrity | no · inert under shuffle & random | positional placeholder · the quiet one |
| 03 · air_gap V18 | from scratch, ~190M | n/a | n/a | optimisation collapse in phase 2 |
Three runs, three different breaks, and the differences are the point. The scale-up announced itself with a collapse; the retrofitted bottleneck announced itself with a flat zero. Loud failures are easy to respect. Run 02 was the dangerous one: utilisation, integrity, loss and even accuracy all said it was working, and only corrupting the channel revealed that its content did nothing. That is the failure mode our causal-necessity test exists for, and it is why, the one time the same test came back positive, we believe it.
It is not only us. "Do Latent Tokens Think?" (arXiv 2512.21711) reached the same conclusion from a different direction, finding that a popular family of latent-reasoning tokens often functions as uninterpretable placeholders leaning on dataset artefacts rather than performing computation. Two independent setups converging on one failure mode is the best evidence that the mode is real, and not a quirk of our code.
Referenced: Do Latent Tokens Think? (2025)
§06 Limits
How hard to lean on these negatives
- Not a controlled ablation. Three separate dead ends on different backbones and tasks that happen to rhyme, not one variable isolated three times.
- Run 03 has no clean metrics. The collapse, the 0% arithmetic and the unusable checkpoint are solid; any finer figure from that run is not, and its ~190M size is computed from the training script, not from a run log.
- Run 02's model is weak in absolute terms (~6% accuracy), so the placeholder finding is a clean relative result (content-invariance) on a weak model, not a statement about strong ones.
- Single runs. None of the three is multi-seed. The qualitative conclusions (inert, non-causal, collapsed) are robust; the exact figures are single-run.
§07 End of log
Why these records exist
The gates in these runs were set before the runs, and the outcomes are reported against them, whichever way they fell; nothing here was rescued in the telling. The records are kept because a negative result produced under a pre-registered gate is evidence, and because run 02's failure mode is invisible without a counterfactual. Anyone building a latent channel will meet it, and the table that exposes it costs one evaluation pass.
◆ Provenance · what each claim rests on
This is a concluded research line, published for its method value. The verdicts above describe research prototypes from that line and nothing else.