PI logo

Abstract CoT, the price of thinking aloud

A model that reasons in sentences pays for every word, and the sentences are not a faithful record of the computation. Both problems point at the same question.

01 The problem

Reasoning written for humans, billed to the machine

When a model reasons step by step, it writes those steps in human sentences, as if narrating its thoughts aloud. The field calls that a chain of thought, abbreviated CoT, and an abstract CoT is the same reasoning carried in something denser than words. The narration was never chosen because it is what a model needs. It was chosen because we can read it, and because a model trained on our text already knows how to produce it.

Two consequences follow, and they pull in the same direction. The narration is long, so it consumes context, latency and money on every single query. And it is a report about the computation rather than the computation itself, so there is no guarantee that what the sentences say is what the model actually did. Whoever pays the bill and whoever has to justify the answer are therefore looking at the same weakness from two sides.

02 Why it matters

The bill is unpredictable and the record is incomplete

Reasoning tokens are generated before the answer, charged like any other output, and largely invisible to whoever wrote the query. Independent measurements of that hidden volume have found it large enough to reverse the ranking of published prices, and variable enough that the cost of the same question asked twice is not the same. Meanwhile the obligation to keep automatically generated logs of a high-risk system's operation is now written into European law.

32%
of model-pair comparisons in which the model with the lower listed price turned out to cost more, over eight frontier reasoning models and twelve tasks. Source: arXiv 2603.23971, 2026
9.7×
the largest spread in thinking tokens measured across repeated runs of one identical query, so the same question need not cost the same twice. Source: arXiv 2603.23971, 2026
<20%
how often, in most settings measured, a reasoning model mentions in its chain of thought a hint that demonstrably changed its answer. Source: Chen et al., arXiv 2505.05410, 2025
6 months
the minimum period for which a deployer of a high-risk AI system must keep its automatically generated logs, under Article 26(6) of the EU AI Act. Source: Regulation (EU) 2024/1689, Article 26(6)

Cost

Verbosity is charged per token on every query, so the form the reasoning takes is a recurring operating cost rather than a one-off design choice.

Predictability

A per-query cost that varies by an order of magnitude for the same input is difficult to budget, and difficult to guarantee under a latency requirement.

Accountability

A log kept for six months is only evidence if it records what actually drove the answer. A narration that omits the decisive influence satisfies the letter and misses the point.

03 Why it is hard

A denser code is easy to obtain and hard to trust

Compressing the reasoning into something more compact than sentences is not the difficult part. The difficult part is showing that the compact thing carries the reasoning, rather than sitting alongside a computation that would have happened anyway.

Compact is not the same as meaningful

A short internal code can be produced by any bottleneck. Whether it carries the computation, or merely accompanies it, is a separate question that compression alone never answers.

The failure looks like success

A model can route around its own channel while every dashboard metric keeps improving. Nothing in ordinary training curves distinguishes a channel that is load-bearing from one that is decoration.

Designing the language is a trap

Nothing guarantees that a code designed by people is good for a model, any more than human grammar is. The alternative, letting one emerge in training, means inspecting something nobody specified.

Density costs legibility

The compression that saves tokens makes the trace harder for a person to read, so a system of this kind owes its auditor a test rather than a transcript.

04 Where it stands

The line is closed. The test it produced is not

We explored this between October 2025 and January 2026, across seven design families. The line is concluded: the mechanism turned out to be public territory, better-resourced laboratories carried it to scale, and our own scale-up broke in phase two and was stopped.

What survived is more transferable than what was attempted. Scramble the internal code, and a channel that genuinely carries the reasoning must break; one that does not, will not. That is a test any system of this kind can be held to, and the notes record the runs it passed, the runs it failed, and the run that passed every dashboard while carrying nothing.