Test-Time Compute Allocation

Concept page. Created 2026-08-16.

The question is no longer how much compute to spend at inference. It is where. Given a fixed hardware budget and a query, which of the many places compute could go should receive the next unit: a longer chain, another parallel sample, a deeper branch off a promising prefix, a stronger model, or nothing at all.

This page exists because the wiki has accumulated at least eight results on the question without a page to hold them, which the lint rule flags as a gap once a concept appears in three or more sources.

2026-09-15: the breadth-over-depth claim gets a scale test and, for the first time, a crossover point you can measure

Elo-per-token analysis (09-15) (arxiv 2609.15309, HuggingFace daily) is the measurement apparatus this page has needed since it was created. An agent that revises, calls tools, branches and decides when to stop chooses its own compute, so performance-against-compute has never been plottable. The method: study open-ended tasks that emit a continuous score for every intermediate submission, track the best solution found at each token budget, and use a Bradley-Terry model (the standard way to turn pairwise win-loss records into one rating) to pool within-task orderings into Elo ratings comparable across tasks with different score scales. Four agents, four benchmarks, sessions up to 100M tokens.

The curve has a shape and the shape has a name. Independent sampling is the theoretically characterized reference, where Elo grows linearly with log compute. Agents beat that line early, which is the justification for scaffolding at all, then decay, then fall below it. The crossing is the scaling inflection point: the per-session budget at which marginal Elo gain equals the independent-sampling reference. Past it, the scaffold is worse than drawing more independent samples and picking the best.

This is a direct confirmation of what this page recorded on 09-08, from a different setup and two orders of magnitude more compute. Revision propagation (09-08), which benchmarked propagating a local edit through every dependent part of a multi-turn artifact, found parallel sampling with selection beats sequential self-revision and beats more reasoning tokens on one attempt, by 2.2 to 9.7 points at fixed budget, and this page read the mechanism as recall-versus-depth. Elo-per-token generalizes it: splitting 100M tokens across parallel sessions sized at the inflection point gains +264 Elo over one long session and +355 over ten short ones. Same bill, different allocation. Two independent results in one week is a pattern, and the page's default should now flip: breadth is the default allocation and depth is what needs justifying.

What is genuinely new is that the crossover is a schedulable quantity. "How long should one session run" stops being a hyperparameter someone picked and becomes something measured per task family. That also makes the number the obvious thing to route on, which gives llm-routing a second lever orthogonal to model choice: not only which model, but how many parallel sessions capped at what budget.

A social-sourced Anthropic result the same day may be the mechanism. It reports that unconstrained test-time compute triggers inverse scaling through distractor fixation, template forcing, spurious-correlation drift and self-correction loops, with one reported drop from 70% to 30% accuracy on dense constraint tracking. Those are failure modes; Elo-per-token is a budget curve. If the marginal token past the inflection point is more likely spent second-guessing a valid intermediate step than finding a better one, the curve bending below the sampling reference is exactly what you would observe. Nobody has tested the link and it is cheap: instrument trajectories on both sides of the inflection point and check whether the named failure modes concentrate past it. That is the sharpest open experiment on this page.

Standing caveat, and it is the one that most limits the practical value. The four benchmarks were chosen because they emit continuous intermediate scores, which is what makes the trajectory observable at all. Production agent work has no such signal, so locating the inflection point in the wild needs a proxy the paper does not supply. And everything is priced in tokens: in real serving, parallel sessions share a KV prefix, so ten sessions do not cost ten times one, and the allocation that wins in tokens may not win in dollars.

2026-09-08: a new task says the field over-buys depth and under-buys breadth, and the deletable-selection-layer pattern gains a setup-layer instance

Revision propagation (09-08) (arxiv 2609.03254, Daisuke Kikuta, NTT, HuggingFace daily) benchmarks a task this page had no instance of: propagating a local edit through every dependent part of an artifact built over a multi-turn conversation, where the dependencies were established in dialogue and appear nowhere in the artifact itself. Holding the budget fixed and enumerating the available forms of test-time compute, it finds parallel sampling with selection beats sequential self-revision and beats spending more reasoning tokens on one attempt, for 2.2 to 9.7 percentage points.

The mechanistic reading is the transferable part: dependency discovery is a recall problem, not a reasoning-depth problem. Any single pass has some probability of overlooking a dependent, and thinking harder in one pass does not reliably fix an omission because the model does not know what it missed. Independent draws give union-like coverage at the recall step. That predicts the gain should scale with the number of dependents rather than with reasoning difficulty, which would explain the wide 2.2-to-9.7 range and is a claim the paper does not make.

It is the second result in four days finding coverage, not depth, was the binding quantity. Locked at the Entrance (09-05) found that reinforcement learning with verifiable rewards destroys up to 67% of a policy's solution coverage at the very first token, and recovers 37% of it for free by interpolating with checkpoints already sitting on disk. Different subfield, same verdict: the field buys sequential depth by default and the scarce quantity is breadth. Given that most production reasoning budgets are spent on longer traces rather than more samples, that is a live allocation error and not a research curiosity.

It also extends this page's discipline thread. Select, Compress, Reinvest (09-05) established that compression pays nothing on its own and only earns accuracy when the freed budget is reinvested (eight query-selected frames beating sixteen uniformly spaced ones by 6.9 points), and CLEAR rations inference compute across a batch with a single shadow price. This paper runs the same protocol on a new task: fix the budget, enumerate the forms of compute, measure which converts into accuracy. That protocol keeps producing this page's most reusable findings and is still rare.

One instance to watch: "selection" is unspecified, and this page has a pattern about that. Picking among parallel candidates requires a selector, and the accumulating pattern here is that purpose-built learned selectors are frequently replaceable by something trivial. Random Attention (09-04) deleted the KV importance scorer and matched the best evictor at 32-43% higher throughput; Select, Compress, Reinvest found a 1990s Orthogonal Matching Pursuit matching every purpose-built video frame selector; ACE (09-08) adds a setup-layer instance by deleting the calibration set and the training step entirely for MoE expert skipping, which is a stronger version of the pattern because a calibration-free method has no distribution-shift failure mode at all. Whether revision-propagation selection is majority voting, an LLM judge, or a trained scorer decides both its cost accounting and whether the result is robust, and the abstract does not say.

Finally, a cost-allocation observation that belongs to the harness rather than the model. The practical takeaway is that for revision requests specifically, the marginal inference dollar should go to parallel candidates rather than a longer trace. That is a per-turn test-time-compute policy switch conditioned on turn type, it sits in the harness rather than in the weights, and nothing ships it today.

The five decisions

Every result below is answering one of these, and almost none of them answer two.

Current state (as of 2026-08-16)

The across-trajectory axis just opened, and it is the first one that is a serving-stack change rather than a model change. Gambit (08-16), from Princeton, MIT CSAIL and Meta AI, formalises test-time reasoning as constrained compute allocation over partial trajectories and shows that the two existing paradigms fail in opposite directions. Independent parallel sampling learns nothing from a trace that collapsed at step three and is capped by KV cache pressure rather than by FLOPs. Subtractive pruning (DeepConf, STEP, Slim-SC) frees that memory and then wastes it, starving the hardware and never shifting the output distribution. Gambit adds the missing half: kill weak traces, immediately re-branch from strong prefixes, keep utilisation high. Up to +6.7 points on HMMT-24, >2x throughput, and up to 68.5% fewer total tokens.

The empirical premise underneath it is worth stating separately, because it constrains everything on this page: successful and failed reasoning trajectories share their opening steps and diverge later, so useful computation is disproportionately concentrated in early intermediate states. If that holds generally, then late-stage compute is systematically the least valuable compute, and every method here is really a way of noticing early.

The whether-to-spend-at-all axis got its first serious treatment the same day. CaRL (08-16) documents universal capability overreach: models handed beyond-capability tasks produce long, expensive, specious derivations rather than stopping, and the miscalibration is systematic. Its hindsight refusal augmentation converts observed failures into refusal supervision, so the boundary is learned from where the model actually fails rather than from a difficulty heuristic. Gambit optimises within an attempt, CaRL decides whether to attempt, and nobody has composed them.

There is a live methodological objection to the entire page. Sampling Luck Masquerades as Allocation Gain (Kurate cs.LG #6, week of 2026-08-16) audits test-time budget allocation for neural combinatorial optimization and argues reported allocation gains are frequently the variance of drawing more samples rather than a real allocation effect. Any method that changes the effective number of independent draws, which is most of them, is exposed. Gambit's partial defence is that it reports lower total token consumption alongside higher accuracy, which sampling luck cannot produce, but it does not run the audit. Nobody on this page has run it.

Aggregation over samples is weakening as an assumption. When Self-Consistency Backfires (Kurate cs.AI #1 this week) finds majority vote hurts on the majority of hard science problems for small models, and Are You Sure You're Sure? (08-16) finds instruction tuning consistently collapses cross-rationale diversity while leaving accuracy roughly unchanged. Self-consistency is the baseline nearly every result here is measured against, and two independent papers this week say the baseline is less sound than assumed, from the aggregation side and from the diversity side respectively.

What the allocation is actually constrained by

Worth naming because papers say "compute budget" and mean different things.

• KV cache memory, which is what caps batch size for long reasoning contexts and is therefore the real constraint on parallel sampling. See KV Cache.
• Hardware utilisation, which is what subtractive pruning sacrifices and Gambit protects.
• Tokens, which is what the customer is billed for, and which does not convert cleanly to dollars across vendors: the same text can cost 34.5% more tokens under one tokenizer than another.
• Wall-clock, which is what an interactive product is constrained by and which almost no paper here reports.

Open problems

1. Run the sampling-luck audit on a reasoning benchmark. It has been run on combinatorial optimization. Until it is run on HMMT or AIME, every allocation gain on this page has an unexamined confound.
2. Compose the gate with the rationer. CaRL at the door, Gambit inside. Both are cheap, both are policy-level, and the combination is unbuilt.
3. Route on capability overreach instead of refusing. CaRL's detector fires on "this model cannot do this," which is a routing signal, not a stopping signal. Escalating to a stronger model is strictly better than quitting for any user willing to pay, and LLM Routing has no paper proposing it.
4. Report wall-clock and dollars, not just tokens. With Optima (08-16) shipping per-task cost and time measurement, the excuse that the instrument does not exist has expired.

Related pages

• KV Cache
• LLM Routing
• Knowledge Distillation
• Speculative Decoding

2026-08-28: a new category, spending test-time compute on the weights

Everything else on this page allocates inference compute: how many samples to draw, how long to think, how to ration a fixed budget across a batch of queries. TTPO (arXiv 2608.27448) spends test-time compute on gradient updates to the model, which this page has no prior entry for.

The mechanism in brief: sample rollouts on unlabeled test questions, take a majority-vote pseudo-label, then treat the two branches asymmetrically because rollouts that disagree with the vote are usually wrong regardless of whether the vote was right. Agreeing rollouts get dense on-policy self-distillation; disagreeing ones get a grouped RL penalty restricted to confident errors. Without labels it matches label-supervised on-policy self-distillation on five competition benchmarks and takes Qwen3-1.7B from 38.0% to 45.2%.

The result that belongs on this page specifically is +25.2% to +36.4% "without thinking." That is a direct substitution between two test-time budgets the field has treated as unrelated: TTPO recovers a large share of chain-of-thought's benefit by moving the spend out of generated reasoning tokens and into a weight update. If it holds, "how much test-time compute" stops being one dial and becomes an allocation problem between reasoning tokens and adaptation steps, which is the same shape as this page's existing batch-rationing results but across a boundary nobody was pricing.

The comparison the paper owes and does not give. Test-time training means N rollouts plus optimizer steps per test distribution. The honest baseline is spending that identical compute on more samples plus majority voting at inference, which needs no gradients, no optimizer state and no infrastructure. Until that number exists, TTPO is a capability claim at an unknown price. Note also that adapting weights at test time breaks reproducibility and complicates rollback and audit, which is why this will likely appear first in batch and offline scoring, where the adapted checkpoint can be pinned, rather than in interactive serving.

Related on the same day: Evolution Strategies vs GRPO is the other half of the Pass@K story.

2026-08-29: a sixth decision, when to spend it

The five decisions in the table above all assume the compute is spent at inference. CritICL (08-29) (arXiv 2608.27455, COLM 2026) argues a large share of it can be moved to a build step. Its premise is that a model family's failure modes are structured and transfer across scale, so you can profile a small model's failures once offline, write natural-language critiques of them, store them, and at inference retrieve the relevant critique as an in-context example. The strong model gets targeted guidance on how it is likely to go wrong for the cost of one retrieval and one generation, and the paper reports being competitive with or better than test-time scaling at significantly fewer generations and lower token cost.

Add a row to the table: "When to spend it" — inference, or a build step? (CritICL, 08-29).

The retrieval objective is the transferable idea. Standard in-context-learning retrieval ranks demonstrations by semantic relevance to the query; CritICL ranks by failure relevance, which example addresses the error this query is likely to induce. Those are different orderings over the same corpus.

Two days, two substitutions in the same direction, no mutual citation. TTPO (08-28) moves the spend from generated reasoning tokens into a weight update and reports +25.2% to +36.4% "without thinking." CritICL moves it from generated tokens into a stored text corpus. The difference is auditability: TTPO's artifact is a per-test-distribution weight delta that breaks reproducibility; CritICL's is a repository you can inspect, version and diff. For anything that has to be audited, that is a materially better shape for the same trade.

It is also one of the few entries on this page the sampling-luck objection does not obviously reach. Sampling Luck Masquerades as Allocation Gain argues reported allocation gains are frequently the variance of drawing more samples. CritICL reports gains at fewer generations, and sampling luck cannot manufacture accuracy while reducing sample count. That is a structural defence, not a run of the audit, but it is more than any other method here has.

Open problem 5 (new): publish the build-step cost. CritICL compares against test-time scaling on generation count, not on total compute including offline profiling. That is the third paper in three days to delete a serving-time dependency without pricing what replaced it, after Self-OPD (K full trajectory rollouts per timestep) and TTPO (N rollouts plus optimizer steps per test distribution). The comparison is cheap and nobody is running it. It finds GRPO exhibits entropy collapse, lifting Pass@1 while flattening Pass@K, where Evolution Strategies lifts both. That matters here because every best-of-K and agentic-sampling pipeline on this page is paid for in Pass@K, and the finding says the standard post-training recipe has been destroying the property those pipelines depend on. It also partly reframes the AIMO 3 result (04-17) that prompt diversity is a dead end for inference-time scaling: the narrowness AIMO 3 could not fix from the prompting side may have been created by the training algorithm. Whether an ES-trained model makes inference-time diversity methods work again is untested and cheap to test.

2026-09-05: a seventh decision, what to do with the saving

Every method above spends a budget. None of them says what happens to a budget you free up, and Select, Compress, Reinvest (09-05) (arXiv 2609.03820) shows that is the decision carrying the accuracy. It runs a controlled study of visual-token allocation in long-video models, holding the frame scorer, prompt boundary, resolution policy and answering model fixed and varying one decision at a time across six training-free selection rules, three benchmarks and two answering models. Halving each frame's spatial budget at fixed timestamps costs at most 0.44 points, which reads as a free compression win. Spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points. Compression on its own buys nothing. It buys something only once the saving is reinvested.

Add a row to the table: "What to do with the saving" — bank it as a lower bill, or reinvest it into more of the thing you were budgeting? (SCR, 09-05).

This retro-explains Gambit (08-16), which is the only prior entry that implicitly answers the question. Gambit's stated advantage over subtractive pruning methods (DeepConf, STEP, Slim-SC) is that they free KV memory and then waste it, starving the hardware; Gambit kills weak traces and immediately re-branches from strong prefixes, keeping utilisation high, for up to +6.7 points on HMMT-24, over 2x throughput and up to 68.5% fewer total tokens. That is reinvestment, in a different modality, without the ablation. SCR supplies the ablation.

It also puts a number on why cross-paper comparison here is unsafe. SCR measures a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget, and it found an implementation bug in its own baseline. A 3.74-point harness artifact is larger than most of the margins the frame-selection literature reports against itself, which is a concrete instance of the methodological objection this page already carries from Sampling Luck Masquerades as Allocation Gain. Different confound, same conclusion: the number in the table is not the number you would get.

Selection beats budget, and the winning selector is not new. Eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points on LongVideoBench's hour-long bin. And Orthogonal Matching Pursuit, an unmodified sparse-approximation algorithm from the early 1990s, matches or comes within a point of every purpose-built selector, across all three benchmarks. Read together with Random Attention (09-04), which deleted the KV-cache scorer entirely and matched the strongest prior evictor at 32-43% higher vLLM throughput, and one-shot on-policy distillation (09-04), where a single training query recovered most of full-data gain, three consecutive days have produced three results where the learned selection layer is the deletable part.

The upstream constraint got located the same day. Locked at the Entrance (09-05) (arXiv 2608.29188) shows RLVR contracts a policy's solution coverage by up to 67%, that the contraction is concentrated at the trajectory's first move (per-token likelihood shifts 11-16x larger before the first operation than during downstream reasoning), and that supplying an unselected entrance prefix restores completion in low-access families from 0.018 to 0.212. This page's stated empirical premise is that successful and failed trajectories share their opening steps and diverge later, so useful computation concentrates in early intermediate states. Both claims put the action at the opening steps, and they have opposite signs: that is where the value is, and it is also where the training recipe has already removed the options. Any parallel-sampling method on this page, run against an RLVR-trained policy, is drawing from a distribution narrowed at exactly the point the draws matter. The cheap fix is on the shelf: late-layer parameter interpolation with early checkpoints recovers 37% of coverage at no pass@1 loss, and the early checkpoints are already on disk.

Open problem 6 (new): report the reinvestment ablation, not the compression ratio. Every token-compression result in this wiki, including OmniScope (07-31) at 3.53x prefill speedup for a 0.35-point drop at 25% retention, reports what was saved and not what the saving bought. SCR says the second number is two to three points larger than the first. Nobody else has run it.

2026-09-06: the process reward model may already be inside the policy

Every search-steering method on this page (best-of-N, step-level beam search, tree search, anything that grades intermediate steps) pays for a process reward model, a separately trained scorer that grades each reasoning step rather than only the final answer. That PRM is a second forward pass on every candidate step, and it is a standing serving-time tax this page has priced into nothing.

Sparse Reward Subsystem (09-06) (arXiv 2602.00986, Guowei Xu of Tsinghua with Mert Yuksekgonul and James Zou of Stanford) suggests the tax is avoidable. Prior work established that LLM hidden states encode reward-related information but read it with black-box probes fitted on the full state, which shows the information is decodable and nothing about where it lives. This paper finds it concentrated in a sparse subset of individual neurons, splitting into two functional types: value neurons, whose activations predict state value (the expected probability that continuing from here yields a correct answer), and dopamine neurons, whose activations encode step-level temporal-difference error. Value neurons are reported robust and transferable across datasets and models with causal rather than correlational evidence, and the paper's own applications are the two this page cares about: value neurons as a confidence predictor, and dopamine neurons as a process reward model guiding inference-time search.

If that holds, the PRM stops being a model you serve and becomes a view over a tensor you already computed. Reading a sparse set of activation indices out of the policy's own forward pass costs a gather. That is the rare entry on this page where a deleted dependency is genuinely replaced by something close to free, against open problems 5 and 6, which both amount to the complaint that methods delete a serving-time dependency and never price the replacement.

The number the paper owes and does not give in the abstract is the accuracy gap against a trained PRM. Without it this is a cost claim with an unknown quality denominator, which is exactly the shape this page has criticized elsewhere.

And there is a specific reason to expect the intrinsic PRM to inherit a known pathology. Locked at the Entrance (09-05) showed that reinforcement learning with verifiable rewards contracts a policy's solution coverage by up to 67%, with per-token likelihood shifts 11 to 16 times larger before the first operation than during downstream reasoning, so the damage is concentrated at the entrance. Value neurons exist, on this paper's own max-entropy-RL argument, because the policy needs value estimates for its own decisions. An RLVR-narrowed policy's value neurons are therefore calibrated to the narrowed distribution, so using them to steer search would steer toward the same entrance the training already selected. That would make the intrinsic PRM cheaper and simultaneously more prone to the exact collapse the parallel-sampling literature exists to escape. The experiment is a value-neuron sweep across an RLVR checkpoint sequence, and the checkpoints are already on disk, the same free-fix observation Locked at the Entrance made about late-layer interpolation.

Fifth consecutive instance of the deletable-selection-layer pattern. Random Attention (09-04) deleted the KV-cache scorer and matched the strongest prior evictor at 32-43% higher vLLM throughput; Select, Compress, Reinvest (09-05) found unmodified Orthogonal Matching Pursuit from the early 1990s matching every purpose-built frame selector; DRACO (09-05) beat its own verifier-trained baseline with closed-form credit redistribution; AutoTraceGT (09-05) recovered 73-91% of hand-built failure taxonomies with a 1967 qualitative coding method. This adds the trained PRM to the list of learned components the system may already contain. Five results, four days, five unrelated subfields, and the wiki should now treat "is the learned scorer load-bearing" as the default first question about any method that has one.

Provenance note. This paper reached the wiki through saved reading rather than the research feed, and it is not new: the arXiv identifier is 2602 and HuggingFace surfaced it on 2026-05-11. It is a four-month-old result recirculated by a viral post that also attributed it solely to Stanford and attached a 50%-reasoning-collapse ablation figure absent from the abstract. Treat the mechanism as well-motivated and the headline numbers as unverified.

2026-09-12: the confidence signal gets a qualifier instead of a defence

TASCO (09-12) (arXiv 2609.11393, Kurate cs.AI #4, absent from HuggingFace) is the first entry on this page that neither defends confidence as an allocation signal nor deletes it. It conditions on it. The standing objection to entropy-minimizing test-time adaptation is that a system can be confidently wrong, so pushing toward higher-confidence states drives it deeper into a wrong trajectory. TASCO's observation is that high confidence that survives a small perturbation is much more likely to be correct than high confidence that collapses under one. It optimizes a lightweight task-level prefix with the network frozen, under two alternative perturbation strategies: Random Perturbation for average-case distributional stability across nearby prefixes, and Sharpness-Aware Perturbation for worst-case local sensitivity. Reported gains in both reasoning accuracy and token efficiency, with behavioral analysis showing it does not prematurely concentrate the predictive distribution, which is the failure mode of a naive entropy minimizer.

This page's standing warning about the unvalidated second estimator applies in a new way, and mostly in TASCO's favour. The warning has been that methods in this family depend on a learned or heuristic selection signal nobody validated, and confidence is the most common such signal. TASCO's claim is narrower and therefore more falsifiable than usual: not that confidence is good, but that confidence-plus-stability is. The missing control is the cheapest one in the family: self-consistency sampling with majority vote, which is also a stability measure and requires no training at all. Without that comparison the contribution could be the prefix optimization or could be the perturbation, and there is no way to tell.

The token-efficiency claim needs one specific audit before it is carried forward. Measuring stability requires running perturbed prefixes, which costs tokens. Whether the reported efficiency gain is net of the perturbation cost is the first thing to check, and it decides whether this is a cost result or only a quality one.

The deployable form is probably not adaptation at all, and that is where this connects to routing. If stability-under-perturbation is a better correctness signal than raw confidence, the cheapest use is escalation triage: answer directly when confidence is stable, escalate to a larger system or a verification pass when confidence is high but fragile. That makes it a feature for the routing page rather than a method for this one, and routing features ship far more easily than anything that mutates inference state.

2026-09-18: the difficulty signal gets measured instead of estimated, and the compute unit turns out to be under-specified

When2Think (09-18) (arXiv 2609.19671) takes the third option on this page's longest-running objection. That objection is the unvalidated second estimator: allocation methods depend on a selection signal nobody validated, usually the model's own confidence, which can be confidently wrong. TASCO (09-12) neither defended nor deleted confidence, it conditioned on it, observing that high confidence surviving a perturbation is far more likely to be correct than high confidence that collapses. When2Think replaces the signal entirely with a measured one.

Instance-level Difficulty-Aware Control (IDAC) shapes the reward using pre-computed reference statistics: per-instance accuracy and token usage, measured offline from the base model. An instance the base model solves reliably and cheaply should be answered directly; one it solves rarely and expensively earns extended reasoning. Offline per-instance accuracy is not an estimate of difficulty, it is a sample of it, which is a categorically stronger signal than anything else on this page. The cost is that it must generalize from measured training instances to unseen ones, and the paper does not isolate how well it does.

The training setup is austere in a way that matters: IDAC combines with verifier-based rewards and batch-wise standardized advantages to give critic-free optimization with no learned reward model and no online reference-model queries. Each omission removes a cost that competing approaches on this page pay. The model learns when to use System 1 (NoThink) versus System 2 (Think) rather than being told by an external router.

The result is a move inside the frontier rather than along it, and that is the claim worth checking. Prior entries here accepted an efficiency tax: reduced computation on easy instances bought with accuracy loss on hard ones. On AIME24, Pass@3 rises 10.0 percent while token usage falls 27.9 percent against the base model, and AIME25 reaches 40.0 percent Pass@3 beating both compression and routing-only baselines. Both axes improving says the base model's allocation was leaving accuracy and tokens on the table. That is a stronger claim than any compression result on this page and it needs independent reproduction before it is relied on. Math-only, verifier-exact, Pass@3 rather than Pass@1, and the cost of computing the reference statistics is not netted against the saving anywhere visible.

The compute unit itself was under-specified, by a factor of five

Sample Count Is Not Enough (09-18) (arXiv 2609.19499) shows that N, the candidate count this entire page uses as its budget, hides a 5x systems difference. Holding N = 8 fixed on GSM8K and varying only the schedule, eight serial calls of one candidate consume 4.64 to 4.86 times the gross GPU-device energy and carry 5.77 to 6.12 times the P95 latency of one batched call of eight, on A100s, replicated across three independently scheduled nodes per model and again on short-output SciQ/V100 runs. Same accuracy, same N, five times the energy.

This is the systems denominator this page has never had. Every entry here reports compute in tokens or candidate counts. Two methods quoting the same N can differ in energy by more than most of the methods on this page differ in accuracy, which means a large share of the efficiency comparisons recorded here are not comparable. The authors' recommendation is the right one and should be adopted as a reading rule for this page: an efficiency claim that does not state its generation schedule is under-specified.

It composes with When2Think rather than competing. When2Think cuts tokens per instance by allocating depth to difficulty. This cuts energy per fixed token budget by scheduling. Orthogonal axes, multiplicative savings, and each paper is blind to the other's. It also puts an order of magnitude on the routing-versus-batching problem llm-routing.md named on 09-14: fragmenting a request stream shrinks batches, small batches cost multiples in energy, and no router on that page subtracts it.

Caveats that keep the 5x honest. Small models only, where per-call overhead is a larger share of total work than on a frontier MoE, so the factor should shrink at scale and the paper does not estimate by how much. Gross board-level draw includes idle, which flatters the batched case. A100 and V100 only. And in production the serving stack batches across users, so the schedule is the platform team's lever rather than the application team's. Where it bites hardest is offline evaluation and batch scoring, where one team's loop really does choose the schedule and those jobs run continuously.

2026-09-20: the compute unit gets specified, and the stop-early control ships in three products

The 09-18 entry concluded that the compute unit this page allocates was under-specified. Chapter 8 of the frontier-inference series (09-20) supplies the specification, and it invalidates most of the accounting on this page. Reasoning workloads now spend 90% to 95% of cluster GPU cycles in decode, where arithmetic intensity is 1.0 to 2.5 FLOPs per byte. At that ratio the FLOPs are not the scarce resource. The unit is bytes moved per generated token, not FLOPs, and any allocation policy denominated in FLOPs is optimising a resource that is not binding.

This has a direct consequence for the breadth-versus-depth question the 09-08 entry opened. Depth (one long chain of thought) is a single sequential decode stream and batches with nothing. Breadth (N parallel candidates) batches perfectly, and Sample Count Is Not Enough (09-18) measured the gap at 4.64 to 4.86x GPU energy for eight serial calls versus one batched call of eight. Under a bytes-moved cost model, breadth is not merely competitive with depth on accuracy, it is structurally cheaper per unit of compute purchased, and the page has been comparing them on a FLOPs basis that hid this. That strengthens the 09-08 claim considerably and from a direction that paper did not argue.

The when decision, opened on 08-29, is now shipped in three production systems at once and the page should stop treating it as an open research question. Zhipu's GLM-5.3 exposes reasoning_effort as an explicit API tier (low, high, max) against up to 128,000 output tokens. Moonshot's Kimi K3 and Qwen 3.8 pair token watchdogs with entropy-driven dynamic thinking budgets that terminate when confidence stops improving rather than at a fixed cap. Both join the difficulty-aware length-control line of work measured on 09-18. Three independent vendors shipping an explicit stop-early control in the same quarter is the industrial answer to a question the literature has not settled, and the interesting gap is now the reverse of the usual one: industry is ahead, and no paper has evaluated whether an entropy threshold or a learned difficulty predictor makes the better stopping rule.

A third failure mode joins the two this page tracks. Beyond over-spending and under-spending, Chapter 8 names execution stochasticity: long reasoning trajectories can enter recursive backtracking loops and consume tokens without terminating. That is not a budget-allocation failure, it is a termination failure, and every budget policy on this page implicitly assumes termination. A watchdog is not an allocation mechanism; it is an admission that the allocation mechanism has no lower bound on progress per token.

2026-10-05: spend on the checker, measure in seconds and dollars

• VeriHarness (10-05): consensus across rollouts hides shared errors; a resolver checks disagreements against workspace evidence and a challenger attacks agreements. +6.2 (Gemini 3.5 Flash) and +6.4 (Opus 4.8) points over one rollout. Second verifier-first result after Mid-Harness (10-02). Majority voting assumes independent errors; same-model rollouts violate that.
• Beyond Token Savings (10-05): compression that cuts tokens can add 20-80% latency. Allocation should be scored on wall-clock and cost per solved task, not tokens.

2026-10-12: make past compute reusable

• GitSwarm (10-12) names compounding inference: test-time compute organized so intermediate work, including failures, persists and is reused. 70.1% of failed research commits were cited by later accepted work.
• Research World Models (10-12): an LLM forecasts an experiment's gain from logged history before it runs. Records plus low reasoning effort (0.892 rank correlation) beat max effort without records (0.648). History is a cheaper input than more thinking.
• Vals AI (10-12): teams add little once the solo agent runs at max effort. Team size and reasoning effort are substitutes, not complements.

2026-10-13: compute dials inside one checkpoint

ELR (10-13) makes loop count a per-request compute knob with no one-loop tax (+7.25 points at four loops on 4B). PreSTO (10-13) makes power sampling 1.7-2.3x cheaper without changing its distribution. SelfCompact (10-13) lets the model decide when to spend tokens summarizing, 30-70% cheaper per question.

2026-10-14: move reasoning off the token stream, and measure "no-think"

• HyperThink (10-14): one hypernetwork pass replaces the thinking trace with a per-query weight update. Hybrid Reasoning Agents switch between latent thoughts and text tool calls (35.5-54.7% fewer tokens). Thinking Inertia: no-think prompts do not stop visible reasoning, so a "thinking off" mode is not a token budget (shorts).
• Cohere's agent teams (routing) treat teams as a new inference-time scaling axis, best spent at the planning bottleneck.

2026-10-15: trim the re-checking, retry the small model, nest the sizes

• DoubtLess (10-15): a small model read under "be brief" and "be verbose" prompts gives a logit difference that steers the big model; Qwen3-32B CoT 31% shorter at equal accuracy, agent thinking tokens -20%, training-free, stacks on brevity-trained models. Savings concentrate after the answer first appears. Second training-free contrastive steering method in two weeks (LoopCD, 10-03).
• Fewer Bits, More Tries (10-15): spend idle parallel compute on verified retries of a compressed model instead of loading a bigger one.
• ShellElastic and TacitVLA (shorts, both HF + Kurate): one pretraining run yields exact nested model sizes at the cost of the widest; reasoning supervision helps actions that never wait for the reasoning (3.4x lower latency). Both are "train with it, serve without it."
• Tension: the monitorability essay (10-15) argues every method that moves reasoning off the token stream trades away legibility.

2026-10-22: the price of compute is live

ATEST and implicit budgeting. Energy for the same 4-sample query rises up to 4.1x with GPU load; ATEST-Online stops sampling on answer stability plus live energy cost (GSM8K high load: 60.0% vs 44.0% for offline calibration; 42x less GPU time including profiling). Implicit budgeting: multi-task prompts get fewer reasoning tokens per task; grouping by fragility saves 43% tokens. Prior allocation work (CLEAR, LEV) assumed a constant compute price.