LLM Routing

Routing in LLM systems means deciding which model (or no model) should handle a given query — with the goal of minimizing cost while meeting quality requirements.

2026-09-19: the router's own cost went to zero, and it took the vendor with it

This page has been organized for months around one constraint: a router must cost less than the decision it makes, or it eats its own saving. Pandora's Router (08-25), which prices the cost of estimating which model to use and derives a closed-form value-of-information policy for when estimation is worth paying for, exists only because estimation was expensive. On 09-16 this page recorded Jev, a closed decision-only model that emits structured choices and probabilities over a predefined option set and never generates text, and the digest filed a prediction: an independent benchmark within 30 days, or treat it as vapor.

It took three days, and the answer broke the frame. Six independent open re-implementations shipped inside 72 hours, and on the one head-to-head anyone published, a 706K-parameter, 2.8 MB open model scored 99.7% against hosted Jev's 83.6% on form filling. Kev-0.5B rebuilds the pattern as a LoRA plus a pointer head on Qwen2.5-0.5B, trainable on a MacBook; sarvam-jev reads typed decisions straight out of an Indic model's logits at 8.8x the speed of making the same model write them; SimpleJev converts any HuggingFace model; a local distillation of DeepSeek V4 Flash into a 4B student hits ~22 ms per decision and beats the teacher's instant mode. The mechanism everyone landed on is the same and it is not exotic: stop sampling, read the logits.

The consequence for this page is not that decisions got cheap to rent. It is that they got cheap to own. A 2.8 MB artifact running on a CPU has no per-call price, so there is no value-of-information trade-off left to compute at the routing layer at all. Pandora's Router's question, "is it worth paying to estimate," does not become "how good is a cheap estimate" as the 09-16 entry guessed. It stops being a question. What remains is purely accuracy and calibration, and calibration is exactly what none of the clones report. A router needs to know when it does not know so it can escalate; 99.7% on form filling says nothing about whether the remaining 0.3% is flagged or confidently wrong.

This also settles the 09-14 batching worry in routing's favour. That entry warned that routing across many small models fragments the request stream and erodes the weight-read amortization that keeps a GPU bill competitive with a per-token bill. A kilobyte-to-megabyte decision model never joins the GPU pool. It runs beside the application. It removes itself from the batching problem rather than solving it.

And a second team arrived at the same destination by a different road on the same day. Cactus's Needle 3 is an 8-29 MB automation model whose every layer-depth from 2 to 20 is a usable sub-network, which also refuses to chat, also returns typed records or tool calls, and also returns an empty list rather than a guess when no tool fits. Logit readout and tiny-model-from-scratch are different mechanisms with one product thesis: the decision layer is a local artifact, not a frontier API call. Two of a kind in one day. A third makes it the page's default assumption.

Still open, and now sharper. The 09-18 digest predicted someone would run Gavel (09-16, which reads the routing signal out of a frozen agent model's mid-layer hidden states with two linear maps, for free) against Jev on skill selection with accuracy, latency and dollars reported together. Cua's number is form filling, and it compares a clone to hosted Jev rather than either to Gavel. But the clone wave is evidence for Gavel's half of the argument: both camps now say the decision signal is already inside a model you are running, and the only live disagreement is which internal surface to read it off, the mid-layer hidden states or the output logits.

2026-09-19: routing prices are not exogenous, and no result on this page is stated as an equilibrium

SFT vs ICL under congestion (2607.14371, Yale) is a small paper with an uncomfortable implication for everything above it. It models the choice between supervised fine-tuning (pay once to train weights) and in-context learning (pay a long prompt every request) on shared GPU capacity, and shows the choice is a congestion game rather than a per-user cost calculation: which method is better depends on what every other tenant chose. Two findings bite. Congestion can flip the ranking that holds in isolation. And equilibrium resource use is non-monotonic in model quality, where better pretraining precision reduces congestion but broader coverage and harder tasks can increase it.

Every routing result catalogued on this page optimizes one request against prices treated as given. Pandora's Router, VoI-MoLE, the handoff tax, the depth-and-width framing: all argmax, none fixed-point. If capacity is shared and prices are endogenous, the optimal policy is an equilibrium. The concrete casualty is the 09-17 depth-and-width entry below, whose headline estimate of roughly $500/day all-frontier down to about $66/day routed is a price quoted from an uncontended world. Width, in particular, is a congestion externality: a dial that many tenants turn at once. The market has already conceded the paper's third finding, that offering both methods never hurts platform profit, with the share of major platforms offering both rising from 9.5% in 2021 to 71.4% in 2025.

2026-09-17: the pool itself becomes a variable, and adding an option turns out to be able to make the system worse

Every result on this page has treated the set of routable options as given and optimized the decision over it. Mo' Models, Mo' Problems (09-17) (arxiv 2609.17306, NVIDIA) makes the set itself the variable and reports a clean negative. Across eight selection strategies (by size, by accuracy, by answer diversity, by error diversity, and combinations) over before-generation routing and after-generation aggregation on hard scientific benchmarks: expanding the candidate pool raises the oracle ceiling and lowers achieved accuracy, frequently below the single best model in the pool. The only strategy that reliably beat a standalone model was selecting candidates within a single model family, and majority vote over several samples of the best single model took Humanity's Last Exam from 29.4% to 32.2% while nearly every heterogeneous mixture declined.

The distinction this page should extract, which the paper implies without naming. Heterogeneity is an asset when a router decides, because a correct route never invokes the weak member, and a liability when a vote decides, because an off-distribution member injects variance that no aggregator tested here can discount. The economic case for mixed pools, argued most cleanly by the AlphaSense token-price analysis (08-14) showing that per-token pricing ranks models backwards on task cost, survives for routing and does not survive for post-hoc aggregation. Most production systems that call themselves multi-agent are doing aggregation.

It is the second result in two days saying that mixing model families changes system behaviour in ways neither member's own evaluation predicts. Emergence World (09-16), eight ten-agent worlds over sixteen days and 850,000+ LLM calls, found that the same model-persona pairing behaved differently in mixed versus homogeneous populations and concluded that model-level alignment is not compositional. NVIDIA now reports that model-level capability is not compositional either, on a far shorter horizon and a far cleaner task. Safety-framed and accuracy-framed, neither citing the other. Two of a kind. A third makes it a pattern, and the implication for this page would be that heterogeneous pools need an explicit compositionality argument rather than a diversity intuition.

The obvious missing mechanism, stated as a research angle. Every aggregator in this wiki weights its members equally. Classical ensemble learning does not: it estimates per-member reliability online from agreement history. If a reliability-weighted vote converts the oracle gap into recoverable accuracy, the problem was the aggregator. If it still loses to the single best model, the problem really is the pool and the field should stop building mixed-model juries.

2026-09-17: routing gets a second coordinate, and three production cost ratios now cluster with no shared method

Ken Huang's depth-and-width essay (09-17) names a distinction this page has been circling without vocabulary. Depth is how much reasoning a task gets, and it is now a per-request dial rather than a model choice: GPT-6 Astra exposes five discrete reasoning.effort levels. Width is how much parallelism a task gets: Kimi K3's Agent Swarm runs up to 300 subagents over a 2.8T/104B-active mixture-of-experts backbone with a 1,048,576-token context and explicit context sharding. His router design, the intelligence compiler, classifies a task as routine, parallel or hard, dispatches, verifies, and only then hands a human the decision. Claimed effect on a planning workload: roughly $500/day all-frontier down to about $66/day routed, explicitly labelled an estimate.

The width axis already has a measurement on this page and the two were not connected. Elo-per-token (09-15), which converts within-task solution quality at each token budget into a cross-task Elo and locates the budget at which an agent's marginal token stops beating an independent sample, showed that re-slicing a fixed 100M-token budget across parallel sessions at that cap buys +264 Elo over one long session and +355 over ten short ones. Huang supplies the vocabulary and the production framing; Elo-per-token supplies the threshold. Width is a routing decision with a measurable optimum, orthogonal to which model handles the query.

And the cost claims are now a cluster that this page should stop reporting as news. Spotify's Portal layer: 90% token reduction by hard-blocking files over 350 lines from the frontier model (09-07). The unverified "End of Model Loyalty" post: an 84/16 cheap-to-ceiling split costing $256 against $4,318, about 17x, with a 30-day verification deadline set on 09-15 that expires 2026-10-15. Huang: about 7.6x. Three independent production routing claims between roughly 7x and 17x, and not one of them publishes the workload mix that produced it. Since every routing saving is dominated by the fraction of traffic that is routine rather than by the router's quality, the ratio without the mix is uninterpretable. The right ask from now on is the mix, not the ratio.

2026-09-17: routing arrives as a consumer product surface, and the margin question comes with it

Anthropic merged Claude Cowork and Claude chat into a single product (Anthropic, The Decoder, Simon Willison). Instead of the user choosing between a chat interface and an agent interface, the model decides on its own whether a request deserves a quick answer or a long background workflow. Rolling out to Pro and Max first.

This page has never had an entry where the routed quantity is the user's own intent, and where the router is invisible to the user by design. Every prior entry routes inside the system: a query to a model, a step to a tier, a skill to a shortlist. Here the product architecture that used to encode the decision (two apps, pick a door) is deleted and the judgment moves inside the model. The commentary that got this right on the day framed it as a pricing question: a single text box spanning five-second replies and hour-long jobs converts the question from "which model" to "how much compute does this deserve," and whoever owns that decision owns the margin on every task.

That is the industry landing exactly where the research has been pointing for a month. Gavel (09-16) showed the routing signal is already present in the frozen model's forward pass and can be read out with two linear maps; Jev (09-16) proposed the opposite answer, a separate near-free model that never generates text; and Huang's depth dial says the effort level is now an API parameter. Merging chat and agent is the product form of the same claim: the decision about how much compute a request deserves is not a user-facing choice, it is a routing problem, and it is now on the critical path of a frontier lab's consumer margin. Every lab that shipped a separate agent product will have to answer it.

2026-09-15: a second budget lever with a measured payoff, and a placement question this page has never asked

Elo-per-token analysis (09-15) (arxiv 2609.15309) gives this page a routing decision it has not had a number for. The paper tracks the best solution an agent has found at each token budget and converts within-task orderings into cross-task Elo via a Bradley-Terry model, then compares against independent sampling, whose Elo provably grows linearly with log compute. Agents beat that reference early and fall below it later, and the crossing is the scaling inflection point. Using that point as the per-session budget and splitting 100M tokens across parallel sessions gained +264 Elo over one long session and +355 over ten short ones.

Stated as a routing claim: how many parallel sessions, capped at what budget, is a cost-at-a-quality-bar decision orthogonal to which model handles the query. This page has always routed across models. It now has an equally well-measured axis to route across allocations, and the inflection point is the threshold. The catch is that the paper prices everything in tokens while this page prices in dollars, and parallel sessions share a KV prefix in real serving, so ten sessions do not cost ten times one. The allocation that wins in tokens may not win in dollars, and that conversion is unwritten.

It also composes with this page's sharpest existing claim in a way neither paper noticed. The Handoff Tax (09-07), across 58,000 agent runs and 36 billion tokens, found that passing a cheap model's full history on escalation recovers under half the quality gap and for Claude costs more than twice starting fresh, while dropping the trajectory and passing only the code edits lifts recovery from 47% to 64% (Claude) and 36% to 84% (GPT). Both results say the accumulated trajectory becomes a liability past a threshold: the Handoff Tax locates the threshold at a model boundary, Elo-per-token at a budget boundary. Nobody has asked whether they are the same quantity in different units, and if they are, a router has one threshold to estimate rather than two.

Separately, a robotics essay poses the question this page has never asked: not which model, but where the model runs. A Brain Too Big to Carry (09-15) derives a placement rule by decomposing a system by loop frequency. Anything above roughly 100 Hz can never leave the device, because a 10ms budget is entirely consumed by a 10-to-50ms wireless round trip; a 5 Hz planner gets 200ms and offloads comfortably. And the binding constraint is jitter rather than latency: a fixed delay can be planned around because you know how far the world moves in it, while irregular arrival cannot, and buffering to the worst case makes every action pay the worst case.

That is a direct methodological criticism of this page. Every routing result here reports cost and quality, some report mean latency. None reports latency variance. If the correct decision boundary for local-versus-remote is set by variance rather than by mean, then the routing literature has been measuring the wrong latency statistic, and the omission is invisible in any benchmark that reports averages. The claim transfers cleanly out of robotics: an interactive agent with a hard per-turn deadline faces the same arithmetic, and so does any router that can fall back to a local small model.

A widely-shared but unverified social claim points at the same economics from the arbitrage side. A post circulating today describes a paper titled "The End of Model Loyalty" arguing that two frontier models now share a million-token window at $10-in/$50-out versus $3-in/$15-out with open weights, that benchmarks overlap on the median task, and that the gap only appears on the hard tail (99.9% against 62.7% completion). Its reported production split was 84% to the cheap model and 16% to the ceiling, for $256 against the $4,318 the flagship alone would have cost. No paper link accompanied the post and the figures are unverified, so this is recorded as a hypothesis rather than a result. It is worth tracking because the 84/16 split and the ~17x saving are precisely the shape this page's AlphaSense study (08-14) predicted when it showed per-token pricing ranks models backwards on task cost. If a real paper carries these numbers, it is the largest production routing saving in the wiki. If no paper surfaces within 30 days, treat the post as invention.

2026-09-09: a router turns out to be a measuring instrument, and that is a second return nobody on this page had counted

This page defines routing as choosing which model handles a query so as to minimize cost at a quality bar. Every result on it, up to and including the largest, treats the routing decision as a cost-control action whose output is a served response. NeoHorse-1 (09-09) (arxiv 2609.08183, TokenRhythm/Infinigence/Tsinghua/Peking/CUHK/Alibaba) treats it as a measurement whose output is a training example.

The argument is short and this page should have made it. A routing harness necessarily records, per turn, the capability it predicted the turn would demand, the service tier it selected, and what happened next. That triple is a labelled capability measurement generated for free by production traffic. NeoHorse-1 converts those records into training data (preserving interleaved reasoning, tool calls and harness context, admitted through structural validation, six-dimensional semantic evaluation and subscene labeling), uses the routing signal in place of a difficulty annotation to order a three-stage SFT curriculum, extends it into routing-guided on-policy distillation, and then lets evaluation feedback set the next training mixture. Across eleven benchmarks spanning harness-based agents, tool use, coding and instruction following: macro-average 58.94 to 64.87 at 4B and 65.60 to 69.04 at 9B, with the post-trained 4B substantially narrowing the gap to the 9B base. The cost claim is that routing telemetry buys roughly a model-size step.

It also produces the sharpest single sentence this page can offer about trajectories, by colliding with the Handoff Tax (09-07). That paper, across 58,000 agent runs and 36 billion tokens, found that passing the cheap model's full history on escalation recovers under half the quality gap and for Claude costs more than twice starting fresh, while dropping the trajectory and passing only the code edits lifts recovery from 47% to 64% (Claude) and 36% to 84% (GPT). So the optimal serving policy is to discard the trajectory at the boundary. NeoHorse-1's optimal training policy is to retain the full interleaved trajectory including harness context. Both are right, and stated together: a trajectory is bad context and good training data. Nobody has said this and it is a clean, load-bearing claim for anyone building a router.

Read against the loop-is-an-asset essay (09-09), which argues the harness is the owned asset and the accumulating trace the compounding one, this page now has a research result and an industry thesis converging on the same claim from opposite ends of the stack, and both landed on the day the industry's dominant story was a dispute over whether private tool sessions had improved somebody else's model. The commercial question that follows is whose telemetry it is, and it is no longer hypothetical.

Two things to hold NeoHorse-1 to. The recursion is asserted, not demonstrated: one closed loop is described and called a prototype, with no second iteration showing the improvement rate itself improving, so read it as bounded self-improvement rather than RSI. And there is no routing-signal ablation: a curriculum ordered by a cheap difficulty heuristic, or randomly, at matched data volume is the control, and without it the honest reading is that routing telemetry may have contributed data volume rather than curriculum structure.

2026-09-07: the mid-session switch gets measured, and the answer is that the context policy has to flip with the direction

The Handoff Tax (09-07), from AWS Agentic AI, is the largest routing measurement this page holds: 58,000 agent runs, 2 million API calls, 36 billion tokens on SWE-bench Verified, across Claude (Haiku to Opus) and GPT families. It measures the thing everybody does and nobody prices, which is switching models in the middle of a running agent.

Escalation, the default move, is close to the worst available. Passing the full chat history when you go from the cheap model to the frontier model recovers less than half the quality gap: 47% for Claude, 36% for GPT. For Claude it costs more than twice what starting on the expensive model from scratch would have cost. And the sunk-cost intuition is backwards: even after paying for the cheap model's work, abandoning the run and restarting fresh on the expensive model is both cheaper and more accurate than continuing.

The design rule is the finding, and it is that the optimal interface reverses with direction.

In one line: strong-model trajectories guide weak receivers; weak-model trajectories burden strong receivers. Context is not universally good, it is directional. The escalation cost compounds rather than adds, because the inherited context makes every remaining frontier step 1.6 to 2.2x more expensive at premium input rates.

This closes the wiki's most concrete unpriced item and makes it bigger than expected. The KV cache page recorded on 08-29 that provider cache entries are keyed to a model, so a mid-session route pays a full cold prefill on the accumulated history, and on the 140K-token median agentic prefix that plausibly exceeds the per-token saving the route was chosen for. The prefill was only one of three terms. There is also a quality penalty and a per-step inflation multiplier, and neither was in this page's model.

It puts cross-model KV sharing (09-02) in an awkward position, and the reconciliation is the sharpest carry-off from both. That translation layer converts one model's KV state into a form another model can consume, and this wiki called it the first mechanism to remove the switch penalty rather than price it. The Handoff Tax says the dominant penalty is not transport, it is that carrying the weak model's reasoning into the strong model is what does the damage. So the mechanism that makes trajectory transport cheap is optimizing the direction in which you should be throwing the trajectory away. Not a contradiction, a directional split: portable KV state is worth a lot on downshift and worth little or negative on escalation. Neither paper cites the other.

It also adds a third requirement to this page's standing per-step-router gap. Since 08-06 the missing piece here has been a router at per-step rather than per-query granularity, and the framing has been that it needs two things: a cheap switch (candidate mechanism: portable KV state, 09-02) and a schedule saying when to switch (candidate: TCR (09-04)'s demonstration that a plan's timeline can be compiled into a routing schedule up front). The third requirement is a switch-direction-dependent context policy. A per-step router that switches cheaply on a good schedule and passes full history every time still pays this tax on every escalation.

Industry shipped the enforcement half the same weekend, and its finding is about mechanism not policy. Spotify's engineering team reports cutting Claude Code token usage 90% with a two-model routing layer called Portal: two cheaper assistant models handle file opening, boilerplate and repetitive code, the frontier model touches only novel reasoning, and files over 350 lines are hard-blocked from the expensive model rather than discouraged. Their stated lesson is that they tried rules-as-instructions first and it did not hold, because both engineers and the model routed around soft guidance. This page has catalogued routing policies for five months and has almost nothing on routing enforcement. Spotify's claim is that at production scale the policy is the easy part.

A practitioner counterweight worth recording, because it is the honest version of the routing pitch. An X article circulating the same day, "LLM Routing Can Cost More Than Not Routing," argues that the canonical setup (a classifier reads the request, cheap model handles easy ones, frontier handles the rest) is exactly the version that breaks in production. Together with the Handoff Tax the message is consistent: the naive per-query classifier router is the configuration with the worst evidence behind it, and the two configurations with real numbers are hard architectural blocks and planned downshift.

2026-09-04: the routing key becomes when, and this page's taxonomy gains a coordinate

TCR (09-04) (arxiv 2609.02367, PKU with Qwen Applications, HKUST, CUHK, SJTU) is filed here for the routed object, not the application. Its domain is script-driven audio-video generation, where joint generators keep audio and video synchronized with each other while both drift off the script's stated timeline. The diagnosis is exact: the timing written in a structured prompt exists only in the prompt's text representation, which is not aligned to either modality's temporal coordinates, so the failure is invisible to every audio-visual synchronization metric. Temporal Context Routing maps script timing onto the shared temporal axis the generator already uses, then routes each prompt's guidance to the matching positions in both modalities. Shot Boundary MAE falls 96%, from 1.11s to 0.042s. Dialogue Accuracy at 0.5s tolerance rises from 28.3% to 84.1%. Quality and audio-visual sync stay comparable, and a user study prefers it on all five dimensions.

Every entry in this page's taxonomy routes content to a computational destination. TRACER (04-17) a query to a model; CaRE (05-11) to a task-axis expert; MISA (05-11) per-head KV; Raven (08-04) a write into a memory slot per token; VI-MoLE (08-05) adapter budget by certified value of information; SMRC-SD (08-10) whether a supervision signal is admissible at all; Safin-1 (09-02) retrieval over persistent capability states. TCR routes a conditioning signal to a temporal position. The destination is a coordinate on a time axis, and the same guidance goes to two modalities at matched coordinates. It is the first entry here whose routing key is when rather than what or which.

The generalization is where the interest lies, and it is not speculative. Any generation process with an explicit timeline and a structured specification has the identical defect: the spec's timing lives in a text encoder while the generation lives on a positional axis, and the two are never introduced. That describes script-driven video, and it equally describes a long-horizon agent plan with deadlines, a staged budget in a streaming pipeline, and any decode process whose instructions include "by step N, have done X." Nobody has built it for any of those, and the one place somebody did build it produced a 96% error reduction.

It answers, constructively and from an unrelated subfield, the complaint Agents Are Not Time-Aware (08-30) raised without a mechanism. That paper found agents systematically mishandle deadlines, relative durations and ordering, because time enters the model as text like everything else and carries no privileged structure. TCR's answer: do not encode time as text and hope the model recovers it; route the timed instruction to the position on the axis where it applies. Neither paper cites the other, and together they make the stronger claim that temporal specifications need a dedicated channel.

It is the second result in three days where the fix is "stop making the model recover structure it was handed." Declarative Attention (09-03) had the model declare its own attention scope inside its chain of thought, parsed by the inference engine like a tool call, cutting attended tokens 52.0% on Gemma-4-31B for a 1.27pp accuracy drop, zero-shot. Both are protocol design rather than modeling: the win comes from giving structure a machine-readable channel instead of leaving it embedded in text for the network to re-derive. This page should watch for the third instance.

And it supplies a candidate for the missing half of the per-step router. This page's standing gap since 08-06 is a router operating at per-step rather than per-query granularity, and the blocker was the prefill penalty on mid-session switching, which cross-model KV sharing (09-02) is the first mechanism to remove rather than price. A per-step compute router needs two things: a cheap switch and a schedule saying when to switch. Portable KV state is the first. A timeline-to-position routing map is a candidate for the second, and TCR is the existence proof that a plan's timeline can be compiled into a routing schedule up front.

Limits. The comparison is against "the baseline" singular on 200 test scripts, and the paper's own related-work survey names MTSS as the closest prior, which supplies temporal links without a dedicated routing mechanism and is therefore the comparison that would separate a large result from an enormous one. One generator, no training or inference cost reported (the routing literature's habitual omission), and the method assumes a script with explicit timings already exists, so the harder problem of inferring the timeline then routing to it is untouched.

Separately, a cost result relevant to every router here. RealSWE (09-04) measures that two of six information categories in a coding request, Environment Information and Reproduction Steps, add tokens with no measurable benefit, while Desired Behavior and Motivation significantly help. Harnesses that auto-collect the first two are paying prefill for measured-inert content, and any cost-per-task router sitting above such a harness is optimizing model choice while the prompt carries dead weight.

2026-09-02: the prefill penalty on mid-session routing gets a mechanism, and a new routable object appears

The single largest unpriced cost of routing on this page may have just become optional. Cross-model KV sharing (09-02) (arxiv 2608.30963, Kurate cs.LG #17) translates the KV state produced by one model into a form a different model can consume, across scale, architecture, attention configuration, tokenizer and model family. On the heterogeneous Llama3.1-70B → Qwen2.5-7B handoff it reaches 44.0% accuracy against 45.7% for native target inference while latency falls from 899ms to 138ms; cross-family Qwen2.5-1.5B → Gemma-2-2B cuts target-side prefill cost by up to 67.05% at 4K context.

Why this matters here specifically. kv-cache.md recorded on 08-29 that provider prompt-cache entries are keyed to a model, so a mid-session route to a cheaper model pays a full cold prefill on the entire accumulated history, and that on the 140K-token median agentic prefix measured from replayed Claude Code and Codex traces this plausibly exceeds the per-token saving the route was chosen for. That page called it "the most concrete unpriced item where this page meets routing," and no formulation on this page prices it, LLMRouter's five-component sequential-decision framing (08-14) included. This paper is the first thing in the wiki that removes the penalty instead of measuring it. The caveat is a regime mismatch worth holding onto: the 67% figure is at 4K, the problem is at 140K, and nobody has reported the handoff at agentic context length.

If it holds, it changes which routing policies are even admissible. Every cost model on this page implicitly assumes routing happens at query boundaries, because mid-session switching carries a prefill penalty nobody wants to pay. Portable context makes per-step routing affordable, which is precisely the granularity this page has wanted since 08-06 and repeatedly said nobody was building: a router that detects an imminent skill switch (per Skill Entropy (08-06), which defined long-horizon difficulty as the cost of transitions between skills rather than the difficulty of any one skill) and escalates only across the boundary. SMRC-SD (08-10) was the first method operating at the right temporal granularity but it gated a supervision signal, not compute. Cheap mid-session handoff is the missing serving primitive that would make a per-step compute router deployable rather than theoretical.

Note the incentive asymmetry, because it predicts the deployment order. Within a vendor family, large-to-small handoff is a margin gain. Across vendors, a portable cache is a margin leak, and current prompt caching bills roughly 1.25x base rate to write and 0.1x to read. Expect intra-family handoff to ship first. That also sharpens the page's 08-11 observation that the cheapest reported routing win is intra-vendor substitution to a previous-generation model, an axis essentially no paper models: within-family KV translation is exactly the mechanism that makes that substitution cheap mid-session, and it is the one case where the model pair is guaranteed to share a tokenizer.

A second, unrelated result adds an object to the routing taxonomy. Safin-1 (09-02) (arxiv 2609.00092) builds on MARCH (Memory-Anchor Routing across Context History), which maintains structured memory states and retrieves history through content-conditioned routing, and supports test-time adaptation of persistent capability states without repeatedly modifying the backbone. Against this page's list, that is a new kind of routed thing. TRACER (04-17) picks a model per query; CaRE (05-11) a task-axis expert; MISA (05-11) per-head KV; Raven (08-04) a write into memory per incoming token; VI-MoLE (08-05) adapter budget by certified value of information; SMRC-SD (08-10) whether a supervision signal is admissible at all. None of those objects persists across queries while remaining independently adaptable. A capability state does.

That makes it the closest structural analogue yet to the thing this page has wanted for two months and nobody has proposed: routing over harnesses. A harness is also a persistent artifact, maintained separately from the weights, that lifts different models by different amounts. Safin-1 does not route over harnesses and does not claim to. But it demonstrates that a durable, separately-adapted, non-weight artifact can be the target of a routing decision inside a shipped architecture, which is the existence proof the harness-routing proposal has been missing. The standing gap is otherwise unchanged and is now over two months old.

And the harness-cost obstruction got worse today, not better. Harness-of-Harness (09-02) reports a 52.25% average relative gain over three standalone coding harnesses and publishes no cost for its own mechanism, the eighth consecutive harness result to do so. Routing over model-harness pairs requires comparable cost-per-success per pair. A meta-harness that wraps three or seventy iterations around an existing harness multiplies its token spend by an unstated factor, which makes that accounting harder. The currency problem this page named on 08-14 via the AlphaSense study, where per-token pricing ranks models backwards because stronger models finish in fewer tokens, and which Optima (08-16) built the infrastructure to fix, remains unsolved on the harness axis specifically.

2026-08-26: a fourth demonstration that the routable unit is the model-harness pair, and still no proposal

Recuris (08-26) supplies the cleanest supporting evidence this page has, because the spread is within one paper. One memory-evolution harness applied to ten models lifts them by materially different amounts: +17.8 on GPT-5.6 Sol, +15.6 on Claude Opus 5, +16.6 on Qwen3.6-27B, +13.5 on Qwen3.6-35B, and the advantage widens with task horizon, reaching +32.2 on the longest tasks. Different models gain differently from the same scaffold, and the gain depends on the task's length. That is exactly the condition under which routing over model-harness pairs dominates routing over models, and it adds a task-dependent term (horizon length) that no formulation on this page carries.

Prior evidence, for the record: A²E (08-11) found no model-harness combination wins across all task types; Evo-Bench (08-11) found synthesized harnesses transfer as reusable reasoning structures across policy models; DarwinX and AutoDesign (08-14) showed harnesses transferring across benchmarks, base models and agent frameworks; Prime Intellect's 153-run study (08-16) reported Kimi K3 differing by 44 steps between two harnesses, roughly the size of the whole Opus-5-versus-Kimi-K3 gap. Today adds a fifth demonstration, from a within-paper ten-model spread.

And a new independent measurement of the swing, from practitioners. Gradient Flow's nine rules (08-26) reports that the same open model showed an 18 percentage point spread between its best and worst harness configuration, and instructs teams to evaluate the model and harness as one system, re-running the evaluation whenever either half changes. That is this page's premise stated as production advice.

Standing gap, unchanged and now nearly two months old: nothing in production routes over harnesses, and no paper proposes it. The demonstrations keep accumulating from every direction — measurement, transfer, evolution, autonomous research, and now a ten-model within-paper spread — and the routers keep routing over models. The concrete unwritten paper remains the one this page named on 08-16: a Pandora-style expensive-inspection policy whose boxes are model-harness pairs and whose inspection cost is a partial rollout.

Current State (as of 2026-08-16)

The cost-metric problem this page named on 08-14 got a product two days later, and two new routing axes opened without anyone calling them routing.

Optima (08-16) resolves the standing prediction, early and by the named actor. Artificial Analysis shipped a platform where you build a benchmark from your own data and workflow and compare models on quality, cost, and time per task rather than a public score. 08-14's Looking Ahead predicted a major leaderboard would make dollars-per-completed-task its headline metric within 60 days and named Artificial Analysis as one of three possible parties. It took 48 hours. This matters here because most routing results on this page, xRouteBench included, score cost in per-token API pricing, and the AlphaSense study (08-14) showed that metric ranks models backwards on real workloads because a stronger model finishes in fewer tokens. Optima's answer to a workload-dependent disagreement is to stop arguing about the representative workload and measure yours. The regime boundary this page recorded, that lightweight routers get more competitive as the cost constraint tightens, is now checkable in the right currency for the first time.

Two same-day datapoints show how wide the error bar on the wrong currency is. DHH ran one identical Rust-rewrite task across five models: $550 for Fable (45 min), $55 for Grok 4.6 (1.5 h), $43 for GPT Sol, $23 for DeepSeek Pro V4 Max (2.5 h), with two models failing to complete at all. A 24x dollar spread, a 3.3x wall-clock spread, and a completion-rate axis on a single task, which is exactly the tri-objective Optima measures and no price table expresses. Separately, Anthropic's Tibo Sottiaux stated publicly that OpenAI's tokenizer is roughly 30% more efficient, with a circulated table putting 493 words at 766 tokens under o200k, 900 under legacy Claude, ~1,170 under a Claude Opus 5 estimate, a 34.5% overall gap and 53.2% on multilingual prose. Two providers quoting the same dollars per million tokens are not quoting the same price, and no router on this page normalises for tokenizer.

Two new routable axes appeared today, neither claimed as routing. Gambit (08-16) scores partial reasoning trajectories mid-flight, kills the weak ones, and re-branches from strong prefixes, cutting tokens up to 68.5% at more than 2x throughput. Under LLMRouter's five-component formulation that is a router whose candidate set is partial trajectories of a single model. CaRL (08-16) detects capability overreach, that a query lies beyond what this model can do, and trains the model to refuse rather than emit an expensive specious derivation. A refusal is the degenerate case of an escalation: the detector that fires is precisely a routing signal, and routing up to a stronger model strictly dominates quitting for any user willing to pay. Nobody has proposed either.

Standing gap, unchanged and now a full month old: nothing in production routes over harnesses. Today added evidence, not proposals. Prime Intellect's 153-run study (08-16) reports every leaderboard row as a model-harness-effort triple and shows Kimi K3 differing by 44 steps between two harnesses, roughly the size of the whole Opus-5-versus-Kimi-K3 gap. The routable unit keeps being demonstrated and the routers keep routing over models.

Prior State (as of 2026-08-14)

The field got its first unified formulation and its first honest cost metric on the same day, and they disagree with each other.

LLMRouter (08-14) does the consolidation work this page has needed since April. It formulates routing as a sequential decision process with five components (context encoders, model encoders, scoring functions, decision rules, learning signals) and shows single-turn, multi-turn, and personalized routing are all instances of it. That is the first frame that makes this page's scattered axes commensurable: MISA (05-11) routing on attention heads, CaRE (05-11) on tasks, Conductor (05-11) as an RL orchestration policy, and VI-MoLE (08-05) on expected value of information are now all different scoring functions over different context encoders, rather than merely adjacent ideas. It ships xRouteBench (generic, memory-augmented, vision, time-series, personalized) plus 16+ routers under one interface, with cost-aware evaluation and automated supervision construction so a new candidate pool does not require restarting.

Three results move this page. Learned routers beat the strongest fixed-model baseline by 14.6% relative, which is the honest comparison most routing papers avoid. Lightweight routers get more competitive as the cost constraint tightens, because a heavy router's own cost eats the savings it produces. That is a genuine regime boundary this page had not named, and it sharpens When Is Routing Meaningful? (07-20) from "does routing pay" to "which router pays at which budget." And user-conditioned routing consistently improves personalization, making identity a real routing feature.

The disagreement is about the cost axis itself, and it is not a small one. LLMRouter scores cost in per-token API pricing. On the same day, the AlphaSense study (08-14) found that metric ranks models backwards: across 246 financial-analysis tasks, GPT-5.6 Sol cost ~13% less than Kimi K3 with ~20% higher quality, and Opus 4.8 cost about half of Kimi K3 with 13% higher quality, despite both carrying higher sticker prices per token. The mechanism is that more capable models use fewer tokens, and the token-count difference swamps the unit-price difference. A routing benchmark scoring cost per token rather than per completed task will therefore rank routers differently from a production system, and possibly wrongly. Note this is contested: Artificial Analysis ranks the same Chinese models as significantly cheaper on its own task set, so the answer is workload-dependent.

Industry has already shipped the architecture the research just formalized. AlphaSense's production system uses its own harness containing a router that splits a single query: an expensive model to plan how the question should be answered, a cheap model to execute that plan. That is LLMRouter's multi-turn cost-aware formulation running in production, reported the same day the formalism was published, by a company with no stake in routing research. The plan-versus-execute split is now the most concretely reproducible routing pattern on this page and it required no routing algorithm at all.

What this does to the $10B question below. The Stripe/OpenRouter framing was "developers want cheaper models." AlphaSense reframes it: developers want the cheapest completion, which is a harder problem, requires measurement infrastructure to solve, and is therefore worth considerably more as a product than a price-comparison table would be.

Still unaddressed, and now with a third instance. No production router routes over harnesses, though A²E (08-11) showed the routable unit is the model-harness pair, and today DarwinX (08-14) demonstrated harnesses transfer unchanged across benchmarks and base models while AutoDesign (08-14) showed one harness lifting seven different model-agent configurations. Harnesses are now demonstrably portable, transferable, optimizable artifacts. Routing over them remains proposed by nobody.

Prior state (as of 2026-08-11)

The market repriced this page's entire subject matter in about two weeks, and the price is $10 billion.

Stripe is in advanced talks to acquire OpenRouter for around $10B, and The Information's reporting is that the bid revealed pre-existing demand rather than creating it: several tech companies were already building routers, Meta's AI incubator is developing an OpenRouter rival specifically to cut its own coding costs, and large software companies including Snowflake have now entered. The texture is the useful part. Requesty, a five-person UK startup routing developer requests across models, was approached by at least 25 companies in two weeks about investment, acquisition or partnership. Its CEO: "The race to optimize is crazy."

The demand driver is named, and it confirms the premise this page's research corpus has assumed since April. Developers want to cut cost by switching to older models from the same vendors (Anthropic, Google, OpenAI) for some tasks, or to cheaper open-weight alternatives such as Kimi, and the pressure rose because agents consume far more tokens than chat did. When Is Routing Meaningful? (07-20) asked exactly when routing pays; the market's answer is that it pays when the price spread across acceptable models is large and token volume is large, and agentic workloads delivered the second condition while the open-weight tier delivered the first.

The underrated detail is intra-vendor substitution. The cheapest reported win is not frontier-to-open-weight, it is frontier-to-previous-generation-frontier from the same vendor. Essentially no paper on this page models that axis, because the literature compares across labs and capability tiers rather than across one vendor's version history. It is also the substitution vendors have the strongest incentive to foreclose, by deprecating older versions faster.

The strategic read, and it is uncomfortable for the research line. OpenRouter's durable asset is the position between a developer and every model vendor, with billing attached, which is the position Stripe already holds for money. If the value is distribution and metering rather than routing policy, then better routing algorithms accrue to whoever holds the position rather than to whoever invents them. Meta building in-house says the same thing from the buy side. This page's thirty-plus method papers, from TRACER (04-17) through Sakana's Fugu ultra-router (07-24) and Google Cloud's LLM Router public preview (08-06), are the commodity layer under a distribution business.

A second, larger routing surface opened on the research side the same day, and nothing deployed addresses it. A²E (08-11) instruments agent harnesses with standardized execution traces and multidimensional metrics, and reports that model-harness combinations vary substantially by task type with no combination winning across all tasks. That is this page's founding premise moved up one level: the routable unit is the model-harness pair, not the model. Evo-Bench (08-11) adds that autonomously synthesized harnesses act as transferable reasoning structures lifting diverse policy models, so harnesses are portable artifacts that could be routed over. No production router routes over harnesses, and no paper on this page proposes it.

Standing gap, now sharper. SemiAnalysis's TileRT piece (08-11) reports that premium "fast modes" prove users will pay more for lower latency at higher gross margin, while the router market exists because developers want to pay less for equivalent work. The market is splitting into a latency-premium tier and a cost-optimized tier with the router as the switch, and no router on this page routes on a latency SLO, which is the same missing primitive kv-cache.md named when it observed that neither vLLM nor SGLang can express a per-request deadline.

Prior State (as of 2026-08-10)

A new object enters the routing taxonomy, and the thing being gated is neither cost nor uncertainty. It is validity. SMRC-SD (08-10) (2608.05219) routes a supervision signal, per turn, on a state-compatibility test. Privileged on-policy distillation lets a synchronized teacher that can see a successful reference trajectory re-score a multi-turn agent at every turn. In an interactive environment that breaks, because the student's own earlier actions keep changing the execution state, so its rollout reaches states the reference never visited and the reference stops being valid guidance for the state actually reached. SMRC-SD verifies per turn whether the student's current state is supported on the reference, distils only at matched states, and at those states builds teacher context conditioned on the state reached rather than on the reference's global path. ALFWorld 0.746 to 0.865, WebShop 0.574 to 0.693, at Qwen3-1.7B.

Placed against this page's existing list, that is a fourth kind of routing decision. TRACER (04-17) picks a model per query. CaRE (05-11) picks a task-axis expert. Raven (08-04) routes a write into memory per incoming token. VI-MoLE (08-05) allocates adapter budget by certified value of information, on the argument that uncertainty tells you the model does not know while routing needs to know whether this adapter would help. SMRC-SD gates whether a supervision signal is admissible at all. The first four ask which option is best; this one asks whether any option applies.

It is also the closest thing yet to the per-step router this page predicted on 08-06 and said nobody was building. Reading Skill Entropy (08-06), which defines long-horizon task difficulty as the cost of transitions between skills rather than the difficulty of any individual skill and measures a clean skill-switching gap across 8 frontier and 4 open-source models, this page argued the interesting router "detects an imminent skill switch and escalates only across the boundary, which is a per-step decision inside one query rather than a per-query decision." SMRC-SD is a per-step decision inside one task episode. It gates supervision rather than compute, so the prediction is not scored, but it is the first method on this page operating at the right temporal granularity, and the machinery it needs (a cheap test for whether the current state is one the reference covers) is structurally the same machinery a boundary-detecting compute router would need.

The industry side moved this week in a way that sharpens the page's 07-30 falsifier rather than resolving it. That falsifier: if a frontier lab ships first-party cross-vendor routing, the aggregation layer becomes a feature rather than a company. Google Cloud shipped exactly that on 08-06 with an undisclosed objective. This week the pressure arrives from a different direction: GitHub retired GitHub Models entirely, the unified multi-provider LLM API whose distinctive feature was that code in GitHub Actions could use the ambient GitHub token to run prompts. Simon Willison, whose own workflow broke on it, reads the shutdown as fitting the pattern where coding-agent usage made free or subsidized tokens prohibitively expensive, and swapped to a metered OpenAI key. That is the first datapoint on this page for a router dying of unit economics rather than of competition. A routing layer that does not meter is a subsidy, and agent workloads are what make the subsidy unpayable.

And a production cost signal that the routing literature has no term for. Characterizing AI-Generated C++ in Production (08-10) tracks 3.52 million code changes over twelve months in one enterprise's brownfield C++ codebase and finds AI-generated code carries a distinct quality profile (interface and coupling burdens, copy and allocation overheads, explicit loops in place of optimized standard-library APIs) that translates into a 5 to 8% increase in compute resource consumption in production. Every routing result on this page competes for single-digit and low-double-digit percentage savings. This is a 5 to 8% cost increase arriving from the tool the same organizations adopted for velocity, and it is invisible to every routing objective in the literature because it lands in the serving bill of the generated code rather than in the cost of generating it.

Prior State (as of 2026-08-06)

Routing stopped being a thing teams build and became a thing they buy, and the first vendor to sell it is one of the models it routes away from. Google Cloud shipped model routing on API Gateway in public preview (08-06): it accepts OpenAI-compatible requests and dispatches them to Gemini, Claude, or OpenAI's open-weight GPT, with rate limiting and token tracking in the same layer. Every routing result on this page has until now been either a research method or third-party middleware. This is the first time a hyperscaler has sold query-level selection across competing vendors as billed managed infrastructure rather than as a feature inside its own model family, and Google is willing to route paying traffic to Anthropic and OpenAI, which prices the gateway position above the model position.

Two things follow for this page. First, the OpenAI-compatible surface means migration cost from an existing integration is close to zero, so this is a distribution play as much as a technical one, and it compresses the margin of every startup whose product is a router. Second, and more awkward given what this page spent 08-05 arguing, no routing objective is published: there is no stated selection signal, no accuracy-versus-cost curve, and no policy documentation. VI-MoLE (08-05)'s complaint was that the field keeps using a calibrated-looking scalar as a decision rule without deriving the rule from a cost model. A managed router with an undisclosed objective, operated by a party that also sells one of the options, is that objection in commercial form and it is unauditable by construction.

A second result today argues the routing axis everyone uses may be the wrong one. Skill Entropy (08-06) defines the difficulty of a long-horizon task as the cost of the transitions between skills rather than the difficulty of any individual skill, and measures a clean skill-switching gap across 8 frontier and 4 open-source models: accuracy falls as a task's skill entropy rises, while a salient size measure fails to track empirical difficulty at all. Every router on this page selects on a property of the query or of the model's uncertainty about it. If transition cost is where the difficulty actually concentrates, the interesting router is one that detects an imminent skill switch and escalates only across the boundary, which is a per-step decision inside one query rather than a per-query decision. Nobody is building that, and it composes cleanly with VI-MoLE's budget allocation, since a skill boundary is exactly the kind of high-value-of-information moment its certificates are meant to price.

One more, from outside the routing literature, that is a routing failure wearing an agent label. Shadow evaluations (08-06) gave frontier agents six days and thousands of dollars of API credits on genuine unpublished research questions, and both runs ended with under 50% of the budget spent and hours remaining, despite being able to monitor usage and being told to spend down. That is a pure allocation failure at the task level, and it is the same object VI-MoLE formalizes as spending a global budget on whichever action buys the most certified risk reduction per unit cost. The routing literature has a formalism for the failure the agent literature just measured, and nobody has connected them.

Prior State (as of 2026-08-05)

Two papers on the same Kurate board argue that the field keeps using a calibrated-looking scalar as a decision rule without ever deriving the decision rule from a cost model. This page has been doing it too.

VI-MoLE (08-05) (2608.02528, Kurate cs.LG #5, tier 1, absent from HuggingFace) states the objection in its title: uncertainty is not enough. Nearly every mixture-of-LoRA-experts router activates more adapters when the model looks unsure, but uncertainty says the model does not know, and routing needs to know whether this adapter would help. Those are different quantities, and a model can be highly uncertain on a query no adapter improves while being confidently wrong on one a specific adapter fixes. VI-MoLE reformulates routing as certified value-of-information allocation: learn counterfactual risk after each expert prefix, convert to simultaneous upper-risk certificates on calibration data (simultaneous matters, because a pointwise bound does not survive being applied thousands of times), then spend a global adapter budget on whichever actions buy the most certified marginal risk reduction per unit cost. It proves simultaneous certificate validity, optimality of greedy allocation under diminishing certified gains, and a regret bound under value-estimation error. The evaluation axes are better chosen than most routing papers': matched-compute accuracy, certificate coverage, risk-coverage tradeoffs, distribution-shift robustness, and tail latency, the metric adaptive-compute work usually omits and the one that decides deployability.

Its Kurate neighbour "Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agents" (cs.LG #3) makes the identical complaint about agent tool acquisition: a relevance score is not a stopping rule. Same board, same week, same structural criticism at two different layers.

This also pairs with the pruning-side result the page already carries. Beyond Geometric Complementarity (07-31), now Kurate cs.LG #1, found expert-subspace similarity cannot determine redundancy, invalidating similarity-derived compression ratios. VI-MoLE says uncertainty cannot determine activation value. Two routing decisions, prune and activate, and the incumbent signal is indicted for both. VI-MoLE's per-adapter marginal-value-per-cost objective is also the within-query analogue of CLEAR's shadow price for rationing compute across a batch; the two compose and neither cites the other.

What makes this actionable now: Mixture-of-Kittens (08-05), Cursor's open-source MoE training megakernel, cuts MoE dispatch cost by 2.37x on isolated layers and 1.41x end to end, and LLaDA MoE v2 (08-05) finds that at scale, larger expert pools win at fixed activated capacity. Cheaper dispatch plus more experts widens the space of routing policies that are affordable, including the unbalanced ones routing-quality research keeps wanting and load-balancing losses keep suppressing.

Prior State (as of 2026-08-04)

The first routing result on this page where the thing being routed is neither a model, a computation, nor a context slot, but a write into memory. Raven (2607.25357, Kurate cs.LG #16 at ai_rating 7.0, the highest on either board this week and absent from HuggingFace entirely, from Arshia Afzal and Volkan Cevher at EPFL with Aviv Bick, Eric Xing and Albert Gu at CMU and Cartesia AI) keeps a fixed set of memory slots inside a linear-time sequence model and, at each step, a learned input-dependent router decays and updates only a selected subset, leaving the rest untouched. It sits between two extremes: state-space models write densely so nothing is evicted but everything interferes, and sliding-window attention writes sparsely with exact in-window recall but evicts by position. Competitive with or better than linear-time baselines on recall-intensive benchmarks, holding at 16x training context length.

Every routing decision this page has tracked is query-conditioned: TRACER (04-17) picks a model per query, CaRE (05-11) picks a task-axis expert, MISA picks per-head KV, Kilo's plan/implement split (06-16) picks by phase. Raven's router fires per incoming token, before any query exists, which makes it structurally admission control against a fixed budget rather than dispatch. That is precisely the shape InMind (07-29) said was missing when it measured the implicit-association blind spot, where retrieval-based memory only surfaces a fact when the fact resembles the query, so a stored tree-nut allergy never fires on a macaron request, and six vector, graph and agentic systems reached at most 14.4% on indirect queries against 84.0% when the memory was simply placed in context. Its stated open problem was deciding which facts must stay visible, decided before the query is known, which cannot be query-conditioned by construction. Raven does not solve InMind's benchmark, which concerns an agent's external store rather than a layer's internal state, but it is the first architecture in this wiki whose routing decision has the right causal structure, and the 07-28 digest prediction that someone would build a route-to-memory tier within 90 days is now partially scored: not at the agent layer where the prediction was aimed, but one level down in the model's own parameters.

On the industry side, the MoE routing configuration turns out to be a network-topology decision that the interpretability literature has been reading as a representational one. SemiAnalysis on Kimi K3 (08-04) derives the MoE communication-to-computation ratio as (P*F)/(6*m*B) * (1 - 1/E), in which the expert intermediate dimension m is the only model-configuration term, so raising m lowers the ratio and lets more communication hide behind computation. LatentMoE compresses routed tokens before dispatch, and halving Kimi's latent input dimension from 7168 to 3584 let active experts double from 8 to 16 at unchanged communication volume. SemiAnalysis reads the same pressure behind DeepSeek V4 Pro, MiniMax M3, MiMo V2.5 Pro and Inkling all raising expert intermediate dimension: as hardware FLOPs rise and expert weights get quantized harder, F grows, so m must grow. Read against Beyond Geometric Complementarity (07-31), which found expert subspaces overlap substantially while routes still beat matched alternatives in every one of 39 factorial cells and concluded that pruning by representational similarity is invalid, the two results are complementary and slightly deflating: expert count and dimension were set by a communication budget, so asking what they mean representationally was asking the wrong question of a number chosen by the interconnect. Kimi K3 also ships quantile balancing, a hyperparameter-free auxiliary-loss-free load balancer that solves for the router bias directly from the distribution of router scores relative to the routing cutoff rather than nudging it by a tuned coefficient, so bias updates shrink naturally as the router balances.

And the industry side gives the page a name for a pattern it has been circling for seven weeks: routing in shipped coding products is decomposition by role, not selection by difficulty. Kilo's second production dataset in five days (10,643 code reviews across 13 models) reports that 32.3% of attributed reviews used a different model than the one that wrote the code, most commonly Step 3.7 Flash writing and Laguna M.1 reviewing, and frames authoring and reviewing as different jobs teams already split. That is the third published production role-split from two vendors, after Kilo's plan/implement split (06-16) and Cursor's planner-worker agent swarm (07-27). Plan, implement, review. This changes how the page should read its own negative results. When is routing meaningful (07-20) found many reported routing gains vanish under honest accounting of the router's own cost and of what a well-chosen single model would have achieved, and IBM's system-cost work (07-15) found routing is mispriced at the system level. Both measure selection by difficulty, which is nearly the entire academic literature, and neither obviously binds role decomposition, because a role split is not choosing the best single model for a query, it is assigning stages of one task to models with different cost-quality profiles. That may be the cleanest available explanation for the gap the 07-30 entry below names between a literature that struggles to justify routing gains and a market pricing OpenRouter at roughly 70x revenue.

Kilo's second finding is a measurement-validity result and belongs with this week's cluster of them. Models agree far more on what is wrong than on how bad it is, so the severity axis every escalation workflow depends on is the least consistent output the models produce. And the widely repeated claim that open-weight reviewers are less secure turns out to be carried by one outlier model: remove it and the gap mostly disappears, with open-weight reviewers matching closed leaders on critical findings, which makes the correct decision model-specific rather than tier-specific. Same shape as Beyond Geometric Complementarity (07-31) below and as Eviction as Estimation (08-03): the instrument everyone trusts is not measuring the quantity it is trusted for. Kilo's own methodological fix is worth borrowing, which is to normalize findings per review before comparing so a reviewer that flags everything does not win on volume.

A second new routing axis today, this one in distribution space. ROPD (08-04) (2607.27081, Kurate cs.AI #18) realigns a maliciously fine-tuned model by modelling the divergence between the aligned and compromised output distributions rather than fitting the attacker's prompt template, which is the template-invariant object and is what buys robustness when the defender cannot observe the attacker's template. The title oversells the routing framing, since nothing dispatches queries between models, so the honest placement is next to multi-head latent control (07-27) rather than in the model-selection family. Meanwhile the cost-side market kept validating what the literature cannot reproduce: Alibaba shipped Qwen3.8-Max priced below Kimi K3 with open weights due in a week, DeepSeek V4 Flash cut coding prices 99%, and Kilo itself was acquired by Anaconda.

Current State (as of 2026-07-31)

The intuitive explanation for why expert routing works has been measured and it does not hold, which invalidates the standard method for deciding which experts to prune. Beyond Geometric Complementarity (2607.28308) separates three quantities that MoE routing evidence routinely conflates: route coherence (does the router pick sensibly), candidate quality (is the picked expert good alone), and candidate-by-context interaction (does its advantage depend on the prefix). Using an Expert Subspace Separation Index, matched-route residuals, and a prefix-controlled 2x2 factorial, it reports three paired contrasts across six MoE architectures.

Expert subspaces overlap substantially, contradicting the geometric-complementarity story, yet actual routes still explain token representations better than matched alternatives. Across 39 factorial cells in OLMoE, Mixtral and DeepSeek, the selected candidate beats the strongest unselected rival in every cell, while the real prefix narrows that advantage everywhere: all interactions negative, every 95% confidence interval below zero. And the geometric narrowing does not imply functional redundancy, because adding later-ranked experts still improves next-token prediction in 24 of 39 frozen-route comparisons, with Top-2 beating Top-1 in all three seeds of a controlled training study. The authors name the joint pattern coherent overlap.

The operational consequence is a pruning result, and it is negative. Expert pruning and expert merging both usually select what to drop by representational similarity. This paper says similarity cannot determine redundancy or pruning value in exactly the regime where it is used. The correct instrument is a frozen-route intervention, which is cheap and directly functional. Any MoE compression ratio selected by subspace similarity should be re-derived.

This is the third layer at which this page's July results have found the same thing. When is routing meaningful (07-20) found many reported model-selection routing gains vanish under honest cost accounting. IBM's system-cost work (07-15) found routing is priced wrong at the system level because the router's own latency and error rate get excluded. Now expert-level routing turns out to work for a reason nobody had verified. In all three cases the mechanism is fine and the published justification for it does not survive controlled measurement, which is a measurement-validity thread rather than a routing thread, and it now runs alongside the wiki's KV-eviction-ablation and agent-benchmark-protocol threads.

On the industry side, routing between open and closed weights got its first published production dataset. Kilo, the application and routing layer that does not train or host models, published deployment data showing open-weight models now carry 79% of its coding workload, with the stated position that closed frontier models still win the hardest problems while open weights win the other 80% on cost, privacy and control, and that "the future isn't open vs closed, it's routing between both" (blog, summary). Their companion test had Kimi K3 for planning plus Grok 4.5 for implementation build the same embedded database as Claude Opus 5 for $1.27 against $31.71, scoring 93 versus 98 with identical crash-test results, and the five-point gap came entirely from tests, docs and code hygiene rather than correctness (blog).

That 25x price gap at a five-point quality cost is the strongest concrete number this page has for the procurement-side routing gain the academic comparisons do not model, and it sits directly against the 07-30 entry below: a literature that struggles to justify routing gains, and a market pricing the aggregation layer at 70x revenue. The reconciliation may simply be that the literature measures accuracy-per-query and the market is buying dollars-per-task, and those diverge by an order of magnitude when the open-weight tier is good enough for four fifths of the work.

Current State (as of 2026-07-30)

The market has put a price on the routing layer, and it does not match what the routing literature says routing is worth. Stripe is reportedly close to acquiring OpenRouter for around $10 billion, roughly 70x the three-year-old startup's ~$140 million annualized revenue (The Information). OpenRouter is a pure-play aggregation and routing layer giving developers access to hundreds of models; revenue is about $12 million a month, up nearly threefold since April, on relatively low costs.

That multiple prices routing as durable infrastructure. This page's two most rigorous recent results argue the opposite. When is routing meaningful (07-20) found that many reported routing gains do not survive honest accounting of what the router itself costs and what a well-chosen single model would have achieved. IBM's model-routing system-optimization work (07-15) argued routing is priced wrong at the system level, because the router's own latency and error rate are usually excluded from the comparison. The gap between a 70x revenue multiple and a literature that struggles to justify routing gains is the sharpest research-versus-industry divergence currently on this page, and it has a clean falsifier: if any frontier lab ships first-party cross-vendor routing, the aggregation layer becomes a feature rather than a company.

Two supporting signals point the same way as the market rather than the literature. Kilo's acquisition by Anaconda came with a Gradient Ventures partner's claim that portfolio companies shifting work off proprietary models save 50-80%, which is a routing gain measured in procurement rather than in benchmark accuracy, and it is a category the academic comparisons do not model. And Cursor Start launched in India at ₹649/month (roughly $7) bundling Grok 4.5 and cloud agents, which is only viable if the underlying model mix is being routed hard against cost.

A new adjacent axis: routing across context windows rather than across models. OpenAI reported that two settings tripled its ARC-AGI-3 score by letting the model retain reasoning across context-window boundaries via canonical compaction. That is a scheduling decision about what state survives a boundary, structurally closer to admission control than to model selection, and it extends the routing-over-context-occupancy axis this page opened on 07-29. It also cost nothing in model capability, which is the part that should worry anyone selling routing as a capability product.

And the most aggressive version of routing is deletion. TurboVLA (07-30) reaches 97.7% on LIBERO at 0.2B parameters and 0.9 GB VRAM by removing the language model from the vision-language-action pathway entirely, rather than routing around it. Read alongside DSPy task-model separation (07-25) and its 550x cost gap, the family of moves is consistent: the large general model is increasingly something to be avoided rather than dispatched to, and routing is one point on that spectrum rather than the endpoint.

Current State (as of 2026-07-29)

A new routing axis arrives from a memory paper, and it is the first one where the thing being routed is neither a model nor a computation but context occupancy. InMind (2607.24368) measures what it calls the implicit-association blind spot: retrieval-based memory only surfaces a stored fact when the fact resembles the query, so a stored tree-nut allergy never fires on a macaron request, because the bridge (macarons use almond flour) is world knowledge the retriever cannot see. Its paired controls isolate the failure precisely. The backbone answers 84.0% of indirect queries when the decisive memory is placed in context, and the same memory systems recall those facts on demand at up to 100%, but when the memory must be retrieved, six vector, graph, and agentic systems reach at most 14.4%. Eightfold embedding dimensionality does not close it.

The paper names the open problem as routing, and specifically as deciding which facts must stay visible, decided before the query is known. Every routing decision this page tracks is query-conditioned: TRACER (04-17) picks a model per query, CaRE (05-11) picks a task-axis expert, MISA picks per-head KV, Kilo's plan/implement split (06-16) picks by phase. Routing-what-stays-resident cannot be query-conditioned by construction, because the whole failure is that the query does not tell you what you need. That makes it structurally an admission-control problem against a fixed context budget, much closer to KV cache admission than to model selection, and it is the first entry on this page that a serving engineer rather than a model-selection engineer would own.

It also scores the prediction the 07-28 digest made, that someone would build a "route to memory before routing to a model" tier within 90 days. InMind is not that system. It is the benchmark that would grade one, and it establishes the headroom at roughly 70 points.

Industry note on the routing-as-product line. Kilo Code published a fresh head-to-head of its Auto Model router against manual model selection on a backend build, with the framing that once a good plan exists, the model that builds it moves cost far more than it moves the result (blog.kilo.ai). That is the same claim as the Kilo plan/implement split (06-16) and the Kilo routing audit (06-07), now stated as a cost thesis rather than a quality one, and it lands in the week enterprises are contesting AI coding-tool renewals over exactly that cost.

Current State (as of 2026-07-25)

Routing stops being a layer in front of the model and becomes a feature inside it, a shipped default at Microsoft, and a $10B acquisition target, all in one week. The page has tracked routing mostly as research plus a thin production layer. Three same-week industry moves change its status.

(1) Anthropic ships routing into the API. Claude Opus 5 (07-24 launch) adds fallback routing (classifier-blocked requests are re-routed to a recommended model based on the request rather than failing) and an explicit effort dial defaulting to high with a fast mode at ~2.5x speed for 2x base price. The effort dial is the productized, per-request form of what CLEAR (06-05, ration reasoning tokens across a batch by a global shadow price) argued for at the batch level, and it makes the Kilo Code audit's (06-07) "more reasoning is not monotonically better" finding actionable per call rather than per deployment.

(2) Cost-per-task, not capability, is now the frontier vendors compete on, and the pool got genuinely diverse. Cursor's CursorBench 3.2 has Opus 5 High at 66.7 for $3.91/task, Fable 5 High at 66.5 for $8.77, and Grok 4.5 High also at 66.7 for $1.51. Three models tied within 0.2 points across a 5.8x cost spread. This is the exact precondition When is routing meaningful (07-20) said routing gains require (a pool genuinely diverse in cost at comparable capability), and it now demonstrably holds at the frontier, not just between a frontier model and a small one. It simultaneously squeezes Sakana's Fugu Ultra v1.1 claim (that a router over commodity models beats Fable 5 without calling it) from both directions: a frontier model now matches Fable 5 at half price from above, and a flash model ties it at a sixth from below.

(3) Microsoft made cost-aware routing the production default, and the aggregation layer got priced. Satya Nadella's MAI post says Microsoft routes production traffic in GitHub Copilot, Excel, and Outlook to its own smaller MAI models whenever they match or beat frontier alternatives on the specific task, keeping frontier models for work that needs them. The Decoder's read is that this is an Azure economics play (the more open and in-house models running on Azure, the less Microsoft pays OpenAI and Anthropic), and notes MAI performs significantly worse in independent benchmarks, which makes the per-task qualification the load-bearing part. Kilo reports its own Auto Efficient mode at 71% of frontier completion rate for 72% lower cost with no custom models. Meanwhile Stripe is reportedly in talks to buy OpenRouter for close to $10B against a $1.3B last valuation, roughly 8x for a routing-and-billing layer that owns no models. The page's long-running claim that routing is cost control rather than a research nicety now has a dollar figure attached to the aggregation layer itself.

(4) The engineering precondition finally gets named. Every result above quietly assumes you can swap the model without breaking the system. DSPy's "separating the task from the model" argument (AI Engineer talk 07-23, ingested 07-25) supplies the thing that makes swappability real and treats it as the architectural decision. Its claim is that a complete task specification needs three languages (natural-language instructions for what should happen, code for what must happen, examples for what good looks like), and once all three exist the model becomes a swappable implementation detail that automated search can optimize. Reported evidence: Shopify cut an AI workload's cost 550x by holding the contract fixed and letting search find the cheapest model that still passed. Read together with Dan Farrelly's same-conference argument that context (models, prompts, tools) has a half-life of weeks while the execution layer lasts years, there is now a three-way convergence, DSPy plus Farrelly plus Microsoft MAI, on one structural claim: isolate the model behind a stable boundary, because the model is the fastest-decaying part of the system.

The open lever is unchanged and now more valuable. Microsoft, OpenRouter, Kilo, and Anthropic are all routing on a quality-floor-or-safety rule. None of them routes on coverage (predict which model catches which failure class), which the 06-07 audit identified as the unexploited gain implied by disjoint coverage. If a quality-floor router is worth $10B, a coverage-aware one is the obvious attack, and DSPy-style search over a fixed contract is the machinery that could find one.

Prior State (as of 2026-07-24)

The routing-as-substitute thesis gets its strongest public claim, from a product. Sakana's Fugu Ultra v1.1 router claims to beat Anthropic's Fable 5 on Sakana's own eval without Fable 5 in the routing pool, plus up to 7.9 points over v1.0, and adds a Claude-Code-compatible endpoint. This is the commercialized descendant of Conductor (Sakana's RL orchestrator over frontier models), and the boldest form of the "a router over commodity models substitutes for one expensive model" idea the page tracks. It must be read against When is routing meaningful (07-20), which argued routing only pays off when the pool is genuinely diverse in cost and capability, exactly the condition a "beat the model we don't even call" claim needs to justify. No independent verification exists yet and the service is EU-unavailable, so this is a claim to falsify, not a result. The Claude-Code endpoint is the concrete part: Fugu is positioning as a drop-in agentic-coding backend competing on economics, the same battlefield as the flash-model price war (Ling 3.0 Flash, Grok 4.5). → summary

Current State (as of 2026-06-18)

The learned orchestrator now confronts its own training cost (SciOrch, 2026-06-18). SciOrch (arxiv 2606.15872) trains a lightweight 8B model to orchestrate frontier commercial LLMs for hard scientific reasoning: it decomposes each question, delegates sub-problems to selected models via API calls, and synthesizes a final answer. It is the fourth datapoint in the page's "routing IS the policy" line after Conductor (05-11, 7B RL orchestrator over frontier workers, beats every individual model at ~3 calls/question), Maestro (05-23, 4B outcome-only RL orchestrator over a frozen model-skill registry, beats GPT-5 on multimodal), and Orchestra-o1 (06-15, 8B DA-GRPO orchestrator with decision-aligned reward, +10.3% on OmniGAIA). What SciOrch adds is a constraint the prior three sidestepped: when every routing action is a paid frontier API call, online RL rollouts are infeasible. Its answer is to decouple exploration from optimization. MCTS (Monte Carlo Tree Search, which explores decision branches by sampling rollouts and backing up their values) mines diverse orchestration trajectories once, the per-node single-turn samples become a reusable offline dataset, and GRPO-style training optimizes the orchestrator on that dataset without further live calls. The result lands on the same side as Conductor and Maestro: the 8B router reaches 56.66% average accuracy on a 240-question SGI-Reasoning + SFE test set, beating the strongest single commercial model by 3.74 points and the strongest multi-agent baseline by 3.33 points, at under half the API cost of typical multi-agent methods. The open economic question shifts from inference cost to training cost: the MCTS mining phase still issues many paid calls, so the amortization point (how many served queries before offline-trained SciOrch beats running a multi-agent baseline online) is the production number the paper leaves unstated. → summary

Current State (as of 2026-06-16)

Capability is phase-localized: three same-day datapoints say decompose the loop and right-size the model per piece. The page's trajectory-routing axis (pick the model per step from trajectory signals) gets its sharpest production instance and two architectural confirmations on one day. Kilo's plan/implement split ran both Claude Fable 5 and GPT-5.5 on the same task: the strong model wrote the clearly better plan (rubric 9.1 vs 8.3), but when both implemented the winning plan, both passed 15/15 acceptance with identical rollout — at $6.30 (GPT-5.5) vs $16.66 (Fable 5). Plan-strong/implement-cheap produced the same service for 59% less. This adds a third finding to the Kilo Code audit (06-07, "more reasoning is not monotonically better" + "coverage is disjoint"): the expensive model earns its cost in the hard-to-verify planning phase; the cheap model suffices for the verifiable implementation phase. FastContext (06-16, Microsoft) is the subtask version — a trained 4B–30B exploration subagent finds code for a fixed solver, cutting tokens up to 60% and lifting resolution up to 5.5% — the coding-agent cousin of S2L-PO's small-model explorer. VibeThinker-3B (06-16) supplies the architecture-level reason: its Parametric Compression-Coverage Hypothesis holds that verifiable reasoning compresses into a small core while knowledge needs broad coverage — so the verifiable, plan-constrained phases (implementation, exploration) are exactly where a small/cheap model is correct, not merely tolerable. The standing tension with the learned-orchestration line (Conductor, Orchestra-o1) sharpens: a hand-specified two-phase split already captures most of the cost win, setting a bar any RL-learned router must beat.

Current State (as of 2026-06-15)

Orchestration is routing, and now it is a learned, modality-aware policy (Orchestra-o1, 2026-06-15). Orchestra-o1 (arxiv 2606.13707) extends the page's "routing IS the policy" thread (Conductor 05-11, RL orchestrator over frontier models; Maestro 05-23, RL-orchestrated model/skill ensemble) into the omnimodal regime where text, image, audio, and video coexist in one task. It does modality-aware task decomposition, online sub-agent specialization, and parallel execution, beating the second-best system by 10.3% on OmniGAIA. The technically interesting piece is DA-GRPO (decision-aligned GRPO), which rewards the orchestrator's routing/decomposition decisions rather than only final-answer correctness, training Orchestra-o1-8B to SOTA among open omnimodal agents. That decision-aligned reward is the orchestration analogue of the RL orchestrators above and joins today's broader GRPO-variant cluster (S2L-PO, AdaSR's HRPO) where three same-day papers each rebuild GRPO's advantage assignment for a different structure. The open unification the page has tracked widens again: a router that jointly emits (which model, which experts, how many tokens, which operator, which sub-agent), all priced by one signal, with the orchestration layer now a learnable policy rather than a hand-built prompt graph. → summary

Current State (as of 2026-06-09)

Input-conditioned compute allocation arrives at two new granularities the same day — and one of them ships to phones. The page has tracked routing across models (Conductor), tasks (CaRE), heads (MISA), recovery strategies (RePoT), and token budgets (CLEAR). Today adds the intra-network operator and per-prompt expert-set granularities. Chiaroscuro Attention (CHIAR-Former, arxiv 2606.08327) routes each token to one of three mixing operators (cheap DCT, RBF, or full self-attention) by its per-token spectral entropy, a complexity signal. Routing collapses to DCT+attention only (the router rejects RBF), and the two-operator variant hits 45% lower perplexity on WikiText-103 at 62.5% fewer attention FLOPs — spending the expensive operator only on hard tokens, the same instinct as sharpness-routed LIVEditor/ISA (05-07). Meanwhile Apple AFM 3 ships per-prompt adaptive compute into hundreds of millions of devices: AFM 3 Core Advanced is a 20B MoE that decides once, up front which experts (and how many active params, 1–4B) to use for the full model depth. This is the consumer-scale product proof of the routing-as-compute-allocation thesis CLEAR (06-05) and the Kilo audit (06-07) argued at the batch and model layers. The open unification widens: a router that jointly emits (which model, which experts, how many tokens, which operator), all priced by one signal. CHIAR-Former's spectral entropy is a candidate signal that could also drive KV decisions (cf. today's FlashMemory LSA).

Current State (as of 2026-06-07)

The production face of the top routing layer (which model), priced in dollars and bug coverage. The Kilo Code audit (2026-06-07) is the field datapoint behind all the research above on the model-selection layer. Same task (a webhook service with 17 planted bugs as answer key), Claude Opus 4.8 at four reasoning levels vs MiniMax M3: M3 found 13/17 for ~$0.07, the cheapest Claude run found the same 13 for $1.30, Claude xhigh/max led at 15/17 but cost 10x+ more. Two findings matter for the concept. First, more reasoning is not monotonically better (Claude medium/high caught an async-transaction bug that xhigh/max missed; max cost 67% more than xhigh for nothing better) — the production echo of the recurring "more compute is not monotonically better" result the page has tracked since the 05-29 hybrid study and CLEAR (06-05). Second, coverage is disjoint: the cheap and expensive runs that tied on count did not catch the same bugs, which argues the optimum is an orchestrated ensemble keyed to bug class (cf. Conductor, MAESTRO), not cheapest-capable selection alone. The urgency is economic: GitHub Copilot moved to usage-based billing on 2026-06-01 and Uber reportedly burned its 2026 AI-coding budget by April, so routing-as-cost-control is now a line item, not a research nicety. Open lever: coverage-aware routing (predict which model catches which issue class) rather than cost-aware routing alone. → summary

Current State (as of 2026-06-05)

A third scarce resource gets routed: the reasoning-token budget across a batch (CLEAR, 2026-06-05). Where the routing thread above is about which model or which recovery strategy, CLEAR (Constrained Latent-utility Equilibrium Allocation for Reasoning, arxiv 2606.03092) routes how many tokens each query in a batch gets to think. It models per-query compute-utility as an S-shaped curve and casts batch-level budget allocation as constrained optimization governed by a global shadow price (the marginal accuracy per token, equalized across funded queries). Its distinctive move is rational abandonment: stop spending on insolvent queries and reallocate to solvable ones near their accuracy-emergence threshold, for up to 3x global accuracy over uniform allocation under scarcity. This is the batch-level, compute-amount counterpart to the compute-location routing the wiki tracked all week (Perplexity local/cloud orchestrator 06-04, NVIDIA OpenShell 06-03) and the per-query over-thinking cut of ThoughtFold (06-04). It also reinforces the recurring "more compute is not monotonically better" finding from a fresh angle: the optimum abandons some queries entirely. The open unification is a router that emits a (model, token-budget) pair priced by one shared shadow price.

Current State (as of 2026-05-31)

Recovery strategy becomes a routed decision (2026-05-31): Adaptive RePoT. RePoT (Recoverable Program-of-Thought) recovers a failed plan from its last verified checkpoint with one extra LLM call, and ships Adaptive RePoT, a rule-based dispatcher that routes between suffix repair (resume from the verified prefix) and fresh retry based on verified-prefix length. The reason it needs a router at all is a capability-scaling pattern: checkpoint repair helps strong models (decisive win on Gemini), ties on GPT-medium/Claude, and hurts the weakest (loses on GPT-mini). This is the same trajectory-level routing as Step-level Optimization for Computer-Use Agents (05-02, which escalated to a frontier model on a Stuck signal), but the routed decision is a recovery strategy and the downside is real, picking the wrong route lowers accuracy. That makes it a sharper test of trajectory routing than the upside-only escalation cascades the wiki has collected so far, and it adds a concrete instance to the recurring "more compute is not monotonically better" finding (cf. the 05-29 hybrid cloud/device study, which reached the same verdict on the accuracy/cost/energy frontier). → summary

Current State (as of 2026-05-27)

MoE design points at both extremes (2026-05-27): MobileMoE and MiniMax-M2. Two papers fix the MoE active/total ratio to a deployment constraint rather than a fixed architecture choice. MobileMoE derives an on-device MoE scaling law under joint mobile memory and compute limits and finds a sweet spot (moderate sparsity, fine-grained + shared experts) that beats on-device dense LLMs at 2-4x fewer FLOPs and runs 1.8-3.8x faster prefill on real phones. MiniMax-M2 goes the other way: a 229.9B-total / 9.8B-active open MoE (~23:1) tuned end-to-end for agentic deployment, where the mini-activation footprint is the economically correct shape for long token-heavy agent trajectories. Both pair with MoE-muP (re-confirmed today on Kurate cs.LG #14, ai_rating 9.0): MoE-muP makes scale-up principled, MobileMoE makes scale-down principled, MiniMax-M2 pushes the active ratio to a deployment extreme. The MoE routing surface now spans pre-training scaling (MoE-muP), per-token activation (BEAM), post-training dynamic conversion (ZeDA), and deployment-constrained sizing (MobileMoE / MiniMax-M2).

Current State (as of 2026-05-19)

Latest addition (2026-05-19): ZEDA post-trained static-to-dynamic MoE conversion. Existing dynamic-MoE methods require pre-training from scratch or task-specific adaptation, which makes them inapplicable to frontier post-trained static MoEs. ZEDA (Zero-Expert Self-Distillation Adaptation) converts a post-trained static MoE into a dynamic one. It injects parameter-free zero-output experts into each MoE layer (the explicit option to skip computation) and adapts via two-stage self-distillation, using the original frozen MoE as teacher with a group-level balancing loss. On Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks, ZEDA eliminates >50% of expert FLOPs at marginal accuracy loss, beats the strongest dynamic-MoE baseline by 6.1 and 4.0 points, and gives ~1.20x end-to-end inference speedup. With ZEDA, the MoE routing surface now has four orthogonal directions: MoE-muP for pre-training scaling (2026-05-17), BEAM for per-token activation (2026-05-16), HodgeCover for resident-count compression (2026-05-18), ZEDA for post-training dynamic conversion. → summary

Prior State (as of 2026-05-16)

Addition (2026-05-16): BEAM dynamic expert-activation masking. Fixed Top-K MoE routing wastes computation on easy tokens and underserves hard ones. BEAM (Binary Expert Activation Masking) replaces it with token-adaptive trainable binary masks: a small gating head per layer learns which experts to activate, with a straight-through estimator carrying gradients through the binarization and an auxiliary regularization on activation count enforcing a budget. Ships with a custom vLLM kernel that exploits the binary structure to avoid the gather-scatter overhead that has historically killed dynamic-K MoE inference. >98% performance retention, up to 85% MoE-layer FLOP reduction, 2.5x faster decoding, 1.4x higher throughput. The routing surface now has seven addressable internal layers (model, adapter, expert-set-per-token, expert-router-per-task, distillation loss, decoding head, latent code) plus the orthogonal profile-design axis. → summary

Prior State (as of 2026-05-15)

Latest additions (2026-05-15): the design space opens on two new axes. Two papers reframe routing from "router mechanism" to "routing target and signal." RouteProfile treats LLM profiling (the structured description of what each candidate model is good at) as an independent design surface with four dimensions: organizational form (flat vs structured), representation type (text vs embeddings vs scalars), aggregation depth (raw vs summary vs deep abstract), learning configuration (frozen vs trainable). Across three representative routers and standard-plus-new-model-generalization settings: structured profiles beat flat ones, query-level signals beat domain-level signals, and generalization to newly introduced models benefits most from structured + trainable profiles. → summary. Dynamic Latent Routing (DLR) moves routing into the LM post-training pipeline itself: joint learning of discrete latent codes, routing policies, and model parameters in a single stage, motivated by the General Dijkstra Search theorem (globally optimal goal-reaching policies recoverable through temporal composition of sub-policies). +6.6 pp mean gain over SFT in low-data fine-tuning; mechanistic analyses show the learned codes have distinct causal roles. → summary. Together with yesterday's MinT (catalog routing) and Orthrus (cache-as-coordination), the routing surface now has six addressable layers: model, adapter (MinT), expert (CARE), distillation loss (Cliff), decoding head (Orthrus), and latent code (DLR), plus an orthogonal profile-design axis (RouteProfile).

Prior state (as of 2026-05-11)

2026-05-11 additions: Routing has now visibly moved inside the model. Three new papers on the same axis: Conductor (Sakana, ICLR 2026) trains a 7B model with RL to orchestrate frontier workers (GPT-5, Claude Sonnet 4, Gemini 2.5 Pro), beating every individual model on GPQA-D / LiveCodeBench / AIME25 at ~3 calls per question; the orchestrator decides topology and per-worker prompt content as a single RL policy, generalizes to unseen agent mixes, forms recursive topologies when allowed to self-call. CaRE (HF) introduces Bi-Level Routing MoE for continual learning: a task-router selection stage above the expert-routing stage, scales to 300+ non-overlapping tasks (where flat MoE routers collapse). MISA (HF) routes the indexer-head axis of sparse attention, treating the 64 indexer heads as an MoE pool. Three papers in 24 hours making the same architectural claim: routing is the policy, not the wrapper. The wiki's three-axis framing (query / provider / trajectory) now needs a fourth: model-internal routing. DAIR.AI's weekly top-papers email (Gmail) independently surfaced Conductor plus HeavySkill (RLVR-trained parallel-deliberation as an inner skill) and described both as "harness wins look like model wins" — same pattern from a third source.

Prior state (as of 2026-05-04)

Routing is an active research and production concern operating on three distinct axes that the May 2026 batch makes explicit:

1. Query-level routing — given a query, pick a model. TRACER (04-17) is the canonical surrogate-routing example.
2. Provider/tier routing inside a harness — pick provider, model, fallback chain, cheap-vs-primary tier per turn. Ken Huang Ch 14 (05-01) is the most concrete public read.
3. Step-level (trajectory) routing — in multi-step agentic systems, pick the model per step based on signals from the trajectory itself. Step-level Optimization for Computer-Use Agents (05-02) is the first concrete mechanism.

Stacked, the three axes form a routing surface: which provider × which tier × which model per step. The wiki has now seen at least one concrete paper / harness analysis on each axis.

Three routing paradigms remain operative:

1. Surrogate routing — cheap classifier handles easy traffic, fall back to LLM for hard cases (TRACER).
2. Capability-based routing — direct queries to different models based on task type or capability match (cheap-model heuristics in Hermes are the production form; learned variants are open).
3. Agent trajectory routing — in multi-step agentic systems, optimize the path through a sequence of model calls, not just individual ones (Step-level Optimization).

Key Papers / Posts

SciOrch (2026-06-18) — A lightweight 8B model trained to orchestrate frontier commercial LLMs for scientific reasoning: decompose the question, delegate sub-problems to selected models via API, synthesize the answer. The distinctive contribution is offline orchestrator training: because each routing action is an expensive paid API call, online RL rollouts are infeasible, so SciOrch mines orchestration trajectories with MCTS, extracts per-node single-turn samples, and optimizes with GRPO-style training on that fixed dataset. Reaches 56.66% average accuracy on a 240-question SGI-Reasoning + SFE set, beating the best single commercial model by 3.74 points and the best multi-agent baseline by 3.33 points at under half the API cost. The first paper in this wiki to treat the training cost of the orchestrator (not just its inference cost) as the binding constraint. → summary

Conductor (Sakana AI, ICLR 2026, surfaced 2026-05-11) — 7B RL-trained orchestrator over frontier LLMs (GPT-5, Claude Sonnet 4, Gemini 2.5 Pro). The model writes natural-language subtasks, assigns each to a worker, and decides which prior outputs that worker sees. Outperforms every individual worker on GPQA-D, LiveCodeBench, AIME25 at ~3 calls per question. Trained against randomized agent pools (generalizes to unseen mixes). Recursive topologies emerge when self-call is allowed. Cross-source confirmed via DAIR.AI weekly email and @burkov retweet. → summary

CaRE: Bi-Level Routing MoE (2026-05-11) — Continual learning with PTM backbones at 300+ tasks. Bi-level routing: a task-router selection stage activates relevant task-specific routers, then an expert-routing stage activates and aggregates expert outputs at every intermediate layer. Introduces OmniBenchmark-1K (the first 100-to-300+ task CIL benchmark). The first task-axis routing primitive in the wiki, structurally similar to MISA's head-axis routing in inference. → summary

TRACER (2026-04-17) — Trains lightweight ML surrogates on an LLM's own production traces (free labeled data). A parity gate activates the surrogate only when it agrees with the teacher above a confidence threshold. Achieves 100% surrogate coverage on a 150-class benchmark using Claude Sonnet 4.6 as teacher. Generates interpretability artifacts for the routing boundary. → summary

Ken Huang Ch 14 — Routing and Provider Abstraction (2026-05-01) — Cross-harness comparison of how Claude Code and Hermes implement routing. Claude Code: compile-time provider abstraction, single fallback model, signature-stripping on retry. Hermes: runtime API-mode auto-detection, ordered fallback chain, live OpenRouter context-window discovery (cached 1 h), switch_model mid-session, conservative choose_cheap_model_route. The most detailed public read of production routing engineering. → summary

Step-level Optimization for Computer-Use Agents (2026-05-02) — Event-driven cascade for GUI agents: small policy by default, escalate to frontier model when learned monitors detect a Stuck pattern (progress stalled) or a Milestone (semantically significant checkpoint). Trajectory-aware routing inside the agent. Modular, no retraining. The first concrete mechanism for the trajectory-level axis. → summary

Ken Huang Ch 15 — Structured Output (2026-05-02) — Cross-harness comparison of schema-constrained generation. Both Claude Code and Hermes converge on tool-use forcing as the portable mechanism. Claude Code: SyntheticOutputTool with Ajv compile, schema-identity caching, retries excluded from agent tool budget, child agents have it stripped. Hermes: extract_structured() with portable tool-choice forcing, plus JSONL trajectory format as infrastructure-level structured output. Schema support is a per-model capability — a sub-axis of routing the Hermes flag exposes but no router yet consumes. → summary

Xiaomi MiMo-V2.5-Pro (2026-05-03) — Open-weight long-horizon coding model claiming 40-60% fewer tokens per task than Claude Opus 4.6. The pricing axis shifts from "capability ceiling" to "tokens-per-task." Mechanism not disclosed; candidates include MoE sparsity, LenVM-style length value heads, CoPD-style distillation, RL chain-of-thought truncation. Tokens-per-task is a routing-relevant signal regardless of mechanism — a router can prefer a model that is 50% as expensive at 95% of the quality. → summary

Key Concepts

• Surrogate model: a cheap ML classifier trained to approximate a more expensive LLM's decisions on a specific task
• Parity gate: a confidence threshold that controls when the surrogate is trusted vs. when to fall back to the LLM
• Coverage: fraction of traffic the surrogate handles vs. falls back to the LLM
• Production traces: labeled input-output logs from a deployed LLM — free training data for a surrogate
• Routing boundary: the region of input space where the surrogate is reliable; interpretability artifacts describe this
• Fallback chain: ordered list of (provider, model) tuples consumed in sequence on failure (Hermes); contrast with single-fallback (Claude Code)
• API-mode auto-detection: inferring the API contract (anthropic_messages / chat_completions / codex_responses) from URL and provider name rather than explicit configuration
• Signature stripping: removing model-specific extended-thinking blocks before retrying with a different model — required for cross-provider fallback to work
• Stuck Monitor / Milestone Monitor: learned signals on agent execution traces; fire when escalation to a stronger model is warranted
• Cheap-model routing: per-turn demotion to a cheap model when the user message is short, single-line, free of code blocks/URLs, free of complexity keywords; conservative by design

Open Problems

• Routing for open-ended tasks (no ground-truth labels to train surrogates)
• Multimodal routing: routing queries across text, image, and video models — Nemotron 3 Nano Omni (05-02) multimodal token reduction is one upstream primitive
• Agent trajectory routing: optimizing multi-step tool-use sequences, not just individual calls — Step-level Optimization is the first concrete attempt; Claw-Eval-Live (05-01) provides the calibration data
• Dynamic routing that adapts as model capabilities and costs change
• Cache-aware routing. Switching models invalidates prompt cache; SemiAnalysis (05-01) showed cache hits drive blended Opus pricing to $0.99/MTok. A router that knows current cache state and routes within-cache aggressively is the obvious efficiency move; nobody has published it.
• Reasoning-mode routing. Compliance vs Sensibility (05-02) shows reasoning mode is a steerable linear direction. A router that picks both model and forced reasoning mode is the deeper control surface.
• MCP server selection as routing. Ken Huang Ch 13 (05-01) made clear that MCP server selection is an explicit routing problem; today the agent picks "whichever first."
• Schema-aware routing. Hermes's ModelCapabilities.structured_output flag (Ch 15, 05-02) is a per-model feature today; routing to a model that lacks native JSON-mode requires falling back to tool-use forcing. The router that consumes this flag does not exist.
• Tokens-per-task pricing as routing signal. Xiaomi MiMo-V2.5-Pro (05-03) reframes the cost axis: a router that knows which model is cheap on this query, rather than which model is cheap in general, is more valuable in long-horizon agents. Concept now has external open-weight competitive pressure.
• Safety-as-routing-constraint. Defense Trilemma (05-02) implies the wrapper around any single model cannot be both utility-preserving and complete. Routing across models with uncorrelated failure modes is a defense-in-depth mechanism the trilemma cannot constrain. Nobody has formalized this as a routing objective.

Cross-axis composition (to track)

• Step-level × Provider/tier: stack the 05-02 step-level cascade with Ch 14 provider routing → two-axis routing surface inside one agent.
• Step-level × Surrogate: train a TRACER-style surrogate on the cheap-tier model in the cascade; 05-02 cascade only needs the small model for routine steps.
• Trajectory routing × Claw-Eval-Live: 05-01 noted that no single frontier model crosses 70% on Claw-Eval-Live; trajectory-aware routing is the cleanest open lever to cross it. Step-level Optimization is the first candidate mechanism.
• Tokens-per-task × Trajectory routing: Xiaomi MiMo-V2.5-Pro (05-03) optimizes the per-step cost; Step-level Optimization (05-02) optimizes per-trajectory routing. Composing them — a cheaper model in routine steps + Stuck/Milestone escalation — is the multiplicative efficiency play that no paper has measured.
• Schema-aware × Provider routing: a router that picks among providers where some support native JSON mode and others require tool-use forcing should bias toward the cheaper mechanism when both succeed. Hermes's flag is the substrate; no consumer exists.

Related Pages

• Inference Efficiency
• KV Cache

Recent additions

• 2026-05-08 — Netflix Tech Blog: State of Routing in Model Serving (Nipun Kumar, Rajat Shah, Peter Chng). Title-level signal only; surfaced via Gmail Medium digest. Production-engineering routing taxonomy from Netflix. Worth a manual read. Stubbed for now, full summary pending.
• 2026-06-04 — Perplexity hybrid local/cloud orchestrator. Productized local-vs-cloud query router shipped to consumers, the third in a week after NVIDIA OpenShell (06-03) and the Cloud-Device Hybrid Agents research framing (05-29). The compute-location axis (local vs cloud) is now the dominant productized routing axis, while research still lacks a benchmark scoring router decisions on cost × latency × decision quality together. The runtime market (NVIDIA, Microsoft, Nous, Perplexity) is being decided by product polish, not measured routing quality.
• 2026-06-04 — Scaling-parametrization for sparse/efficient architectures continues: Gated Delta Networks μP extends the closed-form hyperparameter-transfer programme from MoE (MoE-μP, 05-17) to gated linear attention, and Marin's open MoE recipe reports the empirical dense→MoE speedup ledger (6.7x theoretical / 3.6x realized). "Derive the scaling rule once, never re-sweep at target scale" now spans two architecture families in three weeks.
• 2026-06-08 — Disentangling Agent Self-Evolution: a routing insight wearing a self-evolution costume. It splits self-evolution into harness-updating (an evolver model writes scaffold edits) and harness-benefit (a solver model exploits them), and finds updating is flat across model tiers (a Qwen3.5-9B edits as well as Claude Opus 4.6) while benefit is non-monotonic (mid-tier solvers gain most). The cost-optimal recipe is to route the cheap model to the editor role and reserve frontier capacity for solving, the same "no model is Pareto-dominant, assign by role" logic as the Kilo audit but applied inside a single agent's improvement loop.
• 2026-06-07 — Kilo Code audit: MiniMax M3 vs Claude Opus 4.8. The production datapoint under the whole routing thesis: same code-audit task, MiniMax M3 found 13/17 bugs for ~$0.07 while Claude Opus 4.8 cost 10x+ for the same-to-slightly-better coverage — and the cheap and expensive runs that tied did not catch the same bugs. Empirically grounds the topmost routing axis (which model) in dollars and bug coverage, and argues for coverage-aware routing or ensembling (cf. Conductor, MAESTRO) since no model is Pareto-dominant. Demand-side mirror of CLEAR: with GitHub Copilot on usage billing (06-01) and Uber's 2026 AI-coding budget gone by April, routing-as-cost-control became a line item.

The latent axis opens, and industry ships the phase split (2026-07-27)

Two things landed the same day that belong together.

Multi-Head Latent Control (07-27) is the first router in this wiki that makes an inter-model decision from intra-model evidence. Two lightweight heads read the hidden-state trajectory of a frozen LLM or VLM as it generates. A Capability Head decides whether this model can finish the instance or should defer to a stronger one; a Resolution Head picks among clarify, tool use, abstain, or direct answer. Up to 90.7% fewer large-model calls on AndroidWorld, 27 to 53% fewer on average, plus up to +158% relative gain on tool-use decision quality and 65.5% fewer missed required tool calls. Because the signal is the partial generation rather than the prompt, handoff can fire mid-answer, which no input-side router can do. This adds a fifth routing axis to the page: after model-tier, task/expert, head, and trajectory, there is now latent-trajectory routing.

Note the tension with When Is Routing Meaningful? (07-20), which found learned KNN routers collapse under paraphrase while prompted routing stays stable. MHLC is a learned router and inherits the suspicion, but it conditions on a generation rather than a query, and generation-trajectory paraphrase-invariance is plausibly higher than query-embedding invariance. Untested, and it is the decisive experiment.

Cursor's agent swarm (07-27) is the same split shipped in a product. Cursor separated a planner from workers and had the swarm rebuild SQLite in Rust from documentation alone, no source code and no internet. Every configuration of the new system reached 100% on the test suite; the old swarm, without the split, choked on merge conflicts of its own making. That failure mode is the informative part: the binding constraint in multi-agent coding is write-conflict management, not per-agent code quality, so the planner's real job is partitioning rather than reasoning depth. Nobody has tested whether a cheap partitioner works as well as an expensive one.

Role-based model assignment is now settled, at five independent datapoints in six weeks. Kilo (06-16) plan-strong/implement-cheap at 59% cheaper with identical 15/15 acceptance results; Disentangling Agent Self-Evolution (06-08) harness-updating flat across tiers so cheap models edit scaffolding as well as frontier ones; Conductor (05-11) a 7B orchestrator beating every frontier worker it directs; DSPy/Shopify (07-25) 550x from fixing the contract and searching for the cheapest passing model; and now Cursor plus MHLC. The claim: no frontier model is Pareto-dominant across the roles inside a single task, and the value is in assigning roles, not picking a winner.

The open axis these five sharpen: all of them decide by phase or by instance. None decides by cache state, which the Open Problems list above has flagged as unpublished since May and which MHLC makes more urgent, since routing more often means invalidating prefix cache more often, and discarding a partial generation to hand off means re-prefilling on the larger model.

A sixth axis: adapter-level routing over a frozen base (2026-08-11)

Macaron-V1 (08-11) adds a routing granularity this page has not carried. The five axes above are model-tier, task/expert, attention-head, trajectory, and latent-trajectory. Macaron's Mixture-of-LoRA (MoL) freezes a very large base model, composes specialist LoRA adapters on top of it, and selects exactly one adapter per user turn. The flagship pairs a 744B GLM-5.2 base with four adapters (chat, agent, coding, GenUI); a 50B Qwen3.6 variant runs the same design locally.

Why this axis is different, and it is a cost argument. Every other axis on this page pays a switch penalty. Routing across separate models means separate resident weights and, as the Open Problems list has flagged since May, prefix-cache invalidation whenever the route changes mid-conversation. Multi-Head Latent Control (07-27) sharpened this: because it can hand off mid-generation, it must re-prefill on the larger model, so its 90.7% reduction in large-model calls on AndroidWorld is bought partly with re-prefill cost that the paper does not price. MoL is the first proposal where the base weights and, in principle, the base KV cache stay put and only a small patch swaps. If that holds, it is the cheapest route-switch mechanism anyone has published, and it is the first candidate answer to the routing-by-cache-state question this page has listed as unpublished since May.

The claim is not yet evidenced. No adapter-switch latency and no cache-behaviour measurement appear in the report, which is exactly the figure that would convert this from a plausible cost story into a demonstrated one. Nor is the selection mechanism specified: whether the per-turn choice is learned or prompted, what it costs, and how often it is wrong are all unstated, in a paper whose headline is selection.

Where it sits against the page's existing tensions. At four coarse, semantically distinct roles, MoL is near the easy end of the spectrum When Is Routing Meaningful? (07-20) mapped when it found learned KNN routers collapse under paraphrase while prompted routing stays stable, so paraphrase robustness is plausible here and would not transfer to a finer adapter set. The interesting threshold is adapter count: at four this is specialist selection, and somewhere past a few dozen it becomes a genuine router with a real decision problem. MinT-scale LoRA serving (05-14) already established that serving very large adapter counts against one base is tractable, so the infrastructure exists ahead of the routing policy, which is the same ordering this page has observed repeatedly.

Role-based model assignment reaches six independent datapoints, after Kilo (06-16), Disentangling Agent Self-Evolution (06-08), Conductor (05-11), DSPy/Shopify (07-25), and Cursor plus MHLC (07-27). Macaron is the first to ship the assignment inside one served model rather than across an API boundary, which is what makes it a deployment story and not only a research one.

And the routable unit keeps growing. Macaron versions the model-and-harness together as its unit of improvement. A²E (08-11) reached the same conclusion from measurement, finding no model-harness combination wins across all task types, and Evo-Bench (08-11) found synthesized harnesses transfer across policy models. Three papers on one day treating model and scaffold as a single routable object, and still nothing in production routes over harnesses.

Training-time routing: decaying router noise (2026-08-11)

A mechanism worth recording separately because it concerns how routers are trained, not how they decide. Motif 3 (08-11) perturbs its FP32 router logits with Gaussian noise on a cosine decay schedule: large early, annealed to a floor once expert roles form. The stated rationale is that early routing decisions become self-reinforcing before experts have developed meaningful specialization, so a router can lock in a bad assignment permanently. Their Figure 6b shows maximum tokens per expert falling to the median-load regime by roughly 17B training tokens with noise, against roughly 25B without, and the report claims faster early loss decrease.

Two reasons this matters here. It is a direct treatment for premature routing lock-in, which is the training-time analogue of the load-imbalance problem that inference-time expert-budgeting work like BEAM addresses from the other end. And Hugging Face's Elie Bakouch noted he had not seen router noise defined in a large-model report before, possibly since early Google MoE work, which suggests a technique that was known, dropped, and is now being rediscovered at scale. Motif 3 pairs it with Expert-Specific PolyNorm, which holds a higher effective rank in the expert gate weights than SwiGLU at every layer, meaning a flatter singular-value spectrum and less collapsed specialization. Both are load-balancing mechanisms acting on the router rather than on the loss, which is a different lever from the auxiliary-loss balancing that dominates MoE practice.

The field gets a common formalism, and a cost metric it was measuring wrong (2026-08-14)

Two things landed today that address the two structural weaknesses this page has carried since April: no shared vocabulary, and a cost axis nobody had validated.

LLMRouter (08-14) proposes a single formulation of LLM routing as a sequential decision process with five components: context encoders, model encoders, scoring functions, decision rules, and learning signals. Single-turn, multi-turn, and personalized routing all become instances of it. It ships xRouteBench (generic LLM, memory-augmented, vision, time-series, and personalized routing tasks) plus an open-source library with 16+ representative routers behind one interface, and evaluates jointly on response quality and inference cost rather than quality alone. Three results: learned routers beat the strongest fixed-model baseline by 14.6% relative; lightweight routers get more competitive as the cost constraint tightens, because a heavy router's own cost eats the savings it produces; and user-conditioned routing consistently improves personalization, so identity is a real routing feature.

The immediate value to this page is commensurability. The six axes catalogued above (model-tier, task/expert, attention-head, trajectory, latent-trajectory, adapter) are, under LLMRouter's decomposition, different scoring functions over different context encoders. That is the first frame the wiki has had that makes them comparable rather than merely adjacent. What the formulation does not cover is routing inside one model, where DLR (05-15) and MISA live, and it has no time-of-day term, which DeepSeek's new peak/off-peak pricing (08-14) just made a live variable.

The cost metric problem is worse than this page assumed. The AlphaSense study (08-14) tested models on searching and analyzing a large financial-document corpus and found GPT-5.6 Sol and Opus 4.8 produced better answers at lower total cost than Moonshot's Kimi K3 and Z.ai's GLM-5.2, despite charging $25 and $30 per million output tokens against Kimi's $15. The mechanism is that a stronger model finishes the task in fewer tokens and fewer retries, so unit price and task cost can point in opposite directions. Artificial Analysis, ranking the same models by cost-to-accomplish, puts the Chinese models significantly cheaper, so the two measurements genuinely disagree and the disagreement is about task composition rather than arithmetic.

This is a direct problem for every routing result on this page that optimizes a token-price objective, which is most of them, and for xRouteBench specifically: a benchmark that scores cost per token will rank routers differently from one that scores cost per completed query. It also sharpens the Kilo Code audit (06-07) finding that cheap and expensive runs which tie on score do not catch the same bugs. If task cost is the metric, then routing to the cheap model is only correct when the cheap model actually finishes.

Industry moved on price in the same window. Gemini 3.7 Flash shipped three weeks after 3.6 Flash claiming Claude Sonnet 5 and GPT-5.6 Terra parity at half the price, and Cognition put it in Devin reporting Sonnet-5-level FrontierCode 1.1 performance at less than half the cost with a 50% launch discount through 2026-08-27, noting it is particularly strong on tightly scoped refactors that produce minimal diffs matching repo conventions. That last detail is a routing signal, not a marketing one: it names a task class where the cheap model is not merely acceptable but preferred. Meanwhile Fable 5's slow adoption (six percent of Anthropic tokens sold, per Ramp data) is the demand-side evidence that the frontier-price tier has a ceiling, which is the market condition under which routing stops being an optimization and becomes the default architecture.

Standing gap, unchanged and now three months old: nothing in production routes over harnesses, and no paper proposes it, even though DarwinX and AutoDesign (08-14) both showed a harness transferring across base models and agent frameworks. The routable unit is the model-harness pair and the routers are still routing over models.

Value of information becomes the formalism, and routing turns out to happen at three levels of the stack (2026-08-25)

Three results landed on one day that, read together, change how this page should be organized.

Pandora's Router (08-25) prices the one thing every result above assumed was free. Google DeepMind (Fisch, Trivedi, Huot, Cohen, Kaisers, Lapata, Larson, Eisenstein) observe that all prior routing work, including everything catalogued on this page, estimates each model's expected return on an input and picks the maximizer while treating value estimation as costless with a fixed error profile. It is not. Cheap estimators (embedding predictors) are fast and noisy; accurate ones (fine-tuned scorers with retrieval results or partial reasoning traces) cost real compute. They map this onto the classical Pandora's Box problem (Weitzman 1979, optimal search when inspection costs money) crossed with information value theory (Howard 1966), and derive a closed-form value-of-information policy under a Gaussian signal model: for each specialist and input, an expression says whether refining the estimate beats the cost of refining it. A decentralized variant, Pandora's Bidder, has each model decide independently whether to pay for self-assessment before accepting a price, which is the multi-vendor serving case where no central authority can measure everything.

This is the theory for an empirical result this page recorded five days earlier. LLMRouter (08-14) found that lightweight routers become more competitive as the cost constraint tightens, because a heavy router's own cost eats the savings it produces. That is Pandora's thesis observed experimentally, by a different group, without the formalism. Worth naming the ordering: the measurement came first and the closed form arrived after. Together they also make a meta-point this page should carry explicitly. The AlphaSense study (08-14) showed the field mismeasures the cost of the answer (a stronger model finishes in fewer tokens and retries, so unit price and task cost can point opposite ways). Pandora shows the field ignores the cost of the decision. Two independent corrections in eleven days, both saying the routing literature has been working from an incomplete ledger.

VoI routing for Mixtures of LoRA Experts (08-25) derives the same formalism independently, one level down. Surfaced here because Daniel Whitmore crossed the Kurate rising-author threshold this week; it is LLM-rated (cs.LG #4-#5 across two weeks) and never appeared on HuggingFace. Its target is uncertainty-aware MoLE routers such as CARE, which set the expert count from router concentration and expert disagreement. The named error is confounding uncertainty magnitude with uncertainty reducibility: high uncertainty can mean an unqueried expert holds complementary evidence (recoverable risk, worth paying for) or that the input is irreducibly ambiguous (residual risk, where more experts are pure waste). Existing routers spend identically in both cases. The fix is to route by expected risk reduction against query cost.

The sharpest technical point in that paper is a supervision asymmetry this page should record: because all adapters sit in a shared pool over one frozen backbone, you can observe during training what the unqueried experts would have contributed, giving genuine counterfactual supervision for learning reducibility. Across-model routing has no such signal (you would have to run every model), which means within-model expert acquisition is a fundamentally easier learning problem than the cross-model routing most of this page is about.

It also supplies the mechanism Macaron-V1 (08-11) was missing. This page's recorded critique of Macaron was that its per-turn adapter selection was unspecified: learned or prompted, at what cost, wrong how often, all unstated in a paper whose headline was selection. And it answers the threshold question raised there. Macaron at four coarse semantic adapters is specialist selection; a pool large enough that you must reason about which adapter to query is the genuine router regime. Because only a small patch swaps while base weights and in principle the base KV cache stay put, adapter-level VoI routing remains the best candidate answer to the routing-by-cache-state question this page has listed as unpublished since May.

The reorganizing observation: TileMix (08-25) makes it three levels on one day. TileMix routes numerical precision over hardware-aligned tiles of the attention score matrix, dispatching each tile group through FP16 or INT8 with both paths updating one shared online-softmax state. That is a routing decision below every axis this page has catalogued. So on 2026-08-25 the same decision structure appears at three distinct levels of the stack: across models (Pandora), across adapters inside a model (VoI-MoLE), and across regions of a matrix inside a kernel (TileMix). The six axes this page tracks (model-tier, task/expert, attention-head, trajectory, latent-trajectory, adapter) are all one of those levels. Routing is not a serving-layer pattern. It is the same allocation problem recurring wherever compute can be spent unevenly.

The open problem this creates, and it is a new one. A production serving stack today makes the model choice, the adapter choice, and the precision choice independently, each with its own cost model, none aware of the others. All three are "should I spend more compute to reduce risk on this input." Jointly they are a single constrained allocation problem, and nobody has written that objective down. That is now the highest-value unpublished item on this page.

Standing gap, unchanged and four months old, but Pandora is the first tool for it. Nothing in production routes over harnesses and no paper proposes it, even though A²E and Evo-Bench (08-11), DarwinX and AutoDesign (08-14) all showed harnesses transferring across base models and agent frameworks. Harness selection is exactly the expensive-inspection regime Pandora's formalism targets, since you cannot know whether a harness suits a task without partially running it. A Pandora-style policy whose boxes are model-harness pairs and whose inspection cost is a partial rollout is the concrete unwritten paper.

2026-09-11: a fourth level appears, the denominator changes to watts, and the market moves underneath

Three things landed on one day, and together they do more to this page than any single paper has.

Intelligence per Watt (09-11) (arXiv 2511.07885, Stanford + Together AI, resurfaced across the X feed) adds a fourth level to the stack this page named on 08-25 and changes the objective. That entry recorded the same allocation decision recurring at three levels on one day: across models (Pandora's Router, which prices the cost of estimating which model to use and derives a closed-form value-of-information policy), across adapters inside one model (VoI-MoLE, which separates uncertainty that querying more experts can reduce from uncertainty that it cannot), and across regions of a matrix inside a kernel (TileMix, dispatching FP16 or INT8 per tile of the attention score matrix). Intelligence per Watt routes across devices, local accelerator versus cloud, and it is the first of the four where the resource being allocated is physical rather than financial.

The measurement is serious: 20+ local models under 20B active parameters, 8 accelerators, one million real single-turn queries, with accuracy, energy, latency and power recorded per query. Hybrid local-cloud routing cuts energy, compute and cost 60% to 80% against a batched cloud baseline, which is the efficient comparison, not a naive one. Local serviceable coverage went from 23.2% to 71.3% between 2023 and 2025 while intelligence per watt improved 5.3x. An iPhone 16 Pro achieved roughly 7x higher intelligence-per-watt than workstation GPUs on the same model at the same precision. And the strongest single result for this page: a pool of 20+ local models beat 3 frontier cloud models on 3 of 4 benchmarks when each query was routed to the best member of the pool. Model diversity substituting for model size is the most aggressive version of this page's thesis anyone has measured.

Watts are a more durable objective than dollars, and this page should say so. The AlphaSense study (08-14) showed that unit token price and task cost can point in opposite directions because a stronger model finishes in fewer tokens and fewer retries, which invalidates any router scored on cost per token. Energy is immune to that confusion: a retry burns joules whether or not it burns marked-up tokens, and no vendor can reprice a watt. Every routing result on this page optimizes a quantity its supplier controls. This is the first that does not. The residual is also narrow rather than diffuse: on the hardest reasoning slice about 95% of problems remain unsolved locally even as easy and medium tiers improve fast, so the cloud's remaining job is small and hard, which is precisely the shape that makes a router valuable.

The market moved the same day, and in the direction the page predicted. FT-reported OpenRouter data has proprietary models' share of routed queries falling from roughly 60% to 25% within months, with the routing system directing each request to the lowest-cost model expected to produce an adequate answer, so open models gain usage without any customer choosing them. AT&T is reported moving a fast-growing share of its AI work to cheaper open models at savings up to 80%. This page recorded in August that Fable 5's slow adoption, six percent of Anthropic tokens sold per Ramp data, was demand-side evidence that the frontier-price tier has a ceiling, and named that as the market condition under which routing stops being an optimization and becomes the default architecture. A 35-point share shift in months is that condition arriving. The mechanism is worth naming precisely: the substitution is happening at the router, invisibly, not at the buyer's model-selection screen.

And a product shipped the thesis. Sakana's Fugu Ultra v2 is reported beating Opus 5 on hard reasoning benchmarks (Chartography 48.3 versus 27.3, DeepSWE 74.3, first place on 5 of 8 hard benchmarks) using a router over a pool of cheaper models with no frontier model in the pool at all, with Fugu Max priced at $2/$6 per million tokens, 40-60% below Sonnet and GPT-5.6. Treat the benchmark claims as vendor-reported. The structural point survives the discount: this is the pool-diversity result from the Stanford paper, shipped as a product, in the same week.

What this does to the page's standing open problem. The 08-25 entry said a production serving stack makes the model choice, the adapter choice and the precision choice independently, each with its own cost model, none aware of the others, and that nobody has written the joint objective down. There are now four choices, not three, and the fourth has a different unit. The joint problem is "where should this computation physically live and at what precision," which is also the question DeepSeek V4.1 Flash (09-10) answers architecturally by putting 196B of its parameters on host LPDDR instead of HBM. Device routing, memory-tier placement and precision selection are one allocation problem under a power budget, and that objective is still unwritten.

The harness gap, now five months old, got its first hard evidence. This page has flagged since May that nothing in production routes over harnesses and no paper proposes it, even though harnesses demonstrably transfer across base models, and that the routable unit is the model-harness pair. NVIDIA's SoL-Pi (09-11) does not route, it searches offline for one good harness, cutting tokens 45-49% at about 94% of task score. But its EdgeBench result, where SoL-Pi on GPT-5.6 Sol beats that model's own native Codex harness, is the first direct measurement that the model-harness pairing is not fixed by the vendor and that choosing it is worth real money. That is the empirical precondition a harness router needs, and it is now on the record.

2026-09-12: the router's option set is not what the model card says it is

Provider variance on OpenRouter (09-12) describes a failure mode no result on this page models. Every routing paper here decides between models with different capability and prices the decision in accuracy against cost. An aggregator like OpenRouter decides between nominally identical models, routing one model ID to whichever backend provider is cheapest and available, and the identity does not hold: providers run different serving software with different optimizations and settings, some lack vision capability on models that nominally have it, and reasoning-effort parameters are interpreted differently. The mitigations are provider.only to pin the provider set and the /endpoints method to enumerate who actually serves a model ID.

The severity comes from the silence. A provider without vision support does not return "unsupported"; a differently-mapped reasoning-effort parameter does not error. Your evaluation ran against provider A and your traffic is served by provider B, and nothing in the response says so. The discipline that follows: evaluate against a pinned provider set, deploy against the same set, and treat adding a provider as a version bump requiring re-evaluation.

It puts a boundary underneath the boundary that the Handoff Tax (09-07) priced. That paper found the dominant escalation penalty is not transport cost but that carrying the weak model's reasoning into the strong model is actively harmful, with trajectory-dropping lifting recovery from 47% to 64% on Claude and 36% to 84% on GPT. Its scope was the model boundary. A provider boundary sits underneath it, invisible to the application, and an agent trajectory crossing it is crossing a boundary nobody registered.

It also compounds a cost problem the KV cache page has carried since 08-29, that prompt-cache entries are keyed to a model so mid-session routing voids the cache and re-prefills at full price against a cache-hit rate billed at roughly 10% of base input. The cache is keyed not merely to the model but to the serving instance, so being routed to a different provider on turn two is a cold prefill even though the model ID never changed. On a long agentic prefix that is a large invisible bill.

The gap worth naming: this is a practitioner post with no measurements. Nobody has published how often routing crosses providers in practice, the output-quality delta between two providers on one model ID, or the resulting cache-hit rate. One fixed eval run across every provider serving a single popular model ID would settle it, and it is cheap. LLMRouter (08-14) and Google Cloud's router (08-06) both assume a well-defined option set with measurable characteristics; in a multi-provider aggregator, capability is assumed from the model card, and the model card describes the weights, not the deployment.

2026-09-13: routing over a dependency graph, where the ordering is the product

The Menu Is an Execution Prior (09-13) (arXiv 2609.09395, Kurate cs.AI #19, absent from HuggingFace) is a routing result on an object this page has not previously treated as routable: the tool menu, the short ordered subset of available tools an agent is shown before execution and from which alone it may call.

The failure it names is diagnosable and every existing constructor has it. Rank by request relevance and you reliably surface the tool that performs the user's stated goal. The tools that produce that tool's inputs are described in vocabulary the request never uses, so they rank low, get omitted, or get placed after the consumer. The agent then cannot construct the inputs, or burns turns discovering it needs something not on the menu. Relevance ranking optimizes the marginal utility of each item; a multi-step task needs the joint feasibility of the set.

The mechanism. A state path is a pre-execution route from the observable request state to the desired outcome. The encoder represents which tools can run from the current state, how each tool's outputs satisfy later tools' inputs, and which orderings recur in training paths. The retriever must cover three roles rather than one: an executable entry point, the missing-input producers, and the final action. The reranker then places producers before consumers.

Results. ToolBench online success 0.737 to 0.898, with the agent unchanged, beating retrieval, reranking, generation and routing baselines. The coverage number is the more interesting one: a 32-tool state-path menu covers more complete execution chains than the official 128-tool list, a 4x reduction in menu size at better coverage. The gain persists across executor families of different capacity.

Why this is new for this page. Everything here routes an input to a destination: a query to a model, a token to an expert, a request to a precision tier. This routes to an ordered set whose internal ordering is the product, which is why the authors call the menu an execution prior rather than a selection. It is the first entry where the router's output is a plan sketch handed to the executor as a constraint.

It is also an uncounted token-cost result. Tool schemas live in the prompt prefix on every turn. 128 schemas to 32 is a direct cut to per-turn context, and it lands in exactly the region Ken Huang's prefix-stability argument (09-13) identifies as the cached head of the prompt, where one changed byte at token N invalidates the KV state for everything after N. A smaller menu is cheaper twice: fewer tokens, and a shorter stable prefix is easier to hold byte-identical. Neither source mentions the other's axis.

And it closes a mechanism gap on tool calling. That page has recorded repeatedly that tool-selection accuracy degrades with library size, with better embeddings as the standard mitigation. This says the embedding was never the bottleneck for multi-step tasks, because the missing tools are semantically distant from the request by construction. That is a prediction, not just a benchmark win: embedding-based retrieval should help on single-step tool use and never on chains.

Caveats. ToolBench only. The state path is learned from training paths, so it inherits their coverage bias, and no test uses a library with unseen dependency structure. The encoder and reranker run before every execution and their cost is not accounted, so the net saving from the 4x menu reduction is unquantified. Nothing handles tools whose output-to-input compatibility depends on runtime values rather than types.

2026-09-14: routing gets a price list and a harness hook, and a batching problem nobody has priced

Two items on 09-14 move this page in opposite directions, and the contradiction between them is the useful part.

The price list. How Close Are Open-Source Models to GPT-5-Class Performance (09-14) replaces the "have open models caught up" question with a per-task taxonomy and attaches costs. On the Artificial Analysis Intelligence Index (nine evaluations including Humanity's Last Exam, GPQA Diamond, Terminal-Bench and long-context reasoning) the best open weights trail the best proprietary model by about six points: Claude Fable 5.1 at 66, Kimi K3 and GLM-5.3 at 60, Qwen3.8 at 58. The average hides the shape. Open models are at or above parity on retrieval, embeddings, reranking, structured extraction, classification and OCR; within a few points on general coding and reasoning; clearly behind on hardest math and science, multimodal breadth, and long-horizon agentic work. Qwen3-Embedding-0.6B serves at roughly $0.011 per million tokens, which removes any remaining argument for paying per token for a retrieval layer. Self-host footprints: gpt-oss-20b at 16GB on consumer hardware, Qwen3.8-27B at 14-17GB in 4-bit on one 24GB card, gpt-oss-120b at ~80GB on an H100-class GPU, Kimi K3 at ~1.4TB of VRAM, an 8x B200 node, roughly $32K per month. The framing this page should keep: the choice is not cheap-open versus expensive-API, it is a per-token bill versus a GPU bill plus operations, and utilization is the only variable that decides which is smaller.

This confirms from the commercial side what TRACER (04-17) and the Netflix state-of-routing survey (05-08) established as research, that the right granularity for model selection is the request rather than the deployment, and it extends the cloud-device hybrid thread and Perplexity's hybrid deployment (06-04) by naming the escalation boundary: escalate on hardest-reasoning, multimodal and long-horizon agentic steps, keep the rest local.

The harness hook. LangChain's custom agent harness guide (09-14) states that swapping the model based on task complexity is runtime control implemented as middleware, not a prompt instruction. That is this page's subject matter appearing as a named slot in a production agent framework, which is the first time routing shows up here as shipped scaffolding rather than as a research system or a gateway product.

The unpriced conflict. Per-step routing across many small models assumes model switching is cheap. SemiAnalysis's 4-hi HBM analysis (09-14) spells out that self-hosted serving economics are dominated by amortizing one read of the weights across a large batch, which is why batching is the whole game in inference cost. Routing fragments batches. Every additional model in the pool splits the request stream, lowers per-model batch size, and erodes exactly the amortization that makes a GPU bill competitive with a per-token bill. No routing paper on this page reports the throughput cost of switching granularity, and no serving analysis reports the accuracy cost of coarsening it. That crossover, routing granularity against achieved batch size against cost per successful task, is now the most decision-relevant missing number on this page.

2026-09-16: a fifth level (the skill library), and two opposite answers to "where should the router live"

Gavel / The Router Within (09-16) (arXiv 2609.15982, Tsinghua IIIS) adds a fifth level to the stack this page has been building, and it is the first one where the option set is documents rather than compute. The levels so far: across models (Pandora's Router, 08-25, which prices the cost of estimating which model to use and derives a closed-form value-of-information policy), across adapters inside one model (VoI-MoLE, 08-05, separating reducible from irreducible uncertainty), across regions of a matrix inside a kernel (TileMix, 08-25, FP16 or INT8 per tile), and across physical devices (Intelligence per Watt, 09-11, local accelerator versus cloud under a power budget). Gavel routes across the agent's own skill library.

The mechanism is the important part: the router has essentially no parameters. Two linear maps read the frozen agent LLM's mid-layer hidden states. A glance scores the whole library against compact per-skill banks that one forward pass builds at installation time; a verdict resumes forward passes on the shortlist and reads the model's own likelihood and yes/no judgment, fused as a product of experts. No skill text enters the context before selection. On Qwen3-32B it beats progressive disclosure (the Claude Code and Codex pattern of preloading all skill metadata) and retrieve-and-rerank pipelines carrying 1.2B to 16B external parameters, by up to 13.4 points, rising to 21.9 points when the need for a skill arises mid-rollout, which is exactly where request-vocabulary retrieval is worst.

One result on this page has never appeared before and it changes the shape of the harness-router problem: routing accuracy improves as the backbone improves. Every router catalogued here is a fixed-capability component bolted onto a model. Gavel rides the base model's curve. That is the property a harness router would need, and this page has flagged the harness gap as standing and unaddressed since May.

The second 09-16 item is the opposite answer to the same question, and the contrast is the useful part. Jev (09-16), launched by Diogo Almeida (InstructGPT and RLHF co-author at OpenAI) after two years in stealth, is a model that emits structured decisions and probabilities over a predefined option set and never writes natural language at all. Claimed at 20-200x faster and 40-400x cheaper than getting the same decision out of an LLM, at $0.042 per million input tokens with output tokens free. One independent developer report in the feed: ~5,000 requests for about $2 spanning classification, model routing, intent and steering. All vendor numbers, no benchmark, no paper, and the launch was amplified by roughly a dozen accounts in one day with the cadence of paid promotion, so discount heavily.

But the structural point survives the discount, and it is a direct hit on this page's central economic constraint. Every router must cost less than the decision it makes, or it eats its own saving. That constraint is the entire reason Pandora's Router exists: it formalizes when paying to estimate is worth it. If a decision primitive really is 40-400x cheaper than an LLM call, the value-of-information framing changes from "is estimation worth paying for" to "how good is a near-free estimate," and those are different papers. It also does not worsen the batching problem this page named on 09-14, since a tiny discriminative model does not fragment the request stream the way an extra generative model in the pool does.

The head-to-head that settles it does not exist yet and is cheap. Gavel says the routing signal is already inside the agent model and should be read out for free. Jev says the routing signal should come from a separate purpose-built model that never generates. Same question, opposite answers, same day. One skill-selection benchmark run both ways, with latency and dollars reported alongside accuracy, would resolve it. That is now the most decision-relevant missing experiment on this page, displacing nothing: the routing-granularity-versus-batch-size crossover from 09-14 remains the missing number, and this is the missing comparison.

Composition nobody has tried. Gavel's glance is a cheap full-library scorer and The Menu Is an Execution Prior (09-13) is a reranker that orders a shortlist so input-producing tools precede their consumers. Gavel removes skill text from the context; the menu paper shrinks the set from 128 tools to 32 at better chain coverage. Glance to score, state-path to order, is a coherent two-stage skill-and-tool router and neither group has noticed the other.

2026-09-18: the decision primitive gets its independent numbers, and the routing signal turns out to be transmissible

The 09-16 entry said of Jev that it was "all vendor numbers, no benchmark, no paper," amplified by roughly a dozen accounts in one day with the cadence of paid promotion, and instructed the reader to discount heavily. Two days later the independent reports arrived, and the structural claim survives the discount better than expected. Jev is Diogo Almeida's decision-only model, which emits structured decisions and probabilities over a predefined option set and never writes natural language, claimed at 20-200x faster and 40-400x cheaper than extracting the same decision from an LLM.

The third-party evidence from the X home feed, with its weaknesses stated:

• A WebMCP browser-agent benchmark (@0xidanlevin) reports Jev paired with a cheap LLM solving 100% of tasks at roughly 112x lower model cost than GPT-6 Astra using computer use with code execution, and 245x lower than screenshot-based computer use. Solving 100% means the benchmark saturated, so the cost ratio is the only informative number and the accuracy comparison is void.
• A production-judge comparison (@drewdil) benchmarking against judges already in production: 35 of 36 correct classifying sales-call transcripts into one of 32 call types, at a fraction of the cost and latency. The author is explicit about the shape where it works: picking a label from a fixed list you defined, when the evidence is in the input. That is the most useful characterization anyone has produced, and it is a narrower claim than the vendor's.
• A fraud-detection pipeline (@nutlope) classifying 100 emails in 1.42 seconds, then routing only the uncertain cases to a frontier model: 96/100 correct for about $0.07. This is a cascade, and cascades are the oldest trick on this page; the contribution is that the cheap tier is now nearly free.
• A Jev-compatible open API (@ekzhang1) built on Qwen3.6-35B-A3B with SGLang radix cache for prefill reuse, running 64 tasks in under a second. This is the important one for this page: if an open model plus a radix cache reproduces the useful behaviour, the capability is a serving pattern rather than a proprietary model, and the vendor's pricing is not the constraint.

The 09-16 entry's structural argument is the one that matters and it now has support. Every router must cost less than the decision it makes or it eats its own saving, which is the entire reason Pandora's Router (08-25) exists to price the cost of estimating which model to use. If a decision primitive really is two orders of magnitude cheaper than an LLM call, the value-of-information question changes from "is estimation worth paying for" to "how good is a near-free estimate," and those are different papers. The independent reports do not verify the vendor's 40-400x, but they are consistent with a large enough ratio that the reframing holds.

The practitioner use cases enumerate a routing taxonomy this page has been assembling one paper at a time. One widely-shared list (@amitiitbhu) names: re-ranking retrieved results, tool pruning during context compaction, model routing across small/medium/large, query routing between SQL, vector and graph stores, cache admission deciding which responses are worth caching, and cache TTL prediction. Cache admission and TTL prediction are new to this page and they are real routing decisions that nobody has written a paper about. They are also the cheapest possible integration point, because a wrong answer costs a cache miss rather than a wrong output.

The counter-signal is substantive and should be recorded with the rest. The most-shared application was instant context compaction: score every tool call and drop what is irrelevant, with a screenshot showing a Claude Code session going from 904.7k tokens (90% of a 1M window) to 86.5k. @theo's objection is that this misunderstands compaction: compaction is not a filter, its role is to clean up history to keep the agent focused, used sparingly when context gets long, not constantly to keep context small. The screenshot supports the skepticism on inspection, because a large share of the reduction is 217.3k of MCP tools moving to deferred rather than Jev scoring anything. The mechanism is oversold in the viral version and the debate is unresolved; note also that the same-day harness component ablation found context management's value comes almost entirely from preventing overflow failures, which is closer to @theo's framing than to the aggressive-filtering one.

The head-to-head this page named on 09-16 as its most decision-relevant missing experiment still does not exist. Gavel says the routing signal is already inside the agent model and should be read out of its mid-layer hidden states for free. Jev says it should come from a separate purpose-built model that never generates. One skill-selection benchmark run both ways, with latency and dollars alongside accuracy, would settle it, and the week's evidence has made both sides stronger without comparing them.

But a third result quietly favours Gavel's half of the argument. C2C (09-18) projects one model's KV cache directly into another's with a learned per-layer gate, reporting a 2.5x speedup and better accuracy than text-based agent communication, on the premise that forcing model-internal state through a human vocabulary destroys information. If internal state is rich enough to be worth transmitting between models, it is rich enough to be read for routing inside one, which is exactly Gavel's claim. It is a point on the board for reading internals rather than re-deriving them through a separate model.

Batching, unfinished. The 09-14 entry named routing-granularity-versus-batch-size as the missing number. Sample Count Is Not Enough (09-18) does not supply it but establishes the order of magnitude: at a fixed candidate budget, eight serial single-candidate calls cost 4.64 to 4.86 times the GPU energy of one batched call of eight. Fragmentation is expensive. A router buying accuracy by splitting traffic across four models is paying for part of it in destroyed batch efficiency, and no router on this page subtracts it. Jev's shape is the exception worth noting: a tiny discriminative model does not fragment the generative request stream at all, which is a structural advantage over model-pool routing that has nothing to do with its per-call price.

2026-09-20: the primitive finds its use, and it is not the one this page predicted

The 09-18 entry catalogued a routing taxonomy assembled from practitioner reports and noted that context compaction was the most-shared application while judging its mechanism oversold. A 72-hour community census of the Jev ecosystem (the closed decision-only model that returns a typed choice plus a calibrated probability in one forward pass, never generating text) now settles which uses the market actually picked, and neither of the top two is model routing. The public project list went from 46 entries to 160. Full write-up: the 72-hour census.

The largest project by stars is a context compactor, not a router. fast-jev-compaction (3,169 stars) scores each tool call and each result in an agent transcript and deletes what is no longer needed, leaving the rest byte-identical rather than rewriting it into a summary. Reported: a Claude Code session from 156,000 tokens to 62,000, window utilisation 78% to 31%, 10 of 16 entries kept verbatim. Codex and Pi ports exist. This page has treated routing as a choice among models or among skills. Compaction is routing over the agent's own history, where the option set is "keep this span or drop it" and the decision is made thousands of times per session. That is a higher-frequency routing problem than anything else catalogued here, and the reason the decision primitive suits it is exactly the property the 09-18 entry identified as structurally advantageous: a tiny discriminative model does not fragment the generative request stream.

The second-largest category is evaluation. LangChain published Jev-as-a-Judge, arguing a typed evaluator beats an LLM judge on repeatability and cost because it cannot phrase the same verdict two different ways. A judge is a router whose option set is a rubric. This page has no entry on evaluation as a routing surface and should.

A new category with no literature at all: the database predicate. PostgreSQL and DuckDB extensions expose the model as a row predicate, so WHERE jev(people, 'could work from home') runs with no index and no embedding, at roughly ten seconds per thousand DuckDB rows. That is routing over rows rather than over models, it is the cheapest possible integration point, and nobody has written a paper about it.

Nimble is the datapoint that changes the economics on this page, more than any benchmark this week. Bespoke Labs released an open 9B decision model built in one day: a LoRA fine-tune of Qwen3.5-9B on 2,676 examples, with no distillation from the closed model and no RL, open weights, open data, open recipe. On their own 324-example held-out set: base 66.36%, a 27B base 84.88%, Nimble-9B 90.12%, the closed product 93.21%. Median latency about 106ms on an H100. Pandora's Router (08-25) exists to price the cost of estimating which model to use, on the premise that a router must cost less than the decision it makes. This page has since driven the router's inference cost toward zero. Nimble drives its training cost toward zero too, and that is the half nobody had priced. If 2,676 examples and an afternoon of labelling buys 90% of a frontier product, the question stops being which router to buy and becomes which router to train, and the value-of-information calculus has to be redone with a near-zero fixed cost as well as a near-zero marginal one.

A composite benchmark finally exists, and its design is the contribution. JevBench scores this model class on a geometric mean over Intelligence, Calibration, Speed and Cost, chosen so that an exceptional result on one axis cannot fully compensate for a weak one. Under it, GPT-5.6 Luna posts substantially higher hard-case accuracy and still loses the composite once latency, calibration and price are counted. The 08-14 entry recorded that this field was measuring its cost metric wrong. A geometric mean that refuses to let accuracy buy its way out of a latency problem is the first benchmark shape on this page that matches what a deployment decision actually looks like, and it should be the default form for routing results generally, not just for this model class.

The counter-signals bound the claim and three of them are load-bearing. First, a practitioner report notes the vendor documents low CJK accuracy and that on Browser Use's long-horizon browser task it scored 1/20 against a reasoning model's 17/20, because browser operation needs state-space search and backtracking. The useful shape stays what 09-18 identified: pick a label from a list you defined, when the evidence is in the input. Second, a direct price comparison puts embeddings at $0.02 against the decision model's $0.042, with the recommendation to use embeddings as a pre-filter. That puts the decision model in the expensive tier for high-volume work, inverting the framing this page has been using for a week, and nobody has published the crossover point. Third, the claim that the speed is a decoding property (parallel decoding over a fixed option set) rather than a training property would make the capability a serving feature any inference engine could ship, which is consistent with the 09-18 reproduction on Qwen3.6-35B-A3B plus an SGLang radix cache.

The head-to-head this page has asked for twice now has a third contender. Gavel (09-16) says the routing signal is already in the agent model's mid-layer hidden states and should be read out for free. The decision-primitive camp says it should come from a separate purpose-built model. Probe Guidance (09-20) is a third result on Gavel's side of the argument, from a different subfield entirely: it trains a tiny MLP on a frozen diffusion language model's intermediate hidden states to serve as the weak model in a guidance pair, replacing a full second forward pass. With C2C (09-18), that is three independent results in five days arguing the intermediate representation already carries a signal the field has been paying a second model to recompute. Three is this wiki's threshold for a pattern, so the pattern is now on the record: internal state is an underexploited free resource, and the 160-project ecosystem is betting the opposite way. Neither camp has run the other's experiment, and one skill-selection benchmark run both ways with latency and dollars alongside accuracy would still settle it.

Industry confirmation of the coarse policy, at gateway scale. Vercel AI Gateway data (09-20) reports open-weight models at 78.4% of token volume against closed models' 21.6%, up from roughly 40% in June, while closed models keep the money: Anthropic's five listed models take 36.9% of spending, and DeepSeek V4.1 Flash moves most of the tokens for 5.1% of spending. This is the largest real-traffic confirmation this page has of its founding premise, and it also embarrasses the literature. The deployed policy is not a learned router; it is a floor, send everything cheap unless you have already decided otherwise, and it is capturing most of the available saving. The marginal value of every sophisticated router on this page is whatever is left after that floor runs, and no paper here measures that residual.

2026-09-21: price was never the binding constraint, calibration is

The 09-19 entry argued the decision-primitive moat was gone because a 706K-parameter clone beat hosted Jev 99.7% to 83.6% on form-filling, and the 09-20 census argued the ecosystem had found uses this page did not predict. Day six corrects both by turning attention to the only number that matters for routing, and it is the one nobody was checking. Full write-up: the calibration reckoning.

The live failure. jev-trader, running on a real MON/USDC order book on Kuru with no option to abstain, reported the model at 85 to 88 percent confidence on nearly every call across 1,838 decisions in under ten minutes, finishing down 5.2 percent, with confidence flat and high through the entire losing run. Adding one rule, flatten under confidence 0.55, produced a 2,758-call run ending up 4.3 percent. A model that reports 86 percent on essentially every call is not reporting a probability, it is reporting a constant, and a threshold gate placed on a constant does nothing. This page has treated a calibrated probability as the property that distinguishes a router from a classifier, because escalation is the whole point of a cascade. Today is the first in-the-wild measurement of that property and it failed.

Two mechanism-level defects reported the same day, both of which would corrupt a calibration curve before anyone could draw one. The model is not deterministic: the same prompt returns different probabilities across runs. And permuting the option list substantially changes the probabilities, which is the classic position-bias defect of multiple-choice scoring and exactly what "prefill once, read the option logits" predicts. One of the open clones, Verdict at ~150M parameters, is engineered specifically around overconfident wrong answers and order-flipping, which is independent corroboration that these are the two defects practitioners hit first.

The out-of-domain gap is real and roughly six points, and it changes the commoditization claim. Kev-8B (Apache-2.0, Qwen3 base, LoRA plus pointer head, 40 minutes to train Kev-4B on one H100, 83 minutes for 8B) reports 79.6% out of domain against Jev's 85.7%. Laya at 421M scores 42.5% against 87.0% on a 77-option choice. Nimble reported 90.12 against 93.21 on a held-out set two days ago. The 09-19 claim that the moat is gone was drawn from an in-domain form-filling number and did not generalise. The honest version is narrower: the mechanism is commoditized, the training data is not.

A useful four-school taxonomy of the open wave appeared, and it is the cleanest structural read anyone has produced: (1) read the logits of a stock model with no retraining (SemIf, simple-jev); (2) train a purpose-built small decision model (Laya 421M, Von ~400M, Verdict ~150M); (3) convert an existing LLM with a decision head (Kev, Nimble, Open-Jev 2B/9B); (4) in-paint the answer slots with a diffusion model (OpenJev on DiffusionGemma, ~94ms median on an RTX PRO 6000). The census author's conclusion belongs on this page verbatim in paraphrase: not emitting tokens is easy to copy; accuracy, generalization and probability calibration are what separate the projects.

The naming fight is a measurement fight, and the classifier camp is right. The argument that this is a classifier, made today by Hamel Husain and by the OpenCode founding team, matters because the classifier literature already contains the fixes for every defect reported above: reliability diagrams, expected calibration error, temperature and Platt scaling, abstention thresholds, position-bias correction. Naming it a new category detached it from the measurement apparatus it needs. A mechanistic explainer added the reason a naive "just read the logits" clone should be expected to be worse calibrated than the hosted product: RLHF-induced distribution collapse is precisely why an instruction-tuned model's logprobs are not usable as confidence, which is why the vendor trains with a contrastive-distillation objective instead.

The reinforcement-learning consequence is the sharpest forward-looking claim of the day. A calibrated judge's value is not that it is cheap, it is that it can abstain. RL drives straight into whatever the judge rewards, so a confidently wrong judge gets found and exploited, and making it cheaper and faster only shortens the time to the exploit. Cheap makes RL scalable; calibration makes the signal trustworthy. This page has been pricing routers on cost per decision. It should be pricing them on cost per correctly-abstained decision, and no benchmark here does.

Open problem, now the highest-value one on this page. Six days, 160-plus projects, zero reliability diagrams. The 09-19 digest predicted a calibration curve within 60 days that would look worse than the accuracy numbers. The curve is still owed; the direction is already confirmed through the service entrance.

2026-09-22: the benchmark saturates from above, and routing finally gets its subtrahend

The 09-21 entry established that calibration, not price, is the binding constraint on the decision primitive, off a live trading failure where the model reported 85 to 88 percent confidence on nearly every one of 1,838 calls. Day seven adds the two measurements that entry said were owed, and a third number that is more consequential for this page than anything in the decision-model story. Full write-up: the benchmark ladder and the routing tax.

The ladder, and the frontier model wins it. Open-Jev completed five runs on JevBench's 231 public tasks: Open-Jev 2B 64.94%, Open-Jev 9B 77.49%, hosted Jev 86.58%, GPT-5.6 Luna 89.18%, GPT-6 Astra 100%. Two consequences. First, the open-versus-hosted gap at 9B is 9.1 points, wider than the roughly six points recorded on 09-21, so the 09-21 prediction that the gap holds near six through October is under pressure from the wrong direction, and the 09-19 commoditization claim looks increasingly like an in-domain artifact. Second, a general-purpose generative model saturating the suite voids the accuracy column entirely. The comparison now rests on latency and price alone, which is the category's genuine pitch but not its strong claim. The strong claim was that a purpose-built discriminative head is better at deciding. On this benchmark it places third.

The first production-pipeline number, and it localizes the miscalibration. Milvus reranked an identical candidate shortlist on 80 SciFact queries three ways: no reranking, qwen3.7-text-rerank at +0.0446 nDCG@10, and the decision model at +0.0778, the best of the three on quality. The costs: P50 latency 10.2x higher, cost per run 6.7x higher, largely because the state-plus-questions interface turned 30 candidates into 30 concurrent requests, which is interface-shaped overhead rather than model-shaped. The result this page has been waiting for is the filter experiment. At a 0.5 threshold on the returned probability, top-5 precision reached 49.4% while 18.8% of gold-relevant documents were filtered out. That is a published false-negative rate at the natural operating point, from an independent party, on a public dataset. It is not a reliability diagram, so the 09-21 open problem stays open, but it establishes the shape of the defect: the probability is usable for ranking and unusable for gating. A monotone distortion of a score leaves an ordering intact and destroys a threshold, and that is exactly the pattern of two independent failures in two days.

The number that matters most to this page has nothing to do with decision models. The vendor's own 12-page harness blueprint contains, at step 3: an Opus → Sonnet → Opus route costs 6.19 against 4.15 for pure Opus, because handing back reprocesses the whole context. A three-hop route across a model pool costs roughly 1.5x what never routing costs, and the mechanism is prefix-cache destruction. SemiAnalysis the same day supplies the theory: prefill and midfill differ precisely in whether prior context must be re-read, and decode's economics depend on sharing one tensor read across many queued tokens. This is the subtrahend this page has been missing since 09-14. Every router here reports accuracy gained per dollar of model list price; none subtracts the cache it destroys by switching. If a naive three-hop cascade is already 49 percent more expensive than the frontier model it was avoiding, a large share of published routing savings were measured against a baseline that does not exist in a cached serving stack. The blueprint concedes the point at step 8 by recommending routing by trust rather than by difficulty, a policy that switches rarely. Two further figures from it are new here: reading and searching are 56.2% of tool turns and 46.5% of tokens, while writing code is under 10%.

The mechanism is now public, and it confirms the serving-pattern hypothesis. The "parallel sampler" is sequence packing plus tree attention masks plus typed decision heads: one forward pass scores every candidate, shared prefixes are reused, branches are isolated by the mask, no decoding loop. Every component is a standard inference-engine feature, which closes the question 09-18 opened off the Qwen3.6-35B-A3B reproduction and 09-20 restated as "a decoding property rather than a training property." The capability is a serving configuration. It also predicts the order-sensitivity defect recorded on 09-21: branches sharing a packed prefix under a tree mask is the textbook setup for position leakage.

Latency parity arrives on-device, removing the last structural advantage. Laya ported to CoreML runs 3.7 ms per decision on an M5 Pro with 99.5% of ops on the Apple Neural Engine; the MLX build reports 7-14 ms under 1 GB of RAM. On a 16 GB MacBook Air, local Laya at ~45 ms beat cloud Jev at ~300 ms. Price arguments about $0.042 per million input tokens are moot against zero marginal cost, and a network round trip was the hosted product's remaining moat.

The category has begun distilling itself away. jimothy converts saved decision-model queries into 15-45 MB task-specific classifiers running 10-20x faster in a browser or on a server, with fallback when confidence drops. The decision model becomes a labelling teacher for the small model that replaces it, which is this page's routing thread meeting the distillation thread in a single artifact. Meanwhile JevHarness has an LLM write a task-specific decision policy once, freezes it, and runs only code plus fast decisions at inference, reporting a Pokémon win rate of 25% → 75% over five iterations, which is the first artifact fusing this page with harness engineering.

Ecosystem trajectory confirms the 09-20 forecast. jev-ultrafast at ~15k stars and fast-jev-compaction at ~5.8k (up from 3,169 on 09-20) keep compaction and browser control as the two largest categories. The 09-20 prediction that stars would keep rising while production adoption stalls at evaluation and compaction is holding exactly, and today's evidence sharpens why: every new integration this week was a ranker or an offline labeller, never a gate. Practitioners appear to have priced the miscalibration without measuring it.

Open problems, updated. The reliability diagram is still owed on day seven. The head-to-head against internal-state readout (Gavel) has still not been run, though Harness-Zero (09-22) is a fourth independent result arguing an external component can be absorbed into the model it serves. And a new one, now the most decision-relevant on this page: no routing evaluation anywhere subtracts destroyed prefix-cache value, and the one number we have suggests the correction is large enough to invert results.

2026-09-23: the calibration number lands in the dead zone, and a seventh routing level opens inside attention

The prediction this page has carried since 09-19 resolved today, and it resolved ambiguously, which is the informative outcome. The 09-21 digest restated it with a decision rule: an expected-calibration-error figure for a typed-decision model within 45 days, where ECE under 5% closes the question in the vendors' favour and anything above 15% means every threshold gate shipped this month is decorative. S1 Bench now publishes ECE as a first-class column across 1,999 items and six subsets, 22 days early. The hosted anchor sits at 0.076. Neither branch fires.

What 0.076 means operationally. A threshold set at 0.88 is really admitting somewhere around 0.80 to 0.95. That is fine for a content filter and dangerous for an automated refund. The category's calibration is good enough that nobody has to stop, and bad enough that nobody should stop checking. The board also shows that post-hoc calibration works and is nearly free: "AutoJev 27B + calibration" reaches ECE 0.048 at 0.795 macro accuracy, beating the plain variant's 0.069 at the same accuracy. Poorly calibrated entries exist and are visible (the diffusion variants at 0.166 to 0.178).

The 09-21 trading anomaly and today's board are both correct and do not conflict. The live run reported 85-88% confidence on nearly every one of 1,838 calls while losing 5.2%. An aggregate in-distribution ECE of 0.076 is fully compatible with catastrophic out-of-distribution miscalibration, because ECE averages over an eval set and a market order book is nowhere in it. The operational rule this page should now carry: the published 0.076 is not yours until you have measured it on your own traffic. Full treatment: day eight.

A vindication worth recording. On 09-21 Hamel Husain and the OpenCode founding team argued the category should be called a classifier so it would inherit the classifier literature's measurement apparatus: reliability diagrams, ECE, temperature and Platt scaling, abstention thresholds, position-bias correction. The apparatus arrived within 48 hours of the argument, and the leaderboard's top entries are the ones that applied it.

The primitive's use is confirmed as context control, not model selection. The 09-20 census called this; today's repo lists make it unambiguous. The dominant integrations are context sieves that judge each tool result before it enters context (winnow, fast-jev-compaction), per-turn model routers (jev-codex-router), browser-step choosers, and CLI classifiers. fast-jev-compaction is the most notable: it scores every tool call and result in one request, drops stale ones, and keeps what remains verbatim, which is a direct attack on lossy compaction from the harness side while KVMEM attacks it from the cache side the same day.

The moat narrowed twice in one day. Tinker showed any open LLM serves the interface after a $5, ten-minute fine-tune, on the observation that next-token prediction is already a probabilistic classifier. And constrained decoding delivers 100% schema adherence at full generation speed for under 40 microseconds per step in vLLM V1 and SGLang today, with no second model. So the decision model's type-safety guarantee is not a novel capability. What it adds is the single forward pass and the probability, not the format guarantee. FrontiersMind's Lumma-fev-0.6B and Pydantic's production use of semif for issue labelling at a p>0.8 threshold are the week's evidence that the open tier is now genuinely usable.

A seventh level: routing inside the attention state

ARM, top of this week's Kurate cs.LG board, is a router whose option set is memory slots. A hierarchical Gumbel-Softmax router selects which of a fixed number of slots each piece of context writes into, sigmoid gates blend new against stored content, and nothing is hard-evicted. This page has recorded routing at the query level (which model), the token level (which expert), the adapter level (09-11), the precision level (09-11), the dependency-graph level (09-13) and the skill-library level (09-16). ARM adds routing inside the attention state itself, trained jointly with the model.

Its second router is the more novel one. A learned policy decides at inference how many memory slots to read, spending less on simple context and more on inputs that need deeper reasoning. That is test-time compute allocation applied to memory bandwidth rather than to reasoning tokens, and it is the first instance of an adaptive read-width policy on this wiki.

The price cut strengthens, not weakens, the 09-21 finding

On 09-21 this page recorded that price was never the binding constraint, calibration is, on evidence that the crudest routing policy (send everything cheap) already captures most of the available saving. The 09-22 price war cut GPT-6 Sol and Luna in half, Opus 5.5 by 20%, and Opus cache reads by 60%. If the market keeps cutting prices faster than routers can learn, a learned router's payback period keeps receding, which makes calibration (knowing when not to trust the cheap path) the only axis where routing still earns its keep. See the price-war entry.

2026-09-24: an open decision model, a selector below baseline, and a ceiling on routing itself

• CLM-8B (Stanford/NVIDIA, Apache 2.0) is the first architectural alternative to Jev: a contrastive state-action model (CLIP-style infoNCE) claiming Jev-comparable zero-shot decisions at up to 9x lower latency. Its fine-tuned best-of-N selection results (81.6% DeepSWE Bo4, 87.6% Terminal Bench 2.1 Bo5) come with a detail the release does not highlight: zero-shot Jev as a selector scored below the pass@1 baseline on both (71.1% vs 73.7%; 83.1% vs 84.0%). A decision model used as a verifier can be net negative.
• Cost floor: Together's Jev-like tev1-4B trained for $17, served at $0.042/M input and free output. distil labs found Jev perfect on inbox sorting but 0.79 on pay/don't-pay, where a 4B reason-first fine-tune hit 0.98: the category's boundary is decisions that need no reasoning.
• Capital: The Information reports TypeSafe in early talks for $1B+ at $10B+, against about $200M a week earlier.
• A ceiling on routing: Self-Organizing Agent Teams beat a perfect router over members' independent answers (66.7% vs 59.0% average; +13.4 on AIME 2026). Routing is bounded by the best member's answer; collaboration is not, and the gain tracks demonstrability (ρ = 0.90). This is the first result on this page that bounds the oracle-router upper bound every routing paper reports against.

2026-09-25: the cascade gets a held-out number, and the gate's blind spot

• JEV-as-a-Judge (CMU, 2609.26550) is the first held-out test of the cheap-first-pass cascade this page has argued for since 09-21. Jev's max label probability q gates: accept at q ≥ 0.9, else escalate to GPT-6 Astra. On 510 held-out preference pairs it accepts 53.7%, scores 92.5% vs 93.1%, and pays 56.8% of the fee. Pairwise items are judged in both orders and averaged before gating, which also cancels position bias.
• The blind spot is style. On RM-Bench hard pairs (the rejected answer is more elaborately written) Jev scores 74.8% vs 94.6%, and its confidence stops identifying errors. A calibration-based router fails exactly on well-written wrong answers. That refines the 09-21 claim: calibration is the binding constraint, and aggregate calibration (S1 Bench's ECE 0.076) hides a failure region.
• Thresholds are per-workload. A GPT-5.6-fallback policy frozen at τ = 0.7 accepts 81% but loses 2.35 points.
• The same model helps or hurts depending on the escalation path. As an ungated best-of-N selector (CLM-8B's chart, 09-24) Jev scored below pass@1. As a gated first-pass judge it keeps 99% of frontier accuracy. The router's value is the fallback, not the first model.
• Same-day corroboration: Just Ask Jev screens ten alignment-failure types at median AUROC 0.886 for 63x less than LLM judges. Open tier: GLiNER2.5-Decide (340M encoder, 167 ms on CPU), an open decider-4b topping JevBench, a 150K-example open decision dataset, and AnyJev's three-tier calibration ladder (L1 temperature scaling on 100 to 500 labels).
• Pricing side: Microsoft moved Copilot Autopilot and Code to usage-based billing the same day. Metered billing is the unit that makes a cascade's saving visible to the buyer.

2026-09-26: the cascade needs uncorrelated errors, not just calibration

• Jev vs. LLMs as Rubric Judges (Penn, 2609.29769) is the counter-result to yesterday's cascade. On nine rubric-grading panels, Jev matches three flash-tier LLM judges in most comparisons at 29x to 325x lower cost, and its confidence ranks its own errors. But the LLM judges repeat 96.0% of Jev's most confident errors (50.3% expected under independence), so a cascade gains at most 1.5 points over the best single judge with cross-fitted thresholds and 2.0 with oracle thresholds.
• Reconciling with JEV-as-a-Judge (09-25). CMU's cascade kept 99% of frontier accuracy because it escalated to GPT-6 Astra, whose errors differ from Jev's. Penn escalated to flash-tier models that share them. CMU's own GPT-5.6-fallback policy lost 2.35 points. Refinement of the 09-21 claim on this page: calibration decides how much a cascade saves; error correlation between tiers decides whether it gains anything. A router's fallback should be picked for disagreement with the first stage, not for standalone score.
• Drex (Nace.AI) is the fourth decision-model architecture on this page, described as a diffusion model trained with RL. Drex 1.1 (10B total) leads Decision Index 0.2 at 52.82 vs Jev 51.67, but the per-task table shows a different model: far ahead on causal and fact-verification tasks (CLadder 77.8 vs 45.3, HoVer 73.4 vs 45.7), far behind on knowledge-heavy ones (GPQA Diamond 25.2 vs 71.4, MMLU-Pro 51.1 vs 80.5). RouterBench is a wash. A genuinely different error profile is what today's Penn result says a cascade partner needs.
• The open tier kept filling in: TypeLLM (type-safe typed decisions on any SGLang-served LLM, with shared-prefix KV reuse), AnyJev (Nokia, 719 stars), an open 4B "Lev" on a Qwen backbone, Together's Tev1 0.8B for local Macs, a Core ML GLiNER2.5-Decide at about 4x faster and 5x less peak RAM, and a ten-repo list of Jev-based Claude Code routers.

2026-09-27: the router moves from the request to the task

• Workflow economics (09-27). Ken Huang argues cheap tokens raised agent-step appetite faster than they cut spend, so the moat is deciding which jobs get premium reasoning, measured as cost per successful outcome. A practitioner critique the same day says request-level routers (Cursor's router, OpenRouter plus Jev) are structurally limited: a bare prompt lacks the repo context that sets difficulty, and switching models mid-session breaks the prompt cache. Route at the task level inside the harness instead.
• OpenRouter Fusion is the fan-out mirror of routing: up to eight models answer, an analyst reports agreement, contradiction and blind spots. It productizes the 09-26 finding on this page that tier value comes from uncorrelated errors.
• Open experiment: no one has compared request-level and task-level routing with prompt-cache costs counted.

2026-09-28: a third routing target, depth inside one model

Routers have so far chosen a model or a reasoning-effort setting. LoopFormer makes a looped transformer's loop count adjustable per request on one checkpoint, which gives a router a third target: how many loops. No paper yet trains that policy. Meanwhile the decision-model tier kept moving from claims to measurements (see the 09-28 Media Zone): the CMU judge-cascade paper (09-25 page) went viral, open clones (Julia-1, Lumma-fev, decider) appeared, and a practitioner showed a one-epoch 0.6B head can beat sub-1B deciders on a skewed data mix, a warning about benchmark sensitivity.

2026-09-29: token-level routing moves inside training

MOPD-Router (09-29) is a router whose options are teachers and whose decision happens during training. In multi-teacher on-policy distillation, each student token is scored by whichever teachers in the pool best express their post-training specialty (ExpertAlign), with no domain labels and no router parameters. It beats averaging all teachers by 5.88 points and prompt-level domain routing by 3.95. Two things carry over to this page. First, per-token beats per-prompt, the same granularity argument MoE made for experts. Second, the routing score is read off the candidates' own outputs rather than learned, which fits the 09-19 entry where the router's own cost went to zero. The same day, Continuous Depth Batching (09-29) supplied the serving side for the 09-28 depth target: per-token exits from a looped model can now be batched at up to 99% of the ideal speedup, though nobody has trained the policy that picks the depth.

2026-09-30: the decision interface moves into the serving engine, and speed becomes a tier

• Raschka on Jev (09-30): 96.47% on IMDb's 25k test reviews for about $0.65, roughly equal to a fine-tuned ModernBERT; a Jev-like API is a shared one-output scoring head over (text, instructions, candidate), softmaxed across candidates. He links TypeSafe's undisclosed RLCD to RLCR (Brier penalty on stated confidence).
• SGLang /v1/decisions and /v1/systemone: any served LLM or VLM becomes a scorer/classifier, Jev-SDK compatible. bev-decider-0.4B ships fully open. OpenAI's Decisions API (DevDay) makes it first-party. The router no longer needs its own model or vendor.
• Speed is a new routing target. OpenAI's Ultrafast tier sells up to 8x speed for 6x price, and GPT-6.1 Sol sells near-Astra quality for a fifth of the price (DevDay page). After model, reasoning effort and loop depth (09-28), latency tier is a fourth knob.
• Two decisions per request in production: vLLM office hours (10-01) introduce the llm-d Semantic Classifier: KV-cache routing picks the warm pod, semantic classification picks the model tier.
• Depth routing gets its policy: TaH2 (09-30) learns per-token loop count, the policy the 09-28 entry said did not exist yet.

2026-10-01: the router's training bill gets counted, and harnesses become routing targets

• SaveRouter (10-01) adds the term every prior entry left out: what it costs to learn the router. Running every candidate on historical queries is paid before any saving. Routing quality saturates early, so buying only informative query-model labels (33-41% of the feedback) matches dense routers and reaches break-even 1.9x to 9.5x sooner. The supervision level that minimizes steady-state serving cost differs from the one that pays back fastest. Refinement of the 09-27 "cost per successful outcome" metric: amortized router-training cost belongs in the numerator. Open: no comparison yet against zero-shot decision-model routers (09-30), which need no per-deployment labels.
• A fifth routing target, the harness. Raven (10-01)'s Host Agent routes subtasks to model-plus-harness units. After model, reasoning effort, loop depth (09-28) and speed tier (09-30), the option set now includes the scaffolding itself.
• Routing ships as a product default. GitHub's HydraFusion (in VS Code and the Copilot app's model picker) picks Single, Cascade or cross-family Critique per task; Ollama 0.35 serves Jev-style decision models locally through /v1/systemone (10-01 Media Zone). Token cost vs per-token price: Gemini 4 Argon's low price with 2x tokens per task (10-01) is another case where per-task cost reverses the per-token ranking.
• Distillation-side routing: SAKI (10-01) routes each student token to one of two loss types by a maximal-coupling accept/correct event, a routing decision whose rate equals the total-variation distance.

2026-10-02: describe the query, then route; and decision models get audited

• SeLMRoute (10-02) extracts a probability vector over readable query properties with a decision model, predicts each candidate's performance with a light router, then applies the objective (accuracy or cost). 72.08% on LLMRouterBench vs 69.23% for the best single model; the representation is candidate-independent, which shortens the payback window SaveRouter (10-01) said routers must count.
• Ordinal-scale bias in Jev 1.13 and KEV models: ordinal decisions use only 67 to 76% of the real label spread (26 to 75% at 14 points), with heavy middle-label overuse. Fixable with targeted LoRA (47% to 86%). Any router asking a decision model for a 1-to-N difficulty score inherits this.
• Batch economics: Quail's speed-of-light model puts one AI filter over 5K rows on Qwen3-4B/H100 at about 6.6 s, far below per-row API practice; filter ordering and KV reuse across rows are the levers. GLiDE (Fastino) adds uncertainty-gated reasoning to a decision model, the fifth decision endpoint in a week.
• Practitioner: a builder reported a model router inside a coding agent's harness cut median cost per task 64% with no measurable quality loss (10-02 Media Zone).

2026-10-03: route to a portfolio, pretrain the router, and the gate gets nearly free

FlexRouter, RouteFM and the decision-model wave (10-03).

• FlexRouter optimizes coverage (the chance at least one chosen model is right) with a DPP over quality and redundancy, and sizes the subset per query by greedy log-det gain. Fits best-of-N plus verifier pipelines, where picking correlated models wastes the budget.
• RouteFM pretrains a router across many routing environments to characterize anonymous models from behavior, then adapts by context: +2.23 quality points on held-out MMR-Bench with eight observations per candidate. Answers the router training-bill problem raised by SaveRouter (10-01) by amortization.
• Jev at the edge (academic study): 22.7% to 64.5% lower median decision latency than the fastest hosted LLM, 59.7% to 80.9% lower fees per correct decision on four-field contracts; wide contracts are where substitution stops. Caching gives LLMs nearly the same latency on repeated inputs, so the win is on fresh decisions.
• Price floor: Perplexity pplx-decider-27b open-sourced at 4 cents per million input tokens, free output; Amazon Strands Decider 2B open (106 ms on an RTX 3090); Cloudflare Clef-flash about 39 ms. Five vendors now ship decision endpoints.
• Practitioner: Red Hat, a ~200M classifier within 0.3 points of a 35B model on prompt injection at 6x lower latency; JevBench switched its headline score from composite to capability because speed and cost "are too easy to influence."

2026-10-04: routing by moment and routing between programs

• Claude Code advisor (10-04): escalation triggered by session state (plan boundary, repeated error, completion claim), not query difficulty. Practitioner stack: Jev for mechanical forks, Sonnet 5.5 workers, Opus 5.5 planner, Fable 5.1 advisor. No published cost or lift numbers; prompt-cache impact flagged in the docs.
• Mixture of Self-Improving Branches (10-04): a router picks one specialized harness per input (+34.8% relative on Olympiad math over Meta-Harness). Routing targets now include models, moments and programs.
• Decision models keep landing locally: Cloudflare Clef (27B) and Clef Flash (9B) on Ollama v0.35.1; Foil (10-04) adds a modeling note, that router confidence is a better health signal than load balance inside MoE.

2026-10-05: decision models in six sizes, and turn-priced routing

• Decision 2.0 and turn-priced routing (10-05). vLLM Semantic Router releases Decision 2.0 (0.6B to 27B, open). Ten community Jev builds put decision calls into Claude Code effort selection, Stop hooks, skill routing and Codex subagent model choice; one skill router was removed after 28 of 539 suggestions were used, honest evidence that one-of-N skill routing is not yet reliable.
• Cost logic: cache hits cost the same on Opus 5.5 and Sonnet 5.5 ($0.20 per million), so Opus's per-turn premium falls from ~1.88x at 20K to ~1.26x at 400K context. Route by cost per finished task; escalate with a short fresh handoff, never the whole transcript (re-caching 300K into Opus ~$1.50 vs ~$0.10 for 20K).
• Horace He flagged an inference-pricing distortion apparently driven by OpenRouter's routing (mechanism not captured). Aggregator routing is now a price-setting force, not only a convenience layer.

2026-10-06: the routing layer becomes a budget line

• Enterprise rationing: Microsoft cut projected internal Claude spend by over a third and Meta halved Claude Code seats while pushing in-house tools (subscription economics). Melius upgraded Microsoft on the thesis that enterprises want a "secure wrapper" that routes models to tasks. GPT-6's 100x price spread (Luna $0.10 to Astra $50 per M output) makes per-call routing the main cost lever.
• Planner/executor split in search: FrugalEvo uses a strong model to propose and a cheap one to implement, beating ~$50 systems for under $2.
• Routing for a value objective: Collective Bias Mitigation routes and organizes several LLMs into debate or committee topologies to cut bias (age-bias score 0.25 to 0.10), trading against inference cost (shorts).
• Decision models keep spreading: llm-d's Semantic Classifier (a lightweight routing classifier, explicitly not a guardrail) was presented at vLLM Office Hours #58; a practitioner write-up on training choice-order invariance into JEV-style System One models circulated on X. The biomedical SBERT2S1 paper (arXiv 2610.02486) finds the open RLCD recipe for System One models trails cross-entropy by 2.5-3.0 points because its reward normalization inflates gradient noise 3.6-15x.

2026-10-07: a frontier lab ships the decision primitive, and the audits arrive

• OpenAI Decisions API + five decision-model papers (10-07). OpenAI's endpoint returns yes/no probabilities, category picks or ratings ~10x faster than the Responses API at $0.10/M input. The interface this page has tracked since Jev (09-16) is now first-party at the largest API vendor.
• Works: SearchJev gates search-agent decisions (5.2x faster, 41-74% lower calibration error, BrowseComp-Plus 45% to 54%); LLM-as-Jev shows a stock 4B LLM read through option-ID logits matches dedicated clones with no training.
• New failure axis: option-label bias. Probabilities follow the option's label, not the rule written in its definition (definitions deleted, accuracy unchanged; one prompt-format line flips it). Routing tables whose route names hint at the answer are affected.
• Evidence gap: a SoK of 139 decision-engine paper families finds 50 claim a time budget and 4 measure it. In Open RAN, Jev meets a 1 s budget on 99.8% of calls vs 17.9% and 0% for two hosted LLMs.
• Shift: the question moves from "is a decision model cheaper?" (settled) to "is it specified and evaluated correctly?". SHIFT (10-07) adds per-query harness routing with an explicit execution-cost term (harness internalization).

2026-10-08: the first independent router benchmark comes back negative, and routing moves to the device

• Routing graded (10-08). LMArena ran Jev Router over 4,700+ real agentic sessions. It picks Pareto-efficient models (4 of its top 5) and its steerability (+10%) nearly matches Claude Opus 5.5 High, but calling DeepSeek V4.1 Flash directly gives similar success at 38% lower cost and 1.7x lower median latency (6.18 s vs 3.64 s; P90 30.6 s vs 14.3 s).
• Why: a router pays a per-call toll (classification latency) and, when it switches model mid-session, loses the prefix cache. SambaNova's MiniMax M3 numbers size that cache: cached input 90% cheaper, TTFT 35-88% lower. Practitioner advice (Pachaar): session pinning and model affinity.
• Device routing: Microsoft routes routine Copilot work from Claude Haiku 4.5 to on-device MAI-Code-1.1-Flash (137B/6.8B active, 3-bit, no inference charge); GitHub announced local-model routing under HydraFusion. Liquid AI opened d1 decision models (3B, 600M); Unsloth trains a 0.8B decision model in 4GB VRAM (20.7% to 74.3%).
• Shift: cache-local routing inside one model family (llm-d, 10-07: 2x users, up to 99% lower TTFT) has shown wins; capability routing across vendor families has now been measured once and lost on cost per success. The open question is whether session-state escalation (Claude Code advisor, 10-04) closes that gap.

2026-10-09: escalate on confidence ranking, and pin by session

• System Switch (10-09). A fast decision model acts; a reasoning VLM takes over when confidence is low. Offline deferral gain tracks the actor's AUROC (rank correlation 0.87), not its accuracy; in closed loop no variant finished a level. Second result in two days where good per-decision routing fails per task (after Arena's Jev Router, 10-08).
• Serving side: Dynamo session IDs make session pinning a first-class routing policy, the fix for prefix-cache loss on model switches.
• Harness side: HERMES routes repository components to resident Qwen3-8B models with a strong model only for activation and diagnosis: within 4.5 points of all-frontier at 26.2% lower cost.
• Products: Google's Gemini Agent routes each enterprise task to the most cost-effective model; Laya ships an open local decision engine (about 35 ms vs about 380 ms hosted Jev in one test); PersonTTS treats test-time compute as a three-constraint routing problem.
• Shift: the metric for a cheap router is now AUROC of its confidence, plus cache affinity. Accuracy alone does not predict either cost or escalation value.

2026-10-10: token-level routing gets an engine

• TokenRouter (10-10), cross-source confirmed via social (HF #2 + X). Per-token small/large routing (CITER, R2R, R-Stitch, Co-LLM) finally has a serving system: one subserver per model, async hops, a request parked as pending with its KV intact while the other model writes, and Markov-model-tuned delayed batching. 2.01-64.15x over the stronger baseline; honest range against reasonable serving is 2-3x. The baseline profile is the lesson: small-model inference was 4.2% of each step, cache bookkeeping over 60%.
• Pattern (n=3 in three days): keep the state, move the decision. Arena's Jev Router (10-08) lost on cost partly from cache loss on switches; Dynamo session IDs (10-09) keep a session's cache resident; TokenRouter keeps a request's cache parked across model hops. Routing cost is now dominated by state handling, not router accuracy.
• Product side: Google's Gemini agent ships Smart Routing to the cheapest qualifying model with Claude in the pool and per-project spend caps (industry, 10-10); Copilot will route local vs cloud per task by month end; Liquid's d1-3B ships a W8A8 build for INT8 edge routers (shorts). SanSi (looped decision model) gives one router every budget from one to eight loops.
• Open: TokenRouter measures speed, not quality per hop. Combined with System Switch's AUROC result (10-09), the next paper should report quality-at-cost for token-level routers at a fixed hop rate.

2026-10-11: decision models become a category, and the router learns to abstain

• Decision-model category (10-11). Microsoft-Decision-1 (Qwen3.5-9B post-trained, $0.042/M input, free output, claimed 83.5% at 85 ms over 36 benchmarks) makes Microsoft the fourth large vendor after OpenAI's Decisions API (10-07), Liquid d1 (10-08) and Jev itself. Open side: AnyJev reads next-token logits over allowed answers with no training and debiases option order; Vega settles a ball in a learned energy landscape and abstains when no valley is deep enough.
• Still missing: none of the launches reports AUROC or calibration error, the two metrics System Switch (10-09) showed actually predict escalation value. Microsoft's latency is also contested (one tester: 200-600 ms in practice).
• Memory routing: MemoType proves a single retrieval strategy has a precision ceiling on mixed memory and routes each query to a type-specific strategy (+16.18% Recall@1). Same classify-then-dispatch primitive, applied to agent memory.
• Shift: the cheap gate is now a commodity product with four vendors; the differentiator is calibration, not accuracy.

2026-10-12: the gate goes multimodal and reports calibration

• Decision 3.0, THX-01, JEV-27B-VL (10-12). vllm-sr's Decision 3.0 adds image/screen input in six sizes (0.6B-26B). HAL-X's THX-01 (322M encoder, proper-scoring-rule RL) reports ECE 0.003, the first calibration number from a launch in the category. AutoTrust's JEV-27B-VL passes 1.54M downloads. Two vendors claim the same vision-board lead.
• Pricing link: swarms route narrow child jobs to the cheapest model that passes, which is why per-seat pricing gives way to metering (industry, 10-12).
• Open: still no AUROC from any launch, and no independent calibration audit.

2026-10-13: the gate gets a price, a cloud recipe, and a rack

• Decision-model price war (10-13). SubQ Snap ($0.05/M input, free output, 384K context, <1 s at 100K) and Microsoft-Decision-1 ($0.042/M input, free output, Qwen3.5-9B base, 85 ms p50) put routing gates at roughly 100x below frontier input prices. AWS published a SageMaker recipe serving SemIf (ex-OpenJev) decision scoring on llama.cpp. Liquid's d1 MLX card shows 4-bit quantization shifts calibrated probabilities by up to ~0.2.
• Hardware routing: AMD's Helion agent rack ships with a token router that sends each request to the most cost-effective model (OCP 10-13).
• Fleet routing: Meta routes a variable share of ads traffic to a bigger model depending on idle capacity (efficiency shorts 10-13).
• Depth routing: ELR (10-13) makes loop count a per-request dial inside one checkpoint, a routing decision with no model switch and no KV handoff.
• Open: calibration still has no shared benchmark; Microsoft's own table shows the accuracy leader losing on calibration.

2026-10-14: routing moves from the prompt to the session and the role

• Session routing (10-14). Arena's Max router, trained on 2.5M real Agent Arena sessions, picks a model per multi-turn task: +14.7% net improvement at a median $3.27/task, matching Fable 5.1 (Max) at 37% lower cost. Cohere's controlled agent-team study on ProgramBench: a strong lead with 16 cheap planning workers scores 61.8% vs 63.2% all-strong at $155 vs $510; delegating only exploration and planning matches full delegation at half the cost. IBM's agent factory: all-open models at ~$26 vs $299 proprietary, driven by a ~98% prefix-cache hit rate under llm-d cache-aware routing.
• Looking Ahead status: the 10-08 call that cross-vendor routing would again fail to beat the best single model on cost per success is partially refuted by Max, on Arena's own data. Independent replication still open.
• Tension: Vals AI (10-12) found teams add little once a solo agent runs at max effort; Cohere finds +10 points. Cohere's gain comes from placing the strong model at the bottleneck, i.e., role routing.
• Shift: with gates at ~$0.04/M (10-13), the interesting routing decision is now which role in a session gets the expensive model. Calibration of that decision is still unreported.

2026-10-15: the decision model gets a first-principles architecture

• Jeva (10-15), cross-source confirmed (HF #1 + Kurate cs.AI #3). Names four requirements serialized-text decision models violate: null-aware (tell "none fit" from "unsure"), permutation-invariant, option-scalable, cacheable. Separate encoding of request and options, an immutable cached option memory, symmetric aggregation and a coverage head. Reported: near-perfect null detection, zero permutation flips, thousands of candidates, large latency cuts from cross-request reuse. Authors' numbers.
• Shift: the clone wave (09-19) converged on "stop sampling, read the logits." Jeva argues that is a dead end for routing because a sequence backbone is order-sensitive and cannot cache options. The option-order bias the 10-07 audits found is removed by construction rather than corrected (AnyJev L0, 10-11).
• Coverage head = abstain signal. For a router, "no cheap option covers this request" is the escalate decision. Jeva makes it an explicit output instead of a flat softmax.
• Reading Right, Acting Wrong (same day): across 17 fast decision systems, most wrong next actions follow a correct reading of the state; errors cluster on the same states across systems. That is where an escalation gate should fire.
• Companions: VideoJev (decision heads on video MLLMs, 2.8-3.1x faster via cached and joint inference), JevGPT (System One judges under a System Two research planner). Also Fewer Bits, More Tries (10-15): on verifiable tasks, retrying a small model eight times can replace routing up to a bigger one.
• Still open: Gavel vs Jev head-to-head (asked since 09-16); calibration of the coverage head under quantization.

2026-10-16: the cache enters the routing objective

• Whose Turn Is It? (10-16), Microsoft. Zero-shot per-turn routing for stateful spreadsheet agents. History-aware routers pass +3.3 points at matched offloading, within 1 point of always-expensive. Cheap model's per-task edge shrinks 6.8x to 3.2x on long sessions. Closed-form cache analysis: down-switches recover cache cost within two calls, up-switches never broke even.
• Agentic auctions (10-16), Google DeepMind. First-price auctions among bidding agents: competitive floor (distilled near-peers) drives price of anarchy toward zero; near-independence gives ~1 - 1/e² efficiency. Validated on EmbedLLM.
• Granularity ladder now complete: request (classic routers) → session (Arena Max, 10-14) → role (Cohere, 10-14) → turn (Whose Turn, 10-16).
• Shift: prompt-cache pricing (Argon's 95% cached discount, 10-14) makes switch direction a cost variable. Open: no trained router with an asymmetric switching penalty yet.

2026-10-17: the weak model drafts, the strong model reviews

• SpecAgent, EdgeWise, LEASH (10-17). Three step-level escalation designs on one HF day. SpecAgent: weak model drafts grounded read-only steps, strong model verifies the segment in one prefill and repairs the first bad step; within 1 point of strong-only on τ²-Bench at ~half the memory traffic and wall time. EdgeWise: step-level RL teaches the on-device SLM when to delegate; up to 62% cheaper on SWE-Rebench, 34% on Terminal-Bench 2.1, matching pure-LLM. LEASH (cross-source confirmed HF + Kurate cs.AI #18): training-free, hidden-state similarity to past failures triggers escalation, plus a shadow observer after handoff; Qwen3.5-35B-A3B + LEASH 73.66% at $0.41/task vs always-on Opus 5 72.90%, 19.2% cheaper.
• Granularity ladder extends: request → session (10-14) → role (10-14) → turn (10-16) → step and verified segment (today).
• Three triggers, one pattern (n=3): reviewer verdict, learned delegation policy, latent uncertainty. Practitioner echo: Intern-Decision's confidence handoff (10-16 HF) lifts 73.9% to 87.4% by routing 42 of 111 items up.
• Industry echo: HarnessRouter's UHP routes a task to a harness, not a model, and reports up to 2.1x cost difference from the harness alone.
• Open: no shared benchmark compares the three triggers; LEASH's failure memory needs in-domain labeled trajectories.

2026-10-18: routing reaches the single decision, and the threshold is the product

• JevAgentHarness (10-18), UIUC, surfaced via DAIR.AI on X. A harness sends each routine fork (tool, argument, stop) to a decision model and takes its answer only above a confidence threshold fitted per decision point on the agent's own trajectories. With Gemini 3.5 Flash as the agent: Jev made 27.2% of tool and 20.8% of argument choices, 14.2% cheaper, zero outcome changes across 500 tasks. Fixed 0.5 threshold: -21.5 points; Jev on every tool choice: -19.0.
• Answers the calibration question open since 10-13. Raw decision-model confidence is not usable as-is; a fitted per-point threshold is. Isolated accuracy misranks decision models (5.1-point spread in accuracy vs 57.5-point spread in task success).
• AgentDecision-4B: SFT then GRPO on post-mistake agent states with Jev labels. 91.5% vs 82.0% (Jev) on shifted tasks; per-agent 4B models 81-89% cheaper, up to 2.5x faster, within 3 points. Transfer not shown (tau-bench refit lost 8.1).
• Granularity ladder: request → session (10-14) → role (10-14) → turn (10-16) → step (10-17) → decision (today).
• Decentralized variant: MPT (10-18) trains each agent to emit a bid per step; the highest bidder acts. A learned version of DeepMind's agentic auctions (10-16).
• Industry echo: DeepSeek-V4.1-Flash-1017 reports benchmarks per harness and frames its release as a cost-per-task Pareto move (10-18).
• Open: does a fitted threshold survive a model upgrade of the agent itself? No paper refits after swapping the LLM.

2026-10-19: routing by role inside the agent team

Cohere's coding-agent teams (arXiv 2610.17471) route by job: cheap models are safe as planners, not implementers. A GPT-5.6 Sol lead with 16 Luna planners lands within 1.4 points of an all-Sol team at 70% lower cost; cheap planners recover 47.5% of the gain for 13.9% of the extra cost. Third design in three days after weak-draft/strong-review (10-17) and per-decision thresholds (JevAgentHarness, 10-18). Companion token audit: security agents spend 60.4% of tokens on comprehension; targeted reuse (AVRI) saves 18-24% where generic tricks rarely do. Market side: DeepSeek V4.1-Flash-1017 at $0.3/$1.2 per M tokens is #1 open on Code Arena WebDev. Counterweight: Anthropic's Project Deal shows cheap agents losing money unnoticed when no strong model checks them.

2026-10-20: take the model out of the router

• Meta Thin Orchestrator: production agent system for data-pipeline onboarding. A 1,300-line deterministic orchestrator routes 20+ workflows to 14 isolated specialist subagents; six of seven protocol steps have no LLM reasoning. Prep time 3-4 days to under an hour; ~2/3 of changes land without significant edits. One big agent hallucinated past a context threshold; single-agent alternative 70-80% only with per-task prompts, 10-20% without.
• Ladder update: after weak-draft/strong-review (10-17), fitted per-decision thresholds (10-18) and role routing (Cohere, 10-19), Meta's answer is deterministic routing plus context isolation. Four designs in four days; the common thread is that routing decisions should be cheap, fitted or fixed, never a frontier-model judgment call.
• Depth as a routing target: LEV uses a stopper with a threshold fitted on unlabeled traffic to choose loop depth per question: 15-24% compute saved for under 1 point.
• Role specialists: Hy-Memory-30B-A3B (3B active) matches Opus 5 as the memory writer across four harnesses.
• Serving-side: llm-d's Semantic Classifier (Red Hat) classifies request difficulty at the gateway; vLLM's KV-aware router keys on conversation_id for cache locality (AWS).

2026-10-22: the decision-model audit

Decision models under audit.

• Negative result: a Jev controller at fixed decision points in a Sonnet 5.5 terminal agent raised cost 19% at an identical 73.0% pass rate (pre-registered, 315 trials). Good offline ranking (AUROC 0.78) but miscalibrated (P(complete)=0.50 when 87% passed), so it kept ordering extra verification.
• Positive result, same family: JEV-as-a-Judge (via X) with a threshold frozen in advance beats GPT-6 by 0.9 points at 41% of its fee.
• Reconciliation: the variable is the fitted threshold, as in JevAgentHarness (10-18). Unfitted learned routing can cost more than no routing.
• DecisionPivot: two 27B decision models 2.6 points apart on aggregate differ by 70.1% vs 44.2% on contrast groups; ranking ≠ deciding.
• Harness routing: OO-Harness routes each task to an evolved harness subclass or keeps the base; edge-cloud routing benchmark EC2-Bench (2610.17816) finds stronger edge models shift the best strategy toward edge-controlled cloud calls.
• Market: Cloudflare released clef, an open decision model; Strot compiles a frozen 144M decision artifact; an IPO filer discloses "no model-routing layer" as a risk.

2026-10-23: routing ships in the box, and markets get numbers

• NeMo Switchyard on eight DGX Sparks: a lightweight classifier routes each request or session to a small-model or large-model pool on one cluster; Qwen-only quality on a tool-use benchmark, up to 3x aggregate throughput when traffic shifts small. The simplest rung of this month's ladder, now an NVIDIA product.
• Auction routing resurfaced with simulation numbers (DeepMind, 2610.20587, first covered 10-16): 110 models bidding on 30,000+ EmbedLLM prompts reach 99.2% of optimal value vs 89.4% for routing each prompt cluster to its best model; on the worst cluster the auction keeps 96.0% while the single best model keeps 6.0%. Theory: welfare loss shrinks toward a 1-1/e² bound when several cheap distilled models value a prompt highly.
• Model tier as routing: Claude Code's per-agent model setting (Opus plans, Haiku subagents read) circulated with a $0.23 vs $1.13 example run; the design rule is that the planner never reads raw worker output.
• Reasoning effort as a routing knob: MiniMax disclosed 80% to 68% coding accuracy from max to high effort (summary).
• Routing costs cache: TDS ("The Hidden Cost of LLM Routing") argues mid-session model switches discard prefix cache; OpenAI launched a Decisions API for low-cost per-decision calls.

2026-10-24: route the mode, route the reading

• RouteWeaver (2610.19705): one LLM routing policy chooses the execution mode (single-round, multi-round, agentic) and the models inside it. COMET-GRPO splits trajectory advantage into mode and execution signals; progressive exploration prevents collapse to single-round. 61.4% average, +11.03% relative over the best baseline. No cost numbers.
• Small tool selectors (2610.31483, Meta + CMU): an RL-trained Qwen3-8B (plan, retrieve per substep, select) beats prompted 402B Llama-4-Maverick on ToolBench G3 over 10K+ tools. Tool selection is a routing step that does not need a frontier model.
• Production: Shopify routes reading to small task-trained models (7x faster, ~90% fewer tokens, secondhand); Mubit trains owned small models from traffic; Gemini Smart Routing routes subtasks under spend caps; IF-MAP routes the improvement budget to the most sensitive agent (74% fewer tokens than optimizing all).
• State of knowledge: routing by subtask is now established across research and production (≥6 sources 10-23/24). Open problem: no router reward accounts for prefix-cache retention, which the 10-23 TDS note says model switches destroy.

2026-10-25: route the step size and the context size

• Adaptive Skill Granularity (2610.31461) (Kurate cs.AI #20 + DAIR pick): a controller picks primitive tools, a macro skill, or macro-then-micro fallback per task from task structure, execution state and prior macro failures; a finite-state executor makes recovery explicit. Learned router 100% on a 12-task study where a heuristic fails. One level below RouteWeaver (10-24).
• Context-price cliff (practitioner, unverified): fork cheap-model workers before a context threshold where per-token price reportedly jumps 5x; claimed $6.15 single agent vs $0.85 for nine split workers. Adds a per-worker size constraint to the 10-23 "Opus plans, Haiku reads" pattern.
• Pre-screening: Agent Behavior Maps place 100 LLM-harness pairs by action log-likelihoods without rollouts; a candidate cheap feature for routers.
• Still open: fork-or-route decisions that price the prefix cache they discard.