AI research wiki
October 22, 2026 · daily digest

cere-bro | 2026-10-22

cere-bro | 2026-10-22

The cheapest token is the one you never re-read. Today's savings came from refusing to redo work: agents edit their own KV cache instead of re-prefilling, DeepSeek's serving engine loads its memory table before the first layer runs, and a 2-bit codec shrinks that table 4x. The same day, two audits showed the cheap control layer can cost money. Most harness-evolution gains turned out to be overfitting, and an uncalibrated decision model made an agent 19% more expensive at the same pass rate.

🎯 Today's 5 for you
  1. ReadCache Engineering for Agents. Agents get two KV-cache operations (drop, repos) instead of summarizing history. That is 27.7% cheaper and +1.7 points, with no re-prefill. It sits where KV-cache work meets your top saved theme, harness engineering ([paper](https://arxiv.org/abs/2610.28577)).
  2. ReadDeepInfer + SemiAnalysis on Huawei-DeepSeek. DeepSeek open-sourced the engine behind V4.1-Flash, which runs on both Blackwell and Ascend 950. SemiAnalysis benchmarks the Ascend chip and finds Engram running in production. Serving economics plus the chip story in one read ([repo](https://github.com/deepseek-ai/DeepInfer), [SemiAnalysis](https://newsletter.semianalysis.com/p/huawei-deepseek)).
  3. ReadN-Gram Memories at Two Bits. A 2-bit codec shrinks DeepSeek's Engram table from 203 GB to 51 GB, small enough to stay on the GPU. It is compression aimed at the parameter type that is growing fastest ([paper](https://arxiv.org/abs/2610.21810)).
  4. SkimSpeculative decoding under load. DSpark and DFlash 2 are 2.6-2.8x faster at one request but slower than plain decoding at 512. Specter reuses draft tokens to prefetch MoE experts. Check this before you turn drafting on for batched traffic ([Specter](https://arxiv.org/abs/2511.14102), [EAGLE-4 table](https://x.com/theomlondhe/status/2113162304900747772)).
  5. TrackThe two audits: harness evolution and decision models. About 54% of harness-evolution gains are search-set overfitting. A Jev controller raised agent cost 19% with no gain in pass rate. Read these before trusting any "self-improving harness" number. Skip the "End of Loop Engineering" hype threads ([HDA](https://arxiv.org/abs/2610.22466), [Good Judgment](https://arxiv.org/abs/2610.24649)).

TL;DR

Deep Dives

Cache Engineering for Agents: edit the cache, not the transcript

Summarizing an agent's history saves tokens but destroys the cache. Letting the agent delete and shift cache entries directly saves both.

Source: X Following feed (DAIR.AI), surfaced via social; not on HuggingFace this window Links: Paper · Wiki summary

Edit the cache, not the transcript
Summarizing rewrites the prefix and forces re-prefill. Drop and repos leave the text alone.
flowchart LR
  T["Agent trajectory<br/><small>tools, observations</small>"] --> S["Summarize<br/><small>rewrites prefix</small>"]
  S --> RP["Re-prefill<br/><small>cache invalidated</small>"]
  T --> D["drop + repos<br/><small>agent edits KV</small>"]
  D --> A["AKV engine<br/><small>history-aware reuse</small>"]
  A --> O["Next step<br/><small>27.7% cheaper</small>"]
  classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
  classDef err fill:#ffe3e3,stroke:#e03131,color:#1b1b1b,stroke-width:2px
  classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
  classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
  classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
  class T input
  class S,RP err
  class D loop
  class A core
  class O exit
  linkStyle 1 stroke:#e03131,stroke-width:2px
  linkStyle 4 stroke:#2f9e44,stroke-width:2px
Red is the summarize-and-re-prefill path, amber is the agent's cache operation, purple is the serving engine, green is the saving.

What is it about? Long-running agents pile up tool outputs and observations. Their KV cache (the stored attention keys and values that let the model avoid recomputing past tokens) becomes the main cost. This paper lets the agent manage that cache directly.

What problem does it solve? Until now, agents controlled cost by editing their history: trimming old turns or replacing them with a summary. Any edit to the start of the history invalidates every cached entry after it, so the model must prefill the kept context again. To avoid that cost, agents keep too much, and accuracy suffers when they keep too little.

What's the core novelty? Two cache operations. drop removes chosen entries from future attention. repos shifts the position encodings of the entries that remain, so the sequence stays coherent. The text of the trajectory never changes. That breaks a basic serving assumption: the same tokens can now map to different cache states, depending on which operations ran. So the authors built AKV, a serving engine that tracks that history internally while keeping a normal stateless API outside.

Key takeaways

Gaps in the study The abstract does not say how much repos costs on tasks that need exact positions, or what restoring a dropped entry costs. Benchmarks and models are not named in the abstract.

Industrial implication This is the sixth example in the wiki of "deciding what enters context beats compressing it". The 10-20 digest counted five, all at the harness level: Memorilla, Meta's Thin Orchestrator, CQ2-CARE, CarryOn and Librarian Agents. Cache engineering is the first one below the harness. Expect serving engines to add a cache-operation API next to the chat API within a year. llm-d 1.0's prefix-aware router already captures 97% of available cache reuse on agentic coding traffic. Its next problem is cache states that depend on operation history.

Research angle. Is the agent's own semantic choice of what to drop better than eviction by attention score (H2O-style)? If yes, task structure matters more than attention mass for agent caches. That would change how KV-eviction papers like DropKV (this week's Kurate #1) should be evaluated.

→ Full summary

DeepSeek gives away its serving engine, and SemiAnalysis maps the Huawei stack under it

The engine behind the cheapest frontier-class API is now open source. It targets two chip vendors, one of which is about five years behind in hardware.

Source: X (@deepseek_ai, the window's most-shared post), The Decoder, The Information, Interconnects, SemiAnalysis (full essay via email) Links: DeepInfer repo · SemiAnalysis · Interconnects · Wiki summary

The moat moved from the model to the serving stack, and DeepSeek gave its stack away
The same kernels target Blackwell and Ascend. The vendor underneath becomes swappable.
flowchart LR
  M["DSv4.1-Flash<br/><small>MoE + Engram</small>"] --> K["Kernel layer<br/><small>FlashMLA, DeepGEMM, DeepEP</small>"]
  K --> L["TileLang<br/><small>kernel language</small>"]
  L --> N["Nvidia SM100<br/><small>Blackwell</small>"]
  L --> H["Huawei Ascend 950<br/><small>~5 yrs behind HW</small>"]
  D["DSpark drafter<br/><small>draft length 5</small>"] --> K
  E["Engram table<br/><small>DRAM, prefetched</small>"] --> K
  classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
  classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
  classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
  classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
  class M core
  class K,L loop
  class D,E input
  class N,H exit
Purple is the model, amber is the portable kernel layer, blue are the efficiency add-ons, green are the interchangeable chips.

What is it about? DeepInfer is the inference engine that serves DeepSeek-V4.1-Flash, now released under Apache 2.0. It runs only that model, on Nvidia SM100 (Blackwell) or Huawei Ascend 950. The same day, SemiAnalysis published a long benchmark-driven essay on why DeepSeek and Huawei need each other.

What problem does it solve? Until now, frontier labs kept their serving stacks private, and open engines like vLLM and SGLang had to reverse-engineer each model's tricks. DeepInfer shows the real recipe: DeepSeek's kernels (FlashMLA for attention, DeepGEMM for matrix multiplies, DeepEP for expert communication, plus TileKernels and DeepSelect), attention data-parallelism with balanced context-parallelism, expert parallelism with load balancing, DSpark speculative decoding (a small drafter proposes tokens the big model verifies, default draft length 5) and deterministic sampling. A "bounded replay" feature recomputes only a 140-token suffix on a new prefill. Practitioners note what is missing: no multi-tier KV-cache system, no prefill/decode split across nodes, no replica load balancing.

What's the core novelty? SemiAnalysis's contribution is measurement. On an Ascend 950PR, stock matrix multiply reaches 99.7% of the chip's ceiling on compute-bound BF16 shapes (1,055 tested). Below the roofline ridge (where memory bandwidth, not compute, limits speed) it reaches 41.5% of attainable, against 46.8% for an H100. Absolute speed sits between an A100 and an MI300X. A profile of V4.1-Flash on 32 Ascend 950DT chips shows fused compute-plus-communication kernels (4 communication ops instead of 43 expert dispatches). It also shows DeepSeek's first production use of Engram (a lookup table of n-gram embeddings read by token ID). Because the lookup address is known before the first layer runs, the table can sit in cheap DRAM at no speed cost.

Key takeaways

Gaps in the study DeepInfer omits the KV-cache and disaggregation layers, so the margin estimate cannot be checked from the repo. The Ascend stack works on DeepSeek's supernode setup only: x86-host support is broken and CANN 9.2 was not out when SemiAnalysis wrote.

Industrial implication Inference hosts serving DeepSeek models will adopt DeepInfer within weeks, and DeepSeek token prices across hosts should fall. Combined with TorchTPU and FlyDSL on AMD (Industry Pulse), the lock-in layer is moving from the framework to whoever writes the kernels. A chip vendor without a frontier lab writing kernels for it is now at a disadvantage, which is Huawei's exact problem and DeepSeek's exact offer.

Research angle. The 1.32x-from-2x-compute result makes fused quantization and expert placement the high-value work on Ascend. That claim becomes testable once CANN 9.2 ships.

→ Full summary

N-Gram Memories at Two Bits: a codec for Engram tables

The model does not need the whole memory table in full precision. It needs a short code per row and one shared recipe to rebuild it.

Source: HuggingFace (10-20 list, captured late) Links: Paper · Wiki summary

Store a recipe per row, not the row
Only rows the input touches are rebuilt. The shared trellis is the whole trick.
flowchart LR
  T["Token IDs<br/><small>known before layer 1</small>"] --> A["Row address<br/><small>hash of n-gram</small>"]
  A --> C["2-bit code<br/><small>plus local scale</small>"]
  C --> P["Shared trellis<br/><small>one program, whole table</small>"]
  P --> R["Rebuilt row<br/><small>norm restored</small>"]
  R --> M["Model layer<br/><small>adds memory vector</small>"]
  F["Full fp table<br/><small>203 GB DeepSeek</small>"] -.->|replaced by| C
  classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
  classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
  classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
  classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
  classDef err fill:#ffe3e3,stroke:#e03131,color:#1b1b1b,stroke-width:2px
  class T,A input
  class C,P loop
  class R,M core
  class F err
  linkStyle 3 stroke:#2f9e44,stroke-width:2px
Blue is addressing, amber is the compressed form, purple is what the model consumes, red is the storage it replaces.

What is it about? Conditional memory is a giant lookup table of n-gram embeddings that a model reads by token ID, as in DeepSeek's Engram. It adds capacity without adding much compute. This paper compresses the table.

What problem does it solve? Each input touches only a few rows, but any row might be needed, so the whole table must stay addressable. For DeepSeek that is 203 GB. Until now the choice was to keep it in expensive GPU memory (HBM) or move it to host RAM and prefetch.

What's the core novelty? Conditional Program (CP) stores each row as a 2-bit code plus a local scale. One reconstruction program, a Gaussian trellis with norm restoration, is shared by the whole table. Each row decodes independently, so random access stays cheap and only requested rows are rebuilt.

Key takeaways

Gaps in the study The abstract reports KL, not downstream accuracy at full scale. Trellis decode latency per row, and whether it overlaps with layer compute, is not given. There is no head-to-head against uncompressed host-RAM offload with prefetch.

Industrial implication Two routes to cheap Engram memory now exist. SemiAnalysis shows DeepSeek's route in production: keep the full table in DRAM and prefetch it during layer 0. CP's route: shrink it 4x and keep it in HBM. The 10-16 COUNT paper (n-gram embeddings built from corpus counts on host RAM, no learned table) is a third. For serving stacks with spare HBM and weak host links, CP wins. For stacks with plentiful DRAM, prefetch wins.

Research angle. Does "Gaussian coordinates plus structured rows" also describe cold MoE expert weights? If so, one shared-trellis codec could serve Engram tables and rarely used experts. That would join today's Specter paper, which prefetches experts, into one memory-tiering design.

→ Full summary

TokenGate: skim first, then read closely

A small learned gate drops prompt tokens that later layers never use. First tokens arrive up to 11x sooner with no accuracy loss.

Source: HuggingFace (10-20 list, captured late) Links: Paper · Code · Wiki summary

Skim first, read closely only what survives
The gate learns which tokens later layers never need, so they never enter the KV cache.
flowchart LR
  I["Long prompt<br/><small>repo or document</small>"] --> E["Early layers<br/><small>full sequence</small>"]
  E --> G["TokenGate<br/><small>scores importance</small>"]
  G -->|keep| L["Later layers<br/><small>fewer tokens</small>"]
  G -->|drop| D["Pruned tokens<br/><small>no KV stored</small>"]
  L --> O["First token<br/><small>1.4-11.3x sooner</small>"]
  classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
  classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
  classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
  classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
  classDef err fill:#ffe3e3,stroke:#e03131,color:#1b1b1b,stroke-width:2px
  class I input
  class E,L core
  class G loop
  class D err
  class O exit
  linkStyle 3 stroke:#e03131,stroke-width:2px
  linkStyle 4 stroke:#2f9e44,stroke-width:2px
Amber is the learned gate, red is what never reaches the cache, green is the latency payoff.

What is it about? A drop-in module that lets an existing model skip unimportant input tokens during prefill (the first pass over the prompt, before any output is generated).

What problem does it solve? Models push every input token through every layer. For long inputs like code repositories, that makes time-to-first-token (TTFT) slow and memory-heavy. Prior pruning methods used fixed heuristics because "keep the top K tokens" cannot be trained with gradients.

What's the core novelty? Token-gated attention. Borrowing from mixture-of-experts gating, importance scores re-weight the attention logits during training, so the gate learns end to end. Adapting a model costs about as much as a parameter-efficient fine-tune.

Key takeaways

Gaps in the study The abstract does not say which tasks produce 11.3x versus 1.4x. Multi-turn sessions, where a pruned token can matter later, are untested. Pruning depends on the query, which conflicts with prefix caching across users.

Industrial implication Most KV work decides what to evict after the cache exists. Examples are this week's Kurate picks DropKV (eviction that targets the error each removal causes in the residual output) and TideKV (eviction delayed until chunked prefill finishes). TokenGate decides what never gets written, and cache engineering (above) decides what leaves later. Together they cover both ends of the cache's life. Expect it first in long-document RAG and code-review serving, where prompts are long and not shared.

Research angle. A two-pass design could reconcile pruning with prefix caching: a query-independent prune of the shared prefix, then a query-dependent prune of the suffix only.

→ Full summary

Speculative decoding under load: when drafting stops paying, and what else a draft is good for

At one request, today's best drafters give nearly 3x. At 512 concurrent requests they are slower than no drafter at all.

Source: HuggingFace (Specter, DScout, ECHO, Lafa), AI lab watcher (Liquid, inclusionAI, Google), X feed (EAGLE-4 table, Red Hat), YouTube (Hugging Face "Local AI 202") Links: Specter · DScout · ECHO · Lafa · Liquid DSpark drafter · Ling-3 UBTree · Wiki: papers, drafters under load

The drafter is now a scheduler, not just a guesser
Specter uses draft tokens to predict which experts to fetch. Others redesign who drafts.
flowchart LR
  Q["Quantized draft<br/><small>on-GPU copy</small>"] --> P["Lookahead<br/><small>future expert IDs</small>"]
  P --> F["Prefetch experts<br/><small>PCIe ahead of time</small>"]
  F --> V["Full MoE verify<br/><small>experts already local</small>"]
  V --> O["Tokens out<br/><small>1.5x faster TPOT</small>"]
  H["Host memory<br/><small>offloaded experts</small>"] --> F
  classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
  classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
  classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
  classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
  class Q,H input
  class P,F loop
  class V core
  class O exit
  linkStyle 3 stroke:#2f9e44,stroke-width:2px
Blue holds weights, amber is the scheduling the draft enables, purple verifies, green is the result.

What is it about? Speculative decoding has a small, fast model propose several tokens that the large model checks in one pass. This window brought four papers, three drafter releases and the first clear measurement of when drafting stops helping.

What problem does it solve? Drafter papers usually report speedups at one request at a time. Production serves many requests at once, when the GPU is already busy and verification work is no longer free.

What's the core novelty?

Key takeaways

Gaps in the study The EAGLE-4 crossover comes from a posted table, not a paper in this window. Specter is single-request only. UBTree and DScout report low-concurrency numbers. Nobody yet reports the crossover point for each drafter.

Industrial implication DSpark is now everywhere: DeepInfer's default, Google's new TPU engines, Liquid's drafter. Teams turning it on for batched traffic may lose throughput past a few hundred concurrent requests. The fix is a load-aware draft-length controller, which is a routing decision made from live utilization. This answers the 10-20 thread, where DLite showed drafters lose up to 80% of acceptance on long agent contexts. Context length and concurrency both shrink the gain, and serving stacks need to price both.

Research angle. Specter's idea, draft tokens as a prefetch oracle, should carry over to KV offloading: a draft that predicts which cached blocks attention will hit next could prefetch from CPU or NVMe tiers.

→ Full summary

Decision models under audit: good judgment, bad advice

A small decision model made an agent 19% more expensive at the same pass rate. The same model family, given a pre-fitted threshold, beat GPT-6 as a judge at 41% of the fee.

Source: HuggingFace (Good Judgment, DecisionPivot, System One reranking), X feed (JEV-as-a-Judge) Links: Good Judgment, Bad Advice · JEV-as-a-Judge · DecisionPivot · System One reranking · Wiki summary

Ranking well is not the same as advising well
Jev's signal was informative offline. Miscalibration turned it into extra verification turns.
flowchart LR
  S["Agent state<br/><small>at decision point</small>"] --> J["Decision model<br/><small>P(done) per option</small>"]
  J -->|P=0.50| V["Extra verify<br/><small>work already right</small>"]
  J -->|high P| F["Finish<br/><small>run ends</small>"]
  V --> C["Cost +19%<br/><small>same 73% pass</small>"]
  F --> P["Pass rate<br/><small>unchanged</small>"]
  classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
  classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
  classDef err fill:#ffe3e3,stroke:#e03131,color:#1b1b1b,stroke-width:2px
  classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
  class S input
  class J loop
  class V,C err
  class F,P exit
  linkStyle 1 stroke:#e03131,stroke-width:2px
  linkStyle 2 stroke:#2f9e44,stroke-width:2px
Amber is the decision model, red is the miscalibrated path that costs money, green is the intended exit.

What is it about? Decision models (small models like Jev that return one probability per option instead of generating text) are sold as the cheap routing layer inside agent loops. They decide whether to stop, verify or change approach. Three papers test the claim.

What problem does it solve? Nobody had run a controlled, pre-registered test of whether delegating small agent decisions to a decision model actually saves money.

What's the core novelty? Good Judgment holds one Claude Sonnet 5.5 terminal agent fixed and varies only the controller. The arms are none, fixed rules, Haiku 5.5, Jev, and a placebo that copies Jev's choice frequencies at random.

Key takeaways

Gaps in the study 21 tasks is small, and the cost CI barely excludes 1.0. Good Judgment did not test a recalibrated Jev, which is the obvious fix.

Industrial implication This contradicts JevAgentHarness (10-18), which made agents 14.2% cheaper with no outcome changes by fitting a confidence threshold per decision point. The reconciliation is the fitted threshold: JevAgentHarness and JEV-as-a-Judge fitted one and won, Good Judgment did not and lost. That fits the routing ladder the wiki logged all last week, ending with Meta's deterministic Thin Orchestrator (10-20, a router with no LLM reasoning). Routing decisions should be cheap and fitted or fixed. Cloudflare's new open decision model, clef, ships into this debate. Anyone deploying it as a controller should budget for calibration on their own traffic first.

Research angle. The decisive experiment is a 2x2: raw vs fitted thresholds, crossed with Jev vs a small LLM. If fitting closes the gap for both, the value lives in the fitting, not the model.

→ Full summary

Eleven harness-evolution papers in two days, and the audit that came with them

Agents rewriting their own harness was the busiest topic on HuggingFace. The one paper that checked held-out tasks found most of the gain was memorizing the exam.

Source: HuggingFace (10-20 and 10-21 lists), X feed (OO-Harness, continual-learner audit), Agentic AI newsletter (Harness² replication) Links: Harness-Delta Attribution · HarnessPro · Agent S4 · Self-HPO · HarnessAdapt-RL · OO-Harness · Wiki summary

Most of the harness gain is the exam, not the student
HDA's breakdown of an average significant harness-evolution gain.
flowchart LR
  G["Search-set gain<br/><small>what papers report</small>"] --> O["Overfitting<br/><small>~54% of gain</small>"]
  G --> T["Extra compute<br/><small>~14% of gain</small>"]
  G --> R["Residual<br/><small>upper bound, ~32%</small>"]
  R --> H["Held-out transfer<br/><small>often weaker still</small>"]
  P["Cost-aware search<br/><small>HarnessPro, TEAL</small>"] --> R
  classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
  classDef err fill:#ffe3e3,stroke:#e03131,color:#1b1b1b,stroke-width:2px
  classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
  classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
  class G input
  class O,T err
  class R,H exit
  class P loop
  linkStyle 0 stroke:#e03131,stroke-width:2px
  linkStyle 1 stroke:#e03131,stroke-width:2px
Red is gain that will not survive deployment, green is what might, amber is the design response.

What is it about? Harness evolution has an agent automatically rewrite the prompts, tools, skills and orchestration code around a fixed model (the harness), keeping changes that score better. More than ten such papers landed in two days.

What problem does it solve? Harness Delta Attribution (HDA) asks the question the others skip: how much of a reported gain is real?

What's the core novelty? HDA splits an evolution gain into three parts: overfitting to the search set, gains explained by extra test-time compute, and a residual that bounds what could generalize.

Key takeaways

Gaps in the study Apart from HDA, none of these abstracts reports a held-out test that would pass HDA's protocol. Agent S4's cost comparison uses per-model prices and excludes the cost of evolution itself.

Industrial implication This fills a gap the wiki flagged on 10-17. Every credible self-improvement result then had a frozen, hidden or mechanical evaluator, and HDA now prices what happens without one. A 10-18 viral thread argued the human's job becomes "writing the exam" a small model rewrites the harness against. HDA says the exam is the whole game. The "End of Loop Engineering" threads on X this week oversell; the cost-aware papers (HarnessPro, Agent S4, Épi-turbo at 64% of Codex's cost) are the ones to watch. Anyone buying a self-improving harness should ask for held-out task numbers.

Research angle. Run HDA on HarnessPro and Agent S4. If cost-aware evolution shows a smaller overfitting share, cost pressure works as a regularizer: cheaper harnesses generalize better.

→ Full summary

Beam-1.0: fewer tokens, not cheaper tokens

Reflection's open 501B model scores like its rivals on agent benchmarks while spending 3-4x fewer tokens per attempt, because RL penalized unnecessary tokens.

Source: AI Breakfast, Interconnects, VentureBeat (via email), X (Red Hat FP8 release) Links: Model card · FP8 build · VentureBeat · Wiki roundup

What is it about? Beam-1.0 is a 501B-total, 23B-active mixture-of-experts model (each token uses only a small subset of expert sub-networks) from Reflection AI. It is released under Apache 2.0 with a coding agent CLI (Mirror) and Baseten serving through OpenRouter.

What problem does it solve? Open models competed on price per token. Agent workloads are billed per task, and reasoning models spend tens of thousands of tokens per attempt.

What's the core novelty? RL training penalized unnecessary tokens, so the model was trained for cost per task, not just accuracy.

Key takeaways

Gaps in the study No price yet, and no independent token-efficiency measurement beyond vendor and newsletter figures.

Industrial implication Token efficiency is becoming a trained property, like the 10-19 RL post-train of Qwen3.8-27B that halved median tokens on solved Terminal-Bench tasks (54.9K to 23.8K, via X). For per-task billing, a 3x token cut beats a 3x price cut, because it also cuts latency and KV-cache pressure.

Test-time compute meets the real GPU: ATEST and implicit budgeting

The same four-sample query can cost 4.1x more energy depending on how busy the GPU is. Sampling budgets that ignore this waste power.

Source: HuggingFace (10-21 list) Links: ATEST · Implicit budgeting · Wiki summary

The price of one more sample changes every second
ATEST stops sampling on answer stability and live GPU cost, not a fixed N.
flowchart LR
  Q["Query<br/><small>reasoning task</small>"] --> S["Draw sample<br/><small>one at a time</small>"]
  S --> V["Plurality stable?<br/><small>answer agreement</small>"]
  G["GPU load<br/><small>live energy price</small>"] --> V
  V -->|worth it| S
  V -->|stop| O["Answer<br/><small>within energy budget</small>"]
  classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
  classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
  classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
  classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
  class Q,G input
  class S core
  class V loop
  class O exit
  linkStyle 3 stroke:#f08c00,stroke-width:2px
  linkStyle 4 stroke:#2f9e44,stroke-width:2px
Blue is input and live cost, amber is the stop rule, green is the exit.

What is it about? Two papers on how many reasoning tokens to spend. ATEST adds the live GPU price. Implicit budgeting shows models under-spend when tasks are packed into one prompt.

What problem does it solve? Test-time scaling (sampling several answers and voting) uses fixed budgets that ignore both query difficulty and the changing energy cost under GPU contention.

What's the core novelty? ATEST-Online samples one answer at a time, tracks how stable the majority answer is, learns per-step energy cost online, and stops when another sample is not worth its current price.

Key takeaways

Gaps in the study ATEST tests small models (Phi-3-mini, Qwen2.5-3B) under synthetic load. The 43% saving is on math and linguistic tasks, not agent prompts.

Industrial implication Prior allocation work in the wiki assumed compute has a fixed price: CLEAR (one shadow price across a batch) and LEV (10-20, a stopper that chooses loop depth and saves 15-24% compute). With GPU-hour prices rising (DeepInfra +33-67% from Nov 9) and energy the binding constraint at datacenters, a live price signal in the sampler is the natural next step for any self-hosted reasoning service.

→ Full summary

Shorts: edge kernels, memory-native models, hybrid training, distillation, safety benchmarks

A layered kernel agent beats Claude Code on edge chips with the same model. Memory moves into the model state. And a safety score can swing across its full range by changing 0.03% of parameters.

Source: HuggingFace (10-20 and 10-21), Kurate, AI lab watcher, The Decoder, X feed Links: EdgeKernelBench · MemT-1.0 · Competing Learning · KronLSTM · Ask to Distill · Policy Is Not Capability · TRACE · Wiki: kernels, memory, hybrids, distillation, safety, TRACE

Industry Pulse

Funding, valuations, and compute deals

Global View

Serving cost is now decided by what you avoid redoing, and industry shipped the plumbing the same week research named the problem. The 10-20 digest found five harness-level papers showing that choosing what enters an agent's context beats compressing it. One was CQ2-CARE (picking whole files verbatim beat prompt compressors by 6.3 points). Today the same idea reaches the cache and the weights. Cache engineering edits KV entries instead of rewriting history, TokenGate stops unneeded tokens at prefill, and the 2-bit Engram codec plus DeepSeek's prefetched DRAM table avoid moving memory that is not needed. Industry built the matching pieces in parallel: llm-d 1.0's router keeps 97% of cache reuse, Google's TPU engines offload KV to host memory over TPU Sync, and DeepInfer recomputes only a 140-token suffix on new prefills. The open gap is that cache states which depend on operation history, as cache engineering creates, break the prefix matching every one of these engines relies on.

The cheap control layer needs calibration or an exam, and buyers are not yet asking for either. Today's two audits make the same point. Harness evolution without held-out tasks is mostly overfitting (HDA, about 54% of gains). A decision model without a fitted threshold made an agent 19% dearer (Good Judgment). The winners on the same days had a fitted or frozen threshold: JevAgentHarness (10-18, 14.2% cheaper), JEV-as-a-Judge (beats GPT-6 at 41% of its fee) and Meta's deterministic Thin Orchestrator (10-20). Industry is moving the other way. Cloudflare's clef and the Strot decision artifact ship decision models as drop-in controllers, and an IPO filer listed "no model-routing layer" as a risk. Pragmatic Engineer's 27 companies mostly cannot measure AI ROI at all. Routing and self-improving harnesses are being bought on search-set numbers, which is exactly what HDA warns against.

The hardware story moved from chips to kernels and power, and market prices disagree with each other. SemiAnalysis shows Ascend 950 hardware about five years behind Nvidia but software "at the frontier", because DeepSeek writes its kernels. TorchTPU, FlyDSL on AMD and TileLang on Ascend make the same move for other vendors: lock-in now lives with whoever writes the kernels. On cost, GPU rental prices are rising (DeepInfra +33-67%, Nebius B200 spot up sharply). Yet chip equities lost about $2T and CoreWeave fell 25% in a week, and NVIDIA's own GTC numbers point at power, not silicon: a $300B equipment backlog and 200 GW queued for grid connection. This extends the 10-20 thread, where Amazon warehoused GPUs it could not power while B200 spot prices tripled. Powered, usable capacity is scarce, and research that wrings more from it (Specter's single-GPU MoE, ATEST's load-aware sampling) is pricing the right constraint.

Looking Ahead