KV Cache

The KV cache (Key-Value cache) stores the key and value tensors from the attention mechanism for tokens already processed. This means those tokens don't need to be recomputed on every new generation step — critical for making autoregressive decoding fast.

Addendum, 2026-09-15: the architecture that resolves this week's capacity contradiction, and the workload that creates it

Two SemiAnalysis pieces two days apart put this page in a contradiction, and a third source published the same morning resolves it. Long Live the Short King (09-14) argued that an HBM4 cube exposes 2,048 data I/Os regardless of stack height while price tracks gigabytes, so stop paying for capacity: 12-hi costs 26.3% more system-wide for 10% more throughput. Then Vera Rubin NVL72 agentic inference (09-15) defined the agentic workload as one where, because turn n concatenates turn n-1's output, the cached-input ratio tends toward 1 as the session grows, with the explicit qualifier that this depends on how much storage is available to hold KV tensors. The defining workload of the next hardware generation wants more KV capacity than anything before it, and the supply-side argument says capacity is what you should stop buying.

Ken Huang's Chapter 6 (09-15) is the resolution and neither SemiAnalysis piece names it: KV does not have to live in HBM. Moonshot's Mooncake, deployed in the 2.8T Kimi K3 stack, is a hierarchical KVCache storage engine spanning VRAM, host DRAM, local SSD and a distributed pool, with zero-copy GPUDirect RDMA over InfiniBand or RoCEv2 moving tensors between tiers. Capacity is then bought at the cheapest tier that meets the latency budget and HBM is sized purely for bandwidth, which is exactly what the 4-hi argument asks for. This page should now treat KV capacity and KV bandwidth as separately purchasable quantities, because in a tiered architecture they are.

The chapter's load-bearing argument is against chunked prefill, and it is a live contradiction with what production stacks actually ship. The two-phase asymmetry is stark: prefill is compute-bound at 100 to 250 FLOPs per byte moved from HBM, pushing model-FLOPs utilization to 65-80%; decode is bandwidth-bound at 0.5 to 4 FLOPs per byte with the pipelines idle. Colocated, a long prefill seizes the streaming multiprocessors for hundreds of milliseconds and inter-token latency goes from an optimal 15-20ms to 400-600ms. Chunked prefill (Sarathi-Serve, vLLM v0.6) is the standard mitigation and Huang names three costs: arithmetic intensity collapse (slicing a 32K prefill into 64 chunks cuts tensor-core utilization 35-50%), scheduling bubbles and memory jitter, and asymmetric parallelism constraints. The third is the strongest because it is structural: prefill latency improves with high tensor parallelism while single-token decode is dominated by cross-GPU all-reduce overhead that gets worse at high TP, so no single TP value is right for both and any colocated system is misconfigured for one phase by construction.

vLLM and SGLang ship chunked prefill on by default and regard it as a win, so both positions cannot be right everywhere. They reconcile if the crossover depends on context length and concurrency, which is precisely the plot nobody has published. That plot is the single most useful missing artifact for anyone choosing a serving architecture right now, and it is cheap to produce.

This also reframes the cliff this page recorded on 09-14, and the reframing matters. That entry captured a Kimi K3 production run where cutting the KV budget 36-44% tracked full-memory throughput invisibly until concurrency reached about 70, then dropped ~30% at once when occupancy saturated, with 10x the DRAM reads. Mooncake is the Kimi stack's own storage engine, so that cliff was measured on a system already running the hierarchy. The cliff is therefore not evidence against tiering; it is a measurement of where a tier boundary bites. Every eviction, compression and placement policy on this page is implicitly a policy about which tier a block lands in, and not one of them says so.

Sub-agent bursts are the mechanism that makes the cliff unpredictable. The Rubin piece names them as a defining property of agentic traffic: a session spawns short-lived sub-agents with fresh context, so KV allocation is spiky rather than smooth. Every eviction policy on this page assumes a smooth occupancy curve. A disaggregated architecture answers a burst by migrating KV over the fabric instead of holding headroom in every decode pool, which is cheaper if the network keeps up, and which is why the ConnectX-9 and Spectrum-6 half of NVIDIA's six-product co-design is load-bearing rather than marketing.

Addendum, 2026-09-12: the placement thesis generalizes from the cache to the weights, and gets a workload boundary

kimi-k3-in-c (09-12) is the most extreme instance this page will record of the pattern it named on 09-09, that the cache's frontier has moved from compression to placement. A 176 KB portable C99 engine runs Kimi K3, a 2.78-trillion-parameter mixture-of-experts model, on one CPU at a measured 8.24 GB peak resident set, against a 1.56 TB checkpoint that was not compressed at all. 92 routed layers hold 896 experts each with 16 firing per token, so 82,432 routed experts occupy 1.447 TB, 93% of the checkpoint, and they live on NVMe. A token can require 1,472 expert fetches and reads about 25.83 GB of expert weights. The ~113 GB of always-used weights are repacked into a "trunk" laid out so each of 93 layers is one contiguous read, pinned in RAM as far as RAM allows and streamed otherwise. RAM sets latency, not feasibility: 26.5 s/token at 8 GB, 24.2 at 32 GB, 19.8 at 64 GB, 5.6 at 128 GB+, byte-identical output throughout.

The load-bearing detail for this page is that the exercise only works because the sequence-length-dependent term was already bounded by architecture. 69 of K3's 93 layers use KDA, carrying a fixed-size recurrent state rather than a growing KV cache, and the other 24 use MLA, whose cached latent is far smaller than per-head keys and values. Without that, the cache alone exceeds the 8 GB budget on any nontrivial context and the whole scheme collapses. So the placement trick is downstream of the bounded-memory architecture choice, and this page should stop treating those as two separate branches. Bounded attention state is what buys you the right to demote everything else to a slower tier.

It also supplies the workload boundary this page needs, because FastE (09-12) is a pure compression result that still wins. FastE cuts decoder-backbone FLOPs by 40.11% at 99.53% of full-forward nDCG@10 on NarrativeQA with Qwen3-Embedding-0.6B, training-free, by exploiting depth-dependent prefix redundancy: prefix states are destructive to remove in shallow layers and nearly free to remove deep, so it compresses from the depth at which readout-prefix alignment crosses a threshold, ranking prefix states by the attention the readout position gives them. That works because embedding inference is prefill-only, single-pass, with exactly one output position and no decode trajectory. The rule to carry: placement beats compression when the cache must survive across turns; compression still wins when the forward pass is one-shot. The 09-09 placement thesis was stated too generally and this is its scope.

Two practitioner datapoints from the same day price the problem this page exists for, and both are worth quoting. A practitioner measured the crossover directly: Qwen3.8-27B in FP8 is ~28 GB of weights on an H100, and a long-running coding task at ~0.5M tokens carries a 32 GB BF16 KV cache, larger than the model itself. That is the concrete form of the 97% concurrency collapse this page recorded from the 09-07 inference-physics chapter. Separately Red Hat AI reported serving GLM 5.3 at full 1M context on a single 8xH200 node, which did not fit before, and named the reason it is hard in exactly this page's vocabulary: agentic serving means many concurrent requests each with a long and growing context against a fixed GPU block pool, so the cache runs out of room and something has to give.

Addendum, 2026-09-09 later captures: the placement pattern crosses the three-instance threshold twice over, and cache portability gets a training-free mechanism

Cross-model KV cache transfer (NVIDIA, arxiv 2608.03893) resurfaced on the feed today and it is the cleanest statement yet of why prompt caching and routing are structurally at odds. A provider bills cache hits at roughly 10% of base input rate, but the cache is a function of the producing model's weights, so routing traffic to another model voids it and the whole context is re-prefilled at full price. The paper's empirical finding is that cross-model KV carries substantial linear structure between matched pairs: on Qwen3 14B to 32B a single source layer explains 56% of variance in the target's keys and 32% in values, rising to 79% and 65% with multiple source layers. The mapper is a per-head, per-target-layer closed-form ridge regression calibrated on just 500 FineWeb-Edu sequences of 1,024 tokens, with two design details doing the work: cross-layer selection (each target layer takes the top-k most predictive source layers, and the ablation says this contributes most) and RoPE stripping (fit position-free, re-apply the target's rotation at inference, so one mapper is reusable across context lengths). Reported 2.7 to 25x faster than re-prefill, retaining 73-98% of standalone-prefill accuracy on four of six pairs, stable across multi-turn handoff.

It disagrees with this page's other cross-model translation result about what is achievable. A Universal Context-Reuse Layer (09-02) trains an adapter and claims transfer across scale, architecture, attention configuration, tokenizer and family (Llama3.1-70B to Qwen2.5-7B at 44.0% against 45.7% native, 899ms to 138ms). NVIDIA's is training-free and same-family only, with matched KV head count and per-head dimension, dense full-attention models, cross-family named as future work. So the cheap closed-form mechanism exists inside a family and nobody has shown it exists across families, which is precisely the boundary a production router needs. And NVIDIA reports its failures honestly: two of six pairs degrade sharply, with a nonlinear MLP recovering up to +37 points of HellaSwag retention on them, and no predictor of which pairs will fail, which is disqualifying for router deployment until a pre-flight check exists.

The 09-07 direction rule still governs and neither paper applies it. The Handoff Tax (09-07) found the dominant escalation penalty is not transport cost but that carrying the weak model's reasoning into the strong model is actively harmful, with trajectory-dropping lifting recovery from 47% to 64% (Claude) and 36% to 84% (GPT). Cheap KV portability is worth a lot on downshift and little or negative on escalation, and cross-model transfer papers keep framing themselves as generic mid-conversation switching.

The placement pattern named above now has five instances in two days, not three. Alongside IndexShare, KVMem and TPU-Sync: vLLM published on optimizing for real-world agentic serving, stating that agentic workloads are becoming a major traffic source and that their multi-turn sessions, long contexts and extensive prefix reuse demand optimizations across the serving stack. And widely circulated practitioner writing framed the production KV cache plainly as a storage system (the LMCache design point): a cache hit is no longer token matching but block lookup, rehydration into GPU memory, eviction under storage pressure, cross-worker sharing, and survival across engine restarts, with a request sharing 80% of its prefix served from GPU memory, CPU memory, local SSD or remote storage while only the uncached 20% is computed. The inference engine schedules requests; something else has to manage the cache. That separation is now stated by a serving project, a research paper, a hyperscaler and two independent practitioners in the same 48 hours, and it is the most consequential structural change on this page this quarter.

Current State (as of 2026-09-09): the scorer is not useless after all, it was reading the wrong queries

BeaconKV (09-09) (arxiv 2609.04971, Hanyang and Sungkyunkwan) is the first result on this page to name why recency-based importance scoring fails, rather than to conclude from its failure that scoring should be deleted. Every eviction method here estimates future importance from recent queries. In extended chain-of-thought generation that proxy breaks at a specific moment: certain decoding steps are Thought Revisiting Tokens that re-attend to distant context, typically the plan the model wrote early in the trace. Recent queries carry no signal about those jumps, so a recency evictor deletes the plan right before the model returns to it. The exploitable structure is that the revisiting queries cluster into a small number of similarity groups in embedding space, so a handful of cluster representatives ("beacons") stands in for the whole query history at negligible storage. Training-free, no architecture change. Up to 5.8x memory reduction at near-full-cache accuracy, over 4.3x throughput, across four open-source reasoning models.

This is in direct tension with the strongest result this page recorded five days ago, and the crossover experiment is one table. Random Attention (09-04) deleted the KV importance scorer outright, evicted uniformly at random inside each head, and matched the strongest prior evictor at 32-43% higher throughput in vLLM. This page read that as the decode-side scorer failing to pay for itself. BeaconKV says the scorer was fine and the queries feeding it were wrong. Both cannot be the whole story. Run BeaconKV against random eviction on long-CoT traces at matched retention: if random matches, the beacon machinery is decoration and Random Attention's null generalizes; if BeaconKV wins clearly, Random Attention was measured on traces too short to contain revisiting structure, and trace length is the hidden moderator this page has not been controlling for. This is now the sharpest open question on the compression branch of this page.

The standing "second estimator nobody validated" warning applies without modification. The obvious missing control is retaining a random small subset of historical queries at the same storage cost. If that matches, the finding is "keep some old queries," not "cluster them."

A third arrival this week says the cache's remaining problems are memory-systems problems. Chapter 3 of the inference-physics series (09-09) describes Zhipu's IndexShare, which decouples indexer memory from physical KV cache pages so that speculative rollbacks stop churning the page allocator. Put beside KVMem (09-08), which pages KV across GPU, host RAM and NVMe, and Google's TPU-Sync externalization (09-08), which ships disaggregated KV transfer with DRAM and NVMe pooling, that is three independent groups in two days treating the KV cache as a storage hierarchy rather than as a tensor to shrink. Three is this wiki's threshold for naming a pattern, so: the cache's frontier has moved from compression to placement, and BeaconKV is now the best-argued hot-set policy to pair with a paging system rather than a standalone answer.

The same chapter supplies the physical constant this page should quote from now on. During single-request decode an H100 runs at roughly 1.0 FLOP per byte moved against a roofline knee near 295, so under 0.35% of peak compute. Every memory-side saving on this page is an attempt to move that number, and it explains why a 30-to-50% read reduction is worth reorganizing a serving stack for.

One hazard for the bounded-cache branch got a constructive answer today. Kalman Delta Networks (09-09) recast fixed-size recurrent memory as a linear-Gaussian state-space model so the write strength is weighted by tracked uncertainty rather than by the current token alone. The 09-07 entry below recorded that quantizing such a memory would fail as a 70x-to-300x cliff rather than a taper, because the write-back rule changes the dynamical system. KDN is the first bounded-memory model in which the write-back rule is the learned object, so it should be measurably more quantization-robust than a delta-rule memory at the same bit width. Nobody has run that, and it is cheap.

Current State (as of 2026-09-08): the page gains its first result that makes the cache bigger, and it beats the summarizers

Every method on this page so far shrinks the cache. KVMem (09-08) (arxiv 2609.04852) grows it and moves it down the memory hierarchy, and that is a genuinely new axis. The four compression axes this page tracks are tokens (eviction, sparse selection), heads (MISA, grouped-query, MLA), the upstream sequence (LatentPress, LatentStream) and, as of 09-07, depth (KVShare's cross-layer sharing for an additional 50% to 75%). KVMem is on none of them. It pages overflowed KV state across GPU memory, host RAM and NVMe, indexes the pages with lightweight indexes built in the model's own attention space, and materializes a query-dependent execution view bounded by the native context window. On DeepSWE long-context with Qwen3.8-27B it lifts task success 43.8% → 48.4% against compaction-only context management, and on a 24GB RTX 5090 laptop it virtualizes a 1M-token workspace against a 256K native window at ~50 tokens/s.

The mechanism worth carrying forward is the prefill accounting. A retrieval-based memory system re-prefills every chunk it pulls back into the prompt, so the prefill bill is paid once per retrieval. A paged KV block was prefilled once, when it was live, and never again. That reframes context overflow from a context-length problem into a storage-tier problem, and it means the relevant quantity is addressable workspace per dollar of storage tier rather than context window size.

Eviction and paging are substitutes for the same problem and nobody has composed them. Random Attention (09-04) established that deleting the KV importance scorer entirely and evicting uniformly at random inside each head matches the strongest prior evictor at 32-43% higher throughput in vLLM. That finding is directly load-bearing for KVMem, because it means the hot-set eviction policy inside GPU memory can be free while the paging system handles everything cold. Page cold, evict randomly inside hot. Neither paper cites the other and the composition is one experiment.

It partly closes the mid-session-switch penalty this page has carried since 08-29. That entry recorded that prompt-cache entries are keyed to a model, so routing mid-session to a cheaper model pays a full cold prefill on the accumulated history, which on a 140K-token agentic prefix plausibly costs more than the route saves. Cross-model KV sharing (09-02) supplied the translation layer (Llama3.1-70B → Qwen2.5-7B at 44.0% against 45.7% native, latency 899ms → 138ms). KVMem supplies the storage tier that makes a translated cache worth retaining past the session boundary. A translation layer with nowhere to keep its output only helps within one session.

The industry side shipped the infrastructure for this on the same day, which is as clean a research-industry alignment as this page has recorded. TPU inference externalization (09-08) reports Google externalizing TPU-Sync, its disaggregated KV-cache transfer library, with zero-copy transfers via native PJRTBuffer descriptors, native DRAM offloading, and support for Mooncake Store including DRAM P2P pooling, which aggregates every host's KV storage into one logical pool so any chip can read any other server's cache, plus NVMe pooling on top. KVMem is the algorithm and Mooncake plus TPU-Sync is the infrastructure, and neither mentions the other. Separately, TPUv8i carries 384 MB of on-chip SRAM, 3x the prior generation, sized explicitly to hold the KV cache of reasoning and agentic models on-chip rather than round-tripping to HBM. That is the first time a vendor has publicly budgeted SRAM around the KV cache as the primary resident object, and it should be read as the hardware roadmap agreeing with this page's premise.

What to hold KVMem to. The attention-space index is the load-bearing component and carries no reported ablation against a naive baseline (recency, BM25, an off-the-shelf embedding retriever). This page generalized a warning on 08-26 that methods in this family routinely depend on a second estimator nobody validated; that warning applies here directly. If a dumb index matches it, the contribution is the paging system and the paper is a systems result. Also: ~50 tokens/s is a single-session figure, and NVMe paging under concurrency, where the page cache thrashes across users, is the regime that decides whether this ever serves traffic. Not reported.

The tension with the harness literature is real and should be logged as open. The agent harness engineering page crossed a pattern threshold on 09-07 with five results in two days arguing the raw transcript is the wrong thing for a loop to carry, all of which discard information deliberately: ContextPipe plans the prompt instead of appending (31% fewer tokens), SKILL.state replaces history with structured state (O(T) → O(1) prompt growth), Harness-of-Harness passes an evidence bundle, Iris says the context policy outweighs the model gap, the Handoff Tax says a raw transcript across a model boundary is actively harmful. KVMem discards nothing and beats the baseline that discards. The two camps are plausibly attacking different failure modes, attention dilution versus capacity and recompute, and the crossover experiment (KVMem against SKILL.state on one long-horizon benchmark) has not been run.

Current State (as of 2026-09-07): the page gains a fourth compression axis, and the mid-session-switch penalty it has flagged since 08-29 finally gets a price

Three arrivals today, and two of them are about things this page named as open and could not close.

A fourth axis: depth. The KV Cache Frontier (09-07), Chapter 2 of Ken Huang's inference-physics series and the sequel to the roofline chapter (09-02) this page leans on, reports KVShare: in 60-plus-layer topologies, adjacent layers' KV projections have high cosine similarity, so anchor layers can serve follower layers at 1:2 or 1:4 ratios for an additional 50% to 75% memory reduction. Every method on this page compresses along tokens (eviction, sparse selection), along heads (MISA, grouped-query, MLA), or along the sequence upstream (LatentPress, LatentStream). None of them compress along depth. Depth-wise sharing is multiplicative with all of them rather than competing, which makes it the highest-leverage unexplored axis this page currently records.

And there is a same-day corroboration from the training side that nobody has connected. Don't Drop Dropout (09-07) finds that whole transformer layers can be dropped during pre-training on a tuned schedule for lower loss at equal FLOPs and up to 25% fewer training FLOPs, producing a model that is robust to layer removal at inference. Two independent observations, one at training and one at serving, that the depth axis carries much less independent information than the parameter count implies. The testable consequence: a model pre-trained with layer dropout should be more amenable to KVShare, because it was trained to make adjacent-depth representations interchangeable.

The chapter also supplies the operations context that makes this page's whole 09-03 to 09-04 arc commercially decisive rather than a benchmark exercise, and the number is 97%. A Llama-3-70B-class grouped-query model in FP16 costs 327.68 KB of KV per token, so a single 128K session consumes 41.94 GB. On an 8x H100 node with roughly 450 GB left after weights and workspace, a 4K chat regime sustains 340-plus concurrent streams; the moment users submit 128K coding repositories, concurrency collapses to about 10. That collapse comes from user behaviour, not from a model or hardware change, and it is why a 30-to-50% read reduction is worth reorganizing a serving stack for. MLA is priced there too: a 512-dimensional shared latent plus a 64-dimensional decoupled rotary key for a 93.3% per-token cut, with the decoupling required because rotary embeddings do not commute with the low-rank projection. Caveat: everything past the roadmap is paywalled, so the derivations and the production failure-mode section are announced rather than read. The item most worth chasing is the flagged FP8/FP4 dequantization hazard in the low-rank path, which is an interaction between two of this reader's core areas that nobody has measured.

The 08-29 unpriced item is now priced, and it is worse than this page's cache argument predicted. This page recorded on 08-29 that provider cache entries are keyed to a model, so a mid-session route to a cheaper model pays a full cold prefill on the whole accumulated history, and called that "the most concrete unpriced item where this page meets routing." The Handoff Tax (09-07), from AWS Agentic AI across 58,000 agent runs, 2 million API calls and 36 billion tokens on SWE-bench Verified, measures it. Passing the full history when escalating from a cheap to an expensive model recovers less than half the quality gap (47% Claude, 36% GPT) and for Claude costs more than twice starting with the expensive model from scratch. Restarting fresh is cheaper and more accurate than continuing. The prefill penalty this page identified was only one of three terms: there is also a quality penalty and a per-step context-inflation multiplier of 1.6 to 2.2x on every remaining expensive turn.

That puts cross-model KV sharing (09-02) in an awkward position, and this is the sharpest thing to carry off both papers. This page called that translation layer the first mechanism to remove the switch penalty rather than price it. The Handoff Tax says the dominant penalty is not transport cost, it is that carrying the weak model's reasoning into the strong model is actively harmful: dropping the trajectory and passing only the code edits on disk lifts recovery from 47% to 64% (Claude) and 36% to 84% (GPT). So a mechanism that makes trajectory transport cheap is optimizing the direction where you should be deleting the trajectory. The reconciliation is directional: KV portability is worth a lot on downshift, where the cheap model needs the blueprint and loses 1.6 to 2x more steps without it, and worth little or negative on escalation. Neither paper knows about the other.

Industry shipped the enforcement half the same weekend. Spotify's engineering team reports cutting Claude Code token usage 90% with a two-model routing layer where cheap assistant models handle file opening and boilerplate and files over 350 lines are hard-blocked from the expensive model. Their stated finding is that rules expressed as prompt instructions did not hold and only hard blocks at the routing layer worked. Read with the Handoff Tax: the expensive part of capability routing is not choosing the model, it is controlling what crosses the boundary.

One new hazard for the bounded-cache branch. Maglev (08-16) removes the eviction decision by bounding the cache with a fixed-size recurrent memory, and quantizing that memory is the obvious next efficiency move. When Quantization Breaks Memory (09-07) says the failure would not be a graceful taper: in a recurrent network the state is stored and read back, so the write-back rule changes the dynamical system rather than approximating it. Deterministic 4-bit state storage on a fixed trained GRU raised error roughly 70x and 300x on two targets, because repeated sub-threshold updates are silently discarded while the network keeps proposing change. Increasing state precision can make a fixed recurrent solution worse. Nobody has quantized a bounded LLM KV memory and looked for this, and the same argument applies to linear-attention and state-space hybrids.

Prior State (as of 2026-09-04)

The three-day arc on this page ends with the null hypothesis being run, and not rejected. Yesterday's entry (below) recorded two papers concluding that the scorer deciding which KV entries to read had become the cost it was introduced to remove, and stated the rule as: prefill-side, the win is a better threshold; decode-side, the win is not scoring at all. Random Attention (09-04) (arxiv 2609.03430, Salesforce AI Research) is the third instance in two days and the strongest possible form of the decode-side half. It keeps the prompt, evicts uniformly at random inside each attention head, and computes no score whatsoever. Across four models and six reasoning tasks it matches the strongest prior evictor while serving 32-43% higher throughput in vLLM.

The method is a control condition promoted to a method, and the two ablations are the real contribution. First, the prompt is the fragile part of the cache and that is nearly the whole story: most of the measured gap between competing scorers turns out to be whether that scorer's signal happened to retain the prompt. Score-based selection was never being rewarded for finding important reasoning tokens, it was being rewarded incidentally for protecting the prompt, and protecting the prompt explicitly collapses the differences between scorers. Second, the reasoning trace protects itself at two levels: in the text, because the model restates what it still needs as it works, so a dropped fact is recoverable from a later restatement; and across heads, because each head keeps its own copy, so a random draw that loses an entry in one head very likely retains it in another. Redundancy, not importance, is what makes the reasoning cache compressible.

This page should now revise a premise it has held all year. Every method here has assumed the eviction decision is a selection problem whose quality determines the outcome. On reasoning workloads it is a protection problem: identify the non-redundant region, guard it, and randomize the rest. That reading also retro-explains VaSE (06-03), which evicted stochastically behind a large-magnitude value guard and worked without a mechanism; and it converts the 07-28 impossibility result (deterministic top-k eviction cannot certify the error it creates) from a warning into a non-issue, because the community was certifying a decision that did not need to be made. It is also Eviction as Estimation (08-03)'s critique executed as an experiment.

Unresolved tension with CRISP, and the reconciliation is a hypothesis nobody has tested. CRISP's post-softmax mass cliff proof says attention mass falls onto a near-uniform floor, which implies a sharp signal/floor distinction and a strong reason to select carefully. Random Attention says selection does not matter. The likely reconciliation is regime: CRISP works on the quadratic prefill pass over a 512K prompt, Random Attention on the decode-side reasoning trace, whose self-restating redundancy the prompt does not share. Nobody has run both on the same workload, and Random Attention's own gap statement points the same way: on a workload where the cache is mostly non-redundant prompt-like content, "keep the prompt and randomize the rest" degenerates to "keep everything." The 140K-token agentic prefix is exactly that workload.

One methodological credit worth recording, because this page complained about it yesterday. CRISP quoted attention speedup and Declarative Attention quoted attended-token reduction, both proxies, since skipping non-contiguous KV regions can defeat memory coalescing. Random Attention reports realized serving throughput in vLLM, which makes it the first paper in this three-day cluster to close the theoretical-versus-realized sparsity gap this literature has carried all year.

Two upstream levers arrived on the same board and neither has been stacked with any of the above. LatentPress (09-04) writes history into continuous memory tokens read directly through a frozen decoder's input-embedding interface at 4-16x compression, and LatentStream (09-04) internalizes streaming video history into a fixed-length latent memory. Both shrink the number of positions that get cached at all, upstream of every read-side method here. A 7.7x shorter input and a 43% throughput gain multiply rather than overlap, and the composition is unclaimed.

Still no cost-side news on the pricing question this page opened on 09-02. Anthropic's Fable 5.1 cut cache-read cost 75% to $0.25 per million tokens, and the open item was whether cheap cache reads reduce the incentive to adopt read-avoidance at all. Random Attention makes that question sharper rather than answering it: a method that deletes the scorer lowers compute cost rather than read cost, so it is complementary to a read-price cut rather than substitutable by one.

Prior State (as of 2026-09-03)

Two papers today, from unconnected groups, make the same diagnosis: the mechanism that decides which KV entries to read has become the cost it was introduced to remove. This page has spent months on methods that select a sparse subset of the cache. Every one of them pays a scorer to do the selecting, and both of today's papers say the scorer is now the binding constraint. They then exit in opposite directions, which brackets the design space cleanly enough to state as a rule.

• CRISP (09-03) (arxiv 2609.01925) keeps the scorer and makes it structurally free. Prior dynamic-sparse routing builds a pooled proxy attention map and measures Jensen-Shannon divergence against candidate patterns; CRISP shows the decision is a function of the map's shape and replaces the divergence with C_struct, the post-softmax mass sitting at Vertical-Slash compatible positions, reproducing the same routing decisions while deleting both the pooled matmul and the KL step. Its second and larger contribution is a proof that strictly cumulative coverage thresholds accumulate O(n) background noise at long context, formalized as the post-softmax mass cliff: mass does not taper, it falls off a cliff onto a near-uniform floor, and a cumulative rule cannot distinguish floor from signal. The fix is a sink-aware threshold anchored to the noise floor. 5.30x attention speedup at 512K tokens, up to +28.0pp recovered on retrieval tasks, and it matches or exceeds exact dense attention on retrieval-heavy benchmarks. The speedup is attributed primarily to the noise elimination, not the free router.
• Declarative Attention (09-03) (arxiv 2609.02737, KAIST AI with Google DeepMind) removes the scorer entirely. The observation is that per-step proxy scoring is still O(N), so it reduces the constant factor and never changes the complexity, meaning the cache is still being touched in full every step. DA instead elicits the model to declare its own attention scope inside its chain-of-thought, in three modes, <global>, <focus> on a named region, and <local> on recent output only, which the inference engine parses like a tool call before skipping most of the KV read. Zero-shot, no training, off-the-shelf models: Gemma-4-31B attends 52.0% fewer tokens for a 1.27pp accuracy drop; Qwen-3.6-27B 31.1% fewer for 2.75pp, and the accuracy gap shrinks with model scale.

The rule the pair establishes: prefill-side, the win is a better threshold; decode-side, the win is not scoring at all. CRISP operates on the quadratic dense pass over the prompt, DA on the per-step read during generation. They are complements, not rivals, and nobody has composed them. That composition is the obvious unclaimed experiment on this page, and it is cheap to run because DA needs no training and CRISP is a selection rule.

CRISP retroactively reframes a run of results on this page as having optimized the wrong stage. OasisKV (08-11) treated which entries to fetch as a lookahead prediction problem and prefetched them. CRISP's proof says the harder half was never prediction but thresholding: pair a perfect scorer with a cumulative-coverage budget and you still import O(n) floor, which grows with context. Any method on this page whose budget rule is "retain tokens until cumulative mass reaches X%" inherits that defect, and the retrieval degradation the sparse literature has repeatedly reported at long context now has a structural explanation rather than an empirical one.

It also explains a mechanism TileMix (08-25) exploited without naming. TileMix allocated precision per attention tile on the observation that most tiles carry nearly nothing while a few carry the answer. That is the mass cliff, used as an engineering heuristic. CRISP formalizes the same distribution and derives its noise-accumulation consequence, so tile-importance skew and the post-softmax cliff are one phenomenon with two applications, precision allocation and token selection.

Where the model-keyed boundary stands after two days. Yesterday's entry (below) recorded Cross-Model KV Sharing as the candidate mechanism for making a cache portable across models, deleting a prefill rather than accelerating one. Today's pair deletes work on the decode side. The two-day arc is coherent and worth stating as this page's current frame: yesterday, do not re-read what another model already read; today, do not read what you will not use. Both are instances of the constraint Ken Huang's inference physics (09-02) made non-negotiable, that a 70B FP8 model spends 99.66% of every decode step moving bytes at under 0.3% of peak Tensor Core throughput, so only byte-reducing or trip-skipping optimizations count. DA's headline number is a direct cut to the bytes moved per token, and it is the rare one that raises arithmetic intensity without batching.

Industry moved the same lever the same week, as a price. Anthropic's Fable 5.1 (09-02) cut cache-read cost 75%, to $0.25 per million tokens, which the release says should save heavy agentic workloads up to 45%. That is a provider repricing cache reads downward at the moment two papers make cache reads avoidable. The unpriced item this page should now track is whether cheap cache reads reduce the incentive to adopt read-avoidance at all, because a 75% price cut on the operation is a partial substitute for a 52% reduction in the operation.

Standing gap, unchanged. Neither paper reports realized latency. CRISP quotes attention speedup and DA quotes attended-token reduction, and both are proxies: skipping non-contiguous KV regions can defeat memory coalescing, so a 52% cut in attended tokens is not a 52% cut in decode time. The gap between theoretical and realized sparsity has dogged this literature all year and neither of today's papers closes it.

Prior State (as of 2026-09-02)

The model-keyed cache boundary this page named four days ago as "the most concrete unpriced item where this page meets routing" now has a candidate mechanism, and it is more radical than the problem required. A Universal Context-Reuse Layer for Cross-Model KV Sharing (09-02) (arxiv 2608.30963, Kurate cs.LG #17) trains a translation layer that converts the KV state produced by one model into a representation a different model can consume, across differences in scale, architecture, attention configuration, tokenizer, and model family. The authors name the abstraction context mobility: KV states as transferable computational representations rather than strictly model-local caches.

The three reported settings, in ascending order of how much they should change your mental model:

• Qwen2.5-1.5B → Gemma-2-2B (cross-family): up to 67.05% reduction in target-side prefill cost at 4K context, decoding perplexity close to native.
• Llama3.1-70B → Qwen2.5-7B (heterogeneous): 44.0% accuracy against 45.7% native, latency 899ms → 138ms, roughly 6.5x.
• Qwen2.5-7B → Qwen2.5-1.5B (within-family): LongBench2 27.59% → 34.48%, a 6.89-point gain over the native 1.5B baseline. The small model does better reading a large model's translated KV than it does reading the prompt itself, which is the finding that does not fit any framing on this page.

What this does to the 08-29 entry. That entry recorded three operational facts about provider prompt caching, the third being that cache entries are keyed to a model, so a mid-session route to a cheaper model pays a full cold prefill on the whole accumulated history, and that on the 140K-token median agentic prefix measured from replayed Claude Code and Codex traces this plausibly exceeds the per-token saving the route was chosen for. This is the first mechanism in the wiki that removes that penalty rather than pricing it. The honest caveat is a regime mismatch: the 67% prefill saving is quoted at 4K, and the problem lives at 140K. Until someone reports handoff cost and accuracy at 64K-plus, context mobility is a demonstration rather than a routing primitive.

It narrows the append-never-edit rule's claim. This page records three independent arrivals at "keep the prefix byte-identical": TokenPilot (06-16) as a research result, DeepSeek's Harness v0.1 (08-14) as an engineering commitment alongside a roughly six-fold cache-hit price rise, and the 08-29 practitioner explainer as production advice. Cross-model handoff asserts something much stronger: the prefix does not have to be the same model's to be reusable. If that holds, the operative rule shifts from "byte-identical prefix" to "semantically recoverable prefix," which is a materially different serving contract and a much weaker constraint on harness design.

And it sits in direct tension with the bounded-cache answer. Maglev (08-16) removes the eviction decision entirely by bounding the cache with a fixed-size recurrent memory. Cross-model sharing does the opposite: it keeps the full growing cache and makes it portable. The two cannot both be the right direction, and the reason is the one this page already noted about Maglev and prefix economics. A model carrying a bounded recurrent state has little cached prefix worth transferring, so a mobility layer has nothing to move. Whichever of these wins decides whether prefix stability remains a billable asset.

What is still missing, and it is the same omission as everywhere else. No cost is reported for the translation layer, to train or to run per handoff. No compounding analysis across multiple sequential handoffs in one long session, which is the actual deployment shape. And the ai_rating of 3.5/10 is the lowest in this week's Kurate cs.LG top 20, with the authors themselves calling the evidence initial.

Separately, today supplies the physical reason a saved prefill is worth more than a faster decode. The Physics of LLM Inference (09-02) derives that prefill is the compute-bound phase at 150 to 450 FLOP per byte while decode is memory-bound at 1 to 2, and that on a 70B FP8 model memory movement is 99.66% of every decode step. Eliminating a redundant prefill frees the one phase where the accelerator runs near its ceiling. It also prices this page's central resource: escaping the memory wall on an H100 would need roughly B = 296 concurrent decode streams, and KV cache memory is what stops you getting there. That is the cleanest statement yet of why everything on this page matters, and it means the value of KV capacity work rises with each accelerator generation, because compute has outpaced HBM bandwidth every generation and pushes the required batch size up.

Prior State (as of 2026-08-29)

This page has been conflating four different caches under one word, and a practitioner explainer arriving through saved reading makes the separation. KV vs Prefix vs Prompt vs Semantic Caching (08-29) lays out four layers keyed on four different things: the KV cache (GPU memory, one request, ~40 GB for a 70B at BF16 and 128K context), prefix caching (server side, vLLM's 16-token blocks identified by a hash chain over the parent hash plus token IDs, the scheduler stopping at the first miss), prompt caching (the provider's billed version, ~1.25x base input rate to write and ~0.1x to read), and semantic caching (application layer, embedding nearest-neighbour over stored responses).

Three operational facts this page did not have, all in layer 3. Writes happen only at a breakpoint you placed. On reads the system walks backward through a limited number of blocks and Anthropic caps this at 20, so more than 20 blocks of conversation between two calls pushes the last write out of range. And cache entries are keyed to a model. That last one is a routing result, not a caching result: on the 140K-token median agentic prefix AgentX (07-25) measured from replayed Claude Code and Codex traces, a mid-session route to a cheaper model pays a full cold prefill on the entire accumulated history. Nothing on the LLM routing page prices that, and at that context length it plausibly exceeds the per-token saving the route was chosen for. This is now the most concrete unpriced item where this page meets routing.

The append-never-edit rule reaches three independent arrivals, which crosses this page's pattern threshold. The article's four production failure modes are all prefix-shape rather than capacity: variable content at the front of the prompt, tool-schema reordering, settings rendered into the prompt, and history summarization. Its operative advice, truncate tool outputs in place to keep the prefix byte-identical rather than summarizing history, is the same rule TokenPilot (06-16) derived as a research result (any context edit mutating the prefix forces a full prefill recompute that cancels the token saving) and the same rule DeepSeek's Harness v0.1 (08-14) enforces as an engineering commitment while DeepSeek raised cache-hit prices roughly six-fold. Paper, vendor, practitioner. Same rule.

And it adds a row this page has never had: a cache layer that can be wrong. Every method tracked here is correctness-neutral in intent, and where correctness enters it is as measurement difficulty: the 07-28 impossibility result proved deterministic top-k eviction cannot estimate the error it created, and Compute Globally, Materialize Locally (07-29) showed that observing no accuracy loss after dropping a cached fact does not prove the fact was unnecessary. Semantic caching is a different class entirely. It does not degrade an answer, it returns a different question's answer with an HTTP 200, because embeddings place negated sentences close together and two prompts differing in one operational value score near-identical when the frame dominates. Published similarity thresholds span 0.75 to 0.97, and the spread is the admission: nobody has published a false-hit rate as a function of threshold on a real query distribution. That is the cheapest high-value missing measurement in this area and it needs no GPU.

Note on provenance: this arrived through saved reading rather than the paper feed, and it is the first save in five days after a thirteen-save run on harness and loop engineering. The practitioner layer moved from "how to drive the agent loop" to "what the loop costs at the cache," which is the same transition the research feed made on 08-28 when PILOT (08-28) became the first harness paper to publish a serving-side cost-per-success number.

Prior State (as of 2026-08-16)

Two results this week attack the cache from directions this page has almost no evidence on: the architecture that produces it, and the batch that competes for it.

Maglev (08-16) changes the shape of what gets cached rather than the policy over it. Almost every method on this page is a policy applied to an existing growing cache: what to evict, what to quantize, what to share across heads or layers. Maglev bounds the cache by construction with a fixed-size recurrent memory that still updates every token through the model's full nonlinear depth. It trains two coupled models, a prefiller Q with full history that emits memory targets and a decoder P with only sliding-window attention plus recurrent K/V injection, aligns them with a memory consistency loss, and then serves P alone. It beats both sliding-window and latent recurrent transformer baselines on validation loss and downstream pretraining benchmarks, and parameter sharing between P and Q keeps most of the gain while cutting parameter memory. This is a pretraining-time change, so it is the slowest thing here to reach production, and it is the only one that removes the eviction decision entirely rather than making it smarter.

Gambit (08-16) introduces an eviction unit this page has never had: the whole sequence. In batched reasoning inference the binding constraint is not FLOPs but the KV cache held by N concurrent long traces, and Gambit kills weak traces to fund new branches off strong prefixes, keeping utilisation high while cutting total token consumption by up to 68.5% and more than doubling completion throughput. Every eviction policy on this page assumes one sequence and asks which tokens to drop; Gambit assumes many and asks which sequences to drop. Those compose and nobody has stacked them.

The economics below still stand, and Maglev complicates them in an interesting direction. If a served model carries a bounded recurrent state rather than a growing prefix, then prefix stability, which DeepSeek's repricing just turned into a billing line, protects less value because there is less cached prefix to protect. Maglev does not discuss serving economics at all. The two results point at each other anyway.

Prior State (as of 2026-08-14)

The cache stopped being an implementation detail and became a line item on a customer invoice. DeepSeek raised cache-hit token prices roughly six-fold (08-14) while simultaneously open-sourcing Harness v0.1 under MIT, a harness whose central engineering commitment is never altering previously written history: when something in the conversation changes it appends a correction at the tail rather than editing the prefix, because editing invalidates every cached token after the edit point. DeepSeek raised the price of exactly the thing its own harness is engineered never to lose.

This converts a standing inference on this page into an observable price. The economic note below records SemiAnalysis's finding that frontier lab unit economics depend on >90% prompt-cache hit rates, with Anthropic's blended agentic price for Opus 4.7 near $0.99/MTok against a $5/$25 sticker. That was reasoning about provider margins. A 6x cache-hit price makes the cache a cost customers can see and must architect against, and it moves the local-serving break-even (07-30) toward self-hosting for exactly the repeated-context agent workloads where local serving was already most competitive. Append-only context management is now an economic requirement rather than a best practice, which retroactively makes TokenPilot (06-16) a cost-engineering paper rather than a convenience one.

On the mechanism side, outlier position turned out to be architecturally determined rather than empirical. Massive Activations in Hybrid Linear Attention (08-14) finds that in hybrid models (Qwen3-Next, Qwen3.5, Kimi Linear, Kimi K3, Nemotron-H) massive activations spike immediately before every full-attention layer (pre-attention spikes) and persist through the intervening linear layers as plateaus (inter-spike plateaus), with denser full attention connecting the spikes until the classic full-attention profile is recovered. This is the missing coordinate for the 05-19 activation census, which established that magnitudes vary four orders of magnitude across families and recommended per-family calibration but treated layer index as something to scan. For a hybrid model, spike location is now readable off the architecture config. The quantization consequence is scheduling rather than headroom, and it composes with the MoE-headroom asymmetry the 05-19 page identified. The ablation proving a PAS-aware scale beats a flat family-calibrated one is not in the paper and is the cheapest high-value experiment in this area.

A third result reframes what the cache does not hold. The full-bandwidth transformer (08-14) preserves the KV cache untouched but points out that hidden states in it are depth-frozen: reachable only by layers above where they were produced. Feeding the previous top-layer hidden state back into the bottom of the stack through a gated linear unit gives non-verbalized computation a renewed depth budget, and at 1B/400B tokens buys the equivalent of ~1.5x more training data plus shorter reasoning traces at equal or better accuracy. This page has treated the cache as a thing to compress, evict from, tier, and transmit. Vertical restriction is a property nobody here had named. Open risk: latent feedback creates a step-to-step dependency that speculative decoding assumes away, so composition with speculative decoding is untested.

And the consolidation-cadence lever showed up in agent memory. LycheeMemory V2 (08-14) replaces per-turn memory consolidation with semantic segment-level consolidation and cuts construction tokens 86% on LoCoMo and 75.9% on LongMemEval-S while reaching state of the art on both and not increasing query-time tokens. Combined with ICBQ's block-order finding (08-12) and ReOrder-OPD's prompt-ordering result (08-13), that is three efficiency subfields in three days where the schedule beat the operator. The pattern threshold is crossed. KV eviction cadence, as opposed to eviction policy, is the obvious untested fourth instance and this page's literature is almost entirely about policy.

Prior state (as of 2026-08-11)

Every method on this page decides what to throw away. Today's decides where the cache lives, and that is a different question. OasisKV (08-11) (2608.08097, HuggingFace) decouples full KV storage from HBM entirely: the complete cache sits in a higher-capacity tier such as host or remote memory, and only the entries the next decode step will actually attend to are staged in HBM. The prediction mechanism is the part worth keeping. Speculative decoding's draft tokens are reused as a lookahead probe for which KV blocks the model is about to consult, an attention background pipeline scores blocks against that lookahead, and the winners are prefetched one step ahead of use. On vLLM: within 0.7 points of full attention at a 2,048-token KV budget, 1.69x over dense vLLM on reasoning at 0.1 points of loss, up to 2.1x on multi-GPU long-context serving, and under prefill-decode disaggregation about 2x dense throughput while admitting each request with 6.5 to 9.7x less KV and holding 2.2 to 2.6x less decode-node host memory than full KV transfer.

Why this reorders the page. LOCKS (07-29) attends about 2% of tokens via page-local spectral summaries, MSA (06-12) selects blocks per GQA group then attends exactly, VaSE (06-03) evicts stochastically with a large-magnitude value guard, Conf-KV (05-30) sets a per-step budget from model confidence. All of them are irreversible or quasi-irreversible decisions inside HBM. OasisKV's selection error is a stall, not a loss, because the block is still in the capacity tier. That distinction resolves, in the practical direction, the open question this page raised on 07-29 about whether the 07-28 impossibility result (deterministic top-k eviction cannot estimate the error it created) binds reversible selection as tightly as irreversible eviction: with a full cache resident one tier down, the error is recoverable next step regardless of whether it can be certified.

It is also the algorithm that makes a tiering policy this page has listed as unshipped finally cheap. The agentic memory hierarchy survey (06-07) argued dominant memory traffic shifts from weights to KV cache as context grows; AMD's MoRI roadmap in the 07-25 SemiAnalysis piece targets a tiered HBM → DRAM → NVMe KV cache with scheduler co-design. OasisKV hides the tier-crossing latency behind a prediction instead of paying it on demand, which is the missing piece. Its closest prior relative is FlashMemory-LSA (06-09), which trained a Neural Memory Indexer to predict future chunk demand for a 13.5% average footprint; OasisKV gets an equivalent predictor free from a drafter the stack already runs. Nobody has compared a trained indexer against a speculative drafter at matched prefetch budget.

Two caveats this page should carry. First, Eviction as Estimation (08-03) established that cache-management gains appear specifically where reuse is endogenous and time-separated, which describes agent traces and almost no common benchmark; OasisKV reports reasoning and long-context, not agentic traces, and the AgentX distribution (07-25) at a median 140K input tokens per turn is the workload that decides this. Second, the predictor requires speculative decoding, and the Kimi K3 primer (08-04) reported DSpark speculative decoding does not work with pipeline parallelism, so any model too large for one node loses the predictor entirely. No fallback path is reported.

And the same day, the latency side of the same serving stack gets attacked from the opposite direction, with a finding that changes what this page thinks the wall is. SemiAnalysis on TileRT (08-11) measures that at batch size 1 an 8-GPU HGX B200 has 64 TB/s aggregate HBM bandwidth and GLM-5 at NVFP4 needs about 21 GB of active-parameter traffic per token, implying a roofline near 3,047 tokens/s/user that real GPUs come nowhere near. The gap is kernel launch and synchronization overhead plus flat memory latency, not bandwidth: bandwidth improves 2 to 3x per GPU generation while memory latency has not improved at all, so this gap widens every generation. Compiling the entire decode graph into one persistent kernel reaches 500 tok/s/user on one B200, about 3x a GB300 NVL72 on conventional engines. For this page the consequence is that OasisKV and TileRT pull in opposite directions on the throughput-interactivity curve, and a stack running both would need per-request deadline expressiveness that neither vLLM nor SGLang has, which is the gap AAPT (08-04) exposed when it showed deadline-bound GUI agents scoring 0.00 while decoding during execution.

Prior State (as of 2026-08-10)

A third axis arrives, and it is one this page has never priced: whether the cache is still addressable. WorldTrace (08-10) studies visual persistence in interactive video world models, which carry generated frames forward in a growing KV cache, and finds that once a rollout extends past the training horizon the model can no longer reliably retrieve stored content at all. The content is still there. The failure is lookup: the temporal RoPE (rotary positional embedding, which encodes position by rotating query and key vectors through a position-dependent angle) offsets for far-past entries fall outside the range seen in training, so attention cannot find them. Size is fine, bandwidth is fine, and the memory is unreachable.

The second finding is an arithmetic indictment of a common implementation choice. Naively compressing the cache in the RoPE-rotated space corrupts memory by averaging together incompatible positional phases. Any merge, average, or low-rank summarization applied after the rotation blends vectors whose phases differ, producing a result that points where neither original pointed. WorldTrace's fix is training-free and almost embarrassingly cheap: give each compressed summary slot a distinct, in-distribution virtual position, so position becomes an addressing handle you assign rather than a timestamp you inherit. WorldTrace-Field compresses continuous history for temporal coherence (+15.5%); WorldTrace-Landmark stores verbatim scene traces at detected transitions for episodic recall (+19.5% on its new LoopBench, which requires leaving a scene, taking a long detour, and reconstructing it on return).

This composes with, and slightly reorders, the metric argument at the top of this page. SemiAnalysis on Kimi K3 (08-04) argued cache size is not a standalone property and proposed KV throughput, cache size divided by prefill time, as the honest unit. WorldTrace adds a property that neither size nor throughput observes: a cache can be small, fast to fill, and functionally unreachable. Any compression ratio computed in rotated space is a ratio on a corrupted object, which puts this straight into the page's existing measurement-validity thread alongside Eviction as Estimation (08-03), which found KV-eviction ablations do not measure the quantity they are trusted for.

It also undercuts the premise of position-based eviction. Sliding-window and streaming caches evict by position on the belief that position ordering is what attention consumes. WorldTrace shows position is simultaneously the lookup key, so evicting by position destroys the index at the same moment it frees the memory. And it arrives from the opposite direction to Raven (08-04), which keeps a fixed set of memory slots and routes which subset to write, holding recall at 16x training context: Raven's slots are learned at training time, WorldTrace's positions are assigned at serving time, and both are arguing that memory addresses should be allocated rather than inherited from the token stream. Neither cites the other.

Open, and the obvious next experiment: the result is reported on video world models only. Whether text long-context models show the same addressability collapse past the training horizon, separately from the well-studied length-extrapolation loss degradation, is untested. If they do, every production stack that compresses or offloads KV after rotation is silently corrupting its compressed tier, and the fix is a few lines of position bookkeeping.

An independent witness on the same day, from the understanding side rather than the generation side. StreamArena (08-10), a benchmark of 243 videos averaging 88.8 minutes with open-ended questions, reports a three-way streaming-memory tension in which one corner is that "methods that repeatedly compress visual memory struggle to preserve fine-grained details over time." That is WorldTrace's symptom, measured empirically in a different task, on the same day, with no mutual citation, and WorldTrace's virtual-position assignment is a candidate cause and cure for it. That composition is the most valuable untested item on this page.

Prior State (as of 2026-08-04)

This page finally gets the metric it has been missing, and the same article takes back part of the biggest number on it. SemiAnalysis on Kimi K3 (08-04) refuses to score cache efficiency by cache size, on the argument that size is not a standalone property of a model but a consequence of the whole design: no open-weight model ships static KV compression, architecture affects cache efficiency, and how much HBM is left for cache depends on the parallelism strategy, since wide expert parallelism and tensor parallelism leave very different amounts free. The proposed replacement is KV throughput: KV cache size divided by prefill time (time to first token), at a given sequence length. It is the minimum bandwidth needed to serve reliably under prefill-decode disaggregation, and because prefill time encapsulates architectural efficiency the metric walks through memory-bound and compute-bound regimes as context grows. Reported direction: hybrid linear attention's benefit becomes more pronounced as sequence length increases. This is the first metric in this wiki that prices architecture and cache size together, and it is the natural unit for the offload hierarchy the agentic AI memory hierarchy survey (06-07) described, where dominant memory traffic shifts from weights to KV cache as context grows.

The retraction, and it is important. The local coding model report (07-30) measured Nemotron Cascade 2 holding 262K context in under 2 GB of KV against roughly 40 GB for dense Devstral Small 2, concluded the largest available win is architectural and chosen at model-selection time, and this page ran with it. SemiAnalysis adds the production caveat that measurement could not see: the constant-state property belongs to the forward pass, not to a serving system with prefix caching. Inference engines find cache hits by matching the longest cached token prefix, and a recurrent state at position t cannot be reconstructed from a snapshot at t-k without replay, so arbitrary prefix matching would require snapshotting every position, at which point memory grows with sequence length again. Moonshot's fix is coarse: vLLM snapshots KDA state every 32K tokens plus at prompt boundaries, on the reasoning that agentic turns start at prompt boundaries. Their conclusion, stated plainly, is that linear attentions do not consume a constant amount of KV memory in real serving. The 20x is right for one local session and optimistic for a multi-tenant server with prefix reuse, which is the deployment the datacenter pays for.

The hard practitioner number is a cliff, not a slope. InferenceX replayed an hour of real recorded Claude Code traces at steady state rather than fixed-shape synthetic prompts, which is a methodology worth copying: the trace statistics alone are a useful artifact, at a median 142K input tokens and 444 output tokens per turn over a median 65 turns per session, the short outputs being characteristic of harnesses where even an edit is a tool call. Kimi K3 does not fit on a single B200 node, requiring pipeline parallelism, and DSpark speculative decoding does not work with pipeline parallelism, so that path loses speculation entirely. On B300 the model fits one node and HBM holds 3.25M tokens of KV after weights; throughput climbs with batch size until concurrency exceeds 8, exactly where that budget is exhausted, and then prefix cache hit rate collapses below 10% against a theoretical 95%. That is the same shape as TokenPilot (06-16)'s finding that any context edit mutating the prefix forces a full prefill recompute: agentic serving economics are dominated by whether the prefix cache holds, and the failure mode is a cliff. It also sits against the 07-25 AgentX figures reporting a median 99.2% hit rate, and the reconciliation is concurrency: a high hit rate is attainable below the KV budget and unattainable above it.

On the architecture side, the page's sparse-retrieval-over-a-growing-cache pattern gets a fourth instance, and the first one inside the model's parameters. Raven (08-04) (2607.25357, Albert Gu among the authors) keeps a fixed set of memory slots and routes each token's write to a selected subset, decaying only what it touched, which is the training-time twin of what LOCKS (07-29) argued at serving time: a shared representation across all content destroys the content-specific directions that distinguish neighbours, so give each unit its own summary and only touch what you mean to. Everything else on this page reconstructs a relevance signal from a cache after the fact. Raven decides what enters the state in the first place.

Two operational addenda to the SemiAnalysis numbers above. First, the B300 cliff has a workload twin that inverts the whole framing, and it arrived the same day. AAPT (08-04) shows GUI agents failing transient events because autoregressive decode sits on the decision-time critical path, and fixes it by pre-building a bounded policy tree during idle screen periods so execution needs no generation at all. Success inside a contested window rises 0.50 to 0.79 with zero incorrect actions, while open-loop and predict-and-replan baselines both score 0.00 because they still decode during execution. For that class, decode latency is not a cost axis, it is a correctness axis with a step function: below the deadline you succeed, above it you score zero regardless of how right the answer was. Both readings are true of different agent workloads, and nothing in vLLM or SGLang can express a per-request deadline, so a scheduler batching for throughput is exactly wrong for a deadline-bound agent and fails silently rather than slowly. This page has priced latency in dollars and throughput throughout (DraftExpert (08-03) at 1.45x on phones against the 2x-plus datacenters report, LOCKS halving per-token decode latency); AAPT is the first result where the units are wrong.

Second, on the metric: KV throughput is the better version of the standing gap this page named on 07-30, that nobody publishes cache-per-token. Cache-per-token is architecture-blind and KV throughput is not. The comparison SemiAnalysis does not run, and the one that decides real design choices, is KDA-plus-MLA against the GQA-sparse family (GLM 5.2's DeepSeek Sparse Attention, DeepSeek V4's Compressed Sparse Attention, MiniMax M3's sparse attention, MiMo V3's HySparse) on the same axis at the same sequence lengths.

Current State (as of 2026-08-03)

The most useful eviction paper this year reports that its own method does not beat H2O, and the diagnosis indicts the benchmark suite this whole page is scored on. Eviction as Estimation (08-03) (2607.24667, Kurate cs.AI #13) recasts eviction as estimation of a hidden signal, whether an item will be reused, and puts every deployed method on one axis: the commit lag H, the number of steps a policy waits before committing. StreamingLLM, H2O, SnapKV and learned predictors all sit at H=0. Belady's offline optimum sits at H=infinity and is unobservable. The empty middle is fixed-lag smoothing: wait a bounded number of steps, observe which items a correct near-future prediction attended to (the paper calls this demonstrated utility), then commit. The instantiation, RMM, is training-free and reduces to H2O exactly when the measurement is uniform.

Then the honest part. Run inside NVIDIA's KVPress harness against KVPress's own SnapKV, H2O and StreamingLLM implementations, RMM ties H2O on single-turn QA and loses to both on streaming multi-turn. The cause is exact: on natural text the model is correct about almost every token, so weighting attention by correctness is close to the identity, and demonstrated utility collapses onto accumulated attention. The gain appears only where reuse is endogenous (the model's own earlier output determines later consultation) and separated in time, which describes agent traces and describes no benchmark in common use. Two consequences for this page. First, if your workload looks like natural-text QA, the eviction policy choice is nearly free and you should pick the cheapest one, which is consistent with LOCKS (07-29) winning by making selection cheap rather than better. Second, it sits in genuine tension with Make Each Token Count (05-12), which claimed learned globally calibrated eviction can surpass the full cache: one paper says the scoring function has large headroom, the other says almost none, and the variable that would settle it is whether Make Each Token Count's gains survive the KVPress harness against KVPress's own baselines. Nobody has run that. Third, it pairs orthogonally with the 07-28 impossibility result: randomization buys you attribution of the error you created, fixed-lag smoothing reduces the error you create, and the composition (a Poisson-sampled tail with a smoothed commit) does not exist yet. → summary

The agent-side twin of KAP arrived one day later. ACM: Agentic Context Management (08-03) (2607.23809, Meta and CMU) makes the same argument KAP makes about retrievers, but about the agent itself: the standard token-threshold compactor fires because of how much text exists, not because of what the agent is reasoning about, so it is misaligned with the work by construction. ACM gives the agent context-editing tools so it decides when to compress, offloads what it drops to an external memory system rather than deleting it, and queries that store on demand. Reported effects are lower peak token pressure, longer explorations, and more consistent solutions across independent trials, which is a variance claim almost no agent paper reports. The critical missing number for this page: ACM edits the context repeatedly and reports no prompt-cache or prefill accounting, and TokenPilot (06-16) established that any context edit mutating the prefix triggers a full prefill recompute that cancels the saving. An agent that compacts ten times per session may use fewer tokens and cost more money. → summary

Current State (as of 2026-08-02)

Every selection method on this page re-derives importance from inside the model. The first paper to argue the importance signal was destroyed upstream just landed. KAP (08-02) (2607.24260, Kurate cs.LG #2) names the Knowledge Selection-Runtime Consumption gap: a modern LLM system spends real effort producing structured priors before the prompt exists (ranked evidence, graph topology, multimodal alignment, confidence scores), then serializes all of it into a flat token sequence, at which point the serving backend, the component that actually pays for the context, can only consume the KV state densely and uniformly. The consequence is stated as a scaling problem rather than an inefficiency: improving your retriever makes your serving worse, because richer context enlarges the full-prompt KV footprint and decode-time memory traffic together, even when reasoning depends on a small slice. KAP's fix is to compile those priors into a runtime access plan, a universal intermediate representation that governs physical KV access while leaving logical prompt semantics, model weights and training procedures untouched. GraphSpec is the compiler-executor instantiation. Across 4K-128K long-context QA it holds quality comparable to full-context decoding while cutting proposal-time KV access to 5.5% of source KV state at 128K, and the framing the authors reach for is the right one: it decouples physical KV consumption from prompt length rather than moving a point on the existing curve. There is also a derived phase-boundary model for when plan-guided execution is actually faster, which is unusually disciplined for a serving paper.

This reorders how the page should read its own selection line. MISA (05-11) routes on the indexer-head axis, RTPurbo (05-24) found the retrieval geometry lives in a 16-dimensional subspace, MSA (06-12) scores blocks per GQA group, FlashMemory-LSA (06-09) predicts future chunk demand, and LOCKS (07-29) estimates per-page attention mass from a page-local spectral summary without reading any candidate keys. All five reconstruct a relevance signal from the cache after the fact. In a retrieval-augmented or graph-grounded system that signal already existed. So KAP and LOCKS are complements rather than competitors: LOCKS is the method for when nothing upstream knows anything (a raw 1M-token document, an agent trace), KAP is the method for when something upstream knows a lot. Nobody has built the composition, and the fallback is the load-bearing missing piece, because a compiled plan is deterministic and derived from outside the cache, so it cannot certify its own error (the 07-28 impossibility result) and cannot even notice when the upstream prior was wrong. A retriever that ranked the right document third and a plan that reads the top two produce a confident wrong answer with a clean latency profile.

Two cautions on the number. "Proposal-time" implies a draft-and-verify structure, and if verification reads more of the cache the end-to-end saving is smaller than 5.5% suggests. And the quality claim rests entirely on long-context QA, which has a locatable evidence span and is exactly the regime where LOCKS found baseline selectors look fine before collapsing on long-form reasoning (AIME26, MATH-500). A plan compiled from retrieval structure has no obvious reason to survive diffuse reasoning, and no reasoning benchmark is reported.

Current State (as of 2026-07-30)

Practitioners measured a 20x KV cache gap between architectures, which is roughly an order of magnitude larger than anything this page's software methods report, and it reorders the whole optimization stack. The local coding model report benchmarked 9 models across consumer hardware and found NVIDIA's Nemotron Cascade 2 holding a 262K context window with a KV cache under 2 GB, against Devstral Small 2, a dense model, needing roughly 40 GB for comparable context. A 30B model in the set carries a cache 25x smaller than a comparable dense model. The arithmetic inverts the standard advice, because a 30B with a 2 GB cache is a smaller total footprint at long context than a 14B dense model with a 40 GB cache, so the larger model is the one that fits on the card.

This is independent confirmation, from a setting that shares no cost structure, of the claim the hardware memory survey (06-07) made about the datacenter: as context grows, dominant memory traffic shifts from weights to KV cache, making cache management the binding hardware constraint. It is also the same architectural property PrfaaS (04-22) exploits at the opposite scale, where hybrid-attention models emit cache 13x more slowly (MiMo-V2-Flash at 4.66 Gbps against 59.93 Gbps for a dense baseline), letting prefill be offloaded to another datacenter and the cache shipped over Ethernet. Same cause, measured as GB-resident on one card and as Gbps-on-the-wire across sites.

The sequencing lesson is new and it is uncomfortable for this page. July produced three strong software results on cache management: LOCKS (07-29) matching full-cache quality at 100K+ context while reading about 2% of tokens with 2x faster decode, Error Certificates for KV-Cache Eviction (07-28) proving deterministic top-k eviction cannot estimate its own error, and Sparse Event-KV (07-29) proving that dropping a cached fact and observing no accuracy loss does not prove the fact was unnecessary. All three take the cache as given and manage it better. The practitioner data says the largest single win available today is architectural, chosen at model-selection time, and it is roughly 10x what any of them report. Not an argument against the research line, an argument about order: pick the hybrid-attention model first, then apply page selection to what remains.

A second data point that context handling, not reasoning, is the active constraint. OpenAI reported that two settings tripled GPT-5.6 Sol's ARC-AGI-3 score, both of them about memory rather than capability: allow reasoning to continue across multiple context windows, and apply a canonical compaction implementation so it survives the boundary. Taken at face value, two thirds of the previously measured gap on that benchmark was an artifact of discarding reasoning at context boundaries. The caveat raised on social is fair, that other leaderboard entries were measured under different rules, so this is not a clean comparison. It is still a statement that the harness owned the score.

The cache constraint now has a robotics analogue. TurboVLA (07-30) reaches 97.7% on LIBERO with a 0.2B model at 0.9 GB inference VRAM and 31.2 ms latency by removing the language model from the vision-language-action pathway entirely. Different mechanism, same conclusion the local-model report reaches: total resident memory at inference is what determines deployment class, and architecture moves it by more than an order of magnitude.

Standing gap: nobody publishes cache-per-token. It is absent from every model card, it is the number that decides single-GPU deployability, and the gap between the best and worst answer is 20x.

Current State (as of 2026-07-29)

The eviction literature's evaluation methodology is now under attack from two directions in four days, and neither paper appeared on HuggingFace. Error Certificates for KV-Cache Eviction (07-28) proved that deterministic top-k eviction cannot estimate the error it created, and restored a per-step estimate via randomized Poisson eviction at 0.97 coverage. Compute Globally, Materialize Locally (07-29) (2607.23693, Kurate cs.AI #19) attacks the other half: it shows that observing no accuracy loss after dropping a source event does not show the source was unnecessary. Omitting one earlier observation from an otherwise identical agent history, the model's answer on items sensitive to that observation overwhelmingly still follows the omitted value, although no served span states it. The paper calls this semantic materialization: a downstream event's cached rows act as an independently servable view of a computation whose inputs are gone. So the standard eviction experiment (drop entries, measure accuracy, conclude the entries were expendable) is measuring something real and drawing a causal conclusion that does not follow. One paper says you cannot measure the damage; the other says your measurement of no damage does not mean what you think.

The contract it specifies is unusually actionable and unusually uncomfortable. Materialization can be written on purpose: a deliberately phrased, answer-free event raises donor-aligned recovery from 6% to 51% on Qwen3-8B without ever naming the value, while passively harvesting natural mentions from long dialog yields nothing detectable. But capacity is small (compact state survives, larger payloads decay toward chance) and, worst for anyone trying to design around it, whether a construction writes at all turns on phrasing rather than meaning, so two phrasings the model understands equally well can diverge sharply. The property is not semantic, which means you cannot reason your way to it from content.

This is the correctness question behind the workload AgentX (07-25) measured: at a median 140k-in/396-out ratio the cache is the agent's working memory rather than a decode accelerator, so what survives eviction stops being a throughput question. It also gives the Frozen 12B verified-memory result (07-28) an uncomfortable sibling. That paper's store is explicit, addressed, and auditable. Semantic materialization says a second store exists inside the cache rows, unaddressed and uninspectable, and a serving system now has two memories with different reliability properties.

Meanwhile the selection side gets its cleanest engineering result yet, and it exposes that the field has been benchmarking on the easy half of the distribution. LOCKS (2607.24555, Kurate cs.LG #2) starts from a measured structural fact: attention keys are locally low-rank but globally high-rank, so a shared low-rank basis across the whole cache discards exactly the page-specific directions that distinguish one page from its neighbours. LOCKS gives every page its own spectral summary, resident at about a tenth of cache size, reconstructs within-page logits from it, estimates each page's attention mass by log-sum-exp, and attends only the top pages. The decisive design choice is that selection reads no candidate keys or values at all, so selection cost is independent of how much cache sits on the wrong side of the bandwidth wall, which is what caps methods that score blocks by reading a representative key. At a 2048-token budget it matches FullKV aggregate quality at 100K+ context attending ~2% of tokens, halves per-token decode latency (2.0x at 1M), and ships as a drop-in plugin for unmodified vLLM with batched decode in full CUDA graphs.

The result that should change how this page reads its own history: LOCKS's largest margins are on long-form reasoning (AIME26, MATH-500), where baseline selectors collapse, not on the retrieval and long-document QA where the whole selection line (MISA head-axis, MSA blockwise select-then-attend-exactly, RTPurbo query-dependent budgets) validated. Retrieval tasks have a contiguous evidence span a selector can find; long-form reasoning needs diffuse context and has no reason to work under a proxy tuned on retrieval. That baselines collapse there and a per-page basis does not suggests the field's benchmark habit has been hiding a real capability gap. Note also that LOCKS is deterministic top-k with a log-sum-exp mass estimate, so the 07-28 impossibility result says it cannot certify its own error either. The open distinction nobody has drawn: LOCKS selects rather than evicts, so skipped pages stay resident and a mistake is recoverable next step. Whether the certificate result binds reversible selection or only irreversible eviction is the most useful thing to resolve about either paper.

A fifth non-uniform-budget axis, and it is the first semantic one. OmniDelta (2607.25669, HuggingFace) allocates a fixed retained-token budget across modalities in omni-modal models, and its useful contribution is a negative result: direct query-to-audio or query-to-video similarity is unreliable for the inter-modal split, because similarity measures whether a modality is about the query, not whether the answer requires it. It instead reads query intent against per-modality skill pools, then reallocates within each modality by local complexity and temporal redundancy, composing with any existing pruner. At 25% retention on Qwen2.5-Omni-7B: 22.0% less GPU memory, 1.64x end-to-end speedup. The page's non-uniform axes now read head (MISA), head-role (Forcing-KV), value-magnitude (VaSE), layer-function (Rethinking Efficient Attention), and now modality. The first four are properties of the network and can be calibrated offline the way Tangram (06-16) calibrates its head ranking from ~50 samples. Modality is a property of the question, so it cannot be, which puts OmniDelta with KVServe (05-24)'s online-control-surface framing rather than Tangram's static one.

And the sparse-retrieval-over-a-growing-cache primitive escaped the serving stack entirely this week. Wonder (07-29), an interactive video world model, needs to let a user revisit a place seen minutes ago, so it uses a sparse-attention memory that attends to a small relevant set regardless of true context length. That is structurally MSA's select-then-attend-exactly and LOCKS's cheap-selection claim, arrived at from video generation. Three subfields (long-context serving, agentic memory, interactive world models) converging on one mechanism in a quarter is worth naming: sparse retrieval over a growing cache is becoming the default answer to persistence, whatever the modality.

Current State (as of 2026-07-25)

The measured shape of agentic serving arrives, and it is 140k in / 396 out at a 99.2% cache hit rate. Every KV paper this page tracks is implicitly justified by an assumption about the serving distribution, and the industry-standard proxies (8k-in/1k-out, 1k-in/1k-out) are single-turn with random data. SemiAnalysis's AMD Advancing AI 2026 piece introduces AgentX, an InferenceX scenario built by replaying roughly three months of SemiAnalysis's own Claude Code and Codex traces offline at varying concurrency via AIPerf, developed with WEKA, Inferact, RadixArk, LMCache, Mooncake, NVIDIA, and AMD. The measured distribution: median input sequence length 140k, median output 396, median cache hit rate 99.2% (under an infinite-cache assumption, so an upper bound rather than an attainable number). It includes realistic subagent usage and dynamic workflows that continuously inject uncached context, and it captures inter-turn latency from tool use and user think time, which is what determines KV time-to-live and therefore the actual value of KV offloading.

Three consequences for this page. First, it empirically confirms TokenPilot's (06-16) economic claim that prompt-cache hit rate, not token count, is what clears the bill: at 99.2% hit rate on a 140k prefix, a single prefix-invalidating edit costs more than any plausible token saving. Second, it reframes the whole eviction/compression literature: with a 140k:396 in/out ratio, agentic serving is a prefill-and-retention problem, not a decode-throughput problem, which is the workload argument for prefill/decode disaggregation rather than a systems preference. Third, inter-turn think time is the variable that decides whether a KV block should be offloaded to a cheaper tier or dropped, which is exactly the KV-aware tiering policy memory-hierarchy has listed as an open problem and not-yet-shipped serving feature.

Relatedly, Claude Opus 5 (07-25) fixed one specific prefix-invalidating event at the platform layer: adding or removing tools mid-conversation no longer invalidates the prompt cache. Tool-definition churn was one of the concrete prefix breaks TokenPilot's Ingestion-Aware Compaction was designed to work around from the client side; Anthropic moved it into the API contract.

Disaggregation's composability, not its existence, is now the bottleneck. The same SemiAnalysis piece frames the competitive frontier as composable distributed inference: prefill/decode disaggregation, WideEP, RDMA KV transport, and cluster scheduling working together by default rather than per-model. Concrete numbers: an 8,192-token DeepSeek-R1 FP8-KV prefill moves roughly 290MB of KV over RDMA; moving DeepSeek-R1 (256 routed experts) from EP8 on one node to EP64 across 64 GPUs cuts experts per GPU from 32 to 4, freeing HBM for KV cache and enabling larger concurrent batches, with NVIDIA reporting up to 2.28x higher per-GPU output throughput from Wide-EP. Expert parallelism is therefore a KV-capacity lever, not only a compute-sharding one, which is a link this page had not made explicit. AMD's MoRI H2-2026 roadmap makes the tiering target concrete: a tiered distributed KV cache (HBM → DRAM → NVMe via SPDK/GDS) with scheduler co-design. And the cautionary datapoint: composition failures show up as silent accuracy cliffs, not crashes. An uninitialized reduce buffer in AITER's FP4 MoE kernel produced fluent-but-wrong output scoring 0 on GSM8K (patched June), and EP decode at concurrency 64 in SGLang with the MoRI backend still drops GSM8K to ~80% against a ~94% baseline that holds at every other batch size.

Current State (as of 2026-06-17)

Three structural KV levers the same day: a layer-axis allocation prior, a width-axis saving, and a train/inference KV-consistency rule. (1) Rethinking Efficient Attention in Hybrid Architectures (arxiv 2606.15378) shows long-range retrieval is localized to the full-attention layers, with the efficient (SWA/linear) layers shaping the optimization path — a layer-axis allocation prior the non-uniform-KV line (Tangram, MISA) can exploit directly: compress the efficient layers aggressively, protect the full layers. (2) Variable-Width Transformers (arxiv 2606.18246) narrows the middle layers (wide ends, narrow middle), yielding ~15% smaller KV memory and I/O as a structural saving baked into the architecture, not a serving-time compression. (3) LoopCoder-v2 (arxiv 2606.18023) holds the KV footprint roughly constant across loop count via Shared-KV Gated Sliding-Window Attention (share the first loop's KV, blend global/local), making test-time-compute depth a free knob rather than a linear KV penalty. Separately, GLM-5.2's release surfaced a concrete train/inference KV-consistency result (via @eliebakouch): at the t+2 multi-token-prediction head it deliberately omits the indexer KV of the t+1-predicted value ("indexer sharing"), which is both cheaper and better because it removes a distribution shift on that specific cache entry. The page's principle list gains: KV consistency between training and inference matters as much as raw cache size, and KV budget should follow layer function (retrieval in full layers, compress the rest).

Current State (as of 2026-06-16)

The serving substrate finally catches up to head-non-uniform compression, and the cost target shifts from token count to cache continuity. Two papers land the systems-and-economics layer the page's algorithm-heavy backlog needed. Tangram (arxiv 2606.06302, on vLLM) explains why non-uniform KV compression — heterogeneous per-head budgets, which the page has tracked via MISA (head-axis), Forcing-KV (head-role), and VaSE (value-magnitude) — has stayed in papers: serving stacks assume uniform KV length per head, so heterogeneity traps freed memory as page fragmentation, burns up to 25% of prefill on page reclamation, and inflates decode latency up to 1.7x. Tangram's key empirical claim is that head-wise retention follows a two-level structural regularity (an input-invariant head ranking with narrowly bounded per-head ratios) calibratable offline from ~50 samples — so it resolves statically (Budget Reservation, Ragged Paging, Ahead-of-Time Load Balancing) what prior systems handled dynamically, matching the wrapped method's accuracy at up to 2.6x throughput over full-KV. This is the execution-side complement to KVServe (05-24, compression profile as an online control surface): KVServe picks the profile dynamically, Tangram exploits non-uniform input-invariance to fix it offline. The "input-invariant head ranking from 50 samples" is the same outlier-is-locatable physics as RTPurbo (16-dim retrieval subspace) and LongAct (saliency peaks), now turned into a scheduling primitive.

TokenPilot (arxiv 2606.17016, Zhejiang U et al., in LightMem2) names the economic tension the SemiAnalysis 05-01 note implied: agent context management that optimizes token count alone mutates the prompt sequence, which breaks the prompt prefix the backend was caching and triggers a prefill recompute that cancels the savings. Its fix is dual-granularity — Ingestion-Aware Compaction stabilizes prefixes at the ingestion gate; Lifecycle-Aware Eviction defers structural offload until a segment's residual utility expires on a batch-turn schedule — for 61–87% cost reduction at competitive performance. TokenPilot is the agent-trajectory sibling of KV Packet (04-17, immutable cacheable document packets): KV Packet preserves cacheability for retrieved docs, TokenPilot for the agent's own growing trace. The page's cost axis now reads: token count and prompt-cache hit rate, the latter being the one that actually clears the bill.

Current State (as of 2026-06-12)

The MiniMax-M3 sparse-attention engine gets its paper — and audited numbers. MiniMax Sparse Attention (MSA) (arxiv 2606.13392) is the mechanism behind the MiniMax-M3 release we logged on 06-03 from a vendor blog. MSA is blockwise sparse attention on a GQA backbone: a lightweight Index Branch scores KV blocks and selects a Top-k subset per GQA group, then the Main Branch does exact attention over only those blocks. The paper's measured figures refine the 06-03 marketing claims (≈1/20 compute, ~9x prefill, ~15x decode): 28.4x per-token attention-compute reduction at 1M context on a 109B model, on par with dense GQA, but a more grounded 14.2x prefill / 7.6x decode wall-clock on H800 with the co-designed kernel (exp-free Top-k, KV-outer layout for tensor-core utilization). The compute-vs-wall-clock gap is exactly the kernel-efficiency caveat the 06-03 note flagged. MSA is firmly in the "select blocks, attend exactly" camp the page tracks against training-free eviction (VaSE) and predicted-demand pruning (FlashMemory-LSA) — and it is now the one shipping as open weights, confirming the RTPurbo (05-24) "full attention is intrinsically sparse, useful budget is query-dependent" thesis at production scale.

Current State (as of 2026-06-11)

The video-agent line gets a token-preserving KV reparameterization. InternVideo3 (arxiv 2606.12195) introduces M²LA (Multimodal Multi-head Latent Attention), which compresses the per-token KV state into a low-rank latent while keeping the full token stream. That distinction is the point: the page's prior video-KV methods — VideoMLA (06-02), WorldKV (05-24), StateKV (06-01) — compress mostly by evicting or merging tokens, but InternVideo3 is built around a closed-loop video agent that may re-query an earlier frame via retrieval tools, and you cannot retrieve a token you evicted. So M²LA shrinks the KV state and preserves addressability, trading a smaller-per-token cache for keeping all tokens resident. It is MLA (the DeepSeek low-rank-latent-KV idea) carried into the multimodal, multi-head, long-video, agentic regime, and it pairs with yesterday's input-side Latent Memory (one latent token per evidence item) as the two token-policy extremes for affording a long-horizon multimodal agent's evidence. Open lever: whether the latent KV can be selectively retrieved into rather than kept fully resident, turning compression into retrieval-addressable video memory.

Current State (as of 2026-06-09)

Two opposite answers to the 500K-context memory wall land the same day: drop the KV chunks, or never form them. FlashMemory-DeepSeek-V4 (LSA, arxiv 2606.09079) keeps the cache but prunes it predictively: a Neural Memory Indexer, trained backbone-free as a dual-encoder retrieval model, predicts which KV chunks future queries will need and keeps only those resident. Average physical KV footprint drops to 13.5% of full-context, >90% at 500K, with +0.6% average accuracy across LongBench-v2, LongMemEval, RULER. This is the next step past RTPurbo (05-24, 16-dim retrieval subspace suffices) and MISA (05-11, head-axis indexer routing): LSA routes on a predicted-future-demand axis and trains the indexer without ever loading the backbone. Latent Context Language Models (LCLMs, arxiv 2606.09659) take the opposite route: skip the KV cache entirely with encoder-decoder compression. After an architecture search, they continually pre-train 0.6B-encoder / 4B-decoder models on 350B+ tokens each at 1:4 / 1:8 / 1:16 compression, pushing the Pareto frontier past KV-cache compression on accuracy-efficiency, and serving long-horizon agents that skim the compressed context then expand relevant segments on demand. This vindicates the trained-compressor argument LongAttnComp (06-02) made (trained scorer beats training-free heuristics) and generalizes it to a from-scratch encoder-decoder. The page now holds both poles of long-context efficiency: cache-side predictive eviction (LSA, VaSE, Conf-KV) and input-side trained compression (LCLM, LongAttnComp). Both are answers to the Ken Huang memory survey fact that KV cache, not weights, is the binding memory constraint as context grows.

Current State (as of 2026-06-06)

Redundancy gets killed at the input interface, not the cache (AdaCodec). AdaCodec (arxiv 2606.02569) is the input-side sibling of the page's redundancy theme. Where KV-cache eviction asks "which stored keys/values are redundant given everything in context," AdaCodec asks the same question one step upstream for video MLLMs: which frames are redundant given prior frames. Borrowing predictive coding from video codecs (send a full reference frame only when the scene cannot be predicted; otherwise send compact P-tokens of motion + residuals), it beats the Qwen3-VL-8B per-frame RGB baseline at matched token budget, and at 1/7 the budget (32k vs 224k tokens) surpasses it on all long-video benchmarks while cutting time-to-first-token from 9.26s to 1.62s. This complements the StateKV (06-01) / Echo-Infinity (06-04) video-memory line: those compress the history representation inside the model; AdaCodec compresses the visual tokens before they enter it. Same "don't pay for redundant temporal information" physics, different layer.

Current State (as of 2026-06-04)

Learned memory replaces handcrafted KV scheduling for infinite video (2026-06-04). Echo-Infinity (arxiv 2606.04527) is the video-generation instance of the page's biggest pattern: the move from handcrafted cache management (Make-Each-Token-Count learned retention, VaSE's value guard) to a fully learned one. Autoregressive video generators bottleneck on history; existing systems manage it with predefined KV-cache schedules, fixed-ratio compression, or inference-time RoPE adaptation, all of which lose information and amplify compounding error. Echo-Infinity instead trains a small set of Memory Queries end-to-end with the diffusion transformer; they update via attention + gating whenever frames are evicted from the local window, giving constant compute at any compression ratio independent of video length. A Unified Relative RoPE Recipe (sink frames anchored to id 0, newest frame capped at the pretrained max RoPE id) closes the train-test extrapolation gap, enabling a claimed-first 24-hour, >1.3M-frame real-time rollout. Closest to StateKV (06-01, fixed recurrent state as cross-frame memory) but trained rather than training-free, and the memory-as-prior result (optimized initial query state alone improves quality) is the surprise worth tracking.

Merging is also expert-read-bound (2026-06-04). MergePipe (Access Sets Matter, arxiv 2605.29489) is not a cache method but shares the page's "memory bandwidth is the wall" physics (dMoE, 06-01): at LLM scale the limiting resource in weight-space model merging is reading expert delta blocks, so it recasts merging as a budgeted access-set problem (skip low-norm deltas, bounded error), cutting read I/O an order of magnitude for up to 11x speedup. The "skip low-norm blocks" rule is the same sparse-and-locatable instinct as VaSE's large-magnitude value guard, applied offline to merging.

Current State (as of 2026-06-03)

Eviction finally matches selection on reasoning, via two new axes (2026-06-03): a value-magnitude guard and stochasticity. VaSE (Value-Aware Stochastic KV Cache Eviction, arxiv 2606.03928) resolves the long-standing gap the page has tracked — eviction methods kept losing accuracy to selection-based sparse attention that keeps the full cache. VaSE's two training-free findings: (1) a small fraction of value states carry abnormally large magnitudes and evicting them is catastrophic (the model falls into repetitive reasoning loops), so they must be protected; (2) making the eviction decision stochastic rather than deterministic keeps the surviving cache diverse and raises accuracy at the same budget. On Qwen3 at 4x compression across six reasoning tasks, VaSE beats the strongest eviction baseline by >4% and edges out the SOTA selection method, with FlashAttention2 support and a static memory footprint. The value-magnitude result is the same outlier physics the wiki has now seen from three directions: LongAct (04-18, high-magnitude Q/K mark where attention works) and the quantization-outlier line (OSCAR/TurboQuant, where high-magnitude states resist low-bit quantization). VaSE extends it to the value side and to eviction: the same outlier states that resist quantization also must not be evicted. The eviction control axes now read: learned per-token retention (Make-Each-Token-Count, 05-12), per-step confidence budget (Conf-KV, 05-30), per-head role pruning (Forcing-KV, 05-15), and now value-magnitude guard + stochasticity (VaSE). None has been composed with the others.

Current State (as of 2026-06-02)

New axis (2026-06-02): the per-head KV layout itself, via MLA for video diffusion. VideoMLA (arxiv 2605.30351) is the first study of Multi-Head Latent Attention (MLA) in video diffusion. Every prior video-KV paper the wiki tracks changed what the cache holds (which tokens, which heads, how much budget); VideoMLA changes the layout underneath all of them, replacing per-head keys and values with a single shared low-rank content latent plus a shared decoupled 3D-RoPE positional key, cutting per-token KV memory 92.7% at every layer and lifting throughput 1.23x on a B200. Its most important finding is a correction to MLA folklore: pretrained video attention is not low-rank (99%-energy effective rank far above any practical latent dim), yet MLA still works because the MLA bottleneck dimension — not the pretrained spectrum — determines effective rank. Both spectral and random init fill the full rank budget from step 0; training adapts within it. If this holds, MLA-style compression should transfer to modalities whose attention is not low-rank, decoupling MLA from the spectral-assumption story it was sold with.

Input-side companion (2026-06-02): trained context compression beats training-free. LongAttnComp (arxiv 2606.01336) compresses the 100k+-token prefill input rather than the generation cache, the input-side complement to VideoMLA's generation-side compression. The headline argument is against the training-free orthodoxy: pure attention-heuristic compressors leave large gaps on code reasoning, so LongAttnComp fine-tunes a lightweight cross-attention scorer (plus token-level chunking, token-budget top-p selection, positional reordering, format-agnostic query parser) and a two-stage recipe (NIAH retrieval foundation, then multi-hop/reasoning). It matches or exceeds full-context accuracy on InfiniteBench Code-Debug and transfers across four target models from three families. Same "keep only the load-bearing tokens" instinct as Make-Each-Token-Count (05-12) and the broader efficiency thread, but argues a trained scorer is required where training-free attention heuristics fail.

Current State (as of 2026-06-01)

New axis (2026-06-01): fixed recurrent state as a training-free cross-frame memory. StateKV (Linear Scaling Video VLMs, arxiv 2605.31598) adapts a pretrained long-video VLM to linear-time prefill at inference, with no fine-tuning and no architecture change. It carries cross-frame context in a fixed-capacity, importance-based recurrent state, paired with a second full per-frame cache used only for decoding. This is the linear-attention/state-space idea (compress history into a fixed-size state) applied as a wrapper on top of a model pretrained with quadratic attention. Across three long-video benchmarks and seven models in three families it stays close to full self-attention and beats sliding-window/recency streaming approximations, and cuts prefill FLOPs so a larger model fits a fixed compute budget. Significance: the video KV thread now has compression at the encoder (EarlyTom, 05-30), per-head role pruning (Forcing-KV, 05-15), evicted-KV-as-retrievable-world-memory (WorldKV, 05-24), confidence-driven per-step budget (Conf-KV, 05-30), and now a fixed recurrent state for cross-frame memory. StateKV is the only one that replaces the attention math itself (quadratic to linear) rather than managing what the cache holds.

Memory-bound MoE companion (2026-06-01): block-coherent expert routing. dMoE (arxiv 2605.30876) is not a cache method but shares the binding constraint: in diffusion LLMs with mixture-of-experts, block-parallel decoding plus independent per-token routing forces the machine to load the union of every expert any token in a block selected, making inference memory-bound. Aggregating token-level expert distributions into one block-level distribution drops uniquely activated experts from 69.5 to 14.6, cuts memory 76-80%, and retains 99.11% of performance. The same "memory bandwidth, not FLOPs, is the wall" framing that drives KV compression (SemiAnalysis 05-01 prompt-cache economics) now drives expert routing.

Current State (as of 2026-05-30)

New axis (2026-05-30): confidence-driven per-step cache budget. Conf-KV converts the next-token distribution into a scalar confidence score (one minus normalized entropy) and uses that score to set the cache budget step-by-step: low confidence keeps more context, high confidence prunes hard. Within each step's budget, tokens are ranked by accumulated attention mass plus recency, with a protected recent window for local coherence. Storage is mixed FP16/INT8 and the per-layer budget is pyramidal. On Needle-in-a-Haystack at 32K tokens, retrieval is 91.4% versus 53.8% for sliding-window and 80.6% for H2O. VisualWebArena retains 95.3% of full-KV success at 2.8x lower peak memory. Significance: the wiki now tracks three orthogonal control axes for eviction — Make-Each-Token-Count's learned per-token retention (05-12), Forcing-KV's per-head role pruning (05-15), and now Conf-KV's per-step confidence-driven budget (05-30). All three are forward-or-structure-looking signals beyond the recency / attention-history baseline.

Video-side companion (2026-05-30): in-encoder token compression. EarlyTom shifts video-LLM token compression upstream into the vision encoder itself, where the wiki had previously tracked compression only at the cache (WorldKV, Forcing-KV, Stream-T1) or at the LLM input (MotionCache). The vision encoder dominates time-to-first-token on Video-LLMs; EarlyTom is training-free, cuts TTFT by up to 2.65x and FLOPs by 61% on LLaVA-OneVision-7B at A100, with accuracy comparable to the full-token baseline. The video efficiency stack now has compression at every stage.

Current State (as of 2026-05-27)

New axis (2026-05-27): iterative offline consolidation. Language Models Need Sleep (CMU/UMD, Twitter-surfaced) separates scalable memory from scalable reasoning and adds the missing consolidation axis. Existing SSM-attention hybrids can store long-range info in fixed-size fast weights but degrade as reasoning depth rises even at constant information load: the bottleneck is computational, not capacity. "Sleep" runs N learned recurrent passes over accumulated context offline, writes the result into the SSM fast weights via a local rule, then clears the KV cache. Wake-time prediction reads the consolidated weights at normal latency. Increasing N improves accuracy, most on reasoning-heavy examples, on tasks where a plain transformer and an SSM-attention hybrid both fail. This is the inference-time twin of the agent-memory decoupling: MemForest (05-26) decoupled memory construction from the inference loop; Sleep decouples the expensive folding-in (offline) from fast answering (online). Prior consolidation in the cache thread (δ-mem's single-pass associative state, Make-Each-Token-Count's eviction) was single-pass; Sleep makes it iterative.

Agent-side companion (2026-05-27): state-adaptive recall. SAM: State-Adaptive Memory keeps raw trajectory pages plus compact memory cues that act as handles for intent-driven reconstruction, with recall conditioned on the agent's evolving state, backbone frozen. Together with Sleep and MemForest, three memory framings in two days (weight-level consolidation / data-structure / state-conditioned recall) all reject the flat-global-summary baseline.

Current State (as of 2026-05-24)

Latest additions (2026-05-24): KVServe, Gated DeltaNet-2, WorldKV, RTPurbo. Four KV-relevant entries today, all Tier 1.

KVServe (summary) is the first service-aware adaptive KV cache compression framework for disaggregated LLM serving. It unifies quantization, lossless coding, and data transformations into one modular strategy space; a Bayesian Profiling Engine distills a 3D Pareto candidate set at 50x less offline search; and an online controller combines an analytical latency model with a contextual bandit to pick a profile under runtime SLO and bandwidth constraints. Up to 9.13x JCT speedup in PD-separated serving and 32.8x TTFT reduction in KV-disaggregated serving. The reframe: compression configuration is a first-class control surface, not a static hyperparameter.

Gated DeltaNet-2 (summary) decouples the single scalar gate in the delta-rule family into a channel-wise erase gate b_t and a channel-wise write gate w_t. The single scalar previously controlled two different things (how much old content to erase on the key side, how much new content to commit on the value side). At 1.3B parameters trained on 100B FineWeb-Edu tokens, beats Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants on language modeling, commonsense reasoning, and retrieval, most pronounced on long-context RULER needle-in-a-haystack. The chunkwise WY algorithm absorbs the channel-wise decay into asymmetric erase factors so parallel-training cost matches the prior generation. Code: NVlabs/GatedDeltaNet-2.

WorldKV (summary) is a training-free framework for sustaining a persistent world in autoregressive video diffusion models. World Retrieval stores evicted KV-cache chunks in GPU/CPU memory and retrieves scene-relevant chunks via camera and action correspondence, inserting them back into the native attention window without re-encoding. World Compression prunes redundant tokens per chunk via key-key similarity to an anchor frame, halving per-chunk storage. Matches or exceeds full-KV memory fidelity at roughly 2x throughput on Matrix-Game-2.0 and LingBot-World-Fast. The wiki now tracks KV-as-memory in four video-side papers: MotionCache (05-05), Stream-T1 (05-07), LongLive-2.0 (05-19), WorldKV (05-24).

RTPurbo / Full Attention Strikes Back (summary) argues full-attention LLMs are already intrinsically sparse and convert to sparse in only a few hundred training steps. Three observations: only a small subset of attention heads truly requires full long-context processing; long-range retrieval is governed by a low-dimensional subspace (a 16-dimensional indexer is sufficient); the useful token budget is query-dependent, so dynamic top-p beats fixed top-k. Up to 9.36x prefill speedup at 1M context and roughly 2.01x decode speedup, near-lossless. The 16-dimensional retrieval-subspace finding is the load-bearing claim: the long-context retrieval geometry, learned implicitly during full-attention pretraining, lives in a tiny subspace.

Prior State (as of 2026-05-19)

Additions (2026-05-19): CompactAttention, EndPrompt, LongLive-2.0 NVFP4 KV cache. Three KV-relevant entries today. CompactAttention (summary) attacks the chunked-prefill regime where prior block-sparse machinery loses efficiency because the chunk size caps the Q-length. The structural move: treat 2D block-sparse masks as KV-selection signals rather than direct sparse-kernel execution plans, then convert them via Q-block union and intra-group GQA union into minimal block tables under paged execution. 2.72x attention speedup at 128K context on LLaMA-3.1-8B-Instruct with near-dense RULER accuracy. EndPrompt (summary) extends an LLM's context window using only short training sequences via a two-segment construction (original short context as segment 1, brief terminal prompt as segment 2 with positional indices placed near the target length). Beats LongLoRA and full-length fine-tuning on RULER and LongBench at substantially lower compute, with a RoPE-and-Bernstein-inequality smoothness argument for why sparse positional supervision suffices. LongLive-2.0 (summary) is the wiki's first end-to-end NVFP4 video training and inference stack. Quantizes the KV cache to NVFP4 on Blackwell for memory savings; on non-Blackwell, SP inference with quantized KV cache lowers SP inter-GPU communication. 2.15x training speedup, 1.84x inference speedup, 45.7 FPS at 5B.

Prior State (as of 2026-05-16)

Addition (2026-05-16): Lighthouse Attention pre-training wrapper. Nous Research ships a training-only, kernel-decoupled wrapper around ordinary FlashAttention for causal-transformer pre-training at extreme context length. Queries, keys, and values are pooled symmetrically into a multi-resolution pyramid; a gradient-free top-k cascade selects a hierarchical dense sub-sequence; a sorting pass keeps left-to-right causality. The wrapper is removed in a short recovery phase, leaving a standard dense-attention model. Claimed 1.4-1.7x wall-clock speedup at 98K context and ~17x forward+backward at 512K on a single B200. The structural novelty: pre-training attention selection is now a programmable substrate, the same framing the wiki has applied to the inference-time cache. → summary

Prior State (as of 2026-05-15)

Latest addition (2026-05-15): Forcing-KV for autoregressive video diffusion + async continuous batching. Two pieces of the inference stack land the same day. Forcing-KV finds that attention heads in AR video diffusion models (Self Forcing family) cluster into two stable functional roles across samples and denoising steps: static heads (chunk transitions, intra-frame fidelity) tolerate structured pruning; dynamic heads (inter-frame motion, temporal consistency) require segment-similarity-based pruning. Role-conditioned hybrid compression delivers 29+ fps on single H200 at 30% memory reduction, 1.35-1.50x speedup at 480P scaling to 2.82x at 1080P. The cache thread is now policy-aware in three forms: learned eviction (Make Each Token Count), shared coordination (Orthrus), and head-role compression (Forcing-KV). → summary. The HuggingFace asynchronous continuous batching post is the scheduling-layer complement: three CUDA streams (H2D, compute, D2H), CUDA events for handoff, two parallel buffer slots A/B so the CPU prepares batch N+1 while the GPU computes batch N. GPU utilization rises from 76.0% to 99.4%, 22% generation speedup, no kernel or model changes. → summary

Prior state (as of 2026-05-14)

2026-05-14: Orthrus dual-view diffusion on shared cache. Orthrus runs an autoregressive head and a diffusion head on the same frozen LLM, both attending to a single shared KV cache. The AR head executes pre-fill and populates the cache at full fidelity; the diffusion head reads from that same cache to draft tokens in parallel; an exact-consensus mechanism between the two views makes the output bit-identical to the AR baseline. Up to 7.8x speedup with O(1) memory cache overhead. The structural novelty: the cache is the shared coordination object, not a verification ledger. Composes naturally with Make-Each-Token-Count: same cache, both selectively retained and parallelly read. → summary

Companion (2026-05-14): MMProLong long-context VLM recipe. First long-context VLM training recipe in the wiki. Three findings: long-document VQA beats OCR transcription; balanced sequence-length distribution beats target-length-focused; retrieval is the long-context bottleneck. 5B-token long-context continued pretrain extends Qwen2.5-VL-7B from 32K to 128K with generalization to 256K and 512K. The training-side complement to Make-Each-Token-Count's inference-side claim. Both say: long context rewards balance and structure, not volume. → summary

2026-05-13 additions: δ-mem and FocuSFT. Two papers attack long-context inefficiency from orthogonal angles. δ-mem augments a frozen full-attention backbone with a compact 8x8 associative-memory state updated by the delta rule; its readout produces low-rank corrections to the backbone's attention computation. 1.10x average gain over the frozen backbone, 1.31x on MemoryAgentBench, 1.20x on LoCoMo. Composes with Make Each Token Count: aggressive eviction at the cache, associative signal retained in the small online state. FocuSFT identifies attention sinks as a training-side phenomenon (not just inference), shows that standard long-context SFT lets positional biases starve content tokens of attention budget, and fixes it with bilevel optimization (inner loop sharpens attention via fast-weights, outer loop runs SFT conditioned on sharpened representation; bidirectional context with causal response masking removes the sink-creating asymmetry). Up to +14 points on BABILong, 529x sink-mass reduction. Make Each Token Count and FocuSFT together bracket attention dilution: training-side cause + inference-side fix. → δ-mem summary · FocuSFT summary

Prior addition (2026-05-12): Make Each Token Count. A learned, globally calibrated KV-eviction policy that can surpass the full cache, not just approximate it. The framing flip: in long contexts, the full cache is not the ceiling because irrelevant tokens dilute attention. Lightweight retention gates score each cached entry, a shared final scoring projection calibrates scores across every layer and head, and a single global memory budget lets tokens from different layers, heads, and modalities compete for cache capacity. Theoretical analysis shows that preferentially retaining useful tokens reduces attention dilution. This is the language-model analogue of Stream-T1's content-aware video KV eviction (2026-05-07): both treat eviction as a quality intervention, not a compression tradeoff. Composes with MISA (head-axis routing) and TurboQuant (low-bit quantization). → summary

Prior additions (2026-05-11): Two papers attack long-context inference from different angles on the same day. MISA introduces a Mixture of Indexer Sparse Attention: it treats the 64 query heads inside DeepSeek Sparse Attention's indexer as an MoE pool and routes a small active subset (h=8) per query via cheap block-level statistics. Drop-in, no extra training, 92 percent of the tokens DSA would have selected, 3.82x kernel speedup over the original DSA indexer kernel on H200. The new axis here is the head axis: prior sparse-attention work routed on tokens, MISA routes on indexer heads. UniPrefill ships as a vLLM operator with extended continuous-batching scheduling: block-wise dynamic sparsification at the token level that is architecture-agnostic (works on full attention, linear-and-full hybrids, sliding-window hybrids). Up to 2.1x TTFT with the speedup growing as concurrent request count grows, which is the signature of a serving-system optimization. MDN: Momentum DeltaNet is the substrate-level update inside linear attention: it parallelizes stepwise momentum updates via geometric reordering, then uses spectral analysis of the resulting second-order recurrence to constrain gating for stability. Comparable training throughput to Mamba2 and KDA at 400M and 1.3B, beats Transformer / Mamba2 / GDN across downstream tasks. The recurrent rule is now a research surface, not a fixed substrate.

Prior state (as of 2026-05-07)

KV caching is standard in all production LLM serving. Active research is focused on four problems: (1) making caches reusable across contexts without recomputation, (2) compressing the cache to reduce memory footprint, (3) smarter eviction policies when the cache is full, and (4) extending cache-based acceleration patterns (like speculative decoding) to non-text modalities. The parallel daily digest (04-22) introduced two major KV-focused papers, TurboQuant (ultra-low-bit compression) and PrfaaS (cross-datacenter disaggregation via hybrid attention), signaling that the KV cache is now the primary optimization target in production serving. MotionCache (2026-05-05) extends the same iteration-as-optimization-unit principle to autoregressive video generation, using inter-frame motion deltas to decide which pixels need full denoising. Stream-T1 (2026-05-07) introduces the first content-aware KV eviction policy in the wiki: for streaming video diffusion, KV slots are routed through reward-feedback pathways instead of recency-based eviction. LIVEditor / ISA (2026-05-07) routes attention by Query sharpness, sending high-error queries to full attention and low-error queries to a 0-th order Taylor sparse path, achieving ~60% attention-module latency reduction on video editing. The pattern is now visible across six substrates: text KV reuse (KV Packet), KV quantization (TurboQuant), KV transport (PrfaaS), video denoising reuse (MotionCache), content-aware video KV eviction (Stream-T1), and Query-sharpness sparse attention (ISA). The shared principle is that the iteration unit has heterogeneous information density and should be allocated proportionally.

Economic context (SemiAnalysis 05-01): the unit economics of frontier model labs now depend on >90% prompt-cache hit rates. Anthropic's blended price for Opus 4.7 on agentic workloads is ~$0.99/MTok (vs $5/$25 sticker) because cached input tokens dominate. Cache compression / reuse research is now financial-impact-driven, not just academic.

Key Papers

Make Each Token Count (2026-05-12) — Learned, globally calibrated KV-cache eviction with retention gates per cached entry, a shared final scoring projection that calibrates scores across all layers/heads, and a single unified memory budget across layers, heads, and modalities. The theoretical claim is that the full cache is not optimal in long contexts because irrelevant tokens dilute attention away from useful evidence; selective eviction reduces dilution. Matches or surpasses full-cache inference across long-context language, vision-language reasoning, and multi-turn dialogue benchmarks. First paper in the wiki to formally claim eviction improves quality, not just preserves it. → summary

MISA (2026-05-11) — Mixture of Indexer Sparse Attention. Treats the 64 indexer query heads of DeepSeek Sparse Attention as an MoE pool, a cheap block-level router picks h=8 active heads per query. Reduces per-query indexer cost from O(H^I * L) to O(h * L + H^I * M). Recovers 92 percent of DSA's selected tokens at 8x fewer active indexer heads, 3.82x kernel speedup on H200. Drop-in, no training. The first paper in the wiki to route sparse-attention on the head axis rather than the token axis. → summary

UniPrefill (2026-05-11) — Architecture-agnostic prefill accelerator via block-wise dynamic sparsification, implemented as a continuous-batching operator inside vLLM with native prefill-decode co-processing and tensor parallel. Up to 2.1x TTFT speedup, speedup grows with concurrent request count (a serving-system signature, not a single-request one). Works on hybrid architectures where prior sparse-attention prefill methods degrade. → summary

MDN: Momentum DeltaNet (2026-05-11) — Parallelizes stepwise momentum for delta linear attention via geometric reordering of update coefficients. Spectral analysis of the second-order recurrence constrains gating for stability. Triton kernel matches Mamba2 / KDA training throughput. At 400M and 1.3B, beats Transformer / Mamba2 / GDN on broad downstream evals. First substrate-level update to linear attention recurrent rule the wiki has tracked. → summary

KV Packet (2026-04-17) — Eliminates recomputation-on-reuse entirely. Wraps cached documents as immutable packets with lightweight soft-token adapters (trained via self-supervised distillation) that bridge context shifts. Near-zero FLOPs, lower TTFT than all recomputation-based baselines (CacheBlend, EPIC, SAM-KV). → summary

LongAct (2026-04-18) — Identifies high-magnitude activations in Q/K vectors during long-context processing. These "saliency peaks" (same ones that trouble quantization) are the positions where attention is doing real work. LongAct restricts RL gradient updates to only those weights, yielding ~8% gain on LongBench v2 with universal compatibility across GRPO and DAPO. Bridges the KV saliency insight from quantization research into RL training. → summary

TurboQuant (2026-04-22, via parallel digest) — Google (ICLR 2026). Online vector quantization: randomly rotates input vectors to induce a concentrated Beta distribution, applies optimal scalar quantizers per coordinate, followed by a 1-bit QJL transform on the residual for an unbiased inner product quantizer. Absolute quality neutrality at 3.5 bits/channel; marginal degradation at 2.5 bits/channel; 6x+ KV cache memory reduction. Community integrations with vLLM and llama.cpp appearing despite no official implementation.

PrfaaS / Prefill-as-a-Service (2026-04-22, via parallel digest) — Moonshot AI + Tsinghua. Offloads long-context prefill to standalone compute-dense clusters in separate datacenters, transfers resulting KV cache over Ethernet. Enabled by hybrid-attention models (Kimi Linear, MiMo-V2-Flash, Qwen3.5-397B) that mix full-attention + linear-complexity layers. MiMo-V2-Flash produces KV cache at 4.66 Gbps vs 59.93 Gbps for dense-attention baseline (13x reduction). 54% higher throughput, 50% lower mean TTFT vs homogeneous baselines.

SDVG (2026-04-22) — Extends speculative decoding to continuous video generation. A 1.3B drafter proposes video blocks in 4 denoising steps; ImageReward scores per block using worst-frame aggregation; accepted blocks enter the 14B target's KV cache directly. 1.59x speedup at 98.1% quality; 2.09x at 95.7%. Training-free. → summary

MotionCache (2026-05-05) — Motion-aware caching for autoregressive video generation. Inter-frame differences identify which pixels require full denoising vs which can skip steps. Two-phase schedule: warm-up for semantic consistency, then motion-weighted cache reuse with dynamic update frequencies. 6.28x speedup on SkyReels-V2 (1% VBench drop), 1.64x on MAGI-1 (0.01% drop). Training-free, code public. The video-AR analogue of selective KV-cache reuse: iteration count is the optimization unit. → summary

Stream-T1 (2026-05-07) — Test-time scaling framework for streaming video generation. Three components: Stream-Scaled Noise Propagation (reuse high-quality previous-chunk noise as the prior for the next chunk), Stream-Scaled Reward Pruning (combine short-term spatial assessment with sliding-window long-term coherence), and Stream-Scaled Memory Sinking (route KV-cache evictions through reward-feedback-guided update pathways). The first content-aware KV eviction policy tracked by the wiki: not "which token is oldest" but "which token still anchors downstream quality." → summary

LIVEditor / ISA (2026-05-07) — In-context Sparse Attention for ICL video editing. Two stages: context pre-selection (prune low-saliency context tokens) plus dynamic Query routing (route high-error queries to full attention, low-error to a 0-th order Taylor sparse path). Empirically validates the claim that Query sharpness correlates with attention approximation error. ~60% attention-module latency reduction, near-lossless on EditVerseBench / IVE-Bench / VIE-Bench. The first sharpness-routed sparse attention in the wiki, structurally adjacent to language-side speculative decoding (route by difficulty). → summary

See What I See (2026-06-13) — A new use of the KV cache: not memory to compress or reuse, but a communication channel between different models. When two agents coordinate through text, the sender decodes its hidden state to words and the receiver re-encodes them, a lossy and expensive round trip. This paper (Michigan/NVIDIA/Penn) transfers a sender's KV cache directly into a heterogeneous receiver (e.g. Qwen3-8B → Qwen3-14B) via a lightweight cross-model cache transform trained in two phases (reconstruct, then generate). An information-structure duality drives the design: context-aware transfer (receiver also sees input) needs only sparse reasoning signal, context-unaware transfer (receiver sees nothing) needs dense contextual preservation. Matches or beats text communication at 2-3x lower compute in the context-aware regime, and is the first heterogeneous method that survives the context-unaware regime where priors collapse. Limitation: all directions stay inside the Qwen3 family, so true cross-architecture transfer is unproven. The KV cache is now a memory store, a thing to quantize/evict, and an inter-model wire. → summary

Key Concepts

• Context dependency: KV states computed for a document are specific to the attention context at the time. Reusing them in a new context produces attention distribution mismatch — hence the need to recompute.
• TTFT (Time-to-First-Token): the latency before the model outputs the first token. KV cache reuse directly impacts this.
• Soft-token adapters: trainable lightweight token representations that can modify how a cached KV state interacts with a new context, without recomputing the underlying states.
• Cache eviction: when the KV cache fills up, old entries must be evicted. Policy choices (LRU, saliency-based, etc.) affect quality and memory efficiency.

Hardware context (2026-06-07)

The Ken Huang memory survey (Memory Technology for Agentic AI Workloads) names the hardware fact under this whole page: as context grows, the dominant memory traffic shifts from weights to KV cache, so KV-cache management is the binding hardware constraint, not just a software optimization. This is why every eviction/quantization/offload technique above matters economically in a structurally memory-short market (HBM allocation-driven into 2030). Concrete hardware moves: Micron SOCAMM2 (LPDDR) claims >2.3x time-to-first-token when used for KV-cache offload at 1/3 the power of RDIMM; NVIDIA CMX/BlueField-4 turns SSD into an AI-native ephemeral-KV context tier with KV-aware placement. The open systems direction is KV-aware tiering: deciding per-request which KV blocks live in HBM vs LPDDR/CXL/SSD — the hardware dual of CLEAR's per-query compute rationing. See memory-hierarchy.

The cache becomes a billing surface (2026-08-14)

Everything above treats the KV cache as a technical resource: something to compress, quantize, evict, offload, tier, or ship between models. On 2026-08-13 a provider repriced it, and that adds a category this page did not have.

DeepSeek raised API prices with cache-hit tokens repriced to roughly 6x their prior cost, alongside a new peak/off-peak split where off-peak rates run 50% below peak (effective 16:00 UTC, 2026-08-16). A cache hit is what you pay when a request's prompt prefix is already resident, so the workload that pays most is the agent loop, which re-sends a growing prefix every turn. The Decoder called it the biggest increase in the transition and named the affected pattern precisely: agent workflows that repeatedly read the same files. → summary

The same release ships the mitigation. DeepSeek open-sourced Harness v0.1 under MIT the same day, and per Hugging Face's Elie Bakouch reading the code, its organizing commitment is first-class KV-cache-aware design: previously written history is never altered. When something in the conversation must change, the harness appends a statement describing the modification rather than editing the prefix, because an in-place edit invalidates every cached token downstream and forces a full recompute. Bakouch expects other harnesses to adopt the same convention.

Why this is a new entry rather than a footnote. Every technique above optimizes cache behaviour under a fixed price. This is the first time the wiki has seen the price move against a specific access pattern, which flips the direction of the optimization: instead of "how do I use less cache," the question becomes "how do I avoid invalidating the cache I already have." Prefix stability was previously a latency property owned by a systems engineer. It is now a line item owned by whoever signs the invoice, and append-only history is the cheapest available mechanism for protecting it.

Two consequences worth tracking. The append-only discipline has an unpriced accuracy cost: corrections accumulate at the tail, context grows monotonically, and contradictory statements coexist for the model to reconcile. That is a plausible quality tax paid to preserve a cache hit, and the counterfactual methodology from the Illusion of Visual Tool-Use (08-13), which corrupts a tool return and checks whether the answer moves, would test it directly. And peak/off-peak pricing introduces a scheduling dimension that no routing formulation on the LLM routing page currently models: not which model, but which hour.

This also connects the cache directly to compute economics, where Blackwell-generation capacity cleared 15% above record in Nebius's first auction the same week. Cache-hit repricing at the API layer and spot repricing at the silicon layer are the same scarcity showing up two levels apart.

2026-08-25: precision becomes a spatial decision, and INT8 KV caches stop costing quality

TileMix (08-25) is the first entry on this page where precision is routed rather than set. Every quantization result above picks a format and applies it uniformly to a tensor or a layer. TileMix partitions the query-key score matrix into hardware-aligned tiles and dispatches each tile group through either an FP16 or an INT8 score path, with both paths updating one shared online-softmax state so the output stays a single dense attention result. It routes all legal tile groups, so unlike sparse attention it preserves dense token connectivity. Training-free, and it supports grouped-query attention, variable-length batches, and INT8 key/value caches.

The direct consequence for this page: the INT8-KV-cache quality tax is not fixed. This page has recorded INT8 KV quantization as a memory play whose cost is long-context quality. TileMix recovers exactly that lost quality on LongEval and LV-Eval while improving prefill throughput over FP16 on A100, and it explicitly composes with INT8 KV caches rather than competing with them. So the two stack: you keep the memory saving and buy back the accuracy in the kernel. That is a different relationship than this page has recorded between any two compression techniques, which have generally traded off.

One routing bit governs several adjacent key tiles, and that detail is what makes it a long-context method rather than a demo. Per-tile bits would make the bitmask itself grow with sequence length; scalable grouping keeps the metadata compact where it matters.

Where the honest uncertainty is. The result is prefill-only, on A100 only, and the routing policy is a heuristic whose own overhead is asserted compact rather than deeply ablated. Decode-phase behaviour and end-to-end latency under realistic continuous batching are unmeasured, and those are the numbers that decide whether this reaches production. The Hopper and Blackwell story is also genuinely open, because FP8 gives those parts a third precision point and the tile-geometry argument is hardware-specific by construction. A two-bit mask over FP16/FP8/INT8 is the obvious extension and nobody has published it.

And it reframes the relationship between this page and routing. LLM Routing catalogues six axes, all of which route work to a component. TileMix routes a numerical format to a region of a matrix, which is a routing decision below anything that page recorded. On the same day, Pandora's Router priced value estimation across models and VoI-MoLE priced expert acquisition inside one model. Three levels of the stack, one decision structure, one day. The unasked question is whether a serving stack should be making these three decisions jointly under a single cost budget instead of independently, which is how every deployed system does it today.

Related Pages

• Compute economics (GPU pricing, utilization, durability)
• Agent harness engineering
• Memory Hierarchy for AI
• Knowledge Distillation
• LLM Routing

2026-09-06: the cache-hit ratio turns out not to be the objective

The 08-14 entry above recorded the cache becoming a billing surface, and its practical instruction was protect the prefix: DeepSeek repriced cache-hit tokens roughly 6x and shipped a harness whose organizing commitment is that written history is never altered, because an in-place edit invalidates every downstream cached token.

ContextPipe (09-06) (arXiv 2609.00749) is the first result in this wiki that breaks prefixes on purpose and comes out ahead. It treats context assembly as relational query execution, running a deterministic cache-aware optimizer that chooses the prompt's byte layout (where stable content sits, where cache breakpoints go) against a configurable per-provider cache policy, with eight compaction lifecycle tiers and a ForkPrefix mechanism for sharing a parent agent's prefix with spawned children. On the SWE-bench Pro Qutebrowser subset it cuts total token volume 31%, LLM calls 23% and response time 9% against an append-only baseline, while achieving a lower KV cache-hit ratio, which the authors report rather than hide.

The reason this is not a contradiction is arithmetic, and it is worth stating on this page explicitly. Cache-hit ratio is hits over total tokens sent. The bill is a product of tokens and per-token price. Shrinking the denominator by 31% lowers the ratio and lowers the bill at the same time, because a token that is never assembled costs nothing at any cache tier: not at the per-request KV cache, not at server-side prefix caching, not at provider-billed prompt caching. Those are three of the four layers separated on the four cache layers page (08-29), and none of them can charge for absent tokens.

So this page now carries two opposed disciplines with no measured boundary between them. Append-only wins where the accumulated context was going to be sent regardless and the only live question is whether the prefix survives, which is the agent loop re-reading the same files every turn. Plan-and-compact wins where a real share of the history did not need sending. The crossover depends on context length, on how redundant the history is, and on the cache-price ratio, and DeepSeek's own 6x repricing moved that ratio in a direction that makes breaking a cache hit cheaper than it was, since a hit is now worth six times less than the discount it used to represent. ContextPipe does not cite the repricing and the interaction is unmeasured.

Open problem: publish the crossover sweep. Two axes, context length and cache-price ratio, one benchmark, both policies. It is a cheap experiment that would settle production practice for every agent harness on the harness engineering page, and it is the concrete missing number rather than a research direction.

2026-09-10: context dependency becomes a training-time variable, and the semantic layer finally ships

Three arrivals on one day move this page in three different directions, and two of them attack assumptions the page has treated as fixed.

Context dependency is no longer only a serving problem. The Key Concepts section above names context dependency as the reason cached KV states cannot simply be reused: the states were computed under one attention context, so pasting them into another produces distribution mismatch, "hence the need to recompute." KV-concatenation-aware fine-tuning (09-10) treats that mismatch as a property of the weights rather than of the architecture. Fine-tune the model knowing its caches will be concatenated, then selectively recompute a subset rather than all or none. On RULER at 124k tokens: +9.7 RULER points over a recompute-only baseline with TTFT down 80% against full attention. This is the first entry on this page where the model is trained for the cache policy instead of the cache policy being designed around the model, and it is the natural mitigation for the failure mode NVIDIA's cross-model KV transfer (09-09) left open, where two of six model pairs degraded sharply with no predictor. Partial recompute is an insurance premium on cache reuse, and nobody has yet applied it cross-model.

The semantic layer arrived twice in one day. The four cache layers page (08-29) separated per-request KV cache, server-side prefix caching, provider-billed prompt caching and application-level semantic caching, and flagged the fourth as the least studied. On 09-10 the three-layer caching architecture paper reported a deployed semantic query cache on a single 8-vCPU box (89.3% aggregate keyspace hit rate, 0.1 ms reads, 1.38 MB overhead), and Redis pushed LangCache, a managed semantic-response cache, into the practitioner feed with a claimed 90% API cost reduction. The distinction that matters: a prefix-cache hit still runs the request, since new tokens are processed and the answer is decoded, while a semantic hit removes input tokens, output tokens and decode time together. That is a categorically bigger saving and a categorically bigger correctness risk, and neither artifact publishes a false-hit rate. The missing number for this page is now the semantic cache's error rate as a function of similarity threshold.

And the day's flagship model shipped the depth axis this page called unexplored. DeepSeek V4.1 Flash (09-10) has its own entry; the item for this page is that CSA2's Reuse mode declines to recompute both the KV and the top-K token index at every layer, which is cross-layer sharing of the routing decision on top of cross-layer sharing of the cache. Combined with WRP forward-free depth pruning (09-10), which deletes redundant blocks entirely by comparing projection weights across layers, and KVShare (09-07), which showed adjacent layers' KV projections are highly similar, three independent 09-07-to-09-10 results now say depth contains repeated computation that can be shared, reused, or removed. Three instances is the threshold: this is a pattern, and the open question is whether the three are additive or whether they are all cashing in the same redundancy budget.

Counter-pressure worth recording. Trace as State (09-10) improves long-context reasoning by prepending a prior reasoning trace to the context and re-reading, winning 26 of 27 model-task-metric combinations and taking DeepSeek V4 Pro Preview from 29.2% to 81.8% on GraphWalks Parents. Under the 08-14 repricing this is the maximally cache-hostile operation: prepending invalidates the entire prefix and adds a full extra prefill. It is the third entry on this page (after ContextPipe and now this) where breaking the cache wins, and it is the first where breaking the cache wins while increasing total tokens. The append-only discipline is now clearly a heuristic with a measured exception, not a rule.

2026-09-11: precision becomes the second thing you route inside the cache

HyQuant (09-11) (arXiv 2608.27875, Shanghai Jiao Tong + Xi'an Jiaotong + Xiamen + Tencent Penglai Lab) is the second result in eighteen days to route precision inside attention, and it routes along a different axis from the first. TileMix (08-25) partitioned the query-key score matrix into hardware-aligned tiles and dispatched each tile group through FP16 or INT8 with both paths updating one shared online-softmax state. HyQuant partitions by token identity: a lightweight probe finds vertical-line tokens, the small set of key columns that nearly every query attends to persistently across a long context, keeps those plus a local sliding window in high precision, and quantizes everything else hard. One geometric, one semantic. They are close to orthogonal and nobody has composed them.

The practical detail that decides whether this ships is the fused dequantization. In decode, HyQuant applies the same criticality rule as a KV cache compression policy and fuses KV dequantization into the attention computation. Without fusion you decompress a cache block into registers or shared memory and then attend to it, which reintroduces exactly the memory traffic the compression removed. This page has recorded several compressed-cache methods whose reported memory saving did not translate into a latency saving for precisely this reason.

It is also the gentler cousin of eviction, and the distinction matters for the failure mode. Prior work uses the same persistent-attention structure (attention sinks, heavy hitters) to decide what to discard. HyQuant uses it to decide what to keep at what precision. An evicted token is unrecoverable; a low-bit token is merely noisy. Eviction fails catastrophically when the importance estimate is wrong about a token that later turns out to matter, and HyQuant degrades gracefully in the same situation. That is a real argument for preferring precision allocation to eviction under equal memory budgets, and no paper has run the comparison.

What the abstract does not report is the thing the argument depends on. Near-lossless accuracy is claimed across tasks, models and datasets, with no throughput or memory numbers and "limited overhead" for the pattern probe asserted rather than ablated. The probe runs on the attention map, which is the one tensor you were trying not to materialize, and the vertical-line signal is a long-context artifact whose existence in short-context or heavily-batched serving is unaddressed.

The same day supplied the theory this page has been missing for every quantization entry on it. Why Does Post-Training Quantization Work? (09-11) (Kurate cs.LG #10, absent from HuggingFace) shows that a pretrained model survives aggressive rounding because a layer's newly injected error tends to oppose the error it inherited, so the two partly cancel, and because the LM head preferentially preserves high-ranked token scores. The error budget is therefore not uniform, which is a principled selection criterion for exactly the kind of spatial precision routing TileMix and HyQuant do by heuristic. There is now a new quantization concept page collecting this thread; the KV-specific results stay here.

The open question this page raised on 09-10 recurs one level down. That entry asked whether three independent depth-sharing results (DeepSeek V4.1 Flash's CSA2 Reuse mode, WRP forward-free depth pruning, KVShare's finding that adjacent layers' KV projections are highly similar) are additive or all cashing the same redundancy budget. Token-level and tile-level precision routing pose the identical question about the width axis, and it is cheaper to answer: run both selections in one kernel and check whether the recovered accuracy is the sum or roughly the max.

2026-09-13: the prefix becomes a build-system property, and the cache acquires a fifth surface

Ken Huang's prefix-stable KV caching post (09-13) is the first item on this page where the cache's behaviour is decided by a git repository rather than by a serving policy. The mechanism is documented rather than novel: Anthropic's prompt cache keys on the exact rendered byte prefix up to each breakpoint, cache state is allocated on token-block boundaries, and one changed byte at token N invalidates the block containing N and every block after it, because attention at any position depends on all prior positions. There is no partial credit inside a block and no recovery after one.

What is new is the failure taxonomy, and all three failures are invisible to a human reader. A Last updated: stamp or git branch name injected at the head of a memory file, which is the worst possible placement because it invalidates everything below. A script concatenating .kb/*.md without LC_ALL=C sort, so the byte order differs across machines. Line-ending drift or trailing whitespace, which changes tokenizer IDs while changing nothing visible. The prescription is to treat the context file as a compiled artifact with a deterministic build and a prefix-integrity audit, ordered by volatility: static bootstrap first, architecture in low-churn files, session logs appended at the bottom.

This is a fifth surface for this page. The cache has been tracked here through compression (bit width), eviction (which tokens survive), placement (which memory tier holds them, the 09-09 thesis that kimi-k3-in-c pushed to its extreme on 09-12), and billing (08-14). Add provenance stability: nothing is compressed, evicted or moved, and the entire saving comes from holding bytes identical across turns.

It also partly contradicts this page's 09-06 entry, and the reconciliation is a general rule. That entry concluded "the cache-hit ratio turns out not to be the objective," from ContextPipe (09-06), which cut total token volume 31% while deliberately lowering cache-hit ratio, on the arithmetic that a token never assembled costs nothing at any tier. Huang's argument points the other way because his context is a different object: ContextPipe assembles per query, where assembling less wins; an agent instruction file is assembled on every turn of every session and is identical across all of them, which is the one region where hit ratio and total cost agree. The rule that reconciles them: compress what is assembled per query, freeze what is assembled every query. Neither source splits context by reuse frequency, and that split is the missing variable in both.

The cheap unrun experiment. SoL-Pi (09-11) found four harness optimizations worth 45-49% of tokens, two of which (Online Context Compact, ObservationPack) act on the volatile tail. None acts on the static head and byte-stability was not in the search space. Hold the harness fixed, make the prefix byte-stable, measure the change in cached-read fraction and time-to-first-token. It is orthogonal to all four wins.

There are no numbers anywhere in the source. No cached-read fraction before and after, no TTFT delta, no dollar figure. The block size is described as "commonly discussed" as 1024 tokens for many serving stacks rather than confirmed for Anthropic's. Treat the mechanism as solid and the magnitude as unmeasured.

Adjacent, from the same author one day earlier. Chapter 5 on hardware-aware attention kernels (09-12) is what a prefix costs when it does get recomputed: naive attention on an H100 over 16k tokens runs under 15% Model FLOPs Utilization because the N x N score matrix goes to HBM at 3.35 TB/s instead of staying in SRAM at over 33 TB/s. Prefix stability and kernel quality are the two ends of the same bill, one avoiding the recomputation and one making it cheap, and no published stack reports both.

2026-09-14: the depth-sharing pattern gets a retrofit path, and the fourth instance arrives

The 09-10 entry above named a three-instance pattern: DeepSeek V4.1 Flash's CSA2 Reuse mode declining to recompute both the KV and the top-K token index at every layer, WRP forward-free depth pruning (09-10) deleting whole blocks by comparing attention-output and MLP-down-projection weights across layers with no calibration data, and KVShare (09-07) showing adjacent layers' KV projections are highly similar. Transformer depth contains repeated computation that can be shared, reused or removed. All three required either a new architecture or a destructive edit to the checkpoint.

An external KV-approximation retrofit (09-14) is the fourth instance and the first that leaves the model alone. DeepSeek V4.1 Flash (09-10) buys its prefill saving structurally: 20 causal-encoder layers then 20 decoder layers, with the decoder's input-phase KV projected from the encoder's final state rather than computed layer by layer, about 8B active parameters while reading against 16B while writing, and a compressed global KV around 890 bytes per token. A third-party experiment reported through the X home feed ports that shape onto an unmodified Qwen3-8B: freeze every weight, train one small external module that predicts the back-half layers' KV states from the front half's hidden states, and skip the back-half prefill computation. Reported: prefill time down by close to half with output preserved.

Why this changes the character of the pattern rather than just extending it. The first three results say a newly designed or newly pruned model contains depth redundancy. This one says the redundancy is sitting unclaimed in checkpoints that already exist and are already deployed, and that it can be harvested by adding a small component rather than by editing the model. If it replicates, the addressable set is Llama, Qwen, DeepSeek distillations and every already-fine-tuned vertical model, none of which will be retrained.

It also sharpens the question the 09-10 entry left open. That question was whether the depth-sharing results are additive or all cashing the same redundancy budget. The retrofit makes it a cheap single-checkpoint experiment: score Qwen3-8B's blocks with WRP's weight-redundancy metric, then train the approximation module, and check whether the prefill saving from approximation shrinks on the blocks WRP already calls redundant. Disjoint savings mean the budget is layered; heavy overlap means there is one redundancy with three ways to spend it.

The evidence quality is the weakest on this page and should be stated plainly. No paper, no repository, no independent reproduction, reaching the wiki through a practitioner's post about someone else's experiment. Test scale, context lengths, task coverage and the criterion for "output preserved" are all unpublished, and the failure mode of a bad KV approximation is a quiet quality regression rather than a crash. The curation trail's 09-06 note recorded that the compression step between a paper and a social post is where accuracy damage happens; here there is no paper to compare against at all. The mechanism is consistent with V4.1 Flash's published design and should be treated as sound; the magnitude is one unverified number.

Hardware read-through, from the same day. SemiAnalysis's 4-hi HBM analysis (09-14) ran a production InferenceX benchmark on Kimi K3 across 16 GB300 GPUs with HBM utilization restricted from 92% to 85%, cutting the GPU KV budget 36% for aggregate serving and 44% for disaggregated pairs. Throughput tracked the full-memory run until concurrency passed roughly 70, at which point KV occupancy hit 100% and throughput fell about 30% to preemption, reloading and queueing, with over 10x the reads to DRAM-resident KV at matched concurrency. That is the failure curve this page has been missing for the entire offload thread: KV offload to second-tier DRAM absorbs a large capacity cut almost invisibly, and then fails as a cliff rather than a slope. Any eviction, compression or placement policy on this page is operating on the flat part of that curve, and none of them report where their own cliff is.

2026-09-18: the borrowed number gets published, and the cache acquires a use that is not a cost

DeepSeek-V4.1-Flash's paper (09-18) (arXiv 2609.19969) confirms the figure this page has been carrying second-hand for a week. The 09-14 entry quoted "a compressed global KV around 890 bytes per token" for V4.1 Flash while building the case for an external KV-approximation retrofit on a frozen Qwen3-8B, and flagged that whole thread as the weakest evidence on the page: no paper, no repository, no reproduction. The paper states 890 bytes per token, roughly a quarter of DeepSeek-V4-Flash, and adds a persistent-cache figure the page did not have: about one eighth of the prior generation, via SWA Bounded Replay. The retrofit's mechanism was reasoned against a number that turned out to be right.

SWA Bounded Replay is a new kind of move on this page and deserves its own name. Sliding-window layers only ever attend to a bounded local window, so their KV state is reconstructible from raw tokens by replaying a bounded amount of computation. V4.1-Flash therefore declines to store it persistently at all and regenerates it on demand. Every other persistent-tier technique catalogued here decides where to put cache state; this one decides not to keep it, and to pay compute instead. That is a compute-for-storage trade priced specifically at the tier where storage and interconnect bandwidth bind rather than FLOPs, and it is the first instance on this page.

The three-bills framing is the transferable contribution, more so than any single number. The paper separates prefill compute, runtime HBM-resident KV, and persistent SSD-or-host KV, and compresses the global cache along three orthogonal dimensions at once: values per entry (FP4), entries along the sequence (CSA2 token selection), and layers holding independent caches (CSA2 cross-layer reuse). This page's 09-10 claim that the axes are multiplicative rather than competing is now stated by the lab itself as the design principle. Nearly every technique catalogued here improves one of the three bills and is silent on the other two, which makes most of the numbers on this page non-comparable. Reporting all three should become the default form.

What the paper still does not give, and it is the number that decides everything. The 09-14 SemiAnalysis 4-hi HBM study established that KV offload to second-tier DRAM absorbs a large capacity cut almost invisibly and then fails as a cliff past roughly 70 concurrent sessions, not as a slope. A four-times-smaller cache moves that cliff to higher concurrency; it does not remove it, and the paper reports no concurrency sweep. Until someone publishes where V4.1-Flash saturates, the quarter-footprint claim is not yet a quarter-cost claim. The FP4 choice is also presented as a design decision rather than an ablation, so the split of the 4x between precision, token sparsity and cross-layer reuse is not separable from the paper.

The same-day hardware read-through is the sharpest cost signal on this page in months. SemiAnalysis's Engram offloading study (09-18) reports in passing that NVIDIA despec'd Rubin Ultra from 1024 GB to roughly 200 GB of HBM per chip. Everything on this page is an argument about how to spend a memory budget. That budget just got cut five-fold on the flagship roadmap, which converts cache compression from an optimization into an admission criterion.

Cache-to-Cache: the first entry where the cache is an asset rather than an expense

C2C (09-18), reported as ICLR 2026 from Tsinghua and Infinigence, uses a learned projector to fuse one model's KV cache directly into another's, with a per-layer gate selecting which layers benefit. Claimed: up to 14.2 percent accuracy over the individual models, over 5 percent over text-based agent communication, and a 2.5x speedup from never decoding the intermediate message. Every other entry on this page treats the cache as a cost to be compressed, evicted, quantized, placed on a cheaper tier or shared across depth. The 08-29 four-layers entry separated per-request KV, server-side prefix caching, provider-billed prompt caching and semantic caching, and all four are framings of the cache as an expense. C2C treats it as a communication medium whose contents are worth transmitting. This page has no category for that and now needs one.

It also extends the depth-sharing thread across a boundary nobody has crossed. The four instances recorded on 09-10 and 09-14 (CSA2 Reuse, WRP forward-free depth pruning, KVShare's adjacent-layer projection similarity, and the external retrofit) all share cache state across depth inside one model. C2C shares it across two different models, and its per-layer gate is in effect a measurement of how much structure two models' caches have in common. If adjacent layers within a model are similar enough to substitute for one another, how much do two independently trained models share? That is now a well-posed and cheap question, and neither literature has asked it.

Evidence quality, stated plainly: this is reconstructed from two social posts, one of them the reader's own saved bookmark, with no arXiv ID captured and no benchmark named against the 14.2 percent. The decisive unknown is whether the projector must be trained per source-target pair, which would make it a result rather than a primitive. And there is a property being removed that is not in the cost accounting: text handoff is currently the only auditable point in a multi-agent system, and C2C replaces it with a tensor.

2026-09-20: the demand side of the memory squeeze, and compaction as a cache-adjacent surface

The 09-18 entry recorded SemiAnalysis reporting NVIDIA despec'ing Rubin Ultra from 1024 GB to roughly 200 GB of HBM per chip, and concluded the cut converts cache compression from an optimization into an admission criterion. That was the supply half. The demand half arrived today and it is worse than the page assumed. Ken Huang's Chapter 8 on test-time compute (09-20) reports that frontier reasoning systems now spend 90% to 95% of cluster GPU cycles in autoregressive decode, against 75-80% in prefill for the pre-reasoning generation, with single requests emitting 16,000 to 128,000 internal thinking tokens before answering. A single 64k-token reasoning request is reported to consume over 18 GB of GPU VRAM under standard grouped-query attention.

Put the two numbers together and the picture is stark: at roughly 200 GB per chip, that is about eleven concurrent reasoning sessions before the memory is gone, before counting weights at all. Every technique catalogued on this page is an argument about how to spend a memory budget. The budget was cut five-fold on the flagship roadmap in the same week that the per-request demand was measured at 18 GB. The 09-18 open question about where the concurrency cliff sits after a four-times-smaller cache is now the single most decision-relevant unpublished number on this page.

The arithmetic intensity figure is the one to internalise. Decode operates at 1.0 to 2.5 FLOPs per byte. That is the mechanism behind the MFU collapse the chapter names, and it is also the mechanism behind Sample Count Is Not Enough (09-18), which measured eight serial single-candidate calls costing 4.64 to 4.86 times the GPU energy of one batched call of eight. An unbatched decode step pays full weight-read cost to produce one token. The energy paper measured a consequence; this states the cause, and it means any cost model on this page denominated in FLOPs is measuring a resource that is not scarce.

Three production systems, three different compressions, all aimed at the same bottleneck. DeepSeek-V4-Pro pairs Compressed Sparse Attention and Heavily Compressed Attention for a reported ~90% reduction in reasoning KV caches, and drops the RL critic network via GRPO, which is a training-memory win before it is an algorithmic one. Zhipu's GLM-5.3 exposes reasoning_effort as an API tier and reuses sparse attention indexers across Monte Carlo tree search and process-reward verification (IndexShare), so structure computed during search is not recomputed during scoring. Moonshot's Kimi K3 pairs hybrid linear attention with entropy-driven dynamic thinking budgets. IndexShare is the new shape here: every other reuse technique on this page reuses cache contents across depth, sessions or models. This reuses the index, the structure that decides which entries to attend to, across two different consumers of the same sequence. This page has no category for reusing the selection metadata rather than the cached values, and it now needs one.

Compaction is a cache-adjacent surface the page has been ignoring

The 09-18 entry treated agent context compaction as a routing story and flagged the viral version as oversold. Two days later it is the largest project in the decision-primitive ecosystem by stars (3,169), and it belongs partly on this page. The mechanism scores each tool call and result and deletes the ones no longer needed, keeping the rest byte-identical, reporting a Claude Code session dropping 156,000 tokens to 62,000 with 10 of 16 entries retained verbatim. The reason this is a cache result and not only a prompt result is the 09-13 entry's finding that the prefix is a build-system property: deleting a span in the middle of a transcript invalidates every cached prefix downstream of it, so aggressive mid-history filtering trades prompt tokens for cache misses, and none of the reported numbers price that. A compactor that only ever truncates the tail is cache-friendly; one that deletes arbitrary interior spans is not.

The argument remains unresolved and both sides now have weight. @theo's objection, the most-liked post in the thread at 2,276 likes, is that compaction is reconstruction rather than filtering. The 09-18 harness component ablation found context management's value comes almost entirely from preventing overflow failures, which sits closer to the objection. The experiment that would settle it is cheap and nobody has run it: the same agent, the same task suite, filter-style compaction versus summarise-style, reporting task success and cache-hit ratio rather than token count.

One production report worth recording with the usual discount. A widely-shared engineering writeup describes a 50-agent coding cluster whose monthly API bill hit $48,600, re-reading 1,050,000 tokens of repository files per prompt, then dropping to $4,820 after a four-role serving floor that locks a byte-0 system prefix for a claimed 99.1% KV-cache hit rate and 88% input-token cost reduction, plus Tree-sitter AST elision compressing 1,240 tokens of syntax boilerplate to 95. The prefix-locking and AST-elision mechanisms are both well-founded and consistent with the 09-13 prefix-as-build-artifact entry. The numbers are self-reported marketing prose with no artifact, and the precision of "matched to the cent" is itself a warning sign. Record the mechanism, discard the decimals.

2026-09-21: the million-token baseline gets stated, and every compression ratio on this page acquires a denominator

Chapter 9 of the frontier-inference series supplies the number this page has been quoting ratios against without ever writing down. Storing uncompressed 16-bit K and V tensors for an 80-layer 70B model at 1,048,576 tokens requires 137.44 GB of VRAM for a single user stream, so an 8-way H100 SXM5 node with 640 GB of HBM3 holds two or three concurrent streams before it is full. That reframes every compression result on this page. A 4x KV compression at long context is not an optimization, it is the difference between two concurrent users and eight, and there is no version of a million-token production service that keeps everything in HBM.

Two companion figures. A 1M-token prefill computes N² = 1.0995 × 10¹² query-key dot products per head per layer, which on that model is 5.62 × 10¹⁵ FLOPs for the attention core alone, moving time-to-first-token from hundreds of milliseconds into tens of seconds. And across a million keys the attention distribution disperses toward its maximum entropy of ln(N) ≈ 13.86 nats, which is the mechanical account of "lost in the middle": edge recall holds while retrieval accuracy drops below 30 percent across the middle 10 to 90 percent of the window. The entropy figure matters to this page because it says the middle of a long cache is already close to unread, which is an argument for eviction policies being less lossy than their token counts suggest, and nobody has tested that framing.

The chapter's claim is that 2026 production converged on exactly three mitigations, one per failure mode. YaRN interpolates the low-frequency RoPE channels while preserving high-frequency local syntax and adds a softmax temperature term against the entropy dispersion. Dual-Chunk Attention, as shipped in Qwen 3.8-1M, slices into 8,192-token chunks and decouples intra-chunk local attention from inter-chunk coordinate routing to eliminate positional drift. MInference 2.0 identifies at runtime which of three empirical attention topologies (vertical-slash, block-sparse cluster, slash-only) a head is exhibiting and evaluates only the top-k KV tiles in SRAM through fused Triton kernels, claiming roughly 10x faster million-token prefill.

Two of the three are routing, and this page should say so. DCA routes across sequence positions; MInference routes across attention topologies, making a cheap per-head classification at inference that determines which expensive computation runs. MISA (05-11) and SAS (09-14) both argued the right sparsity pattern is head-dependent and learnable. MInference is the deployed version with the patterns enumerated rather than learned, which is a good empirical check on whether learning them buys anything, and the comparison has not been run.

Caveats. The TTFT benchmarks, 1M retrieval-accuracy figures and sizing templates are paywalled, so the free chapter establishes the problem precisely and the solutions in outline. It is a synthesis of published work from DeepSeek, Moonshot, Zhipu, Alibaba, NVIDIA and Google DeepMind: the arithmetic is checkable and correct, the performance claims are secondhand. And the three mitigations are asserted to be complementary without being shown to compose, which is the missing experiment: sparse attention discards tiles, YaRN reshapes the position spectrum, and DCA changes what a position means across a chunk boundary, so whether the retrieval loss of all three together is the sum of the three separately is exactly what a deployment needs to know.

2026-09-23: four papers in one day, and the cache stops being a flat append-only buffer

The day's structural finding is that every serious KV-cache result now treats the cache as something with internal structure that can be allocated, addressed, paged or fused, rather than as a monolithic buffer to be shrunk. Four entries landed together and they attack four different dimensions of the same object.

Flash-dLLM says the bottleneck was never the policy, it was the bus. Diffusion language models had no working KV cache; prior work added caching and parallel decoding separately, each reporting a speedup. Flash-dLLM's measurement is that when you enable both, GPU memory I/O dominates, because the cache is read and written far more often than it is computed against. An I/O-aware fused KV kernel plus a self-drafting draft-and-verify loop (the dLLM is its own drafter, no auxiliary model) gives 5.1x on GSM8K and 11.0x on HumanEval over Elastic-Cache, training-free. This is the same conclusion SemiAnalysis reached on 09-21 about mixture-of-experts serving clusters, arrived at independently from inside a single GPU on a non-autoregressive architecture: the cost model of modern inference is a memory-traffic model, not an arithmetic model.

KV-COBRA makes the per-head budget the optimization variable. Its claim is that the compression scheme was never the interesting choice; the binding question is how a fixed bit budget splits across attention heads, and within a head between rank truncation and bit-width. Using only standard low-rank projection and scalar quantization, co-optimizing the two per head dominates uniform allocation from 0.5 to 4 bits per dimension, with the largest gains at the lowest bit-rates and no per-token overhead. A fused Hadamard rotation equalizes per-channel variance, and reordering the SVD basis by attention-KL importance rather than by singular value makes the solver query-aware. This is the fourth result in six weeks saying the unit of a cache decision is the head, after the 08-25 spatial-precision entry, the 09-11 precision-routing entry and the 09-14 depth-sharing retrofit. Cache heterogeneity is now established, large, and exploitable exactly where the savings matter most.

ARM replaces eviction with routing, and it is filed on the routing page for a reason. Instead of deciding which tokens to delete, a hierarchical Gumbel-Softmax router decides which of a fixed number of memory slots each piece of context writes into, with sigmoid gates blending new content into stored content, so nothing is hard-deleted. A separate learned policy varies how many slots to read at inference. This directly contradicts the premise of the whole eviction literature, which bets that most cached tokens are worthless. ARM bets the information is worth keeping in compressed form and that "delete" was the wrong primitive. The decisive experiment, matched-budget eviction versus routed memory on the same model and long-context suite, has not been run. The deployment asymmetry is large: ARM requires training, every eviction baseline is training-free.

KVMEM makes the cache durable and pages it. Long-running agents currently either compact history into a lossy summary or re-retrieve the text and pay for the same forward pass twice. KVMEM keeps the computed KV state itself across GPU, RAM and NVMe and pages back only the slices the current step needs. DeepSWE Pass@1 rises 43.8% to 48.4% over compaction-only on Qwen3.8-27B; recovery is 11.4x to 53.8x faster than Compact+RAG; a single 24 GB RTX 5090 sustains a 1M-token workspace at roughly 50 tokens/s. Each step still sees a bounded slice, so this makes the history addressable at a million tokens while the attention stays bounded. This partially resolves the 09-20 entry that named compaction a cache-adjacent surface the page was ignoring: 4.6 points of Pass@1 is the first measured cost of compaction's lossiness on this page. It is also the third entry where the cache is an asset rather than an expense, after Cache-to-Cache made it something worth sending.

The four compose and nobody has composed them. KV-COBRA shrinks the bytes, Flash-dLLM cuts how often they move, KVMEM decides where they live, ARM changes what "keeping" means. The open question the page should carry: does a compressed cache make the fused-kernel win larger (fewer bytes to move) or smaller (less headroom to recover)? That is a cheap experiment and it has not been reported.

The cache-read price fell 60% on 09-22, and that is this page's economics update

Anthropic cut Opus 5.5 cache reads from $0.50 to $0.20 per million tokens, three times deeper than its 20% headline input cut, while OpenAI halved GPT-6 Sol and Luna outright (Luna's cached input is now $0.01/M). Simon Willison's framing is the one to keep: in longer agentic conversations 90%+ of input tokens are processed at cached prices, so the cached-input column is most of the input bill for any agent workload. The 08-14 entry called the cache a billing surface; it is now a competitive weapon, and it is moving faster than the compute price around it. Full treatment in the price-war entry.

One cost of a malformed token that this page never counted

Chapter 10 of the frontier-inference series names a third cost of the parse-retry loop, alongside latency amplification and token burn: KV cache thrashing. Malformed tokens pollute prefix and KV allocations with discarded context fragments, forcing premature evictions for concurrent users. That is a multi-tenant externality rather than a cost borne by the failing request, and no standard observability stack attributes it. It reframes constrained decoding (now under 40 microseconds per step with GPU-accelerated grammar compilers, down from 20 to 50 milliseconds on CPU) from a reliability feature into a serving-efficiency one.

2026-09-24: the cache gets shared across layers and moved onto flash

Two new dimensions of the cache-as-structure thesis from 09-23: sharing across layers, and placement below DRAM.

• HySparse2 (Xiaomi) shares KV at two levels. A YOCO-style bridge builds every cross-decoder cache from self-decoder hidden states, so prefill exits halfway through the network, and sparse layers reuse the full-attention layer's cache and token-level selection indices. At 1M context: 2.92x fewer prefill FLOPs, cache 6.72 GB to 2.69 GB, RULER-v2 at 256K up from 32.61 to 58.45. A same-day survey by @eliebakouch notes DeepSeek V4.1 Flash and MiMo V3 both use YOCO prefill exit, while Qwen 3.8 Next Flash and GLM 5.3 Flash interleave sparse and linear attention 3:1. Prefill exit is now a frontier design choice, not a paper curiosity.
• LM-CXD puts the prefix cache on CXL flash within 1.5x of DRAM. The finding that matters for this page: a stock CXL-SSD is no faster than NVMe because the block interface, not the NAND, is the bottleneck. Making KV chunks device-visible and exposing prefetch progress to the engine gives up to 4.03x lower TTFT. It is the hardware counterpart of KVMEM (09-23).
• Memory Attention implies a K-only cache (not claimed). If V = K + Norm(E[token]), values are recomputable from keys and token IDs with one lookup. Nobody has measured the roughly 2x cache reduction this suggests.
• Disaggregated Quantization raises an interface question for this page: the KV cache is now produced by an NVFP4 prefiller and consumed by a 1-3-bit weight-only decoder. Whether representation mismatch at that handoff costs accuracy is unmeasured.

Open question carried forward: does prefill exit (fewer layers producing the cache) compose with KV-COBRA-style per-head bit allocation, or does sharing one cache across many consumer layers make each head's precision budget binding for all of them?

2026-09-26: the cache has a safety subspace, and a database can plan its lifetime

• Alignment Collapse Under KV Cache Quantization (2606.09864, NeurIPS 2026) is the first result on this page where a KV compression method passes every quality check and still breaks the model. Across eleven models (3.8B to 72B), low-bit KV quantization removes refusals while perplexity barely moves: Mistral-7B loses 15.2% of refusals at 1.03x perplexity. Safety features occupy a low-dimensional subspace 10² to 10³ times more sensitive to quantization noise than the full space. A 20-prompt diagnostic (PCR) picks among three fixes and recovers up to 97% in about 35 GPU-minutes, including on KIVI and on vLLM FP8 KV. Every allocation method on this page (KV-COBRA's per-head bit-rank search, attention-score eviction) optimizes a reconstruction or accuracy target that this paper shows is blind to safety. Rotation-based quantization made safety worse in one setting.
• Quail (CMU + Modal) is a new workload class: KV that belongs to a database row, with a lifetime set by the query plan. It keeps a document's KV resident in HBM only while later operators need it ("rewinding" off each predicate's suffix), evicts short documents first because long ones are quadratically costlier to recompute, and reads a join anchor's KV once per group of partners via two FlashAttention-3 calls merged with log-sum-exp. On one H100: 14x faster and 14x cheaper on the hardest query, KV regret down from 50.3M to 18.0M recomputed tokens. Because the future access pattern is known from the plan, this is the rare setting where near-optimal eviction is achievable. Its weak spot is the one this page knows well: no cross-document prefix caching, 2.3x slower than vLLM on the agent-trace query.

Open question carried forward: does the safety subspace survive the other compressions on this page, such as cross-layer KV sharing (HySparse2) and prefill-exit caches, or is the PCR probe needed after every one of them?

2026-09-27: KV as a residual across loops

FlashLoop (09-27) stores each loop's KV cache in a looped transformer as a low-bit delta from the previous loop's KV, because adjacent-loop KV states converge as recurrence proceeds. Combined with token-sparse updates and sparse attention, it reports up to 6x KV memory reduction with no accuracy loss. This is a new family next to eviction, quantization and cross-layer sharing: compression by temporal (loop-to-loop) redundancy rather than by token importance. A community post also circulated MILO, a mixed-precision KV scheme that gives information-dense blocks more bits (up to 50% memory saving and 1.8x throughput on Qwen2.5 3B/7B on one L4); no paper link was attached, so it is recorded as unverified.

2026-09-29: sparse attention saves reads, and the spilled cache still hits

SemiAnalysis on GLM-5.3 states the limit of sparse attention plainly: DSA (DeepSeek Sparse Attention, a lightning indexer that scores every past token and keeps the top 2,048 for attention) reduces KV bytes read per step, but the indexer needs every token resident, so KV capacity does not shrink. SGLang's HiSparse answers with an LRU hierarchy: on a top-k miss, fetch the entry from host DRAM, overlapping layer N's load with layer N-1's compute (inherited from HiCache), and evict least-recently-used entries back down. The serving numbers show the tier works: on B200, doubling concurrency from 8 to 16 drops GPU-memory prompt reuse from 90.3% to 54.8% while host-memory reuse rises from 6.0% to 40.3%, keeping the overall hit rate above 95%.

Two cache-side details from the same piece. IndexShare has one indexer serve every 4 DSA layers and caches the top-k indices for reuse, cutting indexer cache and FLOPs by 75% and lifting throughput 1.5x to 1.8x; this is the indexer-side analogue of the cross-layer KV sharing logged on 09-24. And MLA's two execution modes cross over by length: vLLM uses MHA mode (fewer FLOPs, more memory traffic) from 2K to about 5K tokens under data parallel and to about 77K under tensor parallel, and MQA mode beyond.

2026-09-30: one transcript, many agents, one cache

KVCMAS and PReCache (09-30) add a third sharing axis after cross-layer (HySparse2, 09-24) and cross-tier (HiSparse, 09-29): cross-agent. When several agents share one base model but differ by system prompt (KVCMAS) or LoRA adapter (PReCache), each used to re-prefill the same growing transcript. Both store the shared cache once plus a low-rank per-agent delta. KVCMAS: 2.0x TTFT, up to 3.7x less peak memory than prior correction methods, exact first-agent cache. PReCache: up to 3.1x TTFT, 2.3x throughput, -1.1 points for the accurate variant. The low-rank-delta idea matches FlashLoop's (09-27) per-loop KV deltas: caches vary along small subspaces. Open: gain over production prefix caching is untested.

Attention-side entries the same day bear on capacity: CoWindow Attention gives each KV head a different slice of distant history (union covers everything) and cuts decode memory per rank 7.6x at 128K, a rare sparse-attention design that shrinks per-rank capacity, not just reads. The 09-29 entry said sparse attention saves reads but not capacity; CoWA is the counterexample, at least per rank. WaveFront Decoding shares KV across loop depths to cut wavefront traffic (4.81x speedup on Huginn-3.5B).

2026-10-01: chunked compression has a phase, and edits break prefix caching

• Periodic Weak Spots (10-01). Chunked KV compression (squeeze each fixed-stride window of tokens into fewer entries) creates a hidden coordinate, the token's offset inside its window. Retrieval accuracy cycles with it, by up to 40 points in large open models, while averages look fine. Reproduced from scratch across compression designs; attention components specialize by phase. This applies directly to the 09-29 entry: GLM-5.3's DSA indexer scores pools of 4 consecutive tokens, and the llama.cpp GLM-5.3-Flash merge uses the same k-pool indexer. Nobody has measured phase sensitivity there. Cheap fixes to test: staggered stride offsets per layer or head.
• Editing the context invalidates the prefix cache. Context Language Models (10-01) let the model rewrite its own context, which breaks prefix reuse. Their Suffix Cache Reuse cuts server compute 35% vs stock SGLang at matched quality. Any editable-transcript design needs this primitive.
• KV4 at near-full precision. PrismQuant (10-01) runs W4A4KV4 on Llama-3.1-70B 0.22 points below full precision; on 8B, 56% less decode peak memory.
• Demand-side signal: Micron's NAND revenue rose about 8x to $14B in 18 months, data-center SSDs about 71% of it (10-01). SSD-tier KV and context stores are being bought at scale.
• Practitioner: CAG (cache-augmented generation, precompute and store the KV for static documents) explained as "pay the prefill once"; a 70B BF16 model needs about 300 KB per token (10-01 Media Zone).

2026-10-02: stateful serving, and 2-bit KV with a learned rotation

• Galahad (10-02). 98.7% of prompt tokens across seven real datasets are text the model already read. Galahad saves KV per byte-matched block and reloads it across requests on vLLM, SGLang and llama.cpp; with a section picker it answers 100/100 on a 97K-token recall test at about 0.6 s and 200 J per question vs 10/100 and 2,754 J without. Restore is bit-identical and fails closed. This moves the "pay the prefill once" idea (CAG, 10-01) from a trick to a serving layer, and keys reuse by bytes rather than only by prefix, a partial answer to the edit-aware caching gap raised by Context Language Models (10-01).
• WUSH-KV (10-02). A calibration-derived rotation (from second-order statistics of both matmul factors) before quantizing keys and values; the value transform folds into weights. At 2 bits in SGLang it matches or beats OSCAR. Second rotation-based low-bit KV result in two days after PrismQuant (10-01).
• Effective attention set. Retrieval Capacity of Self-Attention Under Competition keeps only the top-attended tokens per head and query and measures how many are needed to stay within a loss tolerance. The needed set grows with context but shrinks as a fraction of it, and renormalizing the kept weights cuts it further, a direct input to top-k sparse attention budgets.

2026-10-03: cross-family transfer, and compaction without re-prefill

• HeteroFold (10-03). First prefill-free KV transfer across model families (different tokenizer, depth and KV geometry) with both models frozen: align layers, map the sender cache into receiver space, calibrate. Llama-3.1-8B to Ministral-3-14B at 32K is 10.7x faster than native prefill and 1.18x to 1.47x faster than Dense Latent and KV Ridge; matches text communication on a multi-agent benchmark. Partly answers the 09-18 C2C question of how much cache structure independently trained models share. Sharing ladder is now cross-layer, cross-tier, cross-agent (same base), cross-family.
• KV-streams (10-03). Agentic RL compaction normally rebuilds the cache (re-prefill). KV-streams, a vLLM 0.19 fork, evicts whole 16-token blocks from the live cache, tracks logical RoPE position apart from physical slot, and claims about 2x faster agent RL training. Training-side twin of CLMs' Suffix Cache Reuse (10-01). Repo plus X explainer only.
• FOCUS (Microsoft, 10-03 same page). Training-free selection of the history units future decisions depend on: up to 48% less peak context and +8.9 points task success.
• Open question: HeteroFold's mapping is fitted per model pair. How fast it goes stale after a model update decides whether cross-family caching is a product or a demo.

2026-10-05: fewer tokens is not a cheaper agent

Beyond Token Savings (10-05), UT Austin, ~35,000 agent runs. Splits compaction into how / when / how much. On Terminal-Bench with Qwen, policies using a third of the tokens ran 20-80% longer; step-triggered policies need 10-27% more calls; threshold-triggered ones cut tokens 22-55% at near-baseline call counts; policies do not transfer across models (OTRC: Qwen 51.3%, Devstral 38.7%). Copilot trace: compaction sessions are 44.2% of tokens served, median compaction removes 72.8%. Implication for this page: text-level compaction invalidates the prefix cache and adds prefill and summarizer calls, so KV-level dropping (KV-streams, 10-03) may be the only compaction that is free on latency. Open: does a learned trigger (AutoCompact, 10-04) win on wall-clock, not just accuracy?

2026-10-06: send less cache between agents, store less per token, quantize Keys by what they select

• CacheBack (10-06). Latent communication between agents sends KV caches instead of text, but full caches scale with context x agents. The receiver sends a short need description; the sender filters its KV by attention to it. FanOutQA, Qwen 3: 75% less state sent, +14.7 accuracy, 3.2x lower median latency vs text. Works on dense, Mamba-hybrid and sliding-window models. Extends the cross-agent axis opened by KVCMAS/PReCache (09-30): those dedupe shared prefill, CacheBack prunes what crosses the boundary.
• The Extender (10-06). K and V of every layer read only a thin append-only channel (32 dims per layer), so persistent attention memory is 104x smaller than MHA at 924M (width 1664), matching CORE and beating RULER. Architectural cross-layer sharing; untested against GQA at scale.
• QuantWM (10-06). 2-bit KV for video world models: Key errors are smaller than Value errors but hurt more, because they shift attention logits and change which past tokens are selected. Query-sensitivity-weighted INT2 centroids plus a low-rank correction in the query subspace; up to 6.2x KV compression across five world models. Third 2-bit KV method in a week after PrismQuant (10-01) and WUSH-KV (10-02).
• Practitioner: vLLM 0.30 generalized adaptive verification for variable-length speculative decoding to any draft method via an online acceptance estimator (Red Hat, X); chunked prefill explainers trended. Shift: the page's open question from 10-05 (is KV-level compaction the only latency-free kind?) gains a cross-agent version: CacheBack is KV-level, and it cut latency 3.2x.

2026-10-07: never build the long cache

• Periscope (10-07). For closed-set decisions over long text, probe a √N x √N chunk grid by rows and columns; the cache holds one probe. A 27B model reads 4.5M tokens on one 80GB GPU (296GB for a single pass); 9k top-ranked tokens match the best 32k-1M window read.
• Looped models at fixed points (10-07). Terminal KV sharing: store K/V from the last loop only. At 1.6B a learned depth prior with a 3x smaller cache matches fixed-depth with the full cache.
• HLA-WM (efficiency shorts). Retrieval-addressable linear-attention state: 12x less history memory than full KV at 60 s of video, <=1.6% throughput loss.
• Industry: NVIDIA pitched Dynamo's cluster-wide KV lookup as a way to skip recomputation; Uber credits caching pushed down to sub-agents for flat token costs.

2026-10-08: the cache in RL rollout, in latent form, and in storage

• FP4 KV cache inside RL rollout: TRACE (10-08) runs rollout with FP4 KV and matches BF16 RL quality.
• Shared latent from weight compression: LSP (10-08) ties Q/K/V projectors so the cache stores one narrow latent; 13.5x less weight+KV memory at 128K.
• Cache handoff across model variants: SlimWise (10-08) lets a pruned decoder read the full model's prefill cache with no conversion.
• Shorts: DeCoPrune prunes 85% of video-diffusion KV (4x faster); Nanbeige4.2's two-pass loop doubles the KV cache (efficiency shorts).
• Economics: prefix caching is why routing can cost more than not routing (routing page); Anthropic halved Sonnet 5.5 cache reads; an investor note projects SanDisk data-center revenue ~30x by 2028 on agent KV cache moving to enterprise SSDs (forecast, not guidance).

2026-10-09: the cache becomes a session object, and a storage tier

• Session-aware Dynamo (10-09). One session ID (Claude Code, Codex and OpenCode headers map to it) lets routing, admission and cache placement see a whole agent run: tool-boundary backpressure holds a waiting session's cache instead of evicting it; a shared-pool indexer tells the router about reusable KV in external stores; a proposed KvHint lets the orchestrator state cache intent while vLLM/SGLang decide execution. The serving-side answer to 10-08's finding that routers lose prefix cache when they switch models. NVIDIA's HSTU recommender with FlexKV: up to 5.93x lower latency at batch 8 (100% hit rate, best case).
• galahad-kv (10-09). Byte-exact KV blocks (~16K tokens) on encrypted NVMe: 50M tokens on one H100, reload 2.8-4.3x faster than recompute with 8.8-12.3x less GPU energy; 98/100 recall on the 31B model. Block reuse, not a wider window; retrieval of the right block is unsolved. First energy number on this page for storage-tier reuse.
• STEPQuant (10-09). The KV-quantization playbook applied to the fixed recurrent state of linear-attention hybrids: 6-bit state near FP32 accuracy, >5x state compression, up to 68.7% less serving memory in SGLang.
• Data: a public agentic inference trace set (206B tokens, 12,002 sessions, 1.19M requests) gives eviction and routing research real agent workloads.
• Shift: this page's unit of reuse has moved from prefix (2025) to block (09-30 KVCMAS) to session (today). The next open question is policy: who decides what a session keeps, the harness or the engine? KvHint says both.

2026-10-10: the cache that survives a hop, a turn, and a compaction

• Across model hops: TokenRouter (10-10) parks a routed request as pending with its KV kept, so a switch to the other model is a token append; in the standard baseline, radix-cache update and prefix matching ate 53% of each small-model step.
• Across turns, after eviction: SparseEngine (10-10) runs 15 sparse/eviction/compression/quantization methods under one lifecycle contract; Chain Cache resumes an evicted cache next turn and Prefix-Cache Pruning drops chosen history while keeping the prefix matchable. >10x throughput with eviction, >2.5x decode vs vLLM.
• Across compaction: REMORY (10-10) appends learned soft tokens to the text summary, near full-context quality at 5.2% of input positions.
• Hardware/vendor: Huawei's OceanStor M900 Context Memory Storage pitches shared KV capacity as a storage product (shorts).
• Shift: yesterday's question was who decides what a session keeps. Today adds two answers: the sparse method itself (Chain Cache) and a learned compressor (REMORY). The common rule across all three papers: never discard state the next step will rebuild.

2026-10-11: moving the cache across the network

• Serving shorts (10-11). AMD's MoRI + vLLM work moves KV between disaggregated prefill and decode pods over RDMA, the network-layer version of the 10-09/10-10 "keep the state" results. The kilolayer-NAND perspective (hardware, 10-11) warns the flash tier under offloaded caches gets noisier per layer.

2026-10-13: both halves of the MLA cache, and the SSM state's write traffic

• QuantMLA (10-13). MLA's latent cache has two paths with different error sensitivity: RoPE-path error is amplified, content-path error is not. Path-specific, offline-fusible transforms give the first joint INT4 of both; INT2 content + INT4 RoPE stays competitive; 3.59x physical compression at 128K, 5.17x whole-job throughput vs BF16 under cache pressure.
• ReplaySSM (10-13), NVIDIA + Dao AI Lab. In hybrids the SSM state, not the KV cache, is the decode bottleneck at batch 256 up to ~100K tokens (one Mamba-2 layer's FP32 state = an FP8 KV cache at ~8K tokens). Store checkpoint + recent inputs, write the state once per 16 tokens: kernel 1.46-1.80x, speculative throughput 1.85-2.10x on 120B-class. In production in GLM-5.3-Flash and Kimi-K3.
• Priming (10-13), Amazon. Convert local attention layers to SSMs for <0.5% of pretraining tokens; a 32B hybrid decodes up to 2x faster at 128K. SSMs beat sliding-window attention even at 2x its memory budget.
• ELR (10-13). Looped models keep only the last loop's KV, holding cache at the one-loop level.
• Shift: the cache question has split in two. For attention, it is now about which bits of a latent are fragile (position). For hybrids, it is about write traffic of a fixed-size state. Both papers find the sensitive component and protect only that.

2026-10-14: cache the loop, price the reread

• ResidualQuant (via X, 2610.10381): per-loop KV caches in looped models are nearly redundant; a shared 4-bit anchor plus 2-bit residuals keeps BF16-level MATH500 on Ouro-1.4B at 80.7% less KV, 3.27x peak decode throughput (shorts).
• FlexBudget (AMD): fixes kernel and budget, swaps only the eviction importance signal; a fair benchmark for eviction methods.
• Pricing: Gemini 4 Argon's cached input is 95% off (Argon); IBM's agent factory hit ~98% prefix reuse. Prefix-cache hit rate is now a first-order cost variable for agents.
• Hybrids: LFM2.5 and Ling-3.1-flash keep 75-88% of layers off the KV cache (page).

2026-10-15: the indexer is the new cost center

• DoSA (10-15): in indexer-based sparse attention (DeepSeek-V3.2, GLM-5.2), the indexer still scores every key. DoSA certifies keys out with an upper bound against a seed-set lower bound on the top-K threshold, so only eligible keys get full scores. Lossless (identical selection). Indexer up to 2.74x over SGLang and 1.69x over LiteTopK; TTFT up to 1.94x (GLM-5.2) / 1.50x (DeepSeek-V3.2).
• Pattern: FlashMemory (06-09) cut what the cache stores, SAS (09-14) and CompactAttention (05-19) cut selection error; DoSA cuts selection work. Decode-time speedup is the missing number.
• Long video: FramePack-U packs any-length video into a fixed visual token budget at constant memory (shorts).

2026-10-16: protect the fragile slice, read fewer tokens, train a smaller cache

• KV4-RL (10-16): in async RLVR, high concurrency makes rollouts KV-read bound, so KV quantization beats weight quantization at matched workload. INT4 KV with sinks + recent window kept at INT8: 1.80x rollout, 1.27x end-to-end, within 0.5 points on Qwen3-4B/8B. Naive INT4 KV breaks learning even with TIS.
• JSSA (10-16): training-free token-level sparse attention. Closed-form shared query-key subspace for scoring plus a gather-free per-KV-head kernel under GQA. Up to 6.43x decode throughput at 128K on B200; matches DSA on reasoning without training. Answers the 10-15 "decode-time speedup is missing" note for non-DSA models.
• Latent Compaction (10-16): per-document test-time-trained cache, weights frozen, no access to questions. 96.7% RULER at 32x vs 40.0% for KVZip+.
• Pattern (n=3 in four days): QuantMLA (RoPE path), ReplaySSM (state writes), KV4-RL (sinks + recent) each find a small structurally identified fragile slice and compress the rest.
• Tension: vLLM bitwise-consistent RL (10-14) removes sampler/trainer drift; KV4-RL introduces bounded drift for speed.

2026-10-17: the remote tier has a break-even line, and the cache shrinks at the head

• Remote KV tier (10-17): GLM-5.3 on 8xB300 with PCP8. A remote hit took 0.88 s vs 0.31 s to recompute. Hits pay only above B* = sR (2.53 GB/s) and above ~17K tokens (fixed per-hit cost 0.33 s). Rank-sharded store + per-rank servers (GET 6.3x to 10.76 GB/s) + async load: hit-heavy p99 from 83.32 s to 0.67 s, prefill +14% (+34% with kernel fix). Upstreamed to LMCache.
• Contradiction, hardware-relative: galahad-kv (10-09) found NVMe reload 2.8-4.3x faster than recompute on H100. Faster GPUs raise the bandwidth a tier must beat; storage-tier wins do not transfer automatically to Blackwell.
• VHA + PN-Mix (10-17): VHA serves as plain GQA-2 (2-4x less KV than GQA-8/GQA-4) yet beats GQA and MLA on quality at 30B-A3B; RULER 84.4 vs 73.0. PN-Mix: 2.26-bit keys by RoPE pair-norm saliency, graceful to 1.5 bits.
• Pattern now n=4 (fragile slice): QuantMLA (RoPE path), ReplaySSM (state writes), KV4-RL (sinks + recent), PN-Mix (high pair-norm RoPE pairs). RoPE error amplification appears in two of the four.
• Shipped: NVIDIA Dory1.0-4B ships last-loop KV caching for looped models (page), the ELR (10-13) proposal in a product.

2026-10-19: 16 tokens instead of the cache

Memorilla skips the long cache entirely: documents are encoded once, and a learned writer emits 16 query-specific memory tokens into a frozen LLM's input. The whole collection costs 16 KV positions, and FactKG accuracy rises over full context. Untested on exact recall, where soft-prompt compression usually fails.

2026-10-20: the cache learns to travel both ways

• vLLM push + bi-directional KV transfer (AWS, 10-20): in prefill-decode disaggregation, push mode pre-registers decode blocks during prefill and writes KV over NIXL/EFA RDMA from a background thread: up to 3x TTFT and 30% TPOT at 8 QPS / 16K (Qwen3-32B, 1,744 to ~540 ms). Bi-directional mode keeps the decode node's KV (prompt + answer) for 480 s and returns it to prefill next turn: TTFT 1.6x lower (Qwen3-32B) and 3.3x (GLM-5-FP8) at 20-24K. Upstreamed, accelerator-agnostic.
• Break-even is tier-relative, now n=3: galahad-kv (10-09, NVMe reload beats recompute on H100), remote store on B300 (10-17, only above ~17K tokens), and peer-GPU RDMA here (above 64 tokens). The closer and faster the tier, the lower the line.
• Multi-turn agents are the design target for both serving (this) and drafting (DLite, which drops the drafter's own KV history).
• Open: fixed TTL pins HBM for idle sessions; no results with prefix caching on.

2026-10-22: the agent edits its own cache; the prompt fades by a power law

• Cache Engineering for Agents (via X, 2610.28577): drop (exclude entries from attention) and repos (shift remaining keys' positions) replace summarization, so no re-prefill. Rolling drop of old tool responses: +1.7 points, 27.7% cheaper; AKV engine up to 1.2x throughput, 1.5x P95. Cache states now depend on operation history, so prefix matching is no longer a correct cache key.
• TokenGate (2610.26754): learned token pruning at prefill; 1.4-11.3x TTFT, 1.5-3.4x lower peak memory, no accuracy loss. Decides what never enters the cache.
• MemT-1.0: memory inside the model (Gated DeltaNet-2 + Mixture-of-Storage); 66x less persistent storage than full-context KV at 128K, 1.86x end-to-end.
• Serving side: llm-d 1.0's prefix-aware router captures 97% of available cache reuse on agentic coding; Google's TPU engines add KV offload to host DRAM/Lustre via TPU Sync; DeepSeek's FlashMLA stores quantized KV next to its scales on Ascend (DeepInfer/Huawei). AWS's Laplace 3B replaces the cache with a fixed 29.5 MB linear-attention state.
• Theory: the conditional no-hair theorem shows prompt influence on hidden states decays ~t^-0.6 in trained models.
• Kurate underrated: DropKV (residual-output perturbation for near-optimal eviction, cs.AI #1), MOSAIC (multi-objective cache allocation across layers, cs.LG #1), TideKV (delayed eviction via layer-wise chunked prefill).
• Pattern n=6 on selection beats compression: cache engineering is the first instance below the harness.

2026-10-23: estimate the tail, quantize the state, disclose the regressions

• Stratified Sparse Attention (2610.28441): top-k for high-score tokens, score-guided sampling with inclusion-probability reweighting for the rest. At 50x sparsity on RULER-HARD (Ministral-3-8B): 81.8% to 84.2% at 32K, 63.8% to 69.3% at 128K (94%/85% of dense). Up to 2.96x decode over FlashInfer at 1M. First paper to treat discarded attention mass as something to estimate rather than drop; composes with DoSA (10-15, exact top-K with less scoring).
• PACE (2610.29545): for hybrids the recurrent state is the cache. Round once per few tokens, column-shared scales, 8-bit for long-memory heads; matches 8-bit DAMP at ~2/3 memory. Third low-bit-state paper after STEPQuant (10-09).
• Production disclosure (MiniMax M3.1 Flash talk): Q8/KV4 kept the average flat but cut one sub-score 6.6 points. First public evidence that 4-bit KV regressions hide inside averages; argues for per-slice evals whenever KV is quantized.
• Routing x cache: TDS's "hidden cost of LLM routing" argues switching models mid-session throws away the prefix cache. Combined with Cache Engineering (10-22), the cache is now a cost term in routing decisions.

2026-10-25: search the layer budget, not just the token set

• MOSAIC (2610.25431) (Kurate cs.LG #1, second week flagged; not on HF): a multi-objective search learns per-layer budgets for any eviction method. Beats uniform allocation in 146/154 settings (up to +48 RULER, +11.9 LongBench LCC) and matches uniform quality with up to 4x less cache on Llama-3-8B and Mistral-7B. One search transfers across budgets; middle layers get the slots.
• LIRQuant: W4A4KV4 with learnable rotations only on sensitive channels; 3.5x memory saving over FP16.
• State of knowledge: selection now runs on three axes (which tokens, which layers, which bits). Nobody has searched them jointly; MOSAIC's profile under 4-bit KV is the obvious next test.