KV Cache

The KV cache (Key-Value cache) stores the key and value tensors from the attention mechanism for tokens already processed. This means those tokens don't need to be recomputed on every new generation step — critical for making autoregressive decoding fast.

Current State (as of 2026-09-03)

Two papers today, from unconnected groups, make the same diagnosis: the mechanism that decides which KV entries to read has become the cost it was introduced to remove. This page has spent months on methods that select a sparse subset of the cache. Every one of them pays a scorer to do the selecting, and both of today's papers say the scorer is now the binding constraint. They then exit in opposite directions, which brackets the design space cleanly enough to state as a rule.

• CRISP (09-03) (arxiv 2609.01925) keeps the scorer and makes it structurally free. Prior dynamic-sparse routing builds a pooled proxy attention map and measures Jensen-Shannon divergence against candidate patterns; CRISP shows the decision is a function of the map's shape and replaces the divergence with C_struct, the post-softmax mass sitting at Vertical-Slash compatible positions, reproducing the same routing decisions while deleting both the pooled matmul and the KL step. Its second and larger contribution is a proof that strictly cumulative coverage thresholds accumulate O(n) background noise at long context, formalized as the post-softmax mass cliff: mass does not taper, it falls off a cliff onto a near-uniform floor, and a cumulative rule cannot distinguish floor from signal. The fix is a sink-aware threshold anchored to the noise floor. 5.30x attention speedup at 512K tokens, up to +28.0pp recovered on retrieval tasks, and it matches or exceeds exact dense attention on retrieval-heavy benchmarks. The speedup is attributed primarily to the noise elimination, not the free router.
• Declarative Attention (09-03) (arxiv 2609.02737, KAIST AI with Google DeepMind) removes the scorer entirely. The observation is that per-step proxy scoring is still O(N), so it reduces the constant factor and never changes the complexity, meaning the cache is still being touched in full every step. DA instead elicits the model to declare its own attention scope inside its chain-of-thought, in three modes, <global>, <focus> on a named region, and <local> on recent output only, which the inference engine parses like a tool call before skipping most of the KV read. Zero-shot, no training, off-the-shelf models: Gemma-4-31B attends 52.0% fewer tokens for a 1.27pp accuracy drop; Qwen-3.6-27B 31.1% fewer for 2.75pp, and the accuracy gap shrinks with model scale.

The rule the pair establishes: prefill-side, the win is a better threshold; decode-side, the win is not scoring at all. CRISP operates on the quadratic dense pass over the prompt, DA on the per-step read during generation. They are complements, not rivals, and nobody has composed them. That composition is the obvious unclaimed experiment on this page, and it is cheap to run because DA needs no training and CRISP is a selection rule.

CRISP retroactively reframes a run of results on this page as having optimized the wrong stage. OasisKV (08-11) treated which entries to fetch as a lookahead prediction problem and prefetched them. CRISP's proof says the harder half was never prediction but thresholding: pair a perfect scorer with a cumulative-coverage budget and you still import O(n) floor, which grows with context. Any method on this page whose budget rule is "retain tokens until cumulative mass reaches X%" inherits that defect, and the retrieval degradation the sparse literature has repeatedly reported at long context now has a structural explanation rather than an empirical one.

It also explains a mechanism TileMix (08-25) exploited without naming. TileMix allocated precision per attention tile on the observation that most tiles carry nearly nothing while a few carry the answer. That is the mass cliff, used as an engineering heuristic. CRISP formalizes the same distribution and derives its noise-accumulation consequence, so tile-importance skew and the post-softmax cliff are one phenomenon with two applications, precision allocation and token selection.

Where the model-keyed boundary stands after two days. Yesterday's entry (below) recorded Cross-Model KV Sharing as the candidate mechanism for making a cache portable across models, deleting a prefill rather than accelerating one. Today's pair deletes work on the decode side. The two-day arc is coherent and worth stating as this page's current frame: yesterday, do not re-read what another model already read; today, do not read what you will not use. Both are instances of the constraint Ken Huang's inference physics (09-02) made non-negotiable, that a 70B FP8 model spends 99.66% of every decode step moving bytes at under 0.3% of peak Tensor Core throughput, so only byte-reducing or trip-skipping optimizations count. DA's headline number is a direct cut to the bytes moved per token, and it is the rare one that raises arithmetic intensity without batching.

Industry moved the same lever the same week, as a price. Anthropic's Fable 5.1 (09-02) cut cache-read cost 75%, to $0.25 per million tokens, which the release says should save heavy agentic workloads up to 45%. That is a provider repricing cache reads downward at the moment two papers make cache reads avoidable. The unpriced item this page should now track is whether cheap cache reads reduce the incentive to adopt read-avoidance at all, because a 75% price cut on the operation is a partial substitute for a 52% reduction in the operation.

Standing gap, unchanged. Neither paper reports realized latency. CRISP quotes attention speedup and DA quotes attended-token reduction, and both are proxies: skipping non-contiguous KV regions can defeat memory coalescing, so a 52% cut in attended tokens is not a 52% cut in decode time. The gap between theoretical and realized sparsity has dogged this literature all year and neither of today's papers closes it.

Prior State (as of 2026-09-02)

The model-keyed cache boundary this page named four days ago as "the most concrete unpriced item where this page meets routing" now has a candidate mechanism, and it is more radical than the problem required. A Universal Context-Reuse Layer for Cross-Model KV Sharing (09-02) (arxiv 2608.30963, Kurate cs.LG #17) trains a translation layer that converts the KV state produced by one model into a representation a different model can consume, across differences in scale, architecture, attention configuration, tokenizer, and model family. The authors name the abstraction context mobility: KV states as transferable computational representations rather than strictly model-local caches.

The three reported settings, in ascending order of how much they should change your mental model:

• Qwen2.5-1.5B → Gemma-2-2B (cross-family): up to 67.05% reduction in target-side prefill cost at 4K context, decoding perplexity close to native.
• Llama3.1-70B → Qwen2.5-7B (heterogeneous): 44.0% accuracy against 45.7% native, latency 899ms → 138ms, roughly 6.5x.
• Qwen2.5-7B → Qwen2.5-1.5B (within-family): LongBench2 27.59% → 34.48%, a 6.89-point gain over the native 1.5B baseline. The small model does better reading a large model's translated KV than it does reading the prompt itself, which is the finding that does not fit any framing on this page.

What this does to the 08-29 entry. That entry recorded three operational facts about provider prompt caching, the third being that cache entries are keyed to a model, so a mid-session route to a cheaper model pays a full cold prefill on the whole accumulated history, and that on the 140K-token median agentic prefix measured from replayed Claude Code and Codex traces this plausibly exceeds the per-token saving the route was chosen for. This is the first mechanism in the wiki that removes that penalty rather than pricing it. The honest caveat is a regime mismatch: the 67% prefill saving is quoted at 4K, and the problem lives at 140K. Until someone reports handoff cost and accuracy at 64K-plus, context mobility is a demonstration rather than a routing primitive.

It narrows the append-never-edit rule's claim. This page records three independent arrivals at "keep the prefix byte-identical": TokenPilot (06-16) as a research result, DeepSeek's Harness v0.1 (08-14) as an engineering commitment alongside a roughly six-fold cache-hit price rise, and the 08-29 practitioner explainer as production advice. Cross-model handoff asserts something much stronger: the prefix does not have to be the same model's to be reusable. If that holds, the operative rule shifts from "byte-identical prefix" to "semantically recoverable prefix," which is a materially different serving contract and a much weaker constraint on harness design.

And it sits in direct tension with the bounded-cache answer. Maglev (08-16) removes the eviction decision entirely by bounding the cache with a fixed-size recurrent memory. Cross-model sharing does the opposite: it keeps the full growing cache and makes it portable. The two cannot both be the right direction, and the reason is the one this page already noted about Maglev and prefix economics. A model carrying a bounded recurrent state has little cached prefix worth transferring, so a mobility layer has nothing to move. Whichever of these wins decides whether prefix stability remains a billable asset.

What is still missing, and it is the same omission as everywhere else. No cost is reported for the translation layer, to train or to run per handoff. No compounding analysis across multiple sequential handoffs in one long session, which is the actual deployment shape. And the ai_rating of 3.5/10 is the lowest in this week's Kurate cs.LG top 20, with the authors themselves calling the evidence initial.

Separately, today supplies the physical reason a saved prefill is worth more than a faster decode. The Physics of LLM Inference (09-02) derives that prefill is the compute-bound phase at 150 to 450 FLOP per byte while decode is memory-bound at 1 to 2, and that on a 70B FP8 model memory movement is 99.66% of every decode step. Eliminating a redundant prefill frees the one phase where the accelerator runs near its ceiling. It also prices this page's central resource: escaping the memory wall on an H100 would need roughly B = 296 concurrent decode streams, and KV cache memory is what stops you getting there. That is the cleanest statement yet of why everything on this page matters, and it means the value of KV capacity work rises with each accelerator generation, because compute has outpaced HBM bandwidth every generation and pushes the required batch size up.

Prior State (as of 2026-08-29)

This page has been conflating four different caches under one word, and a practitioner explainer arriving through saved reading makes the separation. KV vs Prefix vs Prompt vs Semantic Caching (08-29) lays out four layers keyed on four different things: the KV cache (GPU memory, one request, ~40 GB for a 70B at BF16 and 128K context), prefix caching (server side, vLLM's 16-token blocks identified by a hash chain over the parent hash plus token IDs, the scheduler stopping at the first miss), prompt caching (the provider's billed version, ~1.25x base input rate to write and ~0.1x to read), and semantic caching (application layer, embedding nearest-neighbour over stored responses).

Three operational facts this page did not have, all in layer 3. Writes happen only at a breakpoint you placed. On reads the system walks backward through a limited number of blocks and Anthropic caps this at 20, so more than 20 blocks of conversation between two calls pushes the last write out of range. And cache entries are keyed to a model. That last one is a routing result, not a caching result: on the 140K-token median agentic prefix AgentX (07-25) measured from replayed Claude Code and Codex traces, a mid-session route to a cheaper model pays a full cold prefill on the entire accumulated history. Nothing on the LLM routing page prices that, and at that context length it plausibly exceeds the per-token saving the route was chosen for. This is now the most concrete unpriced item where this page meets routing.

The append-never-edit rule reaches three independent arrivals, which crosses this page's pattern threshold. The article's four production failure modes are all prefix-shape rather than capacity: variable content at the front of the prompt, tool-schema reordering, settings rendered into the prompt, and history summarization. Its operative advice, truncate tool outputs in place to keep the prefix byte-identical rather than summarizing history, is the same rule TokenPilot (06-16) derived as a research result (any context edit mutating the prefix forces a full prefill recompute that cancels the token saving) and the same rule DeepSeek's Harness v0.1 (08-14) enforces as an engineering commitment while DeepSeek raised cache-hit prices roughly six-fold. Paper, vendor, practitioner. Same rule.

And it adds a row this page has never had: a cache layer that can be wrong. Every method tracked here is correctness-neutral in intent, and where correctness enters it is as measurement difficulty: the 07-28 impossibility result proved deterministic top-k eviction cannot estimate the error it created, and Compute Globally, Materialize Locally (07-29) showed that observing no accuracy loss after dropping a cached fact does not prove the fact was unnecessary. Semantic caching is a different class entirely. It does not degrade an answer, it returns a different question's answer with an HTTP 200, because embeddings place negated sentences close together and two prompts differing in one operational value score near-identical when the frame dominates. Published similarity thresholds span 0.75 to 0.97, and the spread is the admission: nobody has published a false-hit rate as a function of threshold on a real query distribution. That is the cheapest high-value missing measurement in this area and it needs no GPU.

Note on provenance: this arrived through saved reading rather than the paper feed, and it is the first save in five days after a thirteen-save run on harness and loop engineering. The practitioner layer moved from "how to drive the agent loop" to "what the loop costs at the cache," which is the same transition the research feed made on 08-28 when PILOT (08-28) became the first harness paper to publish a serving-side cost-per-success number.

Prior State (as of 2026-08-16)

Two results this week attack the cache from directions this page has almost no evidence on: the architecture that produces it, and the batch that competes for it.

Maglev (08-16) changes the shape of what gets cached rather than the policy over it. Almost every method on this page is a policy applied to an existing growing cache: what to evict, what to quantize, what to share across heads or layers. Maglev bounds the cache by construction with a fixed-size recurrent memory that still updates every token through the model's full nonlinear depth. It trains two coupled models, a prefiller Q with full history that emits memory targets and a decoder P with only sliding-window attention plus recurrent K/V injection, aligns them with a memory consistency loss, and then serves P alone. It beats both sliding-window and latent recurrent transformer baselines on validation loss and downstream pretraining benchmarks, and parameter sharing between P and Q keeps most of the gain while cutting parameter memory. This is a pretraining-time change, so it is the slowest thing here to reach production, and it is the only one that removes the eviction decision entirely rather than making it smarter.

Gambit (08-16) introduces an eviction unit this page has never had: the whole sequence. In batched reasoning inference the binding constraint is not FLOPs but the KV cache held by N concurrent long traces, and Gambit kills weak traces to fund new branches off strong prefixes, keeping utilisation high while cutting total token consumption by up to 68.5% and more than doubling completion throughput. Every eviction policy on this page assumes one sequence and asks which tokens to drop; Gambit assumes many and asks which sequences to drop. Those compose and nobody has stacked them.

The economics below still stand, and Maglev complicates them in an interesting direction. If a served model carries a bounded recurrent state rather than a growing prefix, then prefix stability, which DeepSeek's repricing just turned into a billing line, protects less value because there is less cached prefix to protect. Maglev does not discuss serving economics at all. The two results point at each other anyway.

Prior State (as of 2026-08-14)

The cache stopped being an implementation detail and became a line item on a customer invoice. DeepSeek raised cache-hit token prices roughly six-fold (08-14) while simultaneously open-sourcing Harness v0.1 under MIT, a harness whose central engineering commitment is never altering previously written history: when something in the conversation changes it appends a correction at the tail rather than editing the prefix, because editing invalidates every cached token after the edit point. DeepSeek raised the price of exactly the thing its own harness is engineered never to lose.

This converts a standing inference on this page into an observable price. The economic note below records SemiAnalysis's finding that frontier lab unit economics depend on >90% prompt-cache hit rates, with Anthropic's blended agentic price for Opus 4.7 near $0.99/MTok against a $5/$25 sticker. That was reasoning about provider margins. A 6x cache-hit price makes the cache a cost customers can see and must architect against, and it moves the local-serving break-even (07-30) toward self-hosting for exactly the repeated-context agent workloads where local serving was already most competitive. Append-only context management is now an economic requirement rather than a best practice, which retroactively makes TokenPilot (06-16) a cost-engineering paper rather than a convenience one.

On the mechanism side, outlier position turned out to be architecturally determined rather than empirical. Massive Activations in Hybrid Linear Attention (08-14) finds that in hybrid models (Qwen3-Next, Qwen3.5, Kimi Linear, Kimi K3, Nemotron-H) massive activations spike immediately before every full-attention layer (pre-attention spikes) and persist through the intervening linear layers as plateaus (inter-spike plateaus), with denser full attention connecting the spikes until the classic full-attention profile is recovered. This is the missing coordinate for the 05-19 activation census, which established that magnitudes vary four orders of magnitude across families and recommended per-family calibration but treated layer index as something to scan. For a hybrid model, spike location is now readable off the architecture config. The quantization consequence is scheduling rather than headroom, and it composes with the MoE-headroom asymmetry the 05-19 page identified. The ablation proving a PAS-aware scale beats a flat family-calibrated one is not in the paper and is the cheapest high-value experiment in this area.

A third result reframes what the cache does not hold. The full-bandwidth transformer (08-14) preserves the KV cache untouched but points out that hidden states in it are depth-frozen: reachable only by layers above where they were produced. Feeding the previous top-layer hidden state back into the bottom of the stack through a gated linear unit gives non-verbalized computation a renewed depth budget, and at 1B/400B tokens buys the equivalent of ~1.5x more training data plus shorter reasoning traces at equal or better accuracy. This page has treated the cache as a thing to compress, evict from, tier, and transmit. Vertical restriction is a property nobody here had named. Open risk: latent feedback creates a step-to-step dependency that speculative decoding assumes away, so composition with speculative decoding is untested.

And the consolidation-cadence lever showed up in agent memory. LycheeMemory V2 (08-14) replaces per-turn memory consolidation with semantic segment-level consolidation and cuts construction tokens 86% on LoCoMo and 75.9% on LongMemEval-S while reaching state of the art on both and not increasing query-time tokens. Combined with ICBQ's block-order finding (08-12) and ReOrder-OPD's prompt-ordering result (08-13), that is three efficiency subfields in three days where the schedule beat the operator. The pattern threshold is crossed. KV eviction cadence, as opposed to eviction policy, is the obvious untested fourth instance and this page's literature is almost entirely about policy.

Prior state (as of 2026-08-11)

Every method on this page decides what to throw away. Today's decides where the cache lives, and that is a different question. OasisKV (08-11) (2608.08097, HuggingFace) decouples full KV storage from HBM entirely: the complete cache sits in a higher-capacity tier such as host or remote memory, and only the entries the next decode step will actually attend to are staged in HBM. The prediction mechanism is the part worth keeping. Speculative decoding's draft tokens are reused as a lookahead probe for which KV blocks the model is about to consult, an attention background pipeline scores blocks against that lookahead, and the winners are prefetched one step ahead of use. On vLLM: within 0.7 points of full attention at a 2,048-token KV budget, 1.69x over dense vLLM on reasoning at 0.1 points of loss, up to 2.1x on multi-GPU long-context serving, and under prefill-decode disaggregation about 2x dense throughput while admitting each request with 6.5 to 9.7x less KV and holding 2.2 to 2.6x less decode-node host memory than full KV transfer.

Why this reorders the page. LOCKS (07-29) attends about 2% of tokens via page-local spectral summaries, MSA (06-12) selects blocks per GQA group then attends exactly, VaSE (06-03) evicts stochastically with a large-magnitude value guard, Conf-KV (05-30) sets a per-step budget from model confidence. All of them are irreversible or quasi-irreversible decisions inside HBM. OasisKV's selection error is a stall, not a loss, because the block is still in the capacity tier. That distinction resolves, in the practical direction, the open question this page raised on 07-29 about whether the 07-28 impossibility result (deterministic top-k eviction cannot estimate the error it created) binds reversible selection as tightly as irreversible eviction: with a full cache resident one tier down, the error is recoverable next step regardless of whether it can be certified.

It is also the algorithm that makes a tiering policy this page has listed as unshipped finally cheap. The agentic memory hierarchy survey (06-07) argued dominant memory traffic shifts from weights to KV cache as context grows; AMD's MoRI roadmap in the 07-25 SemiAnalysis piece targets a tiered HBM → DRAM → NVMe KV cache with scheduler co-design. OasisKV hides the tier-crossing latency behind a prediction instead of paying it on demand, which is the missing piece. Its closest prior relative is FlashMemory-LSA (06-09), which trained a Neural Memory Indexer to predict future chunk demand for a 13.5% average footprint; OasisKV gets an equivalent predictor free from a drafter the stack already runs. Nobody has compared a trained indexer against a speculative drafter at matched prefetch budget.

Two caveats this page should carry. First, Eviction as Estimation (08-03) established that cache-management gains appear specifically where reuse is endogenous and time-separated, which describes agent traces and almost no common benchmark; OasisKV reports reasoning and long-context, not agentic traces, and the AgentX distribution (07-25) at a median 140K input tokens per turn is the workload that decides this. Second, the predictor requires speculative decoding, and the Kimi K3 primer (08-04) reported DSpark speculative decoding does not work with pipeline parallelism, so any model too large for one node loses the predictor entirely. No fallback path is reported.

And the same day, the latency side of the same serving stack gets attacked from the opposite direction, with a finding that changes what this page thinks the wall is. SemiAnalysis on TileRT (08-11) measures that at batch size 1 an 8-GPU HGX B200 has 64 TB/s aggregate HBM bandwidth and GLM-5 at NVFP4 needs about 21 GB of active-parameter traffic per token, implying a roofline near 3,047 tokens/s/user that real GPUs come nowhere near. The gap is kernel launch and synchronization overhead plus flat memory latency, not bandwidth: bandwidth improves 2 to 3x per GPU generation while memory latency has not improved at all, so this gap widens every generation. Compiling the entire decode graph into one persistent kernel reaches 500 tok/s/user on one B200, about 3x a GB300 NVL72 on conventional engines. For this page the consequence is that OasisKV and TileRT pull in opposite directions on the throughput-interactivity curve, and a stack running both would need per-request deadline expressiveness that neither vLLM nor SGLang has, which is the gap AAPT (08-04) exposed when it showed deadline-bound GUI agents scoring 0.00 while decoding during execution.

Prior State (as of 2026-08-10)

A third axis arrives, and it is one this page has never priced: whether the cache is still addressable. WorldTrace (08-10) studies visual persistence in interactive video world models, which carry generated frames forward in a growing KV cache, and finds that once a rollout extends past the training horizon the model can no longer reliably retrieve stored content at all. The content is still there. The failure is lookup: the temporal RoPE (rotary positional embedding, which encodes position by rotating query and key vectors through a position-dependent angle) offsets for far-past entries fall outside the range seen in training, so attention cannot find them. Size is fine, bandwidth is fine, and the memory is unreachable.

The second finding is an arithmetic indictment of a common implementation choice. Naively compressing the cache in the RoPE-rotated space corrupts memory by averaging together incompatible positional phases. Any merge, average, or low-rank summarization applied after the rotation blends vectors whose phases differ, producing a result that points where neither original pointed. WorldTrace's fix is training-free and almost embarrassingly cheap: give each compressed summary slot a distinct, in-distribution virtual position, so position becomes an addressing handle you assign rather than a timestamp you inherit. WorldTrace-Field compresses continuous history for temporal coherence (+15.5%); WorldTrace-Landmark stores verbatim scene traces at detected transitions for episodic recall (+19.5% on its new LoopBench, which requires leaving a scene, taking a long detour, and reconstructing it on return).

This composes with, and slightly reorders, the metric argument at the top of this page. SemiAnalysis on Kimi K3 (08-04) argued cache size is not a standalone property and proposed KV throughput, cache size divided by prefill time, as the honest unit. WorldTrace adds a property that neither size nor throughput observes: a cache can be small, fast to fill, and functionally unreachable. Any compression ratio computed in rotated space is a ratio on a corrupted object, which puts this straight into the page's existing measurement-validity thread alongside Eviction as Estimation (08-03), which found KV-eviction ablations do not measure the quantity they are trusted for.

It also undercuts the premise of position-based eviction. Sliding-window and streaming caches evict by position on the belief that position ordering is what attention consumes. WorldTrace shows position is simultaneously the lookup key, so evicting by position destroys the index at the same moment it frees the memory. And it arrives from the opposite direction to Raven (08-04), which keeps a fixed set of memory slots and routes which subset to write, holding recall at 16x training context: Raven's slots are learned at training time, WorldTrace's positions are assigned at serving time, and both are arguing that memory addresses should be allocated rather than inherited from the token stream. Neither cites the other.

Open, and the obvious next experiment: the result is reported on video world models only. Whether text long-context models show the same addressability collapse past the training horizon, separately from the well-studied length-extrapolation loss degradation, is untested. If they do, every production stack that compresses or offloads KV after rotation is silently corrupting its compressed tier, and the fix is a few lines of position bookkeeping.

An independent witness on the same day, from the understanding side rather than the generation side. StreamArena (08-10), a benchmark of 243 videos averaging 88.8 minutes with open-ended questions, reports a three-way streaming-memory tension in which one corner is that "methods that repeatedly compress visual memory struggle to preserve fine-grained details over time." That is WorldTrace's symptom, measured empirically in a different task, on the same day, with no mutual citation, and WorldTrace's virtual-position assignment is a candidate cause and cure for it. That composition is the most valuable untested item on this page.

Prior State (as of 2026-08-04)

This page finally gets the metric it has been missing, and the same article takes back part of the biggest number on it. SemiAnalysis on Kimi K3 (08-04) refuses to score cache efficiency by cache size, on the argument that size is not a standalone property of a model but a consequence of the whole design: no open-weight model ships static KV compression, architecture affects cache efficiency, and how much HBM is left for cache depends on the parallelism strategy, since wide expert parallelism and tensor parallelism leave very different amounts free. The proposed replacement is KV throughput: KV cache size divided by prefill time (time to first token), at a given sequence length. It is the minimum bandwidth needed to serve reliably under prefill-decode disaggregation, and because prefill time encapsulates architectural efficiency the metric walks through memory-bound and compute-bound regimes as context grows. Reported direction: hybrid linear attention's benefit becomes more pronounced as sequence length increases. This is the first metric in this wiki that prices architecture and cache size together, and it is the natural unit for the offload hierarchy the agentic AI memory hierarchy survey (06-07) described, where dominant memory traffic shifts from weights to KV cache as context grows.

The retraction, and it is important. The local coding model report (07-30) measured Nemotron Cascade 2 holding 262K context in under 2 GB of KV against roughly 40 GB for dense Devstral Small 2, concluded the largest available win is architectural and chosen at model-selection time, and this page ran with it. SemiAnalysis adds the production caveat that measurement could not see: the constant-state property belongs to the forward pass, not to a serving system with prefix caching. Inference engines find cache hits by matching the longest cached token prefix, and a recurrent state at position t cannot be reconstructed from a snapshot at t-k without replay, so arbitrary prefix matching would require snapshotting every position, at which point memory grows with sequence length again. Moonshot's fix is coarse: vLLM snapshots KDA state every 32K tokens plus at prompt boundaries, on the reasoning that agentic turns start at prompt boundaries. Their conclusion, stated plainly, is that linear attentions do not consume a constant amount of KV memory in real serving. The 20x is right for one local session and optimistic for a multi-tenant server with prefix reuse, which is the deployment the datacenter pays for.

The hard practitioner number is a cliff, not a slope. InferenceX replayed an hour of real recorded Claude Code traces at steady state rather than fixed-shape synthetic prompts, which is a methodology worth copying: the trace statistics alone are a useful artifact, at a median 142K input tokens and 444 output tokens per turn over a median 65 turns per session, the short outputs being characteristic of harnesses where even an edit is a tool call. Kimi K3 does not fit on a single B200 node, requiring pipeline parallelism, and DSpark speculative decoding does not work with pipeline parallelism, so that path loses speculation entirely. On B300 the model fits one node and HBM holds 3.25M tokens of KV after weights; throughput climbs with batch size until concurrency exceeds 8, exactly where that budget is exhausted, and then prefix cache hit rate collapses below 10% against a theoretical 95%. That is the same shape as TokenPilot (06-16)'s finding that any context edit mutating the prefix forces a full prefill recompute: agentic serving economics are dominated by whether the prefix cache holds, and the failure mode is a cliff. It also sits against the 07-25 AgentX figures reporting a median 99.2% hit rate, and the reconciliation is concurrency: a high hit rate is attainable below the KV budget and unattainable above it.

On the architecture side, the page's sparse-retrieval-over-a-growing-cache pattern gets a fourth instance, and the first one inside the model's parameters. Raven (08-04) (2607.25357, Albert Gu among the authors) keeps a fixed set of memory slots and routes each token's write to a selected subset, decaying only what it touched, which is the training-time twin of what LOCKS (07-29) argued at serving time: a shared representation across all content destroys the content-specific directions that distinguish neighbours, so give each unit its own summary and only touch what you mean to. Everything else on this page reconstructs a relevance signal from a cache after the fact. Raven decides what enters the state in the first place.

Two operational addenda to the SemiAnalysis numbers above. First, the B300 cliff has a workload twin that inverts the whole framing, and it arrived the same day. AAPT (08-04) shows GUI agents failing transient events because autoregressive decode sits on the decision-time critical path, and fixes it by pre-building a bounded policy tree during idle screen periods so execution needs no generation at all. Success inside a contested window rises 0.50 to 0.79 with zero incorrect actions, while open-loop and predict-and-replan baselines both score 0.00 because they still decode during execution. For that class, decode latency is not a cost axis, it is a correctness axis with a step function: below the deadline you succeed, above it you score zero regardless of how right the answer was. Both readings are true of different agent workloads, and nothing in vLLM or SGLang can express a per-request deadline, so a scheduler batching for throughput is exactly wrong for a deadline-bound agent and fails silently rather than slowly. This page has priced latency in dollars and throughput throughout (DraftExpert (08-03) at 1.45x on phones against the 2x-plus datacenters report, LOCKS halving per-token decode latency); AAPT is the first result where the units are wrong.

Second, on the metric: KV throughput is the better version of the standing gap this page named on 07-30, that nobody publishes cache-per-token. Cache-per-token is architecture-blind and KV throughput is not. The comparison SemiAnalysis does not run, and the one that decides real design choices, is KDA-plus-MLA against the GQA-sparse family (GLM 5.2's DeepSeek Sparse Attention, DeepSeek V4's Compressed Sparse Attention, MiniMax M3's sparse attention, MiMo V3's HySparse) on the same axis at the same sequence lengths.

Current State (as of 2026-08-03)

The most useful eviction paper this year reports that its own method does not beat H2O, and the diagnosis indicts the benchmark suite this whole page is scored on. Eviction as Estimation (08-03) (2607.24667, Kurate cs.AI #13) recasts eviction as estimation of a hidden signal, whether an item will be reused, and puts every deployed method on one axis: the commit lag H, the number of steps a policy waits before committing. StreamingLLM, H2O, SnapKV and learned predictors all sit at H=0. Belady's offline optimum sits at H=infinity and is unobservable. The empty middle is fixed-lag smoothing: wait a bounded number of steps, observe which items a correct near-future prediction attended to (the paper calls this demonstrated utility), then commit. The instantiation, RMM, is training-free and reduces to H2O exactly when the measurement is uniform.

Then the honest part. Run inside NVIDIA's KVPress harness against KVPress's own SnapKV, H2O and StreamingLLM implementations, RMM ties H2O on single-turn QA and loses to both on streaming multi-turn. The cause is exact: on natural text the model is correct about almost every token, so weighting attention by correctness is close to the identity, and demonstrated utility collapses onto accumulated attention. The gain appears only where reuse is endogenous (the model's own earlier output determines later consultation) and separated in time, which describes agent traces and describes no benchmark in common use. Two consequences for this page. First, if your workload looks like natural-text QA, the eviction policy choice is nearly free and you should pick the cheapest one, which is consistent with LOCKS (07-29) winning by making selection cheap rather than better. Second, it sits in genuine tension with Make Each Token Count (05-12), which claimed learned globally calibrated eviction can surpass the full cache: one paper says the scoring function has large headroom, the other says almost none, and the variable that would settle it is whether Make Each Token Count's gains survive the KVPress harness against KVPress's own baselines. Nobody has run that. Third, it pairs orthogonally with the 07-28 impossibility result: randomization buys you attribution of the error you created, fixed-lag smoothing reduces the error you create, and the composition (a Poisson-sampled tail with a smoothed commit) does not exist yet. → summary

The agent-side twin of KAP arrived one day later. ACM: Agentic Context Management (08-03) (2607.23809, Meta and CMU) makes the same argument KAP makes about retrievers, but about the agent itself: the standard token-threshold compactor fires because of how much text exists, not because of what the agent is reasoning about, so it is misaligned with the work by construction. ACM gives the agent context-editing tools so it decides when to compress, offloads what it drops to an external memory system rather than deleting it, and queries that store on demand. Reported effects are lower peak token pressure, longer explorations, and more consistent solutions across independent trials, which is a variance claim almost no agent paper reports. The critical missing number for this page: ACM edits the context repeatedly and reports no prompt-cache or prefill accounting, and TokenPilot (06-16) established that any context edit mutating the prefix triggers a full prefill recompute that cancels the saving. An agent that compacts ten times per session may use fewer tokens and cost more money. → summary

Current State (as of 2026-08-02)

Every selection method on this page re-derives importance from inside the model. The first paper to argue the importance signal was destroyed upstream just landed. KAP (08-02) (2607.24260, Kurate cs.LG #2) names the Knowledge Selection-Runtime Consumption gap: a modern LLM system spends real effort producing structured priors before the prompt exists (ranked evidence, graph topology, multimodal alignment, confidence scores), then serializes all of it into a flat token sequence, at which point the serving backend, the component that actually pays for the context, can only consume the KV state densely and uniformly. The consequence is stated as a scaling problem rather than an inefficiency: improving your retriever makes your serving worse, because richer context enlarges the full-prompt KV footprint and decode-time memory traffic together, even when reasoning depends on a small slice. KAP's fix is to compile those priors into a runtime access plan, a universal intermediate representation that governs physical KV access while leaving logical prompt semantics, model weights and training procedures untouched. GraphSpec is the compiler-executor instantiation. Across 4K-128K long-context QA it holds quality comparable to full-context decoding while cutting proposal-time KV access to 5.5% of source KV state at 128K, and the framing the authors reach for is the right one: it decouples physical KV consumption from prompt length rather than moving a point on the existing curve. There is also a derived phase-boundary model for when plan-guided execution is actually faster, which is unusually disciplined for a serving paper.

This reorders how the page should read its own selection line. MISA (05-11) routes on the indexer-head axis, RTPurbo (05-24) found the retrieval geometry lives in a 16-dimensional subspace, MSA (06-12) scores blocks per GQA group, FlashMemory-LSA (06-09) predicts future chunk demand, and LOCKS (07-29) estimates per-page attention mass from a page-local spectral summary without reading any candidate keys. All five reconstruct a relevance signal from the cache after the fact. In a retrieval-augmented or graph-grounded system that signal already existed. So KAP and LOCKS are complements rather than competitors: LOCKS is the method for when nothing upstream knows anything (a raw 1M-token document, an agent trace), KAP is the method for when something upstream knows a lot. Nobody has built the composition, and the fallback is the load-bearing missing piece, because a compiled plan is deterministic and derived from outside the cache, so it cannot certify its own error (the 07-28 impossibility result) and cannot even notice when the upstream prior was wrong. A retriever that ranked the right document third and a plan that reads the top two produce a confident wrong answer with a clean latency profile.

Two cautions on the number. "Proposal-time" implies a draft-and-verify structure, and if verification reads more of the cache the end-to-end saving is smaller than 5.5% suggests. And the quality claim rests entirely on long-context QA, which has a locatable evidence span and is exactly the regime where LOCKS found baseline selectors look fine before collapsing on long-form reasoning (AIME26, MATH-500). A plan compiled from retrieval structure has no obvious reason to survive diffuse reasoning, and no reasoning benchmark is reported.

Current State (as of 2026-07-30)

Practitioners measured a 20x KV cache gap between architectures, which is roughly an order of magnitude larger than anything this page's software methods report, and it reorders the whole optimization stack. The local coding model report benchmarked 9 models across consumer hardware and found NVIDIA's Nemotron Cascade 2 holding a 262K context window with a KV cache under 2 GB, against Devstral Small 2, a dense model, needing roughly 40 GB for comparable context. A 30B model in the set carries a cache 25x smaller than a comparable dense model. The arithmetic inverts the standard advice, because a 30B with a 2 GB cache is a smaller total footprint at long context than a 14B dense model with a 40 GB cache, so the larger model is the one that fits on the card.

This is independent confirmation, from a setting that shares no cost structure, of the claim the hardware memory survey (06-07) made about the datacenter: as context grows, dominant memory traffic shifts from weights to KV cache, making cache management the binding hardware constraint. It is also the same architectural property PrfaaS (04-22) exploits at the opposite scale, where hybrid-attention models emit cache 13x more slowly (MiMo-V2-Flash at 4.66 Gbps against 59.93 Gbps for a dense baseline), letting prefill be offloaded to another datacenter and the cache shipped over Ethernet. Same cause, measured as GB-resident on one card and as Gbps-on-the-wire across sites.

The sequencing lesson is new and it is uncomfortable for this page. July produced three strong software results on cache management: LOCKS (07-29) matching full-cache quality at 100K+ context while reading about 2% of tokens with 2x faster decode, Error Certificates for KV-Cache Eviction (07-28) proving deterministic top-k eviction cannot estimate its own error, and Sparse Event-KV (07-29) proving that dropping a cached fact and observing no accuracy loss does not prove the fact was unnecessary. All three take the cache as given and manage it better. The practitioner data says the largest single win available today is architectural, chosen at model-selection time, and it is roughly 10x what any of them report. Not an argument against the research line, an argument about order: pick the hybrid-attention model first, then apply page selection to what remains.

A second data point that context handling, not reasoning, is the active constraint. OpenAI reported that two settings tripled GPT-5.6 Sol's ARC-AGI-3 score, both of them about memory rather than capability: allow reasoning to continue across multiple context windows, and apply a canonical compaction implementation so it survives the boundary. Taken at face value, two thirds of the previously measured gap on that benchmark was an artifact of discarding reasoning at context boundaries. The caveat raised on social is fair, that other leaderboard entries were measured under different rules, so this is not a clean comparison. It is still a statement that the harness owned the score.

The cache constraint now has a robotics analogue. TurboVLA (07-30) reaches 97.7% on LIBERO with a 0.2B model at 0.9 GB inference VRAM and 31.2 ms latency by removing the language model from the vision-language-action pathway entirely. Different mechanism, same conclusion the local-model report reaches: total resident memory at inference is what determines deployment class, and architecture moves it by more than an order of magnitude.

Standing gap: nobody publishes cache-per-token. It is absent from every model card, it is the number that decides single-GPU deployability, and the gap between the best and worst answer is 20x.

Current State (as of 2026-07-29)

The eviction literature's evaluation methodology is now under attack from two directions in four days, and neither paper appeared on HuggingFace. Error Certificates for KV-Cache Eviction (07-28) proved that deterministic top-k eviction cannot estimate the error it created, and restored a per-step estimate via randomized Poisson eviction at 0.97 coverage. Compute Globally, Materialize Locally (07-29) (2607.23693, Kurate cs.AI #19) attacks the other half: it shows that observing no accuracy loss after dropping a source event does not show the source was unnecessary. Omitting one earlier observation from an otherwise identical agent history, the model's answer on items sensitive to that observation overwhelmingly still follows the omitted value, although no served span states it. The paper calls this semantic materialization: a downstream event's cached rows act as an independently servable view of a computation whose inputs are gone. So the standard eviction experiment (drop entries, measure accuracy, conclude the entries were expendable) is measuring something real and drawing a causal conclusion that does not follow. One paper says you cannot measure the damage; the other says your measurement of no damage does not mean what you think.

The contract it specifies is unusually actionable and unusually uncomfortable. Materialization can be written on purpose: a deliberately phrased, answer-free event raises donor-aligned recovery from 6% to 51% on Qwen3-8B without ever naming the value, while passively harvesting natural mentions from long dialog yields nothing detectable. But capacity is small (compact state survives, larger payloads decay toward chance) and, worst for anyone trying to design around it, whether a construction writes at all turns on phrasing rather than meaning, so two phrasings the model understands equally well can diverge sharply. The property is not semantic, which means you cannot reason your way to it from content.

This is the correctness question behind the workload AgentX (07-25) measured: at a median 140k-in/396-out ratio the cache is the agent's working memory rather than a decode accelerator, so what survives eviction stops being a throughput question. It also gives the Frozen 12B verified-memory result (07-28) an uncomfortable sibling. That paper's store is explicit, addressed, and auditable. Semantic materialization says a second store exists inside the cache rows, unaddressed and uninspectable, and a serving system now has two memories with different reliability properties.

Meanwhile the selection side gets its cleanest engineering result yet, and it exposes that the field has been benchmarking on the easy half of the distribution. LOCKS (2607.24555, Kurate cs.LG #2) starts from a measured structural fact: attention keys are locally low-rank but globally high-rank, so a shared low-rank basis across the whole cache discards exactly the page-specific directions that distinguish one page from its neighbours. LOCKS gives every page its own spectral summary, resident at about a tenth of cache size, reconstructs within-page logits from it, estimates each page's attention mass by log-sum-exp, and attends only the top pages. The decisive design choice is that selection reads no candidate keys or values at all, so selection cost is independent of how much cache sits on the wrong side of the bandwidth wall, which is what caps methods that score blocks by reading a representative key. At a 2048-token budget it matches FullKV aggregate quality at 100K+ context attending ~2% of tokens, halves per-token decode latency (2.0x at 1M), and ships as a drop-in plugin for unmodified vLLM with batched decode in full CUDA graphs.

The result that should change how this page reads its own history: LOCKS's largest margins are on long-form reasoning (AIME26, MATH-500), where baseline selectors collapse, not on the retrieval and long-document QA where the whole selection line (MISA head-axis, MSA blockwise select-then-attend-exactly, RTPurbo query-dependent budgets) validated. Retrieval tasks have a contiguous evidence span a selector can find; long-form reasoning needs diffuse context and has no reason to work under a proxy tuned on retrieval. That baselines collapse there and a per-page basis does not suggests the field's benchmark habit has been hiding a real capability gap. Note also that LOCKS is deterministic top-k with a log-sum-exp mass estimate, so the 07-28 impossibility result says it cannot certify its own error either. The open distinction nobody has drawn: LOCKS selects rather than evicts, so skipped pages stay resident and a mistake is recoverable next step. Whether the certificate result binds reversible selection or only irreversible eviction is the most useful thing to resolve about either paper.

A fifth non-uniform-budget axis, and it is the first semantic one. OmniDelta (2607.25669, HuggingFace) allocates a fixed retained-token budget across modalities in omni-modal models, and its useful contribution is a negative result: direct query-to-audio or query-to-video similarity is unreliable for the inter-modal split, because similarity measures whether a modality is about the query, not whether the answer requires it. It instead reads query intent against per-modality skill pools, then reallocates within each modality by local complexity and temporal redundancy, composing with any existing pruner. At 25% retention on Qwen2.5-Omni-7B: 22.0% less GPU memory, 1.64x end-to-end speedup. The page's non-uniform axes now read head (MISA), head-role (Forcing-KV), value-magnitude (VaSE), layer-function (Rethinking Efficient Attention), and now modality. The first four are properties of the network and can be calibrated offline the way Tangram (06-16) calibrates its head ranking from ~50 samples. Modality is a property of the question, so it cannot be, which puts OmniDelta with KVServe (05-24)'s online-control-surface framing rather than Tangram's static one.

And the sparse-retrieval-over-a-growing-cache primitive escaped the serving stack entirely this week. Wonder (07-29), an interactive video world model, needs to let a user revisit a place seen minutes ago, so it uses a sparse-attention memory that attends to a small relevant set regardless of true context length. That is structurally MSA's select-then-attend-exactly and LOCKS's cheap-selection claim, arrived at from video generation. Three subfields (long-context serving, agentic memory, interactive world models) converging on one mechanism in a quarter is worth naming: sparse retrieval over a growing cache is becoming the default answer to persistence, whatever the modality.

Current State (as of 2026-07-25)

The measured shape of agentic serving arrives, and it is 140k in / 396 out at a 99.2% cache hit rate. Every KV paper this page tracks is implicitly justified by an assumption about the serving distribution, and the industry-standard proxies (8k-in/1k-out, 1k-in/1k-out) are single-turn with random data. SemiAnalysis's AMD Advancing AI 2026 piece introduces AgentX, an InferenceX scenario built by replaying roughly three months of SemiAnalysis's own Claude Code and Codex traces offline at varying concurrency via AIPerf, developed with WEKA, Inferact, RadixArk, LMCache, Mooncake, NVIDIA, and AMD. The measured distribution: median input sequence length 140k, median output 396, median cache hit rate 99.2% (under an infinite-cache assumption, so an upper bound rather than an attainable number). It includes realistic subagent usage and dynamic workflows that continuously inject uncached context, and it captures inter-turn latency from tool use and user think time, which is what determines KV time-to-live and therefore the actual value of KV offloading.

Three consequences for this page. First, it empirically confirms TokenPilot's (06-16) economic claim that prompt-cache hit rate, not token count, is what clears the bill: at 99.2% hit rate on a 140k prefix, a single prefix-invalidating edit costs more than any plausible token saving. Second, it reframes the whole eviction/compression literature: with a 140k:396 in/out ratio, agentic serving is a prefill-and-retention problem, not a decode-throughput problem, which is the workload argument for prefill/decode disaggregation rather than a systems preference. Third, inter-turn think time is the variable that decides whether a KV block should be offloaded to a cheaper tier or dropped, which is exactly the KV-aware tiering policy memory-hierarchy has listed as an open problem and not-yet-shipped serving feature.

Relatedly, Claude Opus 5 (07-25) fixed one specific prefix-invalidating event at the platform layer: adding or removing tools mid-conversation no longer invalidates the prompt cache. Tool-definition churn was one of the concrete prefix breaks TokenPilot's Ingestion-Aware Compaction was designed to work around from the client side; Anthropic moved it into the API contract.

Disaggregation's composability, not its existence, is now the bottleneck. The same SemiAnalysis piece frames the competitive frontier as composable distributed inference: prefill/decode disaggregation, WideEP, RDMA KV transport, and cluster scheduling working together by default rather than per-model. Concrete numbers: an 8,192-token DeepSeek-R1 FP8-KV prefill moves roughly 290MB of KV over RDMA; moving DeepSeek-R1 (256 routed experts) from EP8 on one node to EP64 across 64 GPUs cuts experts per GPU from 32 to 4, freeing HBM for KV cache and enabling larger concurrent batches, with NVIDIA reporting up to 2.28x higher per-GPU output throughput from Wide-EP. Expert parallelism is therefore a KV-capacity lever, not only a compute-sharding one, which is a link this page had not made explicit. AMD's MoRI H2-2026 roadmap makes the tiering target concrete: a tiered distributed KV cache (HBM → DRAM → NVMe via SPDK/GDS) with scheduler co-design. And the cautionary datapoint: composition failures show up as silent accuracy cliffs, not crashes. An uninitialized reduce buffer in AITER's FP4 MoE kernel produced fluent-but-wrong output scoring 0 on GSM8K (patched June), and EP decode at concurrency 64 in SGLang with the MoRI backend still drops GSM8K to ~80% against a ~94% baseline that holds at every other batch size.

Current State (as of 2026-06-17)

Three structural KV levers the same day: a layer-axis allocation prior, a width-axis saving, and a train/inference KV-consistency rule. (1) Rethinking Efficient Attention in Hybrid Architectures (arxiv 2606.15378) shows long-range retrieval is localized to the full-attention layers, with the efficient (SWA/linear) layers shaping the optimization path — a layer-axis allocation prior the non-uniform-KV line (Tangram, MISA) can exploit directly: compress the efficient layers aggressively, protect the full layers. (2) Variable-Width Transformers (arxiv 2606.18246) narrows the middle layers (wide ends, narrow middle), yielding ~15% smaller KV memory and I/O as a structural saving baked into the architecture, not a serving-time compression. (3) LoopCoder-v2 (arxiv 2606.18023) holds the KV footprint roughly constant across loop count via Shared-KV Gated Sliding-Window Attention (share the first loop's KV, blend global/local), making test-time-compute depth a free knob rather than a linear KV penalty. Separately, GLM-5.2's release surfaced a concrete train/inference KV-consistency result (via @eliebakouch): at the t+2 multi-token-prediction head it deliberately omits the indexer KV of the t+1-predicted value ("indexer sharing"), which is both cheaper and better because it removes a distribution shift on that specific cache entry. The page's principle list gains: KV consistency between training and inference matters as much as raw cache size, and KV budget should follow layer function (retrieval in full layers, compress the rest).

Current State (as of 2026-06-16)

The serving substrate finally catches up to head-non-uniform compression, and the cost target shifts from token count to cache continuity. Two papers land the systems-and-economics layer the page's algorithm-heavy backlog needed. Tangram (arxiv 2606.06302, on vLLM) explains why non-uniform KV compression — heterogeneous per-head budgets, which the page has tracked via MISA (head-axis), Forcing-KV (head-role), and VaSE (value-magnitude) — has stayed in papers: serving stacks assume uniform KV length per head, so heterogeneity traps freed memory as page fragmentation, burns up to 25% of prefill on page reclamation, and inflates decode latency up to 1.7x. Tangram's key empirical claim is that head-wise retention follows a two-level structural regularity (an input-invariant head ranking with narrowly bounded per-head ratios) calibratable offline from ~50 samples — so it resolves statically (Budget Reservation, Ragged Paging, Ahead-of-Time Load Balancing) what prior systems handled dynamically, matching the wrapped method's accuracy at up to 2.6x throughput over full-KV. This is the execution-side complement to KVServe (05-24, compression profile as an online control surface): KVServe picks the profile dynamically, Tangram exploits non-uniform input-invariance to fix it offline. The "input-invariant head ranking from 50 samples" is the same outlier-is-locatable physics as RTPurbo (16-dim retrieval subspace) and LongAct (saliency peaks), now turned into a scheduling primitive.

TokenPilot (arxiv 2606.17016, Zhejiang U et al., in LightMem2) names the economic tension the SemiAnalysis 05-01 note implied: agent context management that optimizes token count alone mutates the prompt sequence, which breaks the prompt prefix the backend was caching and triggers a prefill recompute that cancels the savings. Its fix is dual-granularity — Ingestion-Aware Compaction stabilizes prefixes at the ingestion gate; Lifecycle-Aware Eviction defers structural offload until a segment's residual utility expires on a batch-turn schedule — for 61–87% cost reduction at competitive performance. TokenPilot is the agent-trajectory sibling of KV Packet (04-17, immutable cacheable document packets): KV Packet preserves cacheability for retrieved docs, TokenPilot for the agent's own growing trace. The page's cost axis now reads: token count and prompt-cache hit rate, the latter being the one that actually clears the bill.

Current State (as of 2026-06-12)

The MiniMax-M3 sparse-attention engine gets its paper — and audited numbers. MiniMax Sparse Attention (MSA) (arxiv 2606.13392) is the mechanism behind the MiniMax-M3 release we logged on 06-03 from a vendor blog. MSA is blockwise sparse attention on a GQA backbone: a lightweight Index Branch scores KV blocks and selects a Top-k subset per GQA group, then the Main Branch does exact attention over only those blocks. The paper's measured figures refine the 06-03 marketing claims (≈1/20 compute, ~9x prefill, ~15x decode): 28.4x per-token attention-compute reduction at 1M context on a 109B model, on par with dense GQA, but a more grounded 14.2x prefill / 7.6x decode wall-clock on H800 with the co-designed kernel (exp-free Top-k, KV-outer layout for tensor-core utilization). The compute-vs-wall-clock gap is exactly the kernel-efficiency caveat the 06-03 note flagged. MSA is firmly in the "select blocks, attend exactly" camp the page tracks against training-free eviction (VaSE) and predicted-demand pruning (FlashMemory-LSA) — and it is now the one shipping as open weights, confirming the RTPurbo (05-24) "full attention is intrinsically sparse, useful budget is query-dependent" thesis at production scale.

Current State (as of 2026-06-11)

The video-agent line gets a token-preserving KV reparameterization. InternVideo3 (arxiv 2606.12195) introduces M²LA (Multimodal Multi-head Latent Attention), which compresses the per-token KV state into a low-rank latent while keeping the full token stream. That distinction is the point: the page's prior video-KV methods — VideoMLA (06-02), WorldKV (05-24), StateKV (06-01) — compress mostly by evicting or merging tokens, but InternVideo3 is built around a closed-loop video agent that may re-query an earlier frame via retrieval tools, and you cannot retrieve a token you evicted. So M²LA shrinks the KV state and preserves addressability, trading a smaller-per-token cache for keeping all tokens resident. It is MLA (the DeepSeek low-rank-latent-KV idea) carried into the multimodal, multi-head, long-video, agentic regime, and it pairs with yesterday's input-side Latent Memory (one latent token per evidence item) as the two token-policy extremes for affording a long-horizon multimodal agent's evidence. Open lever: whether the latent KV can be selectively retrieved into rather than kept fully resident, turning compression into retrieval-addressable video memory.

Current State (as of 2026-06-09)

Two opposite answers to the 500K-context memory wall land the same day: drop the KV chunks, or never form them. FlashMemory-DeepSeek-V4 (LSA, arxiv 2606.09079) keeps the cache but prunes it predictively: a Neural Memory Indexer, trained backbone-free as a dual-encoder retrieval model, predicts which KV chunks future queries will need and keeps only those resident. Average physical KV footprint drops to 13.5% of full-context, >90% at 500K, with +0.6% average accuracy across LongBench-v2, LongMemEval, RULER. This is the next step past RTPurbo (05-24, 16-dim retrieval subspace suffices) and MISA (05-11, head-axis indexer routing): LSA routes on a predicted-future-demand axis and trains the indexer without ever loading the backbone. Latent Context Language Models (LCLMs, arxiv 2606.09659) take the opposite route: skip the KV cache entirely with encoder-decoder compression. After an architecture search, they continually pre-train 0.6B-encoder / 4B-decoder models on 350B+ tokens each at 1:4 / 1:8 / 1:16 compression, pushing the Pareto frontier past KV-cache compression on accuracy-efficiency, and serving long-horizon agents that skim the compressed context then expand relevant segments on demand. This vindicates the trained-compressor argument LongAttnComp (06-02) made (trained scorer beats training-free heuristics) and generalizes it to a from-scratch encoder-decoder. The page now holds both poles of long-context efficiency: cache-side predictive eviction (LSA, VaSE, Conf-KV) and input-side trained compression (LCLM, LongAttnComp). Both are answers to the Ken Huang memory survey fact that KV cache, not weights, is the binding memory constraint as context grows.

Current State (as of 2026-06-06)

Redundancy gets killed at the input interface, not the cache (AdaCodec). AdaCodec (arxiv 2606.02569) is the input-side sibling of the page's redundancy theme. Where KV-cache eviction asks "which stored keys/values are redundant given everything in context," AdaCodec asks the same question one step upstream for video MLLMs: which frames are redundant given prior frames. Borrowing predictive coding from video codecs (send a full reference frame only when the scene cannot be predicted; otherwise send compact P-tokens of motion + residuals), it beats the Qwen3-VL-8B per-frame RGB baseline at matched token budget, and at 1/7 the budget (32k vs 224k tokens) surpasses it on all long-video benchmarks while cutting time-to-first-token from 9.26s to 1.62s. This complements the StateKV (06-01) / Echo-Infinity (06-04) video-memory line: those compress the history representation inside the model; AdaCodec compresses the visual tokens before they enter it. Same "don't pay for redundant temporal information" physics, different layer.

Current State (as of 2026-06-04)

Learned memory replaces handcrafted KV scheduling for infinite video (2026-06-04). Echo-Infinity (arxiv 2606.04527) is the video-generation instance of the page's biggest pattern: the move from handcrafted cache management (Make-Each-Token-Count learned retention, VaSE's value guard) to a fully learned one. Autoregressive video generators bottleneck on history; existing systems manage it with predefined KV-cache schedules, fixed-ratio compression, or inference-time RoPE adaptation, all of which lose information and amplify compounding error. Echo-Infinity instead trains a small set of Memory Queries end-to-end with the diffusion transformer; they update via attention + gating whenever frames are evicted from the local window, giving constant compute at any compression ratio independent of video length. A Unified Relative RoPE Recipe (sink frames anchored to id 0, newest frame capped at the pretrained max RoPE id) closes the train-test extrapolation gap, enabling a claimed-first 24-hour, >1.3M-frame real-time rollout. Closest to StateKV (06-01, fixed recurrent state as cross-frame memory) but trained rather than training-free, and the memory-as-prior result (optimized initial query state alone improves quality) is the surprise worth tracking.

Merging is also expert-read-bound (2026-06-04). MergePipe (Access Sets Matter, arxiv 2605.29489) is not a cache method but shares the page's "memory bandwidth is the wall" physics (dMoE, 06-01): at LLM scale the limiting resource in weight-space model merging is reading expert delta blocks, so it recasts merging as a budgeted access-set problem (skip low-norm deltas, bounded error), cutting read I/O an order of magnitude for up to 11x speedup. The "skip low-norm blocks" rule is the same sparse-and-locatable instinct as VaSE's large-magnitude value guard, applied offline to merging.

Current State (as of 2026-06-03)

Eviction finally matches selection on reasoning, via two new axes (2026-06-03): a value-magnitude guard and stochasticity. VaSE (Value-Aware Stochastic KV Cache Eviction, arxiv 2606.03928) resolves the long-standing gap the page has tracked — eviction methods kept losing accuracy to selection-based sparse attention that keeps the full cache. VaSE's two training-free findings: (1) a small fraction of value states carry abnormally large magnitudes and evicting them is catastrophic (the model falls into repetitive reasoning loops), so they must be protected; (2) making the eviction decision stochastic rather than deterministic keeps the surviving cache diverse and raises accuracy at the same budget. On Qwen3 at 4x compression across six reasoning tasks, VaSE beats the strongest eviction baseline by >4% and edges out the SOTA selection method, with FlashAttention2 support and a static memory footprint. The value-magnitude result is the same outlier physics the wiki has now seen from three directions: LongAct (04-18, high-magnitude Q/K mark where attention works) and the quantization-outlier line (OSCAR/TurboQuant, where high-magnitude states resist low-bit quantization). VaSE extends it to the value side and to eviction: the same outlier states that resist quantization also must not be evicted. The eviction control axes now read: learned per-token retention (Make-Each-Token-Count, 05-12), per-step confidence budget (Conf-KV, 05-30), per-head role pruning (Forcing-KV, 05-15), and now value-magnitude guard + stochasticity (VaSE). None has been composed with the others.

Current State (as of 2026-06-02)

New axis (2026-06-02): the per-head KV layout itself, via MLA for video diffusion. VideoMLA (arxiv 2605.30351) is the first study of Multi-Head Latent Attention (MLA) in video diffusion. Every prior video-KV paper the wiki tracks changed what the cache holds (which tokens, which heads, how much budget); VideoMLA changes the layout underneath all of them, replacing per-head keys and values with a single shared low-rank content latent plus a shared decoupled 3D-RoPE positional key, cutting per-token KV memory 92.7% at every layer and lifting throughput 1.23x on a B200. Its most important finding is a correction to MLA folklore: pretrained video attention is not low-rank (99%-energy effective rank far above any practical latent dim), yet MLA still works because the MLA bottleneck dimension — not the pretrained spectrum — determines effective rank. Both spectral and random init fill the full rank budget from step 0; training adapts within it. If this holds, MLA-style compression should transfer to modalities whose attention is not low-rank, decoupling MLA from the spectral-assumption story it was sold with.

Input-side companion (2026-06-02): trained context compression beats training-free. LongAttnComp (arxiv 2606.01336) compresses the 100k+-token prefill input rather than the generation cache, the input-side complement to VideoMLA's generation-side compression. The headline argument is against the training-free orthodoxy: pure attention-heuristic compressors leave large gaps on code reasoning, so LongAttnComp fine-tunes a lightweight cross-attention scorer (plus token-level chunking, token-budget top-p selection, positional reordering, format-agnostic query parser) and a two-stage recipe (NIAH retrieval foundation, then multi-hop/reasoning). It matches or exceeds full-context accuracy on InfiniteBench Code-Debug and transfers across four target models from three families. Same "keep only the load-bearing tokens" instinct as Make-Each-Token-Count (05-12) and the broader efficiency thread, but argues a trained scorer is required where training-free attention heuristics fail.

Current State (as of 2026-06-01)

New axis (2026-06-01): fixed recurrent state as a training-free cross-frame memory. StateKV (Linear Scaling Video VLMs, arxiv 2605.31598) adapts a pretrained long-video VLM to linear-time prefill at inference, with no fine-tuning and no architecture change. It carries cross-frame context in a fixed-capacity, importance-based recurrent state, paired with a second full per-frame cache used only for decoding. This is the linear-attention/state-space idea (compress history into a fixed-size state) applied as a wrapper on top of a model pretrained with quadratic attention. Across three long-video benchmarks and seven models in three families it stays close to full self-attention and beats sliding-window/recency streaming approximations, and cuts prefill FLOPs so a larger model fits a fixed compute budget. Significance: the video KV thread now has compression at the encoder (EarlyTom, 05-30), per-head role pruning (Forcing-KV, 05-15), evicted-KV-as-retrievable-world-memory (WorldKV, 05-24), confidence-driven per-step budget (Conf-KV, 05-30), and now a fixed recurrent state for cross-frame memory. StateKV is the only one that replaces the attention math itself (quadratic to linear) rather than managing what the cache holds.

Memory-bound MoE companion (2026-06-01): block-coherent expert routing. dMoE (arxiv 2605.30876) is not a cache method but shares the binding constraint: in diffusion LLMs with mixture-of-experts, block-parallel decoding plus independent per-token routing forces the machine to load the union of every expert any token in a block selected, making inference memory-bound. Aggregating token-level expert distributions into one block-level distribution drops uniquely activated experts from 69.5 to 14.6, cuts memory 76-80%, and retains 99.11% of performance. The same "memory bandwidth, not FLOPs, is the wall" framing that drives KV compression (SemiAnalysis 05-01 prompt-cache economics) now drives expert routing.

Current State (as of 2026-05-30)

New axis (2026-05-30): confidence-driven per-step cache budget. Conf-KV converts the next-token distribution into a scalar confidence score (one minus normalized entropy) and uses that score to set the cache budget step-by-step: low confidence keeps more context, high confidence prunes hard. Within each step's budget, tokens are ranked by accumulated attention mass plus recency, with a protected recent window for local coherence. Storage is mixed FP16/INT8 and the per-layer budget is pyramidal. On Needle-in-a-Haystack at 32K tokens, retrieval is 91.4% versus 53.8% for sliding-window and 80.6% for H2O. VisualWebArena retains 95.3% of full-KV success at 2.8x lower peak memory. Significance: the wiki now tracks three orthogonal control axes for eviction — Make-Each-Token-Count's learned per-token retention (05-12), Forcing-KV's per-head role pruning (05-15), and now Conf-KV's per-step confidence-driven budget (05-30). All three are forward-or-structure-looking signals beyond the recency / attention-history baseline.

Video-side companion (2026-05-30): in-encoder token compression. EarlyTom shifts video-LLM token compression upstream into the vision encoder itself, where the wiki had previously tracked compression only at the cache (WorldKV, Forcing-KV, Stream-T1) or at the LLM input (MotionCache). The vision encoder dominates time-to-first-token on Video-LLMs; EarlyTom is training-free, cuts TTFT by up to 2.65x and FLOPs by 61% on LLaVA-OneVision-7B at A100, with accuracy comparable to the full-token baseline. The video efficiency stack now has compression at every stage.

Current State (as of 2026-05-27)

New axis (2026-05-27): iterative offline consolidation. Language Models Need Sleep (CMU/UMD, Twitter-surfaced) separates scalable memory from scalable reasoning and adds the missing consolidation axis. Existing SSM-attention hybrids can store long-range info in fixed-size fast weights but degrade as reasoning depth rises even at constant information load: the bottleneck is computational, not capacity. "Sleep" runs N learned recurrent passes over accumulated context offline, writes the result into the SSM fast weights via a local rule, then clears the KV cache. Wake-time prediction reads the consolidated weights at normal latency. Increasing N improves accuracy, most on reasoning-heavy examples, on tasks where a plain transformer and an SSM-attention hybrid both fail. This is the inference-time twin of the agent-memory decoupling: MemForest (05-26) decoupled memory construction from the inference loop; Sleep decouples the expensive folding-in (offline) from fast answering (online). Prior consolidation in the cache thread (δ-mem's single-pass associative state, Make-Each-Token-Count's eviction) was single-pass; Sleep makes it iterative.

Agent-side companion (2026-05-27): state-adaptive recall. SAM: State-Adaptive Memory keeps raw trajectory pages plus compact memory cues that act as handles for intent-driven reconstruction, with recall conditioned on the agent's evolving state, backbone frozen. Together with Sleep and MemForest, three memory framings in two days (weight-level consolidation / data-structure / state-conditioned recall) all reject the flat-global-summary baseline.

Current State (as of 2026-05-24)

Latest additions (2026-05-24): KVServe, Gated DeltaNet-2, WorldKV, RTPurbo. Four KV-relevant entries today, all Tier 1.

KVServe (summary) is the first service-aware adaptive KV cache compression framework for disaggregated LLM serving. It unifies quantization, lossless coding, and data transformations into one modular strategy space; a Bayesian Profiling Engine distills a 3D Pareto candidate set at 50x less offline search; and an online controller combines an analytical latency model with a contextual bandit to pick a profile under runtime SLO and bandwidth constraints. Up to 9.13x JCT speedup in PD-separated serving and 32.8x TTFT reduction in KV-disaggregated serving. The reframe: compression configuration is a first-class control surface, not a static hyperparameter.

Gated DeltaNet-2 (summary) decouples the single scalar gate in the delta-rule family into a channel-wise erase gate b_t and a channel-wise write gate w_t. The single scalar previously controlled two different things (how much old content to erase on the key side, how much new content to commit on the value side). At 1.3B parameters trained on 100B FineWeb-Edu tokens, beats Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants on language modeling, commonsense reasoning, and retrieval, most pronounced on long-context RULER needle-in-a-haystack. The chunkwise WY algorithm absorbs the channel-wise decay into asymmetric erase factors so parallel-training cost matches the prior generation. Code: NVlabs/GatedDeltaNet-2.

WorldKV (summary) is a training-free framework for sustaining a persistent world in autoregressive video diffusion models. World Retrieval stores evicted KV-cache chunks in GPU/CPU memory and retrieves scene-relevant chunks via camera and action correspondence, inserting them back into the native attention window without re-encoding. World Compression prunes redundant tokens per chunk via key-key similarity to an anchor frame, halving per-chunk storage. Matches or exceeds full-KV memory fidelity at roughly 2x throughput on Matrix-Game-2.0 and LingBot-World-Fast. The wiki now tracks KV-as-memory in four video-side papers: MotionCache (05-05), Stream-T1 (05-07), LongLive-2.0 (05-19), WorldKV (05-24).

RTPurbo / Full Attention Strikes Back (summary) argues full-attention LLMs are already intrinsically sparse and convert to sparse in only a few hundred training steps. Three observations: only a small subset of attention heads truly requires full long-context processing; long-range retrieval is governed by a low-dimensional subspace (a 16-dimensional indexer is sufficient); the useful token budget is query-dependent, so dynamic top-p beats fixed top-k. Up to 9.36x prefill speedup at 1M context and roughly 2.01x decode speedup, near-lossless. The 16-dimensional retrieval-subspace finding is the load-bearing claim: the long-context retrieval geometry, learned implicitly during full-attention pretraining, lives in a tiny subspace.

Prior State (as of 2026-05-19)

Additions (2026-05-19): CompactAttention, EndPrompt, LongLive-2.0 NVFP4 KV cache. Three KV-relevant entries today. CompactAttention (summary) attacks the chunked-prefill regime where prior block-sparse machinery loses efficiency because the chunk size caps the Q-length. The structural move: treat 2D block-sparse masks as KV-selection signals rather than direct sparse-kernel execution plans, then convert them via Q-block union and intra-group GQA union into minimal block tables under paged execution. 2.72x attention speedup at 128K context on LLaMA-3.1-8B-Instruct with near-dense RULER accuracy. EndPrompt (summary) extends an LLM's context window using only short training sequences via a two-segment construction (original short context as segment 1, brief terminal prompt as segment 2 with positional indices placed near the target length). Beats LongLoRA and full-length fine-tuning on RULER and LongBench at substantially lower compute, with a RoPE-and-Bernstein-inequality smoothness argument for why sparse positional supervision suffices. LongLive-2.0 (summary) is the wiki's first end-to-end NVFP4 video training and inference stack. Quantizes the KV cache to NVFP4 on Blackwell for memory savings; on non-Blackwell, SP inference with quantized KV cache lowers SP inter-GPU communication. 2.15x training speedup, 1.84x inference speedup, 45.7 FPS at 5B.

Prior State (as of 2026-05-16)

Addition (2026-05-16): Lighthouse Attention pre-training wrapper. Nous Research ships a training-only, kernel-decoupled wrapper around ordinary FlashAttention for causal-transformer pre-training at extreme context length. Queries, keys, and values are pooled symmetrically into a multi-resolution pyramid; a gradient-free top-k cascade selects a hierarchical dense sub-sequence; a sorting pass keeps left-to-right causality. The wrapper is removed in a short recovery phase, leaving a standard dense-attention model. Claimed 1.4-1.7x wall-clock speedup at 98K context and ~17x forward+backward at 512K on a single B200. The structural novelty: pre-training attention selection is now a programmable substrate, the same framing the wiki has applied to the inference-time cache. → summary

Prior State (as of 2026-05-15)

Latest addition (2026-05-15): Forcing-KV for autoregressive video diffusion + async continuous batching. Two pieces of the inference stack land the same day. Forcing-KV finds that attention heads in AR video diffusion models (Self Forcing family) cluster into two stable functional roles across samples and denoising steps: static heads (chunk transitions, intra-frame fidelity) tolerate structured pruning; dynamic heads (inter-frame motion, temporal consistency) require segment-similarity-based pruning. Role-conditioned hybrid compression delivers 29+ fps on single H200 at 30% memory reduction, 1.35-1.50x speedup at 480P scaling to 2.82x at 1080P. The cache thread is now policy-aware in three forms: learned eviction (Make Each Token Count), shared coordination (Orthrus), and head-role compression (Forcing-KV). → summary. The HuggingFace asynchronous continuous batching post is the scheduling-layer complement: three CUDA streams (H2D, compute, D2H), CUDA events for handoff, two parallel buffer slots A/B so the CPU prepares batch N+1 while the GPU computes batch N. GPU utilization rises from 76.0% to 99.4%, 22% generation speedup, no kernel or model changes. → summary

Prior state (as of 2026-05-14)

2026-05-14: Orthrus dual-view diffusion on shared cache. Orthrus runs an autoregressive head and a diffusion head on the same frozen LLM, both attending to a single shared KV cache. The AR head executes pre-fill and populates the cache at full fidelity; the diffusion head reads from that same cache to draft tokens in parallel; an exact-consensus mechanism between the two views makes the output bit-identical to the AR baseline. Up to 7.8x speedup with O(1) memory cache overhead. The structural novelty: the cache is the shared coordination object, not a verification ledger. Composes naturally with Make-Each-Token-Count: same cache, both selectively retained and parallelly read. → summary

Companion (2026-05-14): MMProLong long-context VLM recipe. First long-context VLM training recipe in the wiki. Three findings: long-document VQA beats OCR transcription; balanced sequence-length distribution beats target-length-focused; retrieval is the long-context bottleneck. 5B-token long-context continued pretrain extends Qwen2.5-VL-7B from 32K to 128K with generalization to 256K and 512K. The training-side complement to Make-Each-Token-Count's inference-side claim. Both say: long context rewards balance and structure, not volume. → summary

2026-05-13 additions: δ-mem and FocuSFT. Two papers attack long-context inefficiency from orthogonal angles. δ-mem augments a frozen full-attention backbone with a compact 8x8 associative-memory state updated by the delta rule; its readout produces low-rank corrections to the backbone's attention computation. 1.10x average gain over the frozen backbone, 1.31x on MemoryAgentBench, 1.20x on LoCoMo. Composes with Make Each Token Count: aggressive eviction at the cache, associative signal retained in the small online state. FocuSFT identifies attention sinks as a training-side phenomenon (not just inference), shows that standard long-context SFT lets positional biases starve content tokens of attention budget, and fixes it with bilevel optimization (inner loop sharpens attention via fast-weights, outer loop runs SFT conditioned on sharpened representation; bidirectional context with causal response masking removes the sink-creating asymmetry). Up to +14 points on BABILong, 529x sink-mass reduction. Make Each Token Count and FocuSFT together bracket attention dilution: training-side cause + inference-side fix. → δ-mem summary · FocuSFT summary

Prior addition (2026-05-12): Make Each Token Count. A learned, globally calibrated KV-eviction policy that can surpass the full cache, not just approximate it. The framing flip: in long contexts, the full cache is not the ceiling because irrelevant tokens dilute attention. Lightweight retention gates score each cached entry, a shared final scoring projection calibrates scores across every layer and head, and a single global memory budget lets tokens from different layers, heads, and modalities compete for cache capacity. Theoretical analysis shows that preferentially retaining useful tokens reduces attention dilution. This is the language-model analogue of Stream-T1's content-aware video KV eviction (2026-05-07): both treat eviction as a quality intervention, not a compression tradeoff. Composes with MISA (head-axis routing) and TurboQuant (low-bit quantization). → summary

Prior additions (2026-05-11): Two papers attack long-context inference from different angles on the same day. MISA introduces a Mixture of Indexer Sparse Attention: it treats the 64 query heads inside DeepSeek Sparse Attention's indexer as an MoE pool and routes a small active subset (h=8) per query via cheap block-level statistics. Drop-in, no extra training, 92 percent of the tokens DSA would have selected, 3.82x kernel speedup over the original DSA indexer kernel on H200. The new axis here is the head axis: prior sparse-attention work routed on tokens, MISA routes on indexer heads. UniPrefill ships as a vLLM operator with extended continuous-batching scheduling: block-wise dynamic sparsification at the token level that is architecture-agnostic (works on full attention, linear-and-full hybrids, sliding-window hybrids). Up to 2.1x TTFT with the speedup growing as concurrent request count grows, which is the signature of a serving-system optimization. MDN: Momentum DeltaNet is the substrate-level update inside linear attention: it parallelizes stepwise momentum updates via geometric reordering, then uses spectral analysis of the resulting second-order recurrence to constrain gating for stability. Comparable training throughput to Mamba2 and KDA at 400M and 1.3B, beats Transformer / Mamba2 / GDN across downstream tasks. The recurrent rule is now a research surface, not a fixed substrate.

Prior state (as of 2026-05-07)

KV caching is standard in all production LLM serving. Active research is focused on four problems: (1) making caches reusable across contexts without recomputation, (2) compressing the cache to reduce memory footprint, (3) smarter eviction policies when the cache is full, and (4) extending cache-based acceleration patterns (like speculative decoding) to non-text modalities. The parallel daily digest (04-22) introduced two major KV-focused papers, TurboQuant (ultra-low-bit compression) and PrfaaS (cross-datacenter disaggregation via hybrid attention), signaling that the KV cache is now the primary optimization target in production serving. MotionCache (2026-05-05) extends the same iteration-as-optimization-unit principle to autoregressive video generation, using inter-frame motion deltas to decide which pixels need full denoising. Stream-T1 (2026-05-07) introduces the first content-aware KV eviction policy in the wiki: for streaming video diffusion, KV slots are routed through reward-feedback pathways instead of recency-based eviction. LIVEditor / ISA (2026-05-07) routes attention by Query sharpness, sending high-error queries to full attention and low-error queries to a 0-th order Taylor sparse path, achieving ~60% attention-module latency reduction on video editing. The pattern is now visible across six substrates: text KV reuse (KV Packet), KV quantization (TurboQuant), KV transport (PrfaaS), video denoising reuse (MotionCache), content-aware video KV eviction (Stream-T1), and Query-sharpness sparse attention (ISA). The shared principle is that the iteration unit has heterogeneous information density and should be allocated proportionally.

Economic context (SemiAnalysis 05-01): the unit economics of frontier model labs now depend on >90% prompt-cache hit rates. Anthropic's blended price for Opus 4.7 on agentic workloads is ~$0.99/MTok (vs $5/$25 sticker) because cached input tokens dominate. Cache compression / reuse research is now financial-impact-driven, not just academic.

Key Papers

Make Each Token Count (2026-05-12) — Learned, globally calibrated KV-cache eviction with retention gates per cached entry, a shared final scoring projection that calibrates scores across all layers/heads, and a single unified memory budget across layers, heads, and modalities. The theoretical claim is that the full cache is not optimal in long contexts because irrelevant tokens dilute attention away from useful evidence; selective eviction reduces dilution. Matches or surpasses full-cache inference across long-context language, vision-language reasoning, and multi-turn dialogue benchmarks. First paper in the wiki to formally claim eviction improves quality, not just preserves it. → summary

MISA (2026-05-11) — Mixture of Indexer Sparse Attention. Treats the 64 indexer query heads of DeepSeek Sparse Attention as an MoE pool, a cheap block-level router picks h=8 active heads per query. Reduces per-query indexer cost from O(H^I * L) to O(h * L + H^I * M). Recovers 92 percent of DSA's selected tokens at 8x fewer active indexer heads, 3.82x kernel speedup on H200. Drop-in, no training. The first paper in the wiki to route sparse-attention on the head axis rather than the token axis. → summary

UniPrefill (2026-05-11) — Architecture-agnostic prefill accelerator via block-wise dynamic sparsification, implemented as a continuous-batching operator inside vLLM with native prefill-decode co-processing and tensor parallel. Up to 2.1x TTFT speedup, speedup grows with concurrent request count (a serving-system signature, not a single-request one). Works on hybrid architectures where prior sparse-attention prefill methods degrade. → summary

MDN: Momentum DeltaNet (2026-05-11) — Parallelizes stepwise momentum for delta linear attention via geometric reordering of update coefficients. Spectral analysis of the second-order recurrence constrains gating for stability. Triton kernel matches Mamba2 / KDA training throughput. At 400M and 1.3B, beats Transformer / Mamba2 / GDN on broad downstream evals. First substrate-level update to linear attention recurrent rule the wiki has tracked. → summary

KV Packet (2026-04-17) — Eliminates recomputation-on-reuse entirely. Wraps cached documents as immutable packets with lightweight soft-token adapters (trained via self-supervised distillation) that bridge context shifts. Near-zero FLOPs, lower TTFT than all recomputation-based baselines (CacheBlend, EPIC, SAM-KV). → summary

LongAct (2026-04-18) — Identifies high-magnitude activations in Q/K vectors during long-context processing. These "saliency peaks" (same ones that trouble quantization) are the positions where attention is doing real work. LongAct restricts RL gradient updates to only those weights, yielding ~8% gain on LongBench v2 with universal compatibility across GRPO and DAPO. Bridges the KV saliency insight from quantization research into RL training. → summary

TurboQuant (2026-04-22, via parallel digest) — Google (ICLR 2026). Online vector quantization: randomly rotates input vectors to induce a concentrated Beta distribution, applies optimal scalar quantizers per coordinate, followed by a 1-bit QJL transform on the residual for an unbiased inner product quantizer. Absolute quality neutrality at 3.5 bits/channel; marginal degradation at 2.5 bits/channel; 6x+ KV cache memory reduction. Community integrations with vLLM and llama.cpp appearing despite no official implementation.

PrfaaS / Prefill-as-a-Service (2026-04-22, via parallel digest) — Moonshot AI + Tsinghua. Offloads long-context prefill to standalone compute-dense clusters in separate datacenters, transfers resulting KV cache over Ethernet. Enabled by hybrid-attention models (Kimi Linear, MiMo-V2-Flash, Qwen3.5-397B) that mix full-attention + linear-complexity layers. MiMo-V2-Flash produces KV cache at 4.66 Gbps vs 59.93 Gbps for dense-attention baseline (13x reduction). 54% higher throughput, 50% lower mean TTFT vs homogeneous baselines.

SDVG (2026-04-22) — Extends speculative decoding to continuous video generation. A 1.3B drafter proposes video blocks in 4 denoising steps; ImageReward scores per block using worst-frame aggregation; accepted blocks enter the 14B target's KV cache directly. 1.59x speedup at 98.1% quality; 2.09x at 95.7%. Training-free. → summary

MotionCache (2026-05-05) — Motion-aware caching for autoregressive video generation. Inter-frame differences identify which pixels require full denoising vs which can skip steps. Two-phase schedule: warm-up for semantic consistency, then motion-weighted cache reuse with dynamic update frequencies. 6.28x speedup on SkyReels-V2 (1% VBench drop), 1.64x on MAGI-1 (0.01% drop). Training-free, code public. The video-AR analogue of selective KV-cache reuse: iteration count is the optimization unit. → summary

Stream-T1 (2026-05-07) — Test-time scaling framework for streaming video generation. Three components: Stream-Scaled Noise Propagation (reuse high-quality previous-chunk noise as the prior for the next chunk), Stream-Scaled Reward Pruning (combine short-term spatial assessment with sliding-window long-term coherence), and Stream-Scaled Memory Sinking (route KV-cache evictions through reward-feedback-guided update pathways). The first content-aware KV eviction policy tracked by the wiki: not "which token is oldest" but "which token still anchors downstream quality." → summary

LIVEditor / ISA (2026-05-07) — In-context Sparse Attention for ICL video editing. Two stages: context pre-selection (prune low-saliency context tokens) plus dynamic Query routing (route high-error queries to full attention, low-error to a 0-th order Taylor sparse path). Empirically validates the claim that Query sharpness correlates with attention approximation error. ~60% attention-module latency reduction, near-lossless on EditVerseBench / IVE-Bench / VIE-Bench. The first sharpness-routed sparse attention in the wiki, structurally adjacent to language-side speculative decoding (route by difficulty). → summary

See What I See (2026-06-13) — A new use of the KV cache: not memory to compress or reuse, but a communication channel between different models. When two agents coordinate through text, the sender decodes its hidden state to words and the receiver re-encodes them, a lossy and expensive round trip. This paper (Michigan/NVIDIA/Penn) transfers a sender's KV cache directly into a heterogeneous receiver (e.g. Qwen3-8B → Qwen3-14B) via a lightweight cross-model cache transform trained in two phases (reconstruct, then generate). An information-structure duality drives the design: context-aware transfer (receiver also sees input) needs only sparse reasoning signal, context-unaware transfer (receiver sees nothing) needs dense contextual preservation. Matches or beats text communication at 2-3x lower compute in the context-aware regime, and is the first heterogeneous method that survives the context-unaware regime where priors collapse. Limitation: all directions stay inside the Qwen3 family, so true cross-architecture transfer is unproven. The KV cache is now a memory store, a thing to quantize/evict, and an inter-model wire. → summary

Key Concepts

• Context dependency: KV states computed for a document are specific to the attention context at the time. Reusing them in a new context produces attention distribution mismatch — hence the need to recompute.
• TTFT (Time-to-First-Token): the latency before the model outputs the first token. KV cache reuse directly impacts this.
• Soft-token adapters: trainable lightweight token representations that can modify how a cached KV state interacts with a new context, without recomputing the underlying states.
• Cache eviction: when the KV cache fills up, old entries must be evicted. Policy choices (LRU, saliency-based, etc.) affect quality and memory efficiency.

Hardware context (2026-06-07)

The Ken Huang memory survey (Memory Technology for Agentic AI Workloads) names the hardware fact under this whole page: as context grows, the dominant memory traffic shifts from weights to KV cache, so KV-cache management is the binding hardware constraint, not just a software optimization. This is why every eviction/quantization/offload technique above matters economically in a structurally memory-short market (HBM allocation-driven into 2030). Concrete hardware moves: Micron SOCAMM2 (LPDDR) claims >2.3x time-to-first-token when used for KV-cache offload at 1/3 the power of RDIMM; NVIDIA CMX/BlueField-4 turns SSD into an AI-native ephemeral-KV context tier with KV-aware placement. The open systems direction is KV-aware tiering: deciding per-request which KV blocks live in HBM vs LPDDR/CXL/SSD — the hardware dual of CLEAR's per-query compute rationing. See memory-hierarchy.

The cache becomes a billing surface (2026-08-14)

Everything above treats the KV cache as a technical resource: something to compress, quantize, evict, offload, tier, or ship between models. On 2026-08-13 a provider repriced it, and that adds a category this page did not have.

DeepSeek raised API prices with cache-hit tokens repriced to roughly 6x their prior cost, alongside a new peak/off-peak split where off-peak rates run 50% below peak (effective 16:00 UTC, 2026-08-16). A cache hit is what you pay when a request's prompt prefix is already resident, so the workload that pays most is the agent loop, which re-sends a growing prefix every turn. The Decoder called it the biggest increase in the transition and named the affected pattern precisely: agent workflows that repeatedly read the same files. → summary

The same release ships the mitigation. DeepSeek open-sourced Harness v0.1 under MIT the same day, and per Hugging Face's Elie Bakouch reading the code, its organizing commitment is first-class KV-cache-aware design: previously written history is never altered. When something in the conversation must change, the harness appends a statement describing the modification rather than editing the prefix, because an in-place edit invalidates every cached token downstream and forces a full recompute. Bakouch expects other harnesses to adopt the same convention.

Why this is a new entry rather than a footnote. Every technique above optimizes cache behaviour under a fixed price. This is the first time the wiki has seen the price move against a specific access pattern, which flips the direction of the optimization: instead of "how do I use less cache," the question becomes "how do I avoid invalidating the cache I already have." Prefix stability was previously a latency property owned by a systems engineer. It is now a line item owned by whoever signs the invoice, and append-only history is the cheapest available mechanism for protecting it.

Two consequences worth tracking. The append-only discipline has an unpriced accuracy cost: corrections accumulate at the tail, context grows monotonically, and contradictory statements coexist for the model to reconcile. That is a plausible quality tax paid to preserve a cache hit, and the counterfactual methodology from the Illusion of Visual Tool-Use (08-13), which corrupts a tool return and checks whether the answer moves, would test it directly. And peak/off-peak pricing introduces a scheduling dimension that no routing formulation on the LLM routing page currently models: not which model, but which hour.

This also connects the cache directly to compute economics, where Blackwell-generation capacity cleared 15% above record in Nebius's first auction the same week. Cache-hit repricing at the API layer and spot repricing at the silicon layer are the same scarcity showing up two levels apart.

2026-08-25: precision becomes a spatial decision, and INT8 KV caches stop costing quality

TileMix (08-25) is the first entry on this page where precision is routed rather than set. Every quantization result above picks a format and applies it uniformly to a tensor or a layer. TileMix partitions the query-key score matrix into hardware-aligned tiles and dispatches each tile group through either an FP16 or an INT8 score path, with both paths updating one shared online-softmax state so the output stays a single dense attention result. It routes all legal tile groups, so unlike sparse attention it preserves dense token connectivity. Training-free, and it supports grouped-query attention, variable-length batches, and INT8 key/value caches.

The direct consequence for this page: the INT8-KV-cache quality tax is not fixed. This page has recorded INT8 KV quantization as a memory play whose cost is long-context quality. TileMix recovers exactly that lost quality on LongEval and LV-Eval while improving prefill throughput over FP16 on A100, and it explicitly composes with INT8 KV caches rather than competing with them. So the two stack: you keep the memory saving and buy back the accuracy in the kernel. That is a different relationship than this page has recorded between any two compression techniques, which have generally traded off.

One routing bit governs several adjacent key tiles, and that detail is what makes it a long-context method rather than a demo. Per-tile bits would make the bitmask itself grow with sequence length; scalable grouping keeps the metadata compact where it matters.

Where the honest uncertainty is. The result is prefill-only, on A100 only, and the routing policy is a heuristic whose own overhead is asserted compact rather than deeply ablated. Decode-phase behaviour and end-to-end latency under realistic continuous batching are unmeasured, and those are the numbers that decide whether this reaches production. The Hopper and Blackwell story is also genuinely open, because FP8 gives those parts a third precision point and the tile-geometry argument is hardware-specific by construction. A two-bit mask over FP16/FP8/INT8 is the obvious extension and nobody has published it.

And it reframes the relationship between this page and routing. LLM Routing catalogues six axes, all of which route work to a component. TileMix routes a numerical format to a region of a matrix, which is a routing decision below anything that page recorded. On the same day, Pandora's Router priced value estimation across models and VoI-MoLE priced expert acquisition inside one model. Three levels of the stack, one decision structure, one day. The unasked question is whether a serving stack should be making these three decisions jointly under a single cost budget instead of independently, which is how every deployed system does it today.

Related Pages

• Compute economics (GPU pricing, utilization, durability)
• Agent harness engineering
• Memory Hierarchy for AI
• Knowledge Distillation
• LLM Routing