AI research wiki
llms-foundation-models · Tier 2

Attention Mechanisms: Linear, Local-Linear, and Optimizer Codesign

Attention Mechanisms: Linear, Local-Linear, and Optimizer Codesign

The attention read is the Transformer's core computational primitive, and after years of being structurally frozen it is now an active research surface again. Two distinct lines of work are running in parallel: improving the recurrent rule inside linear-attention layers (so they retain history without the quadratic cost), and improving the estimator order of the attention read itself. A third theme cuts across both: the optimizer is not separable from the architecture.

Current State (as of 2026-09-09): the linear-attention write rule gets a confidence term, and it is the first principled one

Linear attention buys constant-memory decoding by replacing the growing KV cache with a fixed-size recurrent memory, and the price is an irreversible online decision at every token: what to write, and how hard to overwrite what is there, before you know what future queries will need. Delta-rule models learn that write strength from the current token embedding. What none of them do is track how confident the memory currently is, so a write into a well-established association is treated exactly like a write into an empty slot.

Kalman Delta Networks (09-09) (arxiv 2609.07816) reformulates recurrent associative memory as a linear-Gaussian state-space model, for which the Kalman filter is the optimal recursive estimator. The transition then propagates the memory state and its uncertainty, so the Kalman gain weights each residual write by accumulated evidence and observation reliability. Delta-style updates fall out as a special case that substitutes a token-wise isotropic surrogate for the predictive covariance and drops covariance tracking entirely, which is a clean statement of what the delta rule was implicitly assuming.

The engineering is the contribution, because exact tracking is the wrong shape for the hardware. A dense state-dependent Riccati recursion is sequential where the entire point of linear attention is an associative scan with logarithmic parallel depth. Two scan-compatible approximations recover it: Diagonal KDN projects each one-step posterior onto the diagonal Gaussian family via online mean-field variational inference, and Isotropic KDN keeps a single uncertainty scalar per head. Both work because their uncertainty recurrences are Mobius maps, which compose associatively. Across controlled pre-training at 750M and 1.3B, KDN variants improve perplexity and mean downstream accuracy over state-of-the-art linear-attention models.

It is the constructive answer to a hazard the wiki logged two days earlier. When Quantization Breaks Memory (09-07) found that in a recurrent network the state is stored and read back, so the write-back rule changes the dynamical system rather than approximating it: deterministic 4-bit state storage on a fixed trained GRU raised error roughly 70x and 300x on two targets because repeated sub-threshold updates are silently discarded while the network keeps proposing change. KDN is the first bounded-memory model where the write-back rule is explicitly the learned object with a confidence term attached, so it should be measurably more quantization-robust than a delta-rule memory at the same bit width (a write the filter already considers low-confidence is one it was going to attenuate anyway). Nobody has run that, and it is cheap and decisive.

Two limits before quoting the perplexity win. 750M and 1.3B controlled pre-training only, and linear-attention advantages have historically compressed at scale. And there is no wall-clock number: scan-compatible is not the same as fast, and the covariance recurrence adds per-token work whose cost is unreported. No long-context evaluation either, which is the regime the fixed-size memory exists for and where a confidence-weighted write should help most.

Current State (as of 2026-08-11)

Attention selectivity and KV compression have been two separate lines on this page. A production-scale model just shipped them as one operator.

Motif 3 (08-11) (2608.09119, Motif Technologies, Korea's Sovereign AI Foundation Model Project) is a decoder-only MoE at 314B total and 13.2B activated per token, with unusually fine-grained sparsity: 384 routed experts per layer, eight selected per token. Its attention contribution is Grouped Differential Latent Attention (GDLA), which fuses differential attention, which suppresses irrelevant context by subtracting two attention maps, with Multi-head Latent Attention, the DeepSeek low-rank latent-KV construction that compresses keys and values into a shared latent. One operator, both properties: sharper reads and a smaller cache. Around it sit modified manifold-constrained hyper-connections, Expert Specific PolyNorm activations, and multi-token prediction. Pretraining: ~12.5T tokens, context to 256K, selective MXFP8 compute and communication, fused memory-efficient kernels, window-aware context parallelism.

Why the fusion is the interesting part. VideoMLA (06-02) established the counterintuitive fact that MLA works even where pretrained attention is not low-rank, because the MLA bottleneck dimension rather than the pretrained spectrum determines effective rank, which decoupled MLA from the spectral story it was sold with and predicted it would transfer broadly. GDLA is that transfer, into a differential-attention backbone at 314B. The paper runs no ablation isolating the two halves, so what the fusion buys over either alone is unmeasured, which is the obvious missing experiment.

It also matters for a gap kv-cache.md has carried since 08-04. SemiAnalysis's Kimi K3 primer proposed KV throughput, cache size divided by prefill time, as the honest efficiency unit and observed that no open-weight model ships static KV compression. Motif 3 does, via GDLA's MLA half. It is therefore a candidate for the comparison this wiki says nobody has run: KDA-plus-MLA against the GQA-sparse family (GLM 5.2's DeepSeek Sparse Attention, DeepSeek V4's Compressed Sparse Attention, MiniMax M3's sparse attention, MiMo V3's HySparse) at matched sequence lengths. The report publishes no cache-per-token figure, which keeps this wiki's standing 07-30 gap open even in the model that would have closed it.

One cross-page echo. Expert Specific PolyNorm activations put source-specific parameters in the feed-forward path while attention stays shared, which is the same partition Poly-OPD (08-06) reached from gradient-agreement measurements (share attention LoRA, keep FFN adapters teacher-specific) and Physics of Multimodal Pretraining (08-06) reached from cross-modality architecture ablations on 13.5B MoE models over 2T tokens. Three independent routes to "attention is a source-agnostic mixing operation and the feed-forward layer is where source-specific knowledge lives," and Motif 3 does not cite either.

Prior State (as of 2026-08-04)

A third line of work opens on this page: write sparsity. And the hybrid recipe finally gets audited at the serving layer, where part of its advertised advantage turns out not to survive.

The architecture side. Raven (2607.25357, Kurate cs.LG #16, ai_rating 7.0, the highest on either board this week, absent from HuggingFace entirely) is from Arshia Afzal and Volkan Cevher (EPFL) with Aviv Bick, Eric Xing and Albert Gu (CMU, Cartesia AI). It frames every linear-time model as one of two extremes in how it writes to memory. SSMs and linear Transformers write densely, updating the whole state per token, so nothing is formally evicted but everything interferes and specific past tokens are hard to recover. Sliding-window attention writes sparsely with explicit per-token representations, so in-window recall is exact, but eviction is decided by position and recall falls off a cliff at the window edge. Raven fills the missing quadrant, sparse write with content-selective gradual decay: a fixed set of memory slots, and at each step a learned input-dependent router decays and updates only a selected subset, leaving the rest untouched. The comparison that isolates the contribution is against Gated Slot Attention and ABC, which already routed over a fixed slot set but still wrote densely to all slots; Raven's claim is that not touching a slot is the load-bearing half, because an unrouted slot stays exactly retrievable. Competitive with or better than prior linear-time baselines on recall-intensive benchmarks, holding at 16x training context length, gains persisting in hybrids.

This is a genuinely new axis for the page. Line 1 improves the recurrent rule (Mamba2/GDN/KDA as online SGD, MDN (05-11) adding momentum, Gated DeltaNet-2 (05-24) decoupling erase from write). Line 2 improves the estimator order of the read (Parallax (05-29), local-constant to local-linear). Raven changes neither: it changes which part of the state gets written at all. MDN improves how a dense update moves, Parallax improves what the read computes, Raven decides most of the state should not be touched. It also reads as the stronger form of what Gated DeltaNet-2 was reaching for, since an unrouted slot needs no erase gate. Composition with the recurrent-rule line (a momentum delta-rule applied only to routed slots) is unrun and is now the second uncombined pair on this page.

The serving side. SemiAnalysis on Kimi K3 prices the hybrid recipe at the kernel and cluster level and produces the page's first real negative result about it. KDA is traced from linear attention through DeltaNet to Gated DeltaNet: the diagonal-matrix expansion of the forget gate gives per-channel memory decay and, as a side effect, enough positional awareness that Kimi removes RoPE from MLA entirely. FlashKDA's derived complexity is linear in sequence length for prefill and constant for decode, with decode traffic dominated by reading and writing the FP32 recurrent state. So far, so good.

Then the qualification, and it matters for every claim on this page about constant-state memory. Prefix caching a recurrent state is structurally hard: engines find cache hits by matching the longest cached token prefix, and you cannot reconstruct state at position t from a snapshot at t-k without replaying, so supporting arbitrary prefix matching would mean caching state at every position, at which point memory grows with sequence length and the entire point is gone. Moonshot's fix is coarse snapshotting, vLLM caching KDA state every 32K tokens plus at prompt boundaries. SemiAnalysis states the consequence flatly: linear attentions like KDA greatly reduce KV cache memory, and realistically during serving they do not consume a constant amount. That is a direct qualification of the 07-30 practitioner measurement of a ~20x KV footprint gap between hybrid and dense models; the 20x is real for a single local session and optimistic for a multi-tenant server with prefix reuse. Raven does not escape this, since it also carries a fixed state to snapshot, but a slot-structured state is addressable and mostly untouched per step, which at least makes differential checkpointing conceivable where a monolithic dense state does not.

The surprise, and a prediction with the author's own data behind it. Kimi K3 keeps MLA as its full-attention layer while every other frontier open-weight model moved to GQA-based sparse attention (GLM 5.2's DeepSeek Sparse Attention, DeepSeek V4's Compressed Sparse Attention, MiniMax M3's MiniMax Sparse Attention, MiMo V3's HySparse). MLA's absorption trick cuts decode compute at the cost of extra prefill compute, which is right for decode-dominant reasoning and wrong for prefill-dominant agentic work. SemiAnalysis predicts K4 drops it, and their own trace corpus supports it: replaying an hour of real Claude Code traces gives a median 142K input tokens against 444 output tokens per turn over a median 65 turns per session, a 320:1 prefill-to-decode ratio.

And Block Attention Residuals now have two independent adopters in two modalities. Kimi K3 replaces the residual stream with softmax attention over the depth axis: each layer attends over previous layers' outputs, with the query a learned per-layer parameter rather than derived from the current token, so every layer gets fine-grained selective access to earlier representations instead of one shared additive channel. Block-chunking cuts communication from O(Ld) to O(Nd), reported at 1.25x compute efficiency, bounded output magnitude where standard residual networks grow with depth, and only 4% pipeline-parallel overhead after cross-stage caching. SANA-Video 2.0 (07-24) used the same mechanism in a video diffusion transformer for a ~12% lift in deep-layer effective rank. Two labs, two modalities, one mechanism: the residual stream is a bottleneck and depth-axis attention is the fix now clears this wiki's bar for an architectural trend rather than one lab's idiosyncrasy.

Three details from the primer worth keeping on the page. First, FlashKDA's derived cost, since Moonshot open-sourced the kernels: two launches, K1 preparing chunk-level tensors in parallel (cumulative decay, decayed Q/K, the (I+L)^-1 inverse via Neumann factorization, the causal mask) and K2 running the chunk recurrence, for 12C^3 + 8C^2 D + 6C D^2 FLOPs and 8C^2 + 22CD bytes per chunk of size C. Decode is roughly 7D^2 FLOPs against 8D^2 bytes. Second, the module details around KDA: short left-padded convolutions on Q, K and V to capture local token dependencies without breaking causality, and L2 norm on Q and K to stabilize the eigenvalues of the transition and output matrices, with the output forget gate implemented as a plain linear transform in K3 where Kimi Linear used a low-rank projection. Third, the 3:1 KDA-to-MLA ratio is now the third independent arrival at the same hybrid band, after SANA-Video 2.0 (07-24) fixed 25% softmax as the video-diffusion quality-efficiency optimum and Ling/Ring-2.6 (06-16) migrated a 1T model in place to 7:1 Lightning Attention to MLA. Three subfields, one design-point neighbourhood, and no theory for why.

And Block Attention Residuals have three adopters, not two. MHAR (07-31) is the third, which puts the depth-axis-attention mechanism at three papers in two weeks.

Current State (as of 2026-07-24)

The hybrid recipe crosses into video, and sparse attention becomes a distributed-scheduling problem. Two same-week papers attack the video-attention bottleneck from the two layers this page tracks. SANA-Video 2.0 (2607.21553, NVIDIA + MIT, Song Han) ports the LLM hybrid recipe (Qwen3-Next, Kimi-Linear, Kimi K3) to a video diffusion transformer trained from scratch: gated linear attention does the O(N) bulk mixing, full-softmax "anchors" at a 3:1 ratio (25% softmax fixed as the quality-efficiency optimum) restore full-rank interactions, and Block Attention Residuals route anchor summaries forward to lift deep-layer effective rank ~12%. It hits softmax-level VBench quality at 3.2x faster forward pass (720p/60s), 120x faster than Wan 2.2 after kernel optimization, 720p on one H100. This is the direct video instance of the 06-16/06-17 finding that retrieval lives in the full-attention layers and the efficient layers shape the path. FVAttn (2607.16190, Tencent WeChat HPC) works the systems layer: training-free Top-p sparse attention gives each head a different workload, which under multi-GPU sequence parallelism becomes a rank-level straggler problem, so FVAttn migrates heavy heads across GPUs via P2P at runtime (imbalance 1.34→1.08, 4.41x over FlashAttention). Together: SANA avoids 75% of softmax by construction, FVAttn load-balances the softmax that remains. The page's estimator-order line (Parallax) and recurrent-rule line (GDN/MDN) now have a distributed-execution sibling.

Current State (as of 2026-06-17)

The mechanism behind the hybrid convergence: retrieval lives in the full-attention layers, the efficient layers just shape the path there. One day after two frontier labs shipped mostly-linear backbones (Nemotron 3 Ultra, Ling/Ring-2.6), Rethinking Efficient Attention in Hybrid Architectures (arxiv 2606.15378) supplies the mechanism study the page asked for. Three findings. (1) Scaling: the efficient-attention choice controls how fast long-context ability emerges, not the final ceiling — hybrids converge given enough training. (2) Mechanism: long-range retrieval is carried by the full-attention layers; efficient attention (SWA, linear mixers) shapes their optimization trajectory. This produces Large-Window Laziness — a bigger sliding-window attention window delays retrieval-head formation in the full layers, because the cheap layers cover for them. (3) Design: applying NoPE (no positional encoding) to only the full-attention layers of a small-window SWA hybrid substantially improves long-context performance at negligible short-context cost. This directly closes the open question 06-16 left ("how much retrieval precision does the 7:1 ratio cost?"): the ratio is safe as long as the full layers stay un-lazy, and it hands the non-uniform KV line a layer-axis allocation prior (compress the efficient layers hard, protect the full layers — see kv-cache). Same-day rhyme: LoopCoder-v2 and Variable-Width Transformers both reject uniform allocation across depth/loops, the same "capacity should follow layer function" prior this paper grounds for attention.

Current State (as of 2026-06-16)

Hybrid linear/full attention becomes the default frontier backbone: two independent labs ship it the same day. The recurrent-rule line the page tracks (Gated DeltaNet-2, MDN) and the linear-attention serving thread (Kimi Linear, MiMo-V2-Flash on the KV cache page) converge into production releases on 2026-06-16. NVIDIA's Nemotron 3 Ultra (arxiv 2606.15007) is a 550B/55B-active MoE with a Mamba (SSM) + attention hybrid backbone, 1M context, NVFP4-native pre-training, claiming ~6x throughput at on-par accuracy — open-sourced end to end. Ant's Ling-2.6 / Ring-2.6 (arxiv 2606.15079) migrates a 1T model in place from full GQA to a 7:1 Lightning Attention (linear) : MLA (low-rank latent KV) hybrid via 10T tokens of continued pre-training. Two things stand out. First, the convergence: SSM-or-linear layers carrying the sequence, full/latent attention layers handling what needs exact mixing, is no longer experimental — it is the assumed shape of a new frontier model. Second, Ling/Ring's pairing of two distinct efficiency families the page separates — linear attention (fixed-state) and MLA (the VideoMLA/DeepSeek low-rank-KV line) — in a fixed 7:1 ratio is an explicit design point on how much exact attention a reasoning model still needs. The "migrate, don't retrain" path (swap the attention type via continued pre-training rather than from scratch) is the practically important claim: it makes linear-attention efficiency adoptable for anyone holding a trained full-attention model. Open question the page can now ask concretely: how much retrieval precision the 7:1 ratio and the Mamba layers cost at 1M context, the usual linear-attention tax, neither release characterizes here.

Current State (as of 2026-06-04)

μP transfer reaches gated linear-attention models (2026-06-04). Unlocking Feature Learning in Gated Delta Networks at Scale (arxiv 2606.04048) supplies the missing theory that lets the recurrent-rule line train at scale without re-tuning. The Maximal Update Parametrization (μP) gives zero-shot hyperparameter transfer across widths for standard Transformers; its derivation did not cover structured state transitions and gating, so it failed to transfer to Gated Delta Networks (GDN). This paper rigorously propagates coordinate-size estimates through GDN's forward pass, gates, and recurrent state, derives the scaling rules, and shows learning-rate transfers cleanly across widths under both AdamW and SGD where standard parametrization fails. This is the third leg of the scale-stable-architecture program: MoE μP (05-17, Kurate cs.LG #14 this week, ai_rating 9.0) did it for mixture-of-experts, Parallax (05-29) showed Muon unlocks local-linear-attention capacity, and now GDN gets its parametrization. The recurrent-rule line (Gated DeltaNet-2 05-24, MDN 05-11) improved the state update; this makes that family trainable at scale. Industry shipped the matching recipe the same morning: the Marin / Open Athena pretraining post reported 6.7x theoretical (3.6x realized) dense→MoE speedup plus a 1.3x AdamH→MuonH optimizer swap, the open-recipe mirror of the parametrization theory. The obvious missing experiment: whether Muon unlocks GDN the way it unlocked Parallax.

Current State (as of 2026-06-02)

Two papers attack the supervision target and the reasoning representation, not the attention read itself, but both bear on representation geometry. NITP (Next Implicit Token Prediction, arxiv 2605.24956) argues standard next-token prediction's one-hot logit-space supervision under-constrains the latent representation space, letting hidden states drift into degenerate, anisotropic geometry. It adds a dense continuous target in representation space — predict the next token's implicit semantic content using the model's own shallow-layer representations as self-supervised targets — and reports +5.7% MMLU-Pro on a 9B MoE at ~2% extra training FLOPs and zero inference cost. The anisotropy/representation-collapse framing connects to the same geometry concerns that motivate interpretability work. GLR (Geometric Latent Reasoning, arxiv 2606.02248) formulates latent reasoning as a path-approximation problem inside the pretrained token-embedding space: a lightweight transition head predicts iterative direction updates, anchored to text CoT traces but free to deviate continuously. On math with Qwen3 it induces substantially shorter generations with no explicit length objective, exposing a new tradeoff between latent compute budget, output length, and accuracy. Both treat the embedding/representation space as the object to shape — NITP during pretraining, GLR during reasoning.

Current State (as of 2026-05-29)

Parallax: local-linear estimator made scalable, and Muon codesign. Parallax reframes softmax attention as a local constant estimator (a kernel-weighted mean) and upgrades it to a local linear estimator (Local Linear Attention, LLA), which nonparametric statistics says has a strictly better bias-variance tradeoff for associative memory. Exact LLA needs a per-query linear solve (a conjugate-gradient inner loop) that is too slow and unstable to pretrain; Parallax removes the solver by learning an extra query-like projector that probes the KV covariance directly, and ships a hardware-aware kernel that raises arithmetic intensity above FlashAttention, pushing attention from memory-bound toward compute-bound. The decode kernel matches or beats FlashAttention 2/3; pretraining at 0.6B and 1.7B shows perplexity gains under both parameter- and compute-matched controls (a Pareto improvement). The secondary headline: Muon unlocks Parallax's capacity where AdamW does not, claimed as the first architecture-optimizer codesign result for an attention mechanism.

This is significant on two axes. On the estimator-order axis, Parallax is a different and arguably deeper move than the recurrent-rule improvements below: it changes what the attention read computes, not just how the linear-layer state evolves. On the optimizer-codesign axis, it converges with the scaling-laws literature (see below).

The two lines of work

Line 1 — recurrent rule inside linear layers (subquadratic, retain history):
   Mamba2 / GDN / KDA : state update = one step of online SGD on a latent objective
   MDN (05-11)        : add MOMENTUM to that update, parallelized without breaking causality
   Delta-Attention    : residual/delta corrections on the linear read

Line 2 — estimator order of the attention read itself:
   softmax attention  : local CONSTANT fit   (kernel-weighted mean of values)
   Parallax (05-31)   : local LINEAR fit      (better bias-variance, solver removed)

Cross-cutting — optimizer is not separable from architecture:
   Optimizer-spectral-scaling (05-23) : Muon scales rare-token rank where AdamW stalls
   Parallax (05-31)                   : Muon unlocks LLA capacity where AdamW does not

Key papers

Parallax (2026-05-29) — Parameterized Local Linear Attention. Replaces exact-LLA's per-query CG solver with a learned covariance probe; hardware-aware kernel matches/beats FlashAttention 2/3; Pareto perplexity gains at 0.6B/1.7B; Muon-architecture codesign. → summary

MDN: Momentum DeltaNet (2026-05-11) — Parallelizes stepwise momentum for delta linear attention without breaking causality, with a spectral-stability analysis constraining the gating. Beats Transformers, Mamba2, GDN at 400M/1.3B. The first paper this month to push the recurrent-rule substrate. → summary

Delta-Attention residuals (2026-05-20) — Residual/delta corrections to the attention read. → summary

Same Architecture, Optimizer-Induced Spectral Scaling Laws (2026-05-23) — Holding architecture and width fixed, Muon realizes near-linear hard-rank scaling on rare-token representations (β=1.02) where AdamW stalls (β=0.44); matched loss does not imply matched representation geometry. The theoretical companion to Parallax's Muon-codesign finding. → summary

Key concepts

  • Local constant vs local linear estimator: softmax attention reads a kernel-weighted average of nearby values (constant fit); local-linear attention fits a line, capturing the local gradient of the value field for better associative recall.
  • Arithmetic intensity: FLOPs per byte of memory traffic. FlashAttention is memory-bound; Parallax deliberately raises arithmetic intensity to enter the compute-bound regime where GPU FLOPs are not bandwidth-starved.
  • Recurrent rule as online SGD: Mamba2/GDN/KDA interpret their state update as one closed-form online-SGD step on an implicit memorization objective; momentum (MDN) is the standard SGD fix applied to that update.
  • Architecture-optimizer codesign: the empirical finding that the right optimizer (Muon) is a function of the architecture, so ablating one while fixing the other mis-measures both.

Open problems

  • Does the local-linear advantage survive scale and long-context retrieval? Parallax tops out at 1.7B and reports no needle-in-haystack / RULER numbers; in-context retrieval is exactly where linear-attention substitutes historically collapse. Raven (08-04) is the first paper to attack this as its primary target, arguing the collapse is a write-policy artifact, but it too reports no RULER numbers and no scale.
  • Why does Muon unlock these architectures? The codesign effect is empirical; no mechanistic account links Muon's update geometry to the local-linear estimator or to rare-token rank.
  • Do recurrent-rule (MDN) and estimator-order (Parallax) gains compose? Nobody has combined a momentum delta-rule with a local-linear read.
  • Do write-sparsity (Raven) and recurrent-rule (MDN) gains compose? Raven decides which slots to write, MDN decides how a write moves. Second uncombined pair on this page, added 08-04.
  • Does a slot-structured state solve the prefix-caching problem? KDA's monolithic dense state has to be checkpointed on a fixed 32K schedule because it is not addressable. Raven's slots are addressable and mostly untouched per step, which makes differential checkpointing conceivable. Nobody has connected the two papers.
  • Does sparse write help or hurt Large-Window Laziness in hybrids? The 06-17 mechanism study found retrieval lives in the full-attention layers and a bigger cheap window delays retrieval-head formation because the cheap layers cover for them. If an efficient layer can itself carry high recall, does retrieval work shift back into it, and does that reintroduce laziness?

Hybrid linear attention has its own outlier morphology (2026-08-14)

This page has tracked hybrid architectures (interleaving cheap linear-attention layers with a few full-attention layers) mainly through what they cost and what they retrieve. Massive Activations in Hybrid Linear Attention LLMs (08-14) adds the first systematic account of what their activations look like, and the structure is architecture-aligned rather than incidental.

Two morphologies. Pre-attention spikes (PAS): massive activations reliably spike in the layer immediately before each full-attention layer. Inter-spike plateaus (ISP): those elevated values can persist through the intervening linear-attention layers rather than decaying. As full attention gets denser, successive spikes connect through plateaus until the picture converges on the stable massive-activation morphology already known from full-attention models. The recurrence is established across five linear-attention architectures, six hybridization configurations, five data domains, and open models from 1.2B to 397B total parameters.

The proposed mechanism is a write-sink-cancel lifecycle governed by cancellation timing: a PAS is written, absorbed, and cancelled locally and fast, while an ISP is the same process with delayed cancellation. Controlled pretraining of GDN-based hybrids to 1.3B shows both emerge early and respond asymmetrically to gating, with full-attention output gating strongly attenuating magnitude but not layerwise organization, and removing GDN gates amplifying only modestly.

Why it belongs on this page and not only on the efficiency page. Massive activations are the reason low-bit quantization of activations is hard: a few enormous values set the dynamic range the whole tensor must fit. Knowing they appear at predictable layer positions in hybrids, immediately before full-attention layers, turns a global precision problem into a per-layer one. That is the same shape of result as the wiki's recurring finding that hybrid capability is not uniformly distributed across layers, and it is the mechanistic complement to the 06-17 Large-Window Laziness study, which found retrieval concentrates in full-attention layers. Retrieval concentrates there and so does the activation outlier structure, which is consistent with the full-attention layers doing the load-bearing work that the linear layers route toward.

Open problem added: does the PAS position predict where per-layer quantization budget should go? Nobody has run a mixed-precision hybrid that assigns higher precision only to PAS-adjacent layers and measures the accuracy-per-bit curve against uniform assignment. The paper localizes the outliers and stops short of exploiting them. Code is at StartluxLabs/Massive-Activations-HLA.

The vertical channel is narrower than the horizontal one (2026-08-14)

Every mechanism on this page so far modifies how a token attends across the sequence. Full-bandwidth transformer (08-14) points at an axis this page has never treated: the channel between decoding steps. At each step a standard model computes a rich top-layer hidden state, samples one token from it, discards the hidden state, and feeds only that single symbol back to the bottom of the stack. Horizontal bandwidth is wide; vertical bandwidth is one token.

The fix is small and structural: fuse the previous top-layer hidden state with the sampled token embedding through a gated linear unit and feed that back as the next input. Non-verbalized computation re-enters the stack with a fresh depth budget instead of having to be re-derived or spelled out as reasoning tokens. The transformer, the KV cache, and the language-modeling objective are all unchanged, which is what makes it deployable rather than a research architecture.

Latent feedback breaks parallel teacher forcing, since step t's input depends on step t-1's output. Their answer is a scheduled multi-pass objective: introduce feedback late in pretraining, mixed with a small fraction of deeper feedback passes for stability. At 1B parameters and 400B tokens it improves validation loss, 5-shot evaluation, math and code generation, and instruction-tuned performance, matching standard transformers trained on roughly 1.5x more tokens at negligible per-token decoding overhead, and producing shorter reasoning traces at equal or better accuracy.

Relation to the looped-transformer line. Looped transformers buy extra effective depth by re-running layers within a step. Full-bandwidth buys it across steps at the cost of one gated fusion. The shorter-reasoning-traces result is the interesting overlap: if latent feedback lets computation persist without being verbalized, then a chunk of what chain-of-thought currently does is being paid for in tokens because the architecture gave it no other place to live. That is a claim about test-time compute, not just architecture, and it makes the direct comparison against a looped model at matched depth budget the obvious missing experiment.

Related pages

2026-09-14: the selector was being trained on the wrong target all along

Every trainable sparse-attention method this page has tracked shares one architectural fact and one training workaround. The fact: a lightweight selector scores context units and a hard Top-K keeps the best few. The workaround: Top-K is discrete, so no gradient reaches the selector from the language-modeling loss, and the field's universal substitute has been distilling the model's own dense attention distribution layer by layer, teaching the selector to rank units the way full attention weighted them.

SAS (09-14) (arXiv 2609.13141) argues that substitute target is systematically misaligned with the deployed objective, and the misalignment is about the budget. Dense attention weight measures how much mass a unit received when the model could afford all of them. Under a budget of K, what matters is a unit's marginal contribution given that only K survive, and a unit can carry real dense weight while being redundant once a few others are selected. The distillation objective has no way to express redundancy, so the budget gets spent on units that look important in isolation.

The fix is to stop blocking the gradient rather than to build a better proxy. SAS injects the selector's continuous scores into the attention logits during training, so the language-modeling loss updates the selector by ordinary backpropagation. No distillation target, no straight-through estimator, no auxiliary loss. Three details decide whether the simple version works: the gate goes inside the softmax in log form, so the selector's score is an additive bias on attention energy rather than a rescaling of an already-normalized distribution; gates are normalized against the always-retained current block, which supplies a fixed reference scale; and scores stay continuous rather than binarized, so the model learns relative priorities and degrades gracefully as the budget tightens. A memory-efficient Triton kernel folds the gate into FlashAttention-style tiled computation, without which you would have to materialize the gated N x N score matrix to differentiate through it.

The reported result is the one the thesis predicts. SAS beats trainable sparse-attention baselines across reasoning, long-context and agentic tasks at every budget, with the largest margins where the budget is tightest. At generous budgets the two orderings mostly agree, because there is room for merely-useful units. At tight budgets only a budget-aware ranking spends well.

This puts SAS in mild tension with the page's other recent selection result. HyQuant (09-11) identifies "vertical-line tokens," the key columns nearly every query attends to persistently, by probing the attention map, and uses that to decide which tokens keep high precision. That probe is a dense-attention-derived importance signal, exactly the family SAS argues is misaligned under a budget. The two are not in direct conflict, because HyQuant allocates bits and SAS allocates slots, but they disagree about whether dense attention weight is a trustworthy target, and nobody has run the comparison.

The measurable quantity nobody has published. SAS's thesis implies a rank correlation between dense-attention weight and budget-conditional marginal contribution, as a function of K. If it is high at large K and collapses at small K, that single curve explains SAS's result and tells every eviction, compression and precision-allocation method on this page and the KV cache page where its importance heuristic stops being trustworthy. It runs on an existing checkpoint.

And it is the third instance of a pattern that now deserves naming: the sparsity idea is never the hard part, the kernel is. MISA (05-11) turned an expensive 64-head indexer into a mixture-of-experts routed to eight heads per query and the result that mattered was a TileLang kernel at 3.82x the original. Sparser, Faster, Lighter (09-13) got over 99% feedforward sparsity from plain L1 regularization and its actual contribution was a sparse packing format plus CUDA kernels. SAS's actual contribution is a one-line training change plus the Triton kernel that makes it affordable. A sparse-attention or sparse-weight paper without its own kernel should be read as a hypothesis, not a result.

2026-09-24: values from a table, prefill that exits early

  • Memory Attention: V = K + Norm(E[token]); no value projection, offloadable per-layer token tables. Perplexity 31.55 to 28.64 at 10B tokens, but at 3x the parameters (1,135M vs 373M), so capacity and architecture are confounded.
  • HySparse2: YOCO-style KV bridging plus token-level sparse KV reuse; the architecture @eliebakouch places alongside DeepSeek V4.1 Flash (YOCO) against Qwen 3.8 Next Flash and GLM 5.3 Flash (3:1 sparse/linear with GDN or KDA).

2026-09-25: scheduling a fixed memory

  • Proteus (NeurIPS 2026): linear-attention memory states are exposed in full from token one, so early tokens sprawl and later ones overwrite. Proteus partitions the state into 16 blocks and unlocks one every N/16 tokens, gating reads and writes. No parameters, no compute, drops into Titans, Comba, SWLA and Hope-Attention; up to +8.4 NIAH at 2x training length. A deterministic counterpart to ARM's learned slot router (09-23). Gated DeltaNet-2, the backbone class it targets, was also accepted to NeurIPS this week.
  • Superposition linearity: averaged embeddings of two prompts produce roughly the average of the two next-token distributions; the property weakens over pretraining and is restored by light fine-tuning; a guided decoder separates two continuations from one pass. Untested: whether two streams can share one KV cache.

2026-09-29: the selector's own cost goes sub-quadratic

PISA (09-29) targets the last quadratic term in block-sparse attention: choosing the blocks. DeepSeek Sparse Attention's lightning indexer, as described in SemiAnalysis's 09-28 GLM-5.3 piece, scores every prior token for every query, so selection is linear per query and quadratic overall. GLM-5.3's IndexShare cut that by 75% by sharing one indexer across four layers. MISA (05-11) and MSA (06-12) shrank the constant. PISA changes the exponent. It pools keys into O(log N) levels, keeps a bounded LogSumExp Top-K at each level, and expands only survivors, for O(N log N) total, with fused Triton kernels that never build the score matrix. It matches the baseline on commonsense reasoning and beats it on retrieval. Its closest ancestor is Lighthouse (05-16), which used a similar pyramid cascade but only during training. Two cautions: greedy pruning at coarse levels cannot recover a missed block, which makes the SAS (09-14) question of what the selector is trained to rank more pressing; and cheaper selection does not shrink the KV cache, which still has to sit in HBM in full.

2026-09-30: cut after scoring, or cut before

MassAlloc and CoWindow (09-30), apparently one group's two papers, split full attention's waste two ways. MALA keeps every QK score and skips post-score value work for tiles with negligible normalized mass, using the online-softmax normalizer already in the kernel (1.6x faster decoding at 128K; matched-work omitted mass 0.0188% vs an oracle's 0.0182%). CoWA is position-defined: shared local and sink windows for all heads, complementary long-range windows per head, so full causal coverage is a property of the ensemble. No indexer to pay for (compare PISA, 09-29, which made the indexer O(N log N)). 3.0x faster decoding, 7.6x lower per-rank decode memory at 128K; duplicated windows of equal size do much worse, so coverage, not window size, is the variable. Both track full attention from 0.6B to 14B. Kernel-side twin: Rowmax-H15 (09-30) approximates the softmax exponent near the row max inside FlashAttention-4.

2026-10-06: a thin KV channel, and a 3D recurrent state

  • The Extender: every layer appends a 32-dim extension to a shared log that alone feeds the K and V projections; persistent attention memory is 104x smaller than MHA at 924M and RULER accuracy is higher. Exact attention with a tiny per-token footprint, a different route from MLA's per-layer latent.
  • Triadic linear attention (shorts): writes key x second-key x value into a 3D state, giving E-fold more state for two extra projections; improves long-context recall for Gated DeltaNet. Joins Proteus (09-25) and ARM (09-23) on fixed-state capacity.

2026-10-09: which hybrid, and what its state costs

Hybrid Mechanics 1.1, RLT, CMM (10-09). A seesaw between hybrid types: linear-attention hybrids gain more from long-context continual pretraining, SWA hybrids extrapolate better without it; Sliding-Window Linear Attention gives 16x training-free extrapolation at 100% NIAH (64K). Sakana's Continuous Memory Machine splits recurrent memory into short- and long-term matrices, joining Proteus (09-25) and triadic states (10-06) on fixed-state capacity. Serving cost of the winning LA hybrid: STEPQuant quantizes its state 5x.

2026-10-13: hybrids become cheap to build and to serve

Priming (10-13): convert a pretrained Transformer's local attention layers (identified by a sliding-window proxy, formally low Hankel rank) into SSMs for <0.5% of pretraining tokens; Gated KalmaNet > Gated DeltaNet > Mamba-2 under controlled conditions; a 32B hybrid matches its source on reasoning at up to 2x decode speed at 128K. ReplaySSM (10-13) removes the SSM-layer decode bottleneck at serving batch sizes. QuantMLA (10-13) shows MLA's decoupled RoPE key is the precision-sensitive part of the latent cache.

2026-10-14: hybrids ship, and depth gets multi-head attention

  • LFM2.5-24B-A2B and Ling-3.1-flash (10-14): open production hybrids at 3:1 (short conv to GQA) and 7:1 (KDA to gated MLA). The ratios Priming and ReplaySSM (10-13) assumed are now shipped. No full-window recall numbers yet.
  • MSAR (10-14): multi-head attention over depth via multi-scale causal convolutions; 1.58x iso-loss compute speedup over standard residuals at 18B/28B MoE (AttnRes 1.35x, mHC 1.30x).

2026-10-22: in hybrids, recurrence starves attention

Competing Learning, KronLSTM, no-hair theorem. In recurrence+attention hybrids the recurrence learns retrieval first and suppresses attention's learning signal; freezing recurrence mid-training lets attention learn retrieval (up to 1B). KronLSTM's Kronecker Delta Rule edits memory along one matrix direction, bounded norm without normalization, beats Gated DeltaNet on long context. Lafa (8B flow LM) and DRLM (draft refinement) push non-AR generation.

2026-10-23

2026-10-24: route or compress the fast-weight MLP

  • MiTA (2602.01219, resurfaced on X): attention as an N-wide fast-weight MLP; efficient attention either routes (top-k, MoE attention, DSA) or compresses (linear attention) it.
  • TTR-Compressor (2610.31574): block summaries as closed-form regression coefficients; better block selection for block-sparse attention on RULER, 55.6% fewer visual tokens (compression shorts).
  • Kimi K3.1 Pro keeps K3's 69 KDA (fixed-state linear attention) + 24 MLA layer hybrid at 1M context (summary).