Attention Mechanisms: Linear, Local-Linear, and Optimizer Codesign
The attention read is the Transformer's core computational primitive, and after years of being structurally frozen it is now an active research surface again. Two distinct lines of work are running in parallel: improving the recurrent rule inside linear-attention layers (so they retain history without the quadratic cost), and improving the estimator order of the attention read itself. A third theme cuts across both: the optimizer is not separable from the architecture.
Current State (as of 2026-08-11)
Attention selectivity and KV compression have been two separate lines on this page. A production-scale model just shipped them as one operator.
Motif 3 (08-11) (2608.09119, Motif Technologies, Korea's Sovereign AI Foundation Model Project) is a decoder-only MoE at 314B total and 13.2B activated per token, with unusually fine-grained sparsity: 384 routed experts per layer, eight selected per token. Its attention contribution is Grouped Differential Latent Attention (GDLA), which fuses differential attention, which suppresses irrelevant context by subtracting two attention maps, with Multi-head Latent Attention, the DeepSeek low-rank latent-KV construction that compresses keys and values into a shared latent. One operator, both properties: sharper reads and a smaller cache. Around it sit modified manifold-constrained hyper-connections, Expert Specific PolyNorm activations, and multi-token prediction. Pretraining: ~12.5T tokens, context to 256K, selective MXFP8 compute and communication, fused memory-efficient kernels, window-aware context parallelism.
Why the fusion is the interesting part. VideoMLA (06-02) established the counterintuitive fact that MLA works even where pretrained attention is not low-rank, because the MLA bottleneck dimension rather than the pretrained spectrum determines effective rank, which decoupled MLA from the spectral story it was sold with and predicted it would transfer broadly. GDLA is that transfer, into a differential-attention backbone at 314B. The paper runs no ablation isolating the two halves, so what the fusion buys over either alone is unmeasured, which is the obvious missing experiment.
It also matters for a gap kv-cache.md has carried since 08-04. SemiAnalysis's Kimi K3 primer proposed KV throughput, cache size divided by prefill time, as the honest efficiency unit and observed that no open-weight model ships static KV compression. Motif 3 does, via GDLA's MLA half. It is therefore a candidate for the comparison this wiki says nobody has run: KDA-plus-MLA against the GQA-sparse family (GLM 5.2's DeepSeek Sparse Attention, DeepSeek V4's Compressed Sparse Attention, MiniMax M3's sparse attention, MiMo V3's HySparse) at matched sequence lengths. The report publishes no cache-per-token figure, which keeps this wiki's standing 07-30 gap open even in the model that would have closed it.
One cross-page echo. Expert Specific PolyNorm activations put source-specific parameters in the feed-forward path while attention stays shared, which is the same partition Poly-OPD (08-06) reached from gradient-agreement measurements (share attention LoRA, keep FFN adapters teacher-specific) and Physics of Multimodal Pretraining (08-06) reached from cross-modality architecture ablations on 13.5B MoE models over 2T tokens. Three independent routes to "attention is a source-agnostic mixing operation and the feed-forward layer is where source-specific knowledge lives," and Motif 3 does not cite either.
Prior State (as of 2026-08-04)
A third line of work opens on this page: write sparsity. And the hybrid recipe finally gets audited at the serving layer, where part of its advertised advantage turns out not to survive.
The architecture side. Raven (2607.25357, Kurate cs.LG #16, ai_rating 7.0, the highest on either board this week, absent from HuggingFace entirely) is from Arshia Afzal and Volkan Cevher (EPFL) with Aviv Bick, Eric Xing and Albert Gu (CMU, Cartesia AI). It frames every linear-time model as one of two extremes in how it writes to memory. SSMs and linear Transformers write densely, updating the whole state per token, so nothing is formally evicted but everything interferes and specific past tokens are hard to recover. Sliding-window attention writes sparsely with explicit per-token representations, so in-window recall is exact, but eviction is decided by position and recall falls off a cliff at the window edge. Raven fills the missing quadrant, sparse write with content-selective gradual decay: a fixed set of memory slots, and at each step a learned input-dependent router decays and updates only a selected subset, leaving the rest untouched. The comparison that isolates the contribution is against Gated Slot Attention and ABC, which already routed over a fixed slot set but still wrote densely to all slots; Raven's claim is that not touching a slot is the load-bearing half, because an unrouted slot stays exactly retrievable. Competitive with or better than prior linear-time baselines on recall-intensive benchmarks, holding at 16x training context length, gains persisting in hybrids.
This is a genuinely new axis for the page. Line 1 improves the recurrent rule (Mamba2/GDN/KDA as online SGD, MDN (05-11) adding momentum, Gated DeltaNet-2 (05-24) decoupling erase from write). Line 2 improves the estimator order of the read (Parallax (05-29), local-constant to local-linear). Raven changes neither: it changes which part of the state gets written at all. MDN improves how a dense update moves, Parallax improves what the read computes, Raven decides most of the state should not be touched. It also reads as the stronger form of what Gated DeltaNet-2 was reaching for, since an unrouted slot needs no erase gate. Composition with the recurrent-rule line (a momentum delta-rule applied only to routed slots) is unrun and is now the second uncombined pair on this page.
The serving side. SemiAnalysis on Kimi K3 prices the hybrid recipe at the kernel and cluster level and produces the page's first real negative result about it. KDA is traced from linear attention through DeltaNet to Gated DeltaNet: the diagonal-matrix expansion of the forget gate gives per-channel memory decay and, as a side effect, enough positional awareness that Kimi removes RoPE from MLA entirely. FlashKDA's derived complexity is linear in sequence length for prefill and constant for decode, with decode traffic dominated by reading and writing the FP32 recurrent state. So far, so good.
Then the qualification, and it matters for every claim on this page about constant-state memory. Prefix caching a recurrent state is structurally hard: engines find cache hits by matching the longest cached token prefix, and you cannot reconstruct state at position t from a snapshot at t-k without replaying, so supporting arbitrary prefix matching would mean caching state at every position, at which point memory grows with sequence length and the entire point is gone. Moonshot's fix is coarse snapshotting, vLLM caching KDA state every 32K tokens plus at prompt boundaries. SemiAnalysis states the consequence flatly: linear attentions like KDA greatly reduce KV cache memory, and realistically during serving they do not consume a constant amount. That is a direct qualification of the 07-30 practitioner measurement of a ~20x KV footprint gap between hybrid and dense models; the 20x is real for a single local session and optimistic for a multi-tenant server with prefix reuse. Raven does not escape this, since it also carries a fixed state to snapshot, but a slot-structured state is addressable and mostly untouched per step, which at least makes differential checkpointing conceivable where a monolithic dense state does not.
The surprise, and a prediction with the author's own data behind it. Kimi K3 keeps MLA as its full-attention layer while every other frontier open-weight model moved to GQA-based sparse attention (GLM 5.2's DeepSeek Sparse Attention, DeepSeek V4's Compressed Sparse Attention, MiniMax M3's MiniMax Sparse Attention, MiMo V3's HySparse). MLA's absorption trick cuts decode compute at the cost of extra prefill compute, which is right for decode-dominant reasoning and wrong for prefill-dominant agentic work. SemiAnalysis predicts K4 drops it, and their own trace corpus supports it: replaying an hour of real Claude Code traces gives a median 142K input tokens against 444 output tokens per turn over a median 65 turns per session, a 320:1 prefill-to-decode ratio.
And Block Attention Residuals now have two independent adopters in two modalities. Kimi K3 replaces the residual stream with softmax attention over the depth axis: each layer attends over previous layers' outputs, with the query a learned per-layer parameter rather than derived from the current token, so every layer gets fine-grained selective access to earlier representations instead of one shared additive channel. Block-chunking cuts communication from O(Ld) to O(Nd), reported at 1.25x compute efficiency, bounded output magnitude where standard residual networks grow with depth, and only 4% pipeline-parallel overhead after cross-stage caching. SANA-Video 2.0 (07-24) used the same mechanism in a video diffusion transformer for a ~12% lift in deep-layer effective rank. Two labs, two modalities, one mechanism: the residual stream is a bottleneck and depth-axis attention is the fix now clears this wiki's bar for an architectural trend rather than one lab's idiosyncrasy.
Three details from the primer worth keeping on the page. First, FlashKDA's derived cost, since Moonshot open-sourced the kernels: two launches, K1 preparing chunk-level tensors in parallel (cumulative decay, decayed Q/K, the (I+L)^-1 inverse via Neumann factorization, the causal mask) and K2 running the chunk recurrence, for 12C^3 + 8C^2 D + 6C D^2 FLOPs and 8C^2 + 22CD bytes per chunk of size C. Decode is roughly 7D^2 FLOPs against 8D^2 bytes. Second, the module details around KDA: short left-padded convolutions on Q, K and V to capture local token dependencies without breaking causality, and L2 norm on Q and K to stabilize the eigenvalues of the transition and output matrices, with the output forget gate implemented as a plain linear transform in K3 where Kimi Linear used a low-rank projection. Third, the 3:1 KDA-to-MLA ratio is now the third independent arrival at the same hybrid band, after SANA-Video 2.0 (07-24) fixed 25% softmax as the video-diffusion quality-efficiency optimum and Ling/Ring-2.6 (06-16) migrated a 1T model in place to 7:1 Lightning Attention to MLA. Three subfields, one design-point neighbourhood, and no theory for why.
And Block Attention Residuals have three adopters, not two. MHAR (07-31) is the third, which puts the depth-axis-attention mechanism at three papers in two weeks.
Current State (as of 2026-07-24)
The hybrid recipe crosses into video, and sparse attention becomes a distributed-scheduling problem. Two same-week papers attack the video-attention bottleneck from the two layers this page tracks. SANA-Video 2.0 (2607.21553, NVIDIA + MIT, Song Han) ports the LLM hybrid recipe (Qwen3-Next, Kimi-Linear, Kimi K3) to a video diffusion transformer trained from scratch: gated linear attention does the O(N) bulk mixing, full-softmax "anchors" at a 3:1 ratio (25% softmax fixed as the quality-efficiency optimum) restore full-rank interactions, and Block Attention Residuals route anchor summaries forward to lift deep-layer effective rank ~12%. It hits softmax-level VBench quality at 3.2x faster forward pass (720p/60s), 120x faster than Wan 2.2 after kernel optimization, 720p on one H100. This is the direct video instance of the 06-16/06-17 finding that retrieval lives in the full-attention layers and the efficient layers shape the path. FVAttn (2607.16190, Tencent WeChat HPC) works the systems layer: training-free Top-p sparse attention gives each head a different workload, which under multi-GPU sequence parallelism becomes a rank-level straggler problem, so FVAttn migrates heavy heads across GPUs via P2P at runtime (imbalance 1.34→1.08, 4.41x over FlashAttention). Together: SANA avoids 75% of softmax by construction, FVAttn load-balances the softmax that remains. The page's estimator-order line (Parallax) and recurrent-rule line (GDN/MDN) now have a distributed-execution sibling.
Current State (as of 2026-06-17)
The mechanism behind the hybrid convergence: retrieval lives in the full-attention layers, the efficient layers just shape the path there. One day after two frontier labs shipped mostly-linear backbones (Nemotron 3 Ultra, Ling/Ring-2.6), Rethinking Efficient Attention in Hybrid Architectures (arxiv 2606.15378) supplies the mechanism study the page asked for. Three findings. (1) Scaling: the efficient-attention choice controls how fast long-context ability emerges, not the final ceiling — hybrids converge given enough training. (2) Mechanism: long-range retrieval is carried by the full-attention layers; efficient attention (SWA, linear mixers) shapes their optimization trajectory. This produces Large-Window Laziness — a bigger sliding-window attention window delays retrieval-head formation in the full layers, because the cheap layers cover for them. (3) Design: applying NoPE (no positional encoding) to only the full-attention layers of a small-window SWA hybrid substantially improves long-context performance at negligible short-context cost. This directly closes the open question 06-16 left ("how much retrieval precision does the 7:1 ratio cost?"): the ratio is safe as long as the full layers stay un-lazy, and it hands the non-uniform KV line a layer-axis allocation prior (compress the efficient layers hard, protect the full layers — see kv-cache). Same-day rhyme: LoopCoder-v2 and Variable-Width Transformers both reject uniform allocation across depth/loops, the same "capacity should follow layer function" prior this paper grounds for attention.
Current State (as of 2026-06-16)
Hybrid linear/full attention becomes the default frontier backbone: two independent labs ship it the same day. The recurrent-rule line the page tracks (Gated DeltaNet-2, MDN) and the linear-attention serving thread (Kimi Linear, MiMo-V2-Flash on the KV cache page) converge into production releases on 2026-06-16. NVIDIA's Nemotron 3 Ultra (arxiv 2606.15007) is a 550B/55B-active MoE with a Mamba (SSM) + attention hybrid backbone, 1M context, NVFP4-native pre-training, claiming ~6x throughput at on-par accuracy — open-sourced end to end. Ant's Ling-2.6 / Ring-2.6 (arxiv 2606.15079) migrates a 1T model in place from full GQA to a 7:1 Lightning Attention (linear) : MLA (low-rank latent KV) hybrid via 10T tokens of continued pre-training. Two things stand out. First, the convergence: SSM-or-linear layers carrying the sequence, full/latent attention layers handling what needs exact mixing, is no longer experimental — it is the assumed shape of a new frontier model. Second, Ling/Ring's pairing of two distinct efficiency families the page separates — linear attention (fixed-state) and MLA (the VideoMLA/DeepSeek low-rank-KV line) — in a fixed 7:1 ratio is an explicit design point on how much exact attention a reasoning model still needs. The "migrate, don't retrain" path (swap the attention type via continued pre-training rather than from scratch) is the practically important claim: it makes linear-attention efficiency adoptable for anyone holding a trained full-attention model. Open question the page can now ask concretely: how much retrieval precision the 7:1 ratio and the Mamba layers cost at 1M context, the usual linear-attention tax, neither release characterizes here.
Current State (as of 2026-06-04)
μP transfer reaches gated linear-attention models (2026-06-04). Unlocking Feature Learning in Gated Delta Networks at Scale (arxiv 2606.04048) supplies the missing theory that lets the recurrent-rule line train at scale without re-tuning. The Maximal Update Parametrization (μP) gives zero-shot hyperparameter transfer across widths for standard Transformers; its derivation did not cover structured state transitions and gating, so it failed to transfer to Gated Delta Networks (GDN). This paper rigorously propagates coordinate-size estimates through GDN's forward pass, gates, and recurrent state, derives the scaling rules, and shows learning-rate transfers cleanly across widths under both AdamW and SGD where standard parametrization fails. This is the third leg of the scale-stable-architecture program: MoE μP (05-17, Kurate cs.LG #14 this week, ai_rating 9.0) did it for mixture-of-experts, Parallax (05-29) showed Muon unlocks local-linear-attention capacity, and now GDN gets its parametrization. The recurrent-rule line (Gated DeltaNet-2 05-24, MDN 05-11) improved the state update; this makes that family trainable at scale. Industry shipped the matching recipe the same morning: the Marin / Open Athena pretraining post reported 6.7x theoretical (3.6x realized) dense→MoE speedup plus a 1.3x AdamH→MuonH optimizer swap, the open-recipe mirror of the parametrization theory. The obvious missing experiment: whether Muon unlocks GDN the way it unlocked Parallax.
Current State (as of 2026-06-02)
Two papers attack the supervision target and the reasoning representation, not the attention read itself, but both bear on representation geometry. NITP (Next Implicit Token Prediction, arxiv 2605.24956) argues standard next-token prediction's one-hot logit-space supervision under-constrains the latent representation space, letting hidden states drift into degenerate, anisotropic geometry. It adds a dense continuous target in representation space — predict the next token's implicit semantic content using the model's own shallow-layer representations as self-supervised targets — and reports +5.7% MMLU-Pro on a 9B MoE at ~2% extra training FLOPs and zero inference cost. The anisotropy/representation-collapse framing connects to the same geometry concerns that motivate interpretability work. GLR (Geometric Latent Reasoning, arxiv 2606.02248) formulates latent reasoning as a path-approximation problem inside the pretrained token-embedding space: a lightweight transition head predicts iterative direction updates, anchored to text CoT traces but free to deviate continuously. On math with Qwen3 it induces substantially shorter generations with no explicit length objective, exposing a new tradeoff between latent compute budget, output length, and accuracy. Both treat the embedding/representation space as the object to shape — NITP during pretraining, GLR during reasoning.
Current State (as of 2026-05-29)
Parallax: local-linear estimator made scalable, and Muon codesign. Parallax reframes softmax attention as a local constant estimator (a kernel-weighted mean) and upgrades it to a local linear estimator (Local Linear Attention, LLA), which nonparametric statistics says has a strictly better bias-variance tradeoff for associative memory. Exact LLA needs a per-query linear solve (a conjugate-gradient inner loop) that is too slow and unstable to pretrain; Parallax removes the solver by learning an extra query-like projector that probes the KV covariance directly, and ships a hardware-aware kernel that raises arithmetic intensity above FlashAttention, pushing attention from memory-bound toward compute-bound. The decode kernel matches or beats FlashAttention 2/3; pretraining at 0.6B and 1.7B shows perplexity gains under both parameter- and compute-matched controls (a Pareto improvement). The secondary headline: Muon unlocks Parallax's capacity where AdamW does not, claimed as the first architecture-optimizer codesign result for an attention mechanism.
This is significant on two axes. On the estimator-order axis, Parallax is a different and arguably deeper move than the recurrent-rule improvements below: it changes what the attention read computes, not just how the linear-layer state evolves. On the optimizer-codesign axis, it converges with the scaling-laws literature (see below).
The two lines of work
Line 1 — recurrent rule inside linear layers (subquadratic, retain history):
Mamba2 / GDN / KDA : state update = one step of online SGD on a latent objective
MDN (05-11) : add MOMENTUM to that update, parallelized without breaking causality
Delta-Attention : residual/delta corrections on the linear read
Line 2 — estimator order of the attention read itself:
softmax attention : local CONSTANT fit (kernel-weighted mean of values)
Parallax (05-31) : local LINEAR fit (better bias-variance, solver removed)
Cross-cutting — optimizer is not separable from architecture:
Optimizer-spectral-scaling (05-23) : Muon scales rare-token rank where AdamW stalls
Parallax (05-31) : Muon unlocks LLA capacity where AdamW does not
Key papers
Parallax (2026-05-29) — Parameterized Local Linear Attention. Replaces exact-LLA's per-query CG solver with a learned covariance probe; hardware-aware kernel matches/beats FlashAttention 2/3; Pareto perplexity gains at 0.6B/1.7B; Muon-architecture codesign. → summary
MDN: Momentum DeltaNet (2026-05-11) — Parallelizes stepwise momentum for delta linear attention without breaking causality, with a spectral-stability analysis constraining the gating. Beats Transformers, Mamba2, GDN at 400M/1.3B. The first paper this month to push the recurrent-rule substrate. → summary
Delta-Attention residuals (2026-05-20) — Residual/delta corrections to the attention read. → summary
Same Architecture, Optimizer-Induced Spectral Scaling Laws (2026-05-23) — Holding architecture and width fixed, Muon realizes near-linear hard-rank scaling on rare-token representations (β=1.02) where AdamW stalls (β=0.44); matched loss does not imply matched representation geometry. The theoretical companion to Parallax's Muon-codesign finding. → summary
Key concepts
- Local constant vs local linear estimator: softmax attention reads a kernel-weighted average of nearby values (constant fit); local-linear attention fits a line, capturing the local gradient of the value field for better associative recall.
- Arithmetic intensity: FLOPs per byte of memory traffic. FlashAttention is memory-bound; Parallax deliberately raises arithmetic intensity to enter the compute-bound regime where GPU FLOPs are not bandwidth-starved.
- Recurrent rule as online SGD: Mamba2/GDN/KDA interpret their state update as one closed-form online-SGD step on an implicit memorization objective; momentum (MDN) is the standard SGD fix applied to that update.
- Architecture-optimizer codesign: the empirical finding that the right optimizer (Muon) is a function of the architecture, so ablating one while fixing the other mis-measures both.
Open problems
- Does the local-linear advantage survive scale and long-context retrieval? Parallax tops out at 1.7B and reports no needle-in-haystack / RULER numbers; in-context retrieval is exactly where linear-attention substitutes historically collapse. Raven (08-04) is the first paper to attack this as its primary target, arguing the collapse is a write-policy artifact, but it too reports no RULER numbers and no scale.
- Why does Muon unlock these architectures? The codesign effect is empirical; no mechanistic account links Muon's update geometry to the local-linear estimator or to rare-token rank.
- Do recurrent-rule (MDN) and estimator-order (Parallax) gains compose? Nobody has combined a momentum delta-rule with a local-linear read.
- Do write-sparsity (Raven) and recurrent-rule (MDN) gains compose? Raven decides which slots to write, MDN decides how a write moves. Second uncombined pair on this page, added 08-04.
- Does a slot-structured state solve the prefix-caching problem? KDA's monolithic dense state has to be checkpointed on a fixed 32K schedule because it is not addressable. Raven's slots are addressable and mostly untouched per step, which makes differential checkpointing conceivable. Nobody has connected the two papers.
- Does sparse write help or hurt Large-Window Laziness in hybrids? The 06-17 mechanism study found retrieval lives in the full-attention layers and a bigger cheap window delays retrieval-head formation because the cheap layers cover for them. If an efficient layer can itself carry high recall, does retrieval work shift back into it, and does that reintroduce laziness?
Hybrid linear attention has its own outlier morphology (2026-08-14)
This page has tracked hybrid architectures (interleaving cheap linear-attention layers with a few full-attention layers) mainly through what they cost and what they retrieve. Massive Activations in Hybrid Linear Attention LLMs (08-14) adds the first systematic account of what their activations look like, and the structure is architecture-aligned rather than incidental.
Two morphologies. Pre-attention spikes (PAS): massive activations reliably spike in the layer immediately before each full-attention layer. Inter-spike plateaus (ISP): those elevated values can persist through the intervening linear-attention layers rather than decaying. As full attention gets denser, successive spikes connect through plateaus until the picture converges on the stable massive-activation morphology already known from full-attention models. The recurrence is established across five linear-attention architectures, six hybridization configurations, five data domains, and open models from 1.2B to 397B total parameters.
The proposed mechanism is a write-sink-cancel lifecycle governed by cancellation timing: a PAS is written, absorbed, and cancelled locally and fast, while an ISP is the same process with delayed cancellation. Controlled pretraining of GDN-based hybrids to 1.3B shows both emerge early and respond asymmetrically to gating, with full-attention output gating strongly attenuating magnitude but not layerwise organization, and removing GDN gates amplifying only modestly.
Why it belongs on this page and not only on the efficiency page. Massive activations are the reason low-bit quantization of activations is hard: a few enormous values set the dynamic range the whole tensor must fit. Knowing they appear at predictable layer positions in hybrids, immediately before full-attention layers, turns a global precision problem into a per-layer one. That is the same shape of result as the wiki's recurring finding that hybrid capability is not uniformly distributed across layers, and it is the mechanistic complement to the 06-17 Large-Window Laziness study, which found retrieval concentrates in full-attention layers. Retrieval concentrates there and so does the activation outlier structure, which is consistent with the full-attention layers doing the load-bearing work that the linear layers route toward.
Open problem added: does the PAS position predict where per-layer quantization budget should go? Nobody has run a mixed-precision hybrid that assigns higher precision only to PAS-adjacent layers and measures the accuracy-per-bit curve against uniform assignment. The paper localizes the outliers and stops short of exploiting them. Code is at StartluxLabs/Massive-Activations-HLA.
The vertical channel is narrower than the horizontal one (2026-08-14)
Every mechanism on this page so far modifies how a token attends across the sequence. Full-bandwidth transformer (08-14) points at an axis this page has never treated: the channel between decoding steps. At each step a standard model computes a rich top-layer hidden state, samples one token from it, discards the hidden state, and feeds only that single symbol back to the bottom of the stack. Horizontal bandwidth is wide; vertical bandwidth is one token.
The fix is small and structural: fuse the previous top-layer hidden state with the sampled token embedding through a gated linear unit and feed that back as the next input. Non-verbalized computation re-enters the stack with a fresh depth budget instead of having to be re-derived or spelled out as reasoning tokens. The transformer, the KV cache, and the language-modeling objective are all unchanged, which is what makes it deployable rather than a research architecture.
Latent feedback breaks parallel teacher forcing, since step t's input depends on step t-1's output. Their answer is a scheduled multi-pass objective: introduce feedback late in pretraining, mixed with a small fraction of deeper feedback passes for stability. At 1B parameters and 400B tokens it improves validation loss, 5-shot evaluation, math and code generation, and instruction-tuned performance, matching standard transformers trained on roughly 1.5x more tokens at negligible per-token decoding overhead, and producing shorter reasoning traces at equal or better accuracy.
Relation to the looped-transformer line. Looped transformers buy extra effective depth by re-running layers within a step. Full-bandwidth buys it across steps at the cost of one gated fusion. The shorter-reasoning-traces result is the interesting overlap: if latent feedback lets computation persist without being verbalized, then a chunk of what chain-of-thought currently does is being paid for in tokens because the architecture gave it no other place to live. That is a claim about test-time compute, not just architecture, and it makes the direct comparison against a looped model at matched depth budget the obvious missing experiment.