Looped Transformers / Iterative Latent Depth

Looped (or "universal") transformers add computation by applying the same shared block repeatedly, rather than stacking many distinct layers. The promise is parameter efficiency (one block, reused) and adaptive depth (loop more on hard inputs). The recurring cost is that naive sequential looping inflates latency and KV cache linearly with loop count, and that more loops do not monotonically help.

Current state (as of 2026-09-02)

The loop-count question is settled to two, by a second paper that got there independently and brought a mechanism. SMELT (09-02) (arxiv 2609.01343, Tsinghua / ByteDance Seed / M-A-P / TokenWave.AI) is the first study to compare looped against unlooped models while matching three budgets simultaneously: per-token FLOPs, total non-embedding parameters, and KV cache. MoE is what makes that feasible, because it decouples total parameters from per-token FLOPs, so you can narrow the hidden dimension to hold FLOPs fixed and then raise the expert count to recover capacity. The surviving recipe is loop the middle half of the layers twice, leaving early and late layers single-pass. Scaled across four sizes to 54B non-embedding parameters with a separate Chinchilla-style law fit per architecture, it saves 6.8 to 18.0% of training FLOPs on the compute-optimal frontier, with the advantage largest on Code and growing with sequence length and in-context example count.

Why this is the important entry on this page. The page's headline finding since 06-17 has been LoopCoder-v2's empirical result that two loops is optimal and three-plus regress, with a diagnosis (later loops give oscillatory low-diversity updates while the positional-mismatch cost keeps accruing) but no account of why the second loop is productive. SMELT supplies one: the second visit reduces the attention sink, the pile-up of attention mass onto a few early or delimiter tokens carrying almost no information, and redirects that mass toward content-relevant tokens. Two papers, two and a half months apart, different scales, dense versus MoE, loose versus strict budget control, same loop count, and now one mechanism. This crosses from "an empirical ceiling somebody found" to "a property of the architecture." It also explains the shape of the gains: attention sinks cost most when there is a lot of context to discriminate among, which is why the advantage grows with sample length and in-context examples rather than being a fixed offset.

Two of this page's three open tensions move.

• "Saturation vs adaptivity" is now half-resolved and better posed. SMELT lands hard on the fixed-depth side and adds a dimension the tension did not have: loop depth should vary by layer position, not by input. LoopWM's claim for adaptive per-step depth is untouched but now looks like the minority position. The sharpened open question: nobody has ablated adaptive-per-input against fixed-middle-half looping on the same task under matched budgets, and until someone does, "loop the middle twice" is the default and adaptivity is the burden of proof.
• "KV cost is the gating constraint" is superseded by a stronger framing. The page recorded that looping was only practical because LoopCoder-v2 shared KV across loops via gated sliding-window attention. SMELT does not need a KV-sharing trick, because it treats KV cache as a budget to hold constant rather than a cost to mitigate. That reframing matters beyond looping: KV size bounds the longest servable context, so a looped model that quietly needs a larger cache is not a drop-in replacement at serving time regardless of its loss curve. Matching KV is a precondition for the loop's benefit being a well-defined quantity at all.

The unresolved axis is now latency, not loss. Looping serializes computation, so matched FLOPs is not matched wall-clock, and SMELT reports the frontier saving in training FLOPs without addressing serving latency. Concurrent MoE-looping work (MoEUT, LoopMoE) matched wall-clock time instead. Those two literatures are optimizing different objectives and no head-to-head exists, which is the cleanest experiment anyone could run on this page right now.

It also puts a claim on scaling-laws.md. SMELT's frontier saving is only legible because it fits a separate law per architecture; one law across both would average the effect away. That is a third kind of failure of the received law, distinct from Skaling's misspecified additive form (08-10) and LLaDA's cross-objective transfer failure (08-05): a claim that architecture belongs inside the law.

Prior state (as of 2026-06-17)

The loop became a first-class scaling axis this week — three papers, three domains, one shared claim: iterative latent depth is orthogonal to width and data.

• LoopCoder-v2 (arxiv 2606.18023) — a 7B Parallel Loop Transformer (PLT) coder trained from scratch on 18T tokens. PLT makes loop count cheap via cross-loop position offsets (CLP) and shared-KV gated sliding-window attention (the KV is reused across loops, so the cache does not grow linearly). The headline empirical result: two loops is optimal (SWE-bench Verified 43.0→64.4, Multi-SWE 14.0→31.0), and three-plus loops regress. Diagnosis: loop 2 supplies the productive refinement; later loops give oscillatory, low-diversity updates while the fixed CLP positional-mismatch cost keeps accruing.
• Looped World Models (LoopWM, arxiv 2606.18208) — the first looped architecture for world simulation. One shared block iteratively refines the latent environment state, claiming up to 100x parameter efficiency with adaptive loop depth scaled to each prediction step's complexity.
• Solve the Loop: Attractor Models for Language and Reasoning (Fein-Ashley & Rashidinejad, Kurate cs.LG #6, ai_rating 7.5) — frames reasoning as convergence to an attractor of a looped/recurrent dynamic, the theoretical sibling of the two architecture papers.

This crosses the wiki's ≥3-papers threshold for declaring a pattern.

Open tensions

• Saturation vs adaptivity. LoopCoder finds fixed deep looping saturates hard at two loops for code. LoopWM claims adaptive per-step loop depth is a feature. The unresolved question: does adaptive depth dodge the oscillatory regression LoopCoder hits past loop 2, or just hide it? No paper yet ablates adaptive vs fixed looping on the same task.
• KV cost is the gating constraint. Looping is only practical because LoopCoder shares KV across loops (gated SWA). This ties the loop line to the wiki's KV-sharing thread (Raschka's MHC/compressed-attention survey, 05-17): sharing KV across loops is the same instinct as sharing it across heads/layers.
• Test-time-compute framing. Looping is a form of test-time compute that does not lengthen the output. That makes it a different lever from chain-of-thought; the LoopCoder result (a hard ceiling on useful loops) parallels the broader finding that scaling inference compute is non-monotonic, echoing the Extrapolation Cliff (05-14, a closed-form threshold past which on-policy distillation collapses).

Related concepts

• attention-mechanisms — shared-KV gated SWA is the enabling attention design.
• kv-cache — KV-across-loops sharing is what keeps looping affordable.
• Variable-Width Transformers — the width-allocation efficiency axis, orthogonal to looped depth.

2026-09-13: "infinite depth" arrives and it is a different axis, so the two-loop ceiling survives

RLT (09-12) (Yifan Zhang, technical report, 263 GitHub stars within a day) pairs a causal encoder that builds global key-value memory with a recurrent decoder that carries its final hidden state and its layerwise sliding-window attention cache across every prompt and response token, never resetting at the serving boundary. The carried state is H_t = (s_t, C_t^D) with transition (s_t, C_t^D) = D_phi( Merge(e_t, s_{t-1}) ; M_{<=t}, C_{t-1}^D, t ). With 48 encoder and 48 decoder layers, 96 logical blocks execute per token while the temporal path traverses 48t decoder blocks after t tokens. The report is explicit that "infinite depth" means an extensible temporal path, not infinite work within a token, and that realized reasoning gains, hardware efficiency and RL scaling all "remain to be established."

It does not contradict this page's settled finding, and the reason is the entry. LoopCoder-v2 (06-17) found two loops optimal, three-plus regressing, and SMELT (09-02) reproduced that count independently under simultaneously matched FLOPs, parameters and KV cache, with a mechanism (the second visit reduces the attention sink). Both are statements about re-reading the same input. RLT never revisits a token's representation. The page's headline should now read: two loops is optimal per token, and the depth-per-token ceiling does not bound depth-per-sequence. These are two different quantities and the literature has been using one word for both.

It satisfies SMELT's KV precondition by construction rather than by argument. SMELT's methodological contribution here was that KV size bounds the longest servable context, so matching KV is a precondition for a loop's benefit being well-defined. RLT's decoder cache is bounded by the sliding window W rather than by sequence length, so its depth growth genuinely costs no decoder KV growth. Encoder memory is the conventional growing one.

And it is the first entry on this page to treat serving latency as a design constraint rather than an afterthought. The 09-02 entry closed by naming latency as the unresolved axis: looping serializes computation, matched FLOPs is not matched wall-clock, SMELT reported training-FLOP savings and said nothing about serving, and concurrent MoE-looping work matched wall-clock instead with no head-to-head anywhere. RLT's second design principle is "parallel work around a recurrent core": batch known-token encoder work and independent decoder updates, reuse weights and memory, checkpoint activations, all while preserving the reference computation. Whether that recovers the serialization loss is precisely what is unmeasured.

Cross-page note. A carried fixed-size recurrent state instead of a growing per-token cache is the same structural bet as the KDA layers in kimi-k3-in-c (09-12), where 69 of 93 layers carry a fixed-size recurrent state and that is what kept a 2.78T-parameter model inside 8.24 GB. Two unrelated projects in two days concluded that the way to make long contexts cheap is to stop storing per-token state.

The caveat is the whole evaluation section. There are no experiments. No loss curve, no benchmark, no ablation, no wall-clock number, no matched-budget baseline. The report says so. There is also an unaddressed risk: a state carried across every token with no reset can drift, and nothing in the transition is a contraction.

2026-09-27: the latency gap gets its first answer, and a second vote for sparse loops

FlashLoop (09-27) (arXiv 2609.29812, ELLIS Tübingen / MPI-IS) is the first entry on this page that attacks serving cost directly. The 09-02 entry named latency as the unresolved axis because SMELT reported only training-FLOP savings. FlashLoop finds three "lazy update" patterns across loops (few tokens change, attention changes sit in a small stable set of key columns, adjacent-loop KV residuals quantize well) and exploits them training-free: up to 1.64x end-to-end speedup and 6x less KV memory, lossless on the paper's benchmarks. The motivating number is the page's strongest evidence yet that parameter efficiency is not serving efficiency: Ouro-2.6B with four loops needs 48 GiB of KV and 27 s of prefill at 32K, against 4 GiB and 3 s for Llama-3.1-8B.

Sparse Layers are Critical to Scaling Looped LMs (09-27) (arXiv 2605.09165, NeurIPS 2026) independently confirms SMELT's MoE result: dense looped models scale worse than standard transformers, Looped-MoE models scale better, because different experts fire on each pass through the shared layers. It adds that loop boundaries are better early-exit points than arbitrary layers.

Updated state. Two independent papers now say looping needs sparse layers to scale, and one shows the serving overhead is mostly redundant and removable. The open tension is their interaction: FlashLoop was tested on dense looped checkpoints, and token-sparse updates may conflict with per-pass expert routing in a Looped-MoE model. The vision-side twin is TWT (09-27), which fuses redundant ViT layer phases at half depth.

2026-09-28: the loop count becomes a per-request dial

LoopFormer (ICLR 2026, resurfaced 09-27) fixes the failure that made loop count a training-time constant: most looped models collapse when run at a different depth than they were trained at. It conditions each loop on its position and step size and trains short trajectories to land where long ones land (shortcut consistency). With FlashLoop (09-27) making each loop cheaper, the serving picture now has both halves: cheaper loops and a choice of how many. Open: nobody has yet shown elastic depth on MoE loops, which Sparse Layers are Critical (09-27) says is how looped models scale, and no one has trained a router to pick the loop count per input.

2026-09-29: the batching problem gets solved, and full looping wins

Continuous Depth Batching (09-29) solves the serving problem that made adaptive depth a paper saving: tokens that exit after different loop counts cannot share a forward pass, so vLLM-style engines could not batch them. CDB rebuilds batches between loop steps, schedules looped and non-looped parts separately, manages looped KV caching, and predicts exits ahead of time so the next batch is ready. On Ouro 1.4B and Huginn 3.5B it reaches up to 99% of the estimated maximum speedup. The architectural finding is the part to keep: fully looped models serve best, because large non-looped pieces (embedding, LM head, unshared blocks) complicate scheduling. With FlashLoop (09-27) (cheaper loops) and LoopFormer (09-28) (robust at any loop count), the serving stack now has all three parts. CDB's exit predictor is a partial answer to the 09-28 open question. It forecasts exits for scheduling but does not choose depth, so a trained per-input loop-count router is still missing. The untested combination is CDB on a Looped-MoE model, which Sparse Layers are Critical (09-27) says is how looped models scale.

2026-09-30: the loop count gets a learned policy, and decoding gets speculative

TaH2 and WaveFront Decoding (09-30). TaH2 post-trains a per-token iteration decider with online labels (does one more loop lower this token's loss?). On AIME its accuracy-per-compute-doubling slope is 2.74 vs 1.79 for the non-looped baseline and it beats the baseline's peak by about 3.4 points at matched compute; its gain grows with depth (+2.8 at 2 to +3.9 at 8) where Ouro and Huginn plateau. This resolves, at token level, the 09-28 Looking Ahead that a small decider would learn a looped model's loop count within 90 days. WFD uses shallow loops as drafts and deep loops as verifiers in the same batched call (2.42x Ouro-2.6B, 3.54x Huginn-3.5B, 4.81x with cross-recurrence KV sharing), training-free. The serving stack now has four pieces: cheaper loops (FlashLoop), batchable exits (CDB), speculative decoding (WFD), learned depth (TaH2). Still all dense; the MoE-looping results (SMELT, Sparse Layers) are untested against mixed-depth batching.

2026-10-01: step size becomes the fifth knob, and depth trades knowledge for reasoning

TAPS, recurrence design, REST (10-01). TAPS adapts each loop's update size online from an exact split of loss sensitivity into steady progress and fluctuation; training-free accuracy gains, and up to 1.56x wall-clock speedup at matched accuracy when trained in. What Makes Recurrence Effective finds extra loops help reasoning beyond the training horizon but hurt knowledge, harder problems do not reliably gain more, placement of unshared layers matters, and channel-wise history-state injection plus timestep conditioning beats initial-state re-injection. REST supervises latent thoughts with four losses (causality, minimality, separability, stability), +7.5 points. Updated state: the knobs are now loop cost (FlashLoop), robustness to depth (LoopFormer), batchable exits (CDB), learned per-token depth and speculative drafts (TaH2, WFD), and update size (TAPS). Open: nobody learns depth and step size jointly, and the knowledge-vs-reasoning trade predicts TaH2's AIME gains will shrink on knowledge-heavy evals.

2026-10-02: a joint scaling law for loops and experts

Loop Scaling Laws (10-02) fit recurrence and sparsity together with size and data, via a bounded, sparsity-conditional mapping from loops to effective parameters. Sparsity gives about 3x active-parameter efficiency, recurrence about 2x total-parameter efficiency on reasoning, and at trillion-token scale a law-tuned looped MoE matches a non-looped MoE about twice its size at matched training compute. This formalizes the 09-27 "sparse layers are critical" result and partly answers the 09-30 prediction about matched-compute wins (reasoning, not yet code or agents). Looped models now have training laws, batching (CDB), exit policies (TaH2) and step control (TAPS); still no released production model.

2026-10-03: early loops as a free contrastive signal

LoopCD (10-03) contrasts the final loop's prediction with an earlier loop's, a training-free contrastive decoding that needs no second model. Ouro-2.6B-Thinking AIME 2024 pass@1 61.88% to 73.33%; halving loops still matches unguided full depth, cutting forward FLOPs 22.5% to 48.2%. With WaveFront (09-30, early loops as drafts), the second result treating recurrence as a source of free weak predictions.

2026-10-04: how to shape a looped MoE

Foil (10-04), Kurate cs.LG. At fixed stored experts, expert compute and effective depth, flatten (E, D, L) to (2E, D/2, 2L) so every routing decision sees a bigger pool, and untie attention per pass while experts and routers stay shared. Loss improves monotonically with flattening at 100B tokens (most flattened: -0.012 nat). Routing confidence tracks healthy expert use better than load balance. Third loop paper in three days after Loop Scaling Laws (10-02) and LoopCD (10-03); open question is latency, since a 1-layer 16-pass model is sequential by construction.

2026-10-06: per-loop routers, and 9 to 12 stable loops

LOOM (10-06). Looped MoE stalls at two loops because hidden-state variance grows each pass and routers keep picking the same experts. Fixes: scaled residual updates, input re-injection each loop, per-loop routers, and a looping residual. Stable to 9-12 loops at 100M-1.7B; 700M near-iso-FLOP best at 5 loops (PPL 18.36 to 16.54); 1.7B peaks at 9 loops (zero-shot 42.4% to 47.7%, not FLOP-matched). Contradiction with Foil (10-04), which shares routers across passes and flattens the expert pool instead: LOOM says shared routers collapse. Possibly reconciled by Foil's larger pool; unresolved. Fourth looped-MoE paper in five days; still no latency numbers.

2026-10-07: fixed points make loops cheap everywhere

Looped Models Done Right II, ALoDLM, LiFT (10-07). Train loops to converge to fixed points (learned depth prior + orthogonal input injection) and four costs drop: truncated backprop, terminal KV sharing (3x smaller cache at equal quality, 1.6B), a 1.79x faster-prefill student, and 2x faster RL gradients from saved states. ALoDLM gives diffusion LMs per-token loop counts and beats matched AR baselines at 1.7B and 8B; LiFT loops a DiT core for images (-3.34 FID vs DiT-XL/2 with ~60% fewer params). Foil (10-04) re-listed on HF. The LOOM-vs-Foil router contradiction (10-06) is still untested head-to-head.

2026-10-08: a field guide to "looped"

AlphaSignal's explainer (efficiency shorts) separates depth, time and flow loops. Nanbeige4.2-3B (Apache 2.0, 22 layers x 2 passes) costs ~2x block compute and 2x KV cache for a 4B file at ~75% of standard token efficiency; an independent 140M test of the Recurrent Looped Transformer lost to a plain transformer at ~20x the GPU-hours. Pachocki (OpenAI): Astra's computation-graph depth is within 2x of GPT-4. Practical rule: budget a looped model's memory by applications, not parameters, and pair it with terminal KV sharing (10-07) or Hybrid Latent Attention (10-07).

2026-10-09: the RLT paper itself, and a contradiction

Recurrent Looped Transformer (10-09). Parallel encoder plus recurrent decoder fed by the previous token's final state: parity from 40 to 256 bits at 100%, S5 tracking 97% at 8x length. Contradicts in spirit the independent 140M test on 10-08 where RLT lost to a plain Transformer at ~20x GPU-hours; the paper's wins are algorithmic, the test was language modeling. Chunked feedback (every 4 tokens) restores parallelism for parity but drops S5 from 100% to 20%. Berkeley also posted a looped-transformer paper on implicit reasoning (X, details not captured).

2026-10-10: loops as a residual-guarded budget knob

InfiLoop adds a loop-native residual (content weighting plus temporal decay, constant memory) so a 7M looped model keeps improving past 20,000 effective steps (Sudoku-Extreme 97.9%, ARC-AGI-2 pass@2 13.6%). SanSi turns a looped LM into a typed decision model readable after every loop (1-8), "System 1.5": 13.5 points over a non-looped twin. Both in efficiency shorts (10-10).

2026-10-13: no one-loop tax

Elastic Latent Recurrence (10-13). Gated state update with learned step size, mixed loop counts per optimizer update, and loop-index/budget conditioning: at one loop a 4B dense / 30B-A3B MoE hybrid is -0.52 / +1.04 vs matched baselines; four loops add +7.25 / +4.75. Only the last loop's KV is kept, so cache stays at one-loop size. Reusing first-loop MoE routes on later loops cuts mixed-depth update time 11.7%. Revises the 10-08 practical rule (budget ~2x KV and ~75% token efficiency): if ELR reproduces, neither cost applies. Koi-4B released.

2026-10-15: the safety cost of the loop, stated plainly

Monitorability essay (10-15). Understanding AI lays out the trade: looping layers lets a 100-layer model approach a 200-layer one with less memory, and also lets it think without writing. Astra's system card reports a substantial drop in CoT monitorability; an independent test gives it 8.6x better odds than Fable 5.1 of solving a reasoning task with no CoT. Pachocki's counter: Astra's computation-graph depth is within 2x of GPT-4. OpenAI's next flagship is reported to loop too. The 10-10/10-13 "loops as a budget knob" results are the efficiency side of the same dial.

2026-10-16: LatentLoop

LatentLoop: in a twice-applied shared MoE block the first visit matters less; compress its sequence, keep full resolution on the second, spend savings on width. 19.0% predicted compute savings vs vanilla MoE, 7.9% vs looped baseline, across 15 configs up to 10^21 FLOPs.

2026-10-17: a looped model ships

• NVIDIA Nemotron-Labs Dory1.0-4B (page): 8 input + 20 shared recurrent + 8 output layers, input re-injected each loop, 1/2/4 loops from one checkpoint, last-loop KV caching (no cache overhead at any loop count), native NVFP4 pretraining, 256K context. SWE-bench Verified 50.3 / 59.5 / 61.8 at 1 / 2 / 4 loops vs 61.0 for Qwen3.5-9B. Gains flatten from 2 to 4 loops.
• Research-to-release in days: ELR (10-13, last-loop KV), ResidualQuant (10-14, per-loop caches are redundant), LatentLoop and Elastic (10-16). Elorian Glimpse applies latent recurrence to visual representations (page). X notes "a hot week for looped Transformers" (Princeton/UPenn work, not ingested).

2026-10-20: the sandwich wins, and depth goes on demand

LEV + Looped Models Done Right.

• Looped Models Done Right (Part I): controlled comparison of recurrence placement, input injection and state design. Huginn-style prelude/recurrent/coda with input injection beats Ouro on all ten benchmarks at 730M/336B tokens, and the lead transfers to MoE (8B-A0.8B, 500B tokens) with more balanced expert routing. 75% fewer resident parameters than a 112-layer feedforward MoE at matched FLOPs. Controlled evidence for the design Dory1.0-4B (10-17) shipped.
• LEV: looped decision model, one checkpoint answers at any depth; 2.6B goes 63.2% to 76.2% from depth 1 to 4 on JevBench; per-question stopper saves 15-24% compute for <1 point.
• Social: TheTuringPost's "10 looped-transformer papers" was among the most-shared AI posts this window.

2026-10-23: loops in diffusion LMs and recommenders

Looped UDLMs: looping early layers per denoising step reaches baseline NLL with 2.8-3.3x fewer training FLOPs; sampling small loop counts more often cuts inference loops 12%. LoopRec adds per-item exit gates in sequential recommendation. Both match LEV (10-20) in making depth a learned, per-input quantity.