Speculative Decoding

A class of lossless acceleration techniques: a cheap "draft" produces candidate tokens (or blocks), an expensive "target" verifies them via exact rejection sampling, and the verified prefix is committed. The target's output distribution is preserved — quality is unchanged.

Current State (as of 2026-09-09): the technique moves from serving to training, and the physics finally gets stated

Two arrivals, and between them they change what this page is a page about.

Online Draft Co-Training (09-09) (arxiv 2609.07108, NVIDIA NeMo RL, code) applies speculative decoding to RL post-training rather than to serving. Rollout generation dominates the cost of RL post-training, and speculative decoding accelerates it, but the RL setting breaks the assumption every other result on this page rests on: the target is not frozen. As the policy updates, a fixed draft goes stale, acceptance falls, and the speedup decays exactly as training proceeds. Co-training the draft online fixes that, and the paper's contribution is the distributed systems work that lets it run at scale. Two obstacles, both parallelism-layer rather than modelling: context parallelism has no branch attention (fixed by merging rank-local branch attention into packed load-balanced zigzag ring attention), and pipeline parallelism puts the target's intermediate features on other stages (fixed by TapChannel, a side transport that leaves the pipeline schedule untouched). Reported: co-trained drafts closely track the policy baseline, substantial rollout and end-to-end speedups up to 122B, strong scaling at 256K tokens with significant memory savings over prior CP implementations.

This is the third distinct position in three days on where the drafter comes from, and the only one that engages a moving target. Uno (09-07) deletes the drafter by training diffusion weights to sample the model's own autoregressive distribution in parallel. Don't Drop Dropout (09-07) deletes it from the training side, so the model's shallow prefix drafts for itself. This one keeps a separate drafter and drops the frozen-target assumption instead.

It also picks up a diagnosis this page has carried unaddressed since June. Bebop (06-11) found multi-token-prediction acceptance is near-linearly bounded by model entropy, so acceptance collapses during RL precisely when rollouts are most expensive. That was a diagnosis with no system attached. Online co-training is a plausible treatment. Neither paper cites the other, and the composition (co-train against a total-variation objective rather than cross-entropy, per Bebop's fix) is untested and obvious.

One caution the 07-31 audit forces. A drafter that systematically overshoots the target's probabilities is the named collapse condition for collaborative verification. A draft co-trained on the target's own current features is at higher risk of correlated overshoot than an independent drafter, not lower. Acceptance and speedup are reported; the overshoot diagnostic is not.

Chapter 3 of the inference-physics series (09-09) supplies the number that should preface everything on this page. During single-request autoregressive decode the GPU streams every active parameter from HBM for one matrix-vector product and discards it, giving roughly 1.0 FLOP per byte moved against an H100 SXM5 roofline knee at about 295 FLOPs per byte (989 TFLOP/s of 16-bit tensor-core compute against 3.35 TB/s HBM3). That is under 0.35% of peak compute. Speculative decoding is not an optimization layered on a well-used machine; it is the correction for a machine running at one three-hundredth of capability, and a 3x speedup moves utilization from 0.35% to about 1%.

The chapter also grounds the batch-size argument this page made on 09-07 and could not explain. That entry noted every acceleration number here is quoted at low batch, where free parallelism exists, and that Uno claiming a win at the largest batch the device supports was the first entry to assert its speedup survives the regime that matters. Chapter 3 explains why the regime differs: as concurrency rises, weights amortize across requests, the roofline position moves, and the free parallelism speculative decoding exploits is already being consumed, with production engines dynamically throttling speculative depth in response. The mechanisms it names: DeepSeek-V4 MTP (sequential draft heads with shared unembedding, trained as a pre-training auxiliary loss, therefore a zero-overhead drafter at inference), EAGLE/EAGLE-2 (draft on the second-to-top hidden state rather than tokens, +25% acceptance over token-level drafts, with entropy-driven dynamic tree expansion and 2D tree attention masks), Medusa heads, and Zhipu IndexShare (decouple indexer memory from physical KV pages so rollbacks do not churn the allocator). Caveat: everything past the roadmap is paywalled, so the rejection-sampling proof, the kernels and the concurrency derivation are announced rather than read.

Native MTP is the direct counterpoint to online co-training, and the comparison is the cheapest decisive experiment in this area. If draft heads are trained as a pre-training auxiliary objective, they inherit the model's updates for free during post-training, which is exactly the staleness problem NVIDIA built a distributed system to solve for an external drafter. Nobody has run a co-trained external draft against native MTP heads under the same RL run.

Current State (as of 2026-09-07): two groups delete the draft model on the same day, and one of them says the speedup survives large batch

This page has spent the year on how to make drafters cheaper. Today two independent results say the drafter is optional, and they get there by opposite routes.

Uno (09-07) (arxiv 2609.04010, Institute of Foundation Models with UIUC, Cornell Tech, Harvard and Cerebras) splits the parameters in two. The autoregressive weights define the model's distribution and train under ordinary next-token prediction. A lightweight set of diffusion weights, added by a short distillation phase, draws several tokens in parallel from that same autoregressive distribution. Because the object being sampled is the autoregressive distribution itself, the speedup is lossless without a verification pass, and an existing open-weight model can be upgraded rather than retrained. Reported: beats leading speculative-decoding methods at every evaluated batch size including the largest the device supports, up to 3x over the base model, with the 8B Uno outperforming the 26B DiffusionGemma and the proprietary Mercury 2 on agentic tool use, coding and long-context reasoning.

Don't Drop Dropout (09-07) (arxiv 2609.05275) arrives at self-speculative decoding from the training side. Pre-training with layer dropout on a tuned schedule produces a model that is robust to having layers removed, so the model's own shallow prefix becomes the drafter with no separate network at all. It reports up to 1.5x inference speedup at negligible accuracy loss from early exit, intermediate-layer skipping and self-speculative decoding combined, alongside up to 25% fewer training FLOPs.

The batch-size claim is the one that should change how this page reads its own numbers. Every acceleration result recorded below is quoted at low batch, where the accelerator has idle capacity and free parallelism exists. Production agentic serving runs at whatever batch the memory allows, and that is exactly where speculative decoding's advantage gets eaten by verification contention. Uno claiming a win at the largest batch the device supports is the first entry here to assert that its speedup survives the regime that matters, and it is the claim a follow-up should attack first.

Two distinctions worth holding, because "lossless" now means two different things on this page. A speculative decoder is lossless because it checks: exact rejection sampling verifies every draft token at runtime. Uno is lossless because it was trained to be: the diffusion sampler is trained to draw from the autoregressive distribution, and nothing at runtime confirms that it did. Those are different reliability guarantees, and they come apart precisely under distribution shift, which is where a serving system is least able to notice.

Against DraftExpert (08-03), Uno sharpens rather than contradicts. DraftExpert's durable structural claim was that once MoE weights are paged, speculation and memory prefetch are the same computation, because a drafter trained to agree with the target's router is a prefetch oracle you were already paying for. That argument requires a drafter with a router opinion. Uno has none, so on a paged on-device MoE it supplies parallelism without supplying a prefetch signal, and whether that is a net loss on memory-constrained hardware is untested and is the obvious composition experiment.

The two of today's results are themselves composable and nobody has said so. Layer-dropout pre-training and an Uno diffusion sampler operate on different axes (depth and sequence position), so an autoregressive model pre-trained for depth robustness and then upgraded with parallel diffusion sampling has two orthogonal parallelism sources. Cerebras authors appear on both papers, which makes the non-composition slightly more conspicuous rather than less.

Practitioner context, same day. An interactive NeurIPS Education Track tutorial on speculative decoding ("how it evolved, when it stays lossless, and what's next") circulated as the day's highest-scoring inference item on the reader's X feed, framing the technique as running under nearly every hosted LLM. That is the adoption baseline both of today's results are trying to displace.

Prior State (as of 2026-08-03)

The first number from the regime where the weights do not fit, and it is much smaller than the datacenter numbers below. DraftExpert (08-03) (2607.24434, Kurate cs.LG #20) targets end-device MoE inference where routed expert weights exceed accelerator memory and are staged on demand from CPU RAM to a GPU or from Flash to a mobile NPU. Under that constraint the two assumptions this page has always relied on both break. Growing the draft expert set improves accuracy but triggers extra expert loading, so a better drafter is a slower one. And verifying a k-token block activates the union of the experts those k tokens route to, so verification is no longer close to one target step. The paper calls the second effect expansion and builds around bounding it.

Three mechanisms. A fixed-footprint drafter: one lightweight draft expert per layer resident on the accelerator, so the drafter is always shared experts plus top-1 routed expert plus the resident draft expert regardless of uncertainty. Confidence-expansion truncation, which stops drafting on a memory-traffic criterion rather than a probability one, and is the first concrete answer on this page to the long-standing content-adaptive k open question below, though with a different objective than anyone expected. And target-expert prefetching, made possible because the draft expert is self-distilled from the frozen target on residual, logit/token and router-agreement signals, so the drafter predicts which experts the target will pick. Results on DeepSeek-V2-Lite and Moonlight-16B-A3B across both offload paths: 1.45x average decode throughput, 84 to 87% draft acceptance, 86 to 88% prefetch hit rate, output exactly verified and therefore lossless.

Two things to carry forward. The gap between 1.45x here and the 2x-plus in the fits-in-memory results below is the price of paging, and on-device MoE marketing quietly assumes it away. And the durable structural claim: once weights are paged, speculation and memory prefetch are the same computation, because a drafter trained to agree with the target's router is a prefetch oracle you were already paying for. Any on-device runtime running speculation without wiring the drafter's router output into its paging layer discards the free half. Caution on the acceptance figure: tinygrad's profile below measured long-context acceptance at 90.5% on repetitive synthetic prompts and roughly 64% on real code, and DraftExpert reports no prompt-distribution breakdown. → summary

Current State (as of 2026-08-01)

Speculative decoding is now the stated mechanism behind frontier-lab pricing, which makes yesterday's audit urgent rather than academic. OpenAI attributes its GPT-5.6 price cuts to infrastructure work rather than a new model: speculative decoding raising token generation 15% and cutting GPU serving cost 20%, cascading into an 80% price cut on GPT-5.6 Luna (to $0.20 per million input, $1.20 per million output), 20% off Terra, and Auto-review in ChatGPT and Codex CLI moved to Luna for a 10x cost reduction. AI Breakfast's aggregate framing is a 13x drop for the same intelligence, since Luna now matches March's GPT-5.4 flagship at one-thirteenth the token price. Independently, tinygrad published a full serving profile for DeepSeek-V4-Flash-0731 on two RTX Pro 6000 Blackwell GPUs using DSpark K5 fixed-depth speculative decode with W4A8 kernels and an fp8 KV cache: ~245 tok/s sustained single-user, median TPOT ~2.8 ms, TTFT ~0.4 s, beating the model's own runbook figure of 217 to 220 tok/s. → summary

Neither party states whether its verifier is lossless, which is exactly the gap the 07-31 audit opened. When speculative decoding was a 1.5x convenience, an unmeasured quality tax in the distribution tails was tolerable. It is now the mechanism behind the price of a frontier endpoint, so the distribution being silently rewritten is the distribution most production traffic samples from. The audit's two-question checklist (which family is your verifier in, and what is the draft-over-target overshoot) is unanswered by both the largest commercial deployment and the most detailed public practitioner profile.

A new reporting failure this page should track: acceptance rate is workload-dependent by 27 points. tinygrad's own footnote reports DSpark acceptance at 90.5% on repetitive random-token prompts and approximately 64% on real code, same model, same scheme. Every speculative-decoding speedup on this page is a function of acceptance, and almost none of them state the prompt distribution acceptance was measured on. A minimal convention (acceptance on synthetic, code, natural prose, multi-turn chat) would cost nothing and would make the literature comparable for the first time.

Open composition question. W4A8 weights, an fp8 KV cache and a speculative draft all perturb the output distribution and every paper studies one at a time. The interaction that should matter most is between draft quantization and the acceptance test, because a quantized draft is systematically miscalibrated rather than merely noisier, and miscalibration in the draft-over-target direction is precisely the overshoot condition the 07-31 audit named as the collapse mode for collaborative verification.

Current State (as of 2026-07-31)

The lossy branch of this page has been audited, and the verdict is that it was being reported wrong. Revisiting Lossy Verification in Speculative Decoding (2607.26627) analyses the distributions that lossy verification schemes actually sample from and finds two things the individual method papers did not report.

First, the taxonomy collapses. Every published lossy verifier reduces to one of two families: truncation-based (accept a draft token when its probability clears a threshold, meant to approximate top-p or top-k sampling from the target) and collaborative (blend draft and target probabilities). Methods presented as distinct mechanisms are superficially different and mechanically the same.

Second, each family has one named failure. Truncation-based schemes suffer distributional distortion severe enough that performance can fall below the true truncation-sampling baseline, which is the damning comparison: the method produces worse output than the exact version of the distribution it claims to approximate, so the trade is not speed-for-a-little-quality, it is speed for a bug. Collaborative schemes are governed by overshoot, the amount by which the draft's probability for a token exceeds the target's. Bound it and quality holds; leave it uncontrolled and output collapses. Critically, the degradation is described as unstable, meaning it appears on some prompts and not others, so a benchmark mean is the wrong instrument for detecting it.

This answers the open question this page left on VIA-SD. The 06-12 entry below closed with "Open question for review: whether the slim-verifier regeneration path is exactly lossless or an approximation." It is an approximation, and it is a collaborative-family mechanism, so the overshoot condition applies. VIA-SD's reported rejection-rate improvements of 0.10 to 0.22 and its 10 to 20% gain over strong SD baselines need re-reporting under this diagnostic before they can be read at face value.

It also changes how this page should frame its own scope. The page defines speculative decoding as lossless and then tracks four axes of generalization, three of which preserve the guarantee (GRAFT on the draft side, Draft-OPD on draft training, SPD on scheduling) and one of which does not. The correct framing going forward is that lossless SD and lossy verification are two different techniques with two different accounting problems, and lumping them together is what allowed lossy methods to be scored on a speedup number while their quality cost went unmeasured.

Shared shape with Bebop (06-11). Bebop found multi-token-prediction acceptance is near-linearly bounded by model entropy, so acceptance collapses during RL exactly when rollouts are most expensive, and fixed it by optimising total variation directly rather than cross-entropy. Both papers find that a speculative-decoding quantity everyone reports as a scalar is really a function of the output distribution's shape, and both fix it with a distributional correction. That is now two of the page's most useful results arriving by the same route, and it suggests the remaining unexamined scalars on this page (acceptance rate, block length, speedup) are worth re-deriving the same way.

Cheapest available action for anyone serving a lossy variant: log draft-probability minus target-probability for each accepted token and inspect the tail rather than the mean. One subtraction per token, and it is the statistic the paper names as the predictor of collapse.

Current State (as of 2026-06-12)

A fourth axis opens: graded verification cost via intra-model routing. VIA-SD (arxiv 2606.12243) observes that the accept/recompute binary is wasteful because many rejected tokens sit in a "middle zone" — wrong as drafted, but correctable by a model far smaller than the full verifier. It carves a slim verifier out of the full verifier via intra-model routing (no new model, no retraining) and inserts it as a middle tier: tokens route to accept (high confidence), slim-verifier regeneration (medium), or full recompute (uncertain), with the tiering grounded in a KL-divergence decomposition of verification. Result: rejection rates down 0.10–0.22, 10–20% over strong SD baselines, 2.5–3x over non-drafting decode, and it drops into existing SD frameworks without training changes. This is the verify-side mirror of GRAFT (05-20), which made the draft side cheaper by retrieving instead of generating — together they push SD toward "spend the minimum compute that preserves the target distribution at every step, on both sides." Open question for review: whether the slim-verifier regeneration path is exactly lossless or an approximation.

Current State (as of 2026-06-11)

The MTP-acceptance collapse during RL now has a mechanism, and it is an entropy bound. Bebop (arxiv 2606.12370) studies why multi-token-prediction drafting — the embedded-drafter idea Nemotron 3 Super introduced (04-21) — loses its speedup when used to accelerate RL rollouts, the application opened by Speculative Decoding for RL Rollouts (04-30). The answer: MTP acceptance is negatively, near-linearly bounded by model entropy, and RL deliberately raises entropy to explore, so acceptance falls exactly when the rollout stage is most expensive. Three fixes: (1) probabilistic rejection sampling of draft tokens absorbs the entropy disturbance far better than greedy draft sampling; (2) a new end-to-end total-variation (TV) loss directly optimizes the multi-step rejection-sampling acceptance rate, where cross-entropy/KL are suboptimal, lifting acceptance to up to 95% (~10% gain); (3) train the MTP head once before RL — pre-RL TV-loss training holds acceptance steady across the whole run, so no costly online MTP updating. Up to 25% extra throughput and up to 1.8x end-to-end async-RL speedup on Qwen3.5/3.6/3.7. The deeper implication: rollout speed and policy-stability tuning are coupled — anything that holds entropy down (e.g. a tighter trust region like DRPO, 06-10) should raise MTP acceptance for free.

Current State (as of 2026-06-02)

Two same-day papers move speculative decoding past the "build a better draft architecture" era (EAGLE3, DFlash) into better training objectives and better system scheduling.

Training axis: on-policy distillation for the drafter. Draft-OPD (arxiv 2605.29343) diagnoses that SFT-built draft models plateau because of an offline-to-inference mismatch: the drafter trains on fixed target trajectories but is judged on the blocks it proposes under its own policy. Naive on-policy distillation fails because draft models cannot roll out reliably alone, and target-assisted rollout destroys the on-policy signal. Draft-OPD's fix: target-assisted rollout for stable continuations, but replay drafting from the verification-exposed error positions, so the drafter learns from target feedback on both accepted and rejected proposals. Over 5x lossless acceleration for thinking models, +23% over EAGLE-3, +13% over DFlash. This is the same covariate-shift lesson the wiki logs in TA-OPD (06-01), DRIFT (06-01), and DAgger-for-LLM-agents (05-14), now applied to the drafter — see knowledge-distillation.md.

System axis: pipeline-parallel, zero-bubble speculation. SPD (arxiv 2605.30852) replaces multi-token prediction (whose difficulty escalates with depth and adds serial drafting latency) with pipeline parallelism: partition the target into n stages, process n tokens in parallel, and aggregate intermediate features across pipeline depths to predict the next token in parallel with the target's pipeline step. Result: bounded prediction difficulty, higher acceptance, and zero latency bubbles (no idle stages) in single-sequence decode. Draft-OPD improves draft quality; SPD removes draft latency. Together they show the field's gains are migrating from architecture to objective and scheduling.

Current State (as of 2026-04-30)

Speculative decoding has crossed three axes of generalization in April 2026:

The pattern is now: wherever a generator has a slow target and a credible cheap proposer plus a verification signal that preserves target behavior, speculation applies. The draft does not need to be exactly compatible with the target's architecture or even produce the same kind of output (e.g., SDVG's drafter produces video blocks, not tokens) — it only needs to produce candidates that the verification signal can accept or reject.

Key Papers

Nemotron 3 Super (2026-04-21) — Embedded Multi-Token Prediction heads as the speculative drafter. The target model is its own drafter: an MTP head proposes future tokens during the same forward pass that produces the next token, eliminating the external draft model. → summary

SDVG (2026-04-22) — Speculative decoding for autoregressive video generation. Token-level rejection replaced by ImageReward-based quality routing. 1.3B drafter proposes blocks; accepted blocks enter the 14B target's KV cache directly. 1.59–2.09× speedup at 95.7–98.1% quality. → summary

Speculative Decoding for RL Rollouts (2026-04-30, NVIDIA) — Lossless integration of speculative decoding into NeMo-RL + vLLM for RL post-training. Target policy is the verifier; log-probs and policy gradients are computed against target. Draft alignment to rollout distribution is the dominant variable; sweet spot at k=3. 1.77× generation, 1.41× per-step at 8B; 2.5× end-to-end projection at 235B on 2048 GB200s. → summary

Key Concepts

• Lossless acceleration: target's output distribution is preserved; no quality regression.
• Draft alignment: the draft must approximate the target's distribution well; misaligned drafts (e.g., chat draft for a math rollout) lose most of the speedup. The draft initialization matters more than online adaptation once aligned.
• k = speculation depth: number of tokens the draft proposes per round. Higher k = higher upside per accept, but also more wasted draft work on rejection. Empirically k=3 is the sweet spot at 8B reasoning workloads; k=5–7 can be net-negative.
• Verification signal: exact rejection sampling (text), quality router with worst-frame aggregation (video), policy-target log-probs (RL rollouts). The signal must be cheap relative to the target generation cost.
• MTP head as embedded drafter: a small head on the target model that produces multi-token proposals. Eliminates the external draft model and stays automatically aligned because it is part of the target.
• Speculation under policy drift: in RL training, the target policy moves with each gradient step. Weight synchronization between target and draft is required to keep alignment.

Open Questions

• Content-adaptive k: empirically the optimal k varies with rollout phase (predictable math rollouts vs branchy reasoning). A learned k-schedule would beat the fixed-k regime.
• Biased speculation: current methods preserve losslessness. A drafter that deliberately proposes near high-reward regions (rather than approximating the current policy) could trade losslessness for sample efficiency in RL training.
• Cross-modal speculation: SDVG showed the verification signal does not have to be probability matching. The next test is whether speculation works for audio diffusion (worst-window quality routing?) or 3D synthesis.
• Composition with consumer-GPU pipeline parallelism (RoundPipe, 05-01): NeMo-RL spec dec is a generation-during-training optimization; RoundPipe is a training-loop optimization for consumer hardware. Combining them on a consumer cluster could shrink small-lab post-training cost dramatically.
• Composition with token-level value modeling (LenVM, 05-01): LenVM tells the model when to stop generating; speculative decoding makes each step cheaper. They should multiply — but no paper has measured the combined effect.

Related Pages

• KV Cache — speculation populates the target's KV cache cheaply
• Knowledge Distillation — drafts can be distilled, MTP heads can be co-trained
• RL for LLMs — rollout cost dominates, speculation is the answer

2026-09-18: the draft model becomes optional

Uno (09-18) (arXiv 2609.04010) keeps this page's defining guarantee and deletes its defining cost. Every technique catalogued here improves the draft, the verification, or the acceptance rate, and all of them presuppose a separate draft model that must be trained against the target, kept in sync when the target updates, and held resident alongside it. That operational burden, not the acceptance rate, is what has kept speculative decoding out of many serving stacks.

Uno decouples parameters into autoregressive weights trained with ordinary next-token prediction and lightweight diffusion weights learned in a short distillation phase. The diffusion pathway proposes several tokens in parallel, and a sampler family called Ψ-Spec accepts them in a way that provably reproduces the autoregressive model's own distribution. Same losslessness guarantee this page has always required, no second network. The released checkpoint is an Apache-2.0 LoRA adapter over an existing 7B base, so adoption is an adapter load rather than a migration, and the paper states Uno can be trained from scratch or retrofitted onto existing open-weight autoregressive models.

The claim that deserves the most scrutiny is the one the abstract underplays: higher throughput than leading speculative-decoding methods at EVERY evaluated batch size, including the largest the device supports, with up to 3x over the base model. Most decode-acceleration on this page loses its advantage as batching improves GPU utilization, because the spare capacity speculation exploits disappears. A method that holds its gain at maximum batch is behaving differently from everything else here, and that is either the most important result in the paper or a measurement artifact. It is the first thing to check in a reproduction.

Evidence quality: a vendor announcement plus an abstract, reaching the wiki through the X home feed as the day's top-ranked post. No independent reproduction. Note also a discrepancy worth carrying: the announcement quotes up to 2.2x and the abstract up to 3x, against different baselines (diffusion methods versus the base autoregressive model), so neither number should be cited without saying which.

Composition nobody has tried. Uno reduces the number of sequential decode steps; DeepSeek-V4.1-Flash (09-18) reduced the bytes per step to 890 per token by compressing precision, tokens and layers at once. Orthogonal axes, and they should multiply.

2026-09-30: the draft model is the model's own early loops

WaveFront Decoding (09-30) is training-free self-speculative decoding for looped LMs. Intermediate loop outputs are the drafts; weight sharing lets drafts at shallow depth and verification at full depth run in one batched recurrent call along a diagonal wavefront, so drafting and verification overlap instead of alternating. 2.42x on Ouro-2.6B, 3.54x on Huginn-3.5B, 4.81x with cross-recurrence KV sharing, across six Spec-Bench categories. The looped architecture gives speculative decoding a free draft model and a free batching trick.

2026-10-03: the draft is the agent's own history

AgSpec (10-03) is retrieval-based speculative decoding tuned for coding agents: drafts are copied from the session trajectory, workspace files indexed in the agent's output format, and a global corpus; draft length is capped per agent offline and adapted online. Up to 4.37x throughput at batch 1 and 4.76x at batch 16 over autoregressive decoding, beating EAGLE-3 in most settings. Third draft-model-free route in three weeks (09-18 optional draft, 09-30 WaveFront early loops, now retrieval from the agent's own text).

2026-10-09: skip verifications while the draft is confident

DLoop (10-09), NAVER. Run several drafting stages before one verification while the drafter stays confident; loop-aware training lets parallel drafters condition on their own hidden states for unverified tokens. +5-41% wall-clock across EAGLE-3, DFlash, Domino, DSpark and MTP modules, lossless. Partial answer to this page's "content-adaptive k" open question, and the first adaptive-depth method for parallel drafters. Open: does the gain hold at high batch?

2026-10-10: speculation in diffusion LMs, and token routing as its general case

SpecFold reuses parent computation across near-identical draft branches in diffusion-LLM multi-branch verification (up to 1.99x over vanilla) (shorts). TokenRouter (10-10) serves token-level routing, where the small model's tokens are kept rather than only verified; its parked-KV design is what a speculative engine would need to let draft and target swap roles.

2026-10-13: repairing parallel drafts, prefetching samplers, and rollback for SSMs

Speculation shorts (10-13). DCOCO keeps top-k candidates per slot of a diffusion drafter's block and scores neighbour compatibility in parallel: up to 6.6x over vanilla, +6-23% over the best rival at 4-32 concurrent requests. PreSTO applies speculation to power sampling's Metropolis-Hastings chain (1.71-2.33x median, distribution provably unchanged). Entropy/CKA layer selection beats EAGLE-3's default target layers. ReplaySSM (10-13) closes this page's hybrid-model gap: SSM rollback becomes a buffer truncation, 1.85-2.10x speculative throughput on 120B-class hybrids. Open: does DCOCO's gain stack with DLoop's (10-09) verification skipping?

2026-10-16: DFlash2 ships with Muse Spark Open

Muse Spark Open: NVIDIA's DFlash2 drafts a 7-token block in one non-causal pass with a pairwise path selector over top-k candidates. Acceptance length 3.73 greedy / 3.01 at t=1 on MT-Bench, 3.30 / 2.78 at 32K input, 2.16 on high-entropy prompts. Confirms the recurring finding that acceptance drops on long, high-entropy agent contexts. Next-Concept Modeling is a non-autoregressive alternative (10-31x faster than diffusion LMs at small scale).

2026-10-17: depth-conditioned parallel drafting, and speculation over agent steps

• DSpine (Tencent, page): parallel drafter where each position receives its predecessor's predicted feature at every draft layer, so causality unfolds through depth. Qwen3-8B average acceptance 7.83 vs 7.55 (DFlash2) and 7.44 (DSpark) at T=0; 6.98 vs 6.53 at T=1. Acceptance only, no wall-clock under batching.
• SpecAgent (page): the draft-then-verify pattern lifted to tool-calling steps. Weak model drafts grounded steps, strong model verifies the segment in one prefill. The verifier decodes only verdicts.
• Still open from 10-16: acceptance on long agentic traces.

2026-10-20: the drafter's context, three ways

DLite, ReaDy, FlashMap, Speculators v0.9.

• DLite answers the 10-16/10-17 open question: drafters lose up to 80% of acceptance length on long agentic contexts; dropping drafter history (reuse target hidden states, local prediction, light sequential head) gives acceptance 2-6, up to 70% over DSpark.
• ReaDy reads hybrid-linear-attention recurrent states per draft position: 3.16x over AR on Qwen3.8-27B on one B200, +8.5% acceptance over DSpark.
• FlashMap (X, 2610.24365): flow-map drafter generates the block in one pass while modelling inter-token dependence; up to 6.1x lossless on Qwen3-8B, +7% average over matched DFlash.
• Tension: DLite removes history, ReaDy adds (compact) history. Reconciled if the cost is the drafter keeping its own KV, not context access.
• Tooling: vLLM Speculators v0.9: fused loss kernel up to 2x faster drafter training forward; FP8 hidden-state storage halves disk.

2026-10-22: drafters under load, and drafts as a prefetch oracle

Specter, DScout, ECHO, Lafa and drafters under load.

• Crossover number: EAGLE-4's table (via X): DSpark and DFlash 2 at 2.6-2.8x at batch 1 fall below vanilla at 512 requests on 4x B300 (14.2K/16.6K vs 21.2K tok/s). EAGLE-4's load-adaptive tree: 3.59x low load, 1.34x high. Liquid's DSpark drafter: 2.38x / 2.69x / 1.71x at concurrency 1 / 32 / 256.
• Specter: a quantized on-GPU draft predicts future expert IDs so offloaded MoE experts are fetched ahead; 1.50x (up to 1.88x) TPOT, single GPU.
• DScout: throughput = accepted tokens / per-round latency; acceptance-weighted diffusion-drafter training, 3.84x/4.14x on Qwen3-4B/8B.
• ECHO: backbone's own last layer predicts every horizon; model is its own drafter, holds alignment on high-entropy text (answers the 10-16 open question from the training side).
• Lafa: 8B blockwise flow LM from AR checkpoints with <30B tokens; self-speculative 2.54x.
• UBTree (Ling-3 drafter, lab watcher): bigram-selected draft trees beat DSpark by ~0.3-0.7x end-to-end.
• Production: DeepInfer defaults to DSpark with draft length 5; Google's TPU engines ship five drafter types; Red Hat's suffix-decay curve (87.5% to 39.5% acceptance by token 6).
• Open: a load-aware draft controller as a routing problem.

2026-10-23: acceptance length decides whether drafting survives load

Red Hat's DFlash2 drafter for Gemma 4 31B (5-layer Qwen3 backbone, trained with Speculators) drafts up to 7 tokens and has 4-7 accepted on average, keeping about 1.4x throughput at high concurrency on coding/math (model card). Against the 10-22 EAGLE-4 table (DSpark and DFlash 2 slower than plain decoding at 512 requests), the reconciliation is acceptance length: long accepted runs amortize verification under batching, short ones do not. MiniMax's M3.1 Flash talk adds the accounting rule: count draft, verify and rejection cost, then retries and tool calls (summary).

2026-10-25: flow-map LMs as drafters (social signal)

Sander Dieleman (DeepMind) highlighted flow-map language models as speculative drafters: they capture token dependencies in 1-2 passes, so one pass drafts a coherent block. He also flagged another 8B blockwise continuous diffusion LM adapted from an AR checkpoint with flow-map distillation and self-speculative decoding (same recipe as Lafa, 10-22). Inception's Mercury 2.5 diffusion LLM is now on OpenRouter. No paper links captured; track for numbers under load.