Speculative Decoding

A class of lossless acceleration techniques: a cheap "draft" produces candidate tokens (or blocks), an expensive "target" verifies them via exact rejection sampling, and the verified prefix is committed. The target's output distribution is preserved — quality is unchanged.

Current State (as of 2026-08-03)

The first number from the regime where the weights do not fit, and it is much smaller than the datacenter numbers below. DraftExpert (08-03) (2607.24434, Kurate cs.LG #20) targets end-device MoE inference where routed expert weights exceed accelerator memory and are staged on demand from CPU RAM to a GPU or from Flash to a mobile NPU. Under that constraint the two assumptions this page has always relied on both break. Growing the draft expert set improves accuracy but triggers extra expert loading, so a better drafter is a slower one. And verifying a k-token block activates the union of the experts those k tokens route to, so verification is no longer close to one target step. The paper calls the second effect expansion and builds around bounding it.

Three mechanisms. A fixed-footprint drafter: one lightweight draft expert per layer resident on the accelerator, so the drafter is always shared experts plus top-1 routed expert plus the resident draft expert regardless of uncertainty. Confidence-expansion truncation, which stops drafting on a memory-traffic criterion rather than a probability one, and is the first concrete answer on this page to the long-standing content-adaptive k open question below, though with a different objective than anyone expected. And target-expert prefetching, made possible because the draft expert is self-distilled from the frozen target on residual, logit/token and router-agreement signals, so the drafter predicts which experts the target will pick. Results on DeepSeek-V2-Lite and Moonlight-16B-A3B across both offload paths: 1.45x average decode throughput, 84 to 87% draft acceptance, 86 to 88% prefetch hit rate, output exactly verified and therefore lossless.

Two things to carry forward. The gap between 1.45x here and the 2x-plus in the fits-in-memory results below is the price of paging, and on-device MoE marketing quietly assumes it away. And the durable structural claim: once weights are paged, speculation and memory prefetch are the same computation, because a drafter trained to agree with the target's router is a prefetch oracle you were already paying for. Any on-device runtime running speculation without wiring the drafter's router output into its paging layer discards the free half. Caution on the acceptance figure: tinygrad's profile below measured long-context acceptance at 90.5% on repetitive synthetic prompts and roughly 64% on real code, and DraftExpert reports no prompt-distribution breakdown. → summary

Current State (as of 2026-08-01)

Speculative decoding is now the stated mechanism behind frontier-lab pricing, which makes yesterday's audit urgent rather than academic. OpenAI attributes its GPT-5.6 price cuts to infrastructure work rather than a new model: speculative decoding raising token generation 15% and cutting GPU serving cost 20%, cascading into an 80% price cut on GPT-5.6 Luna (to $0.20 per million input, $1.20 per million output), 20% off Terra, and Auto-review in ChatGPT and Codex CLI moved to Luna for a 10x cost reduction. AI Breakfast's aggregate framing is a 13x drop for the same intelligence, since Luna now matches March's GPT-5.4 flagship at one-thirteenth the token price. Independently, tinygrad published a full serving profile for DeepSeek-V4-Flash-0731 on two RTX Pro 6000 Blackwell GPUs using DSpark K5 fixed-depth speculative decode with W4A8 kernels and an fp8 KV cache: ~245 tok/s sustained single-user, median TPOT ~2.8 ms, TTFT ~0.4 s, beating the model's own runbook figure of 217 to 220 tok/s. → summary

Neither party states whether its verifier is lossless, which is exactly the gap the 07-31 audit opened. When speculative decoding was a 1.5x convenience, an unmeasured quality tax in the distribution tails was tolerable. It is now the mechanism behind the price of a frontier endpoint, so the distribution being silently rewritten is the distribution most production traffic samples from. The audit's two-question checklist (which family is your verifier in, and what is the draft-over-target overshoot) is unanswered by both the largest commercial deployment and the most detailed public practitioner profile.

A new reporting failure this page should track: acceptance rate is workload-dependent by 27 points. tinygrad's own footnote reports DSpark acceptance at 90.5% on repetitive random-token prompts and approximately 64% on real code, same model, same scheme. Every speculative-decoding speedup on this page is a function of acceptance, and almost none of them state the prompt distribution acceptance was measured on. A minimal convention (acceptance on synthetic, code, natural prose, multi-turn chat) would cost nothing and would make the literature comparable for the first time.

Open composition question. W4A8 weights, an fp8 KV cache and a speculative draft all perturb the output distribution and every paper studies one at a time. The interaction that should matter most is between draft quantization and the acceptance test, because a quantized draft is systematically miscalibrated rather than merely noisier, and miscalibration in the draft-over-target direction is precisely the overshoot condition the 07-31 audit named as the collapse mode for collaborative verification.

Current State (as of 2026-07-31)

The lossy branch of this page has been audited, and the verdict is that it was being reported wrong. Revisiting Lossy Verification in Speculative Decoding (2607.26627) analyses the distributions that lossy verification schemes actually sample from and finds two things the individual method papers did not report.

First, the taxonomy collapses. Every published lossy verifier reduces to one of two families: truncation-based (accept a draft token when its probability clears a threshold, meant to approximate top-p or top-k sampling from the target) and collaborative (blend draft and target probabilities). Methods presented as distinct mechanisms are superficially different and mechanically the same.

Second, each family has one named failure. Truncation-based schemes suffer distributional distortion severe enough that performance can fall below the true truncation-sampling baseline, which is the damning comparison: the method produces worse output than the exact version of the distribution it claims to approximate, so the trade is not speed-for-a-little-quality, it is speed for a bug. Collaborative schemes are governed by overshoot, the amount by which the draft's probability for a token exceeds the target's. Bound it and quality holds; leave it uncontrolled and output collapses. Critically, the degradation is described as unstable, meaning it appears on some prompts and not others, so a benchmark mean is the wrong instrument for detecting it.

This answers the open question this page left on VIA-SD. The 06-12 entry below closed with "Open question for review: whether the slim-verifier regeneration path is exactly lossless or an approximation." It is an approximation, and it is a collaborative-family mechanism, so the overshoot condition applies. VIA-SD's reported rejection-rate improvements of 0.10 to 0.22 and its 10 to 20% gain over strong SD baselines need re-reporting under this diagnostic before they can be read at face value.

It also changes how this page should frame its own scope. The page defines speculative decoding as lossless and then tracks four axes of generalization, three of which preserve the guarantee (GRAFT on the draft side, Draft-OPD on draft training, SPD on scheduling) and one of which does not. The correct framing going forward is that lossless SD and lossy verification are two different techniques with two different accounting problems, and lumping them together is what allowed lossy methods to be scored on a speedup number while their quality cost went unmeasured.

Shared shape with Bebop (06-11). Bebop found multi-token-prediction acceptance is near-linearly bounded by model entropy, so acceptance collapses during RL exactly when rollouts are most expensive, and fixed it by optimising total variation directly rather than cross-entropy. Both papers find that a speculative-decoding quantity everyone reports as a scalar is really a function of the output distribution's shape, and both fix it with a distributional correction. That is now two of the page's most useful results arriving by the same route, and it suggests the remaining unexamined scalars on this page (acceptance rate, block length, speedup) are worth re-deriving the same way.

Cheapest available action for anyone serving a lossy variant: log draft-probability minus target-probability for each accepted token and inspect the tail rather than the mean. One subtraction per token, and it is the statistic the paper names as the predictor of collapse.

Current State (as of 2026-06-12)

A fourth axis opens: graded verification cost via intra-model routing. VIA-SD (arxiv 2606.12243) observes that the accept/recompute binary is wasteful because many rejected tokens sit in a "middle zone" — wrong as drafted, but correctable by a model far smaller than the full verifier. It carves a slim verifier out of the full verifier via intra-model routing (no new model, no retraining) and inserts it as a middle tier: tokens route to accept (high confidence), slim-verifier regeneration (medium), or full recompute (uncertain), with the tiering grounded in a KL-divergence decomposition of verification. Result: rejection rates down 0.10–0.22, 10–20% over strong SD baselines, 2.5–3x over non-drafting decode, and it drops into existing SD frameworks without training changes. This is the verify-side mirror of GRAFT (05-20), which made the draft side cheaper by retrieving instead of generating — together they push SD toward "spend the minimum compute that preserves the target distribution at every step, on both sides." Open question for review: whether the slim-verifier regeneration path is exactly lossless or an approximation.

Current State (as of 2026-06-11)

The MTP-acceptance collapse during RL now has a mechanism, and it is an entropy bound. Bebop (arxiv 2606.12370) studies why multi-token-prediction drafting — the embedded-drafter idea Nemotron 3 Super introduced (04-21) — loses its speedup when used to accelerate RL rollouts, the application opened by Speculative Decoding for RL Rollouts (04-30). The answer: MTP acceptance is negatively, near-linearly bounded by model entropy, and RL deliberately raises entropy to explore, so acceptance falls exactly when the rollout stage is most expensive. Three fixes: (1) probabilistic rejection sampling of draft tokens absorbs the entropy disturbance far better than greedy draft sampling; (2) a new end-to-end total-variation (TV) loss directly optimizes the multi-step rejection-sampling acceptance rate, where cross-entropy/KL are suboptimal, lifting acceptance to up to 95% (~10% gain); (3) train the MTP head once before RL — pre-RL TV-loss training holds acceptance steady across the whole run, so no costly online MTP updating. Up to 25% extra throughput and up to 1.8x end-to-end async-RL speedup on Qwen3.5/3.6/3.7. The deeper implication: rollout speed and policy-stability tuning are coupled — anything that holds entropy down (e.g. a tighter trust region like DRPO, 06-10) should raise MTP acceptance for free.

Current State (as of 2026-06-02)

Two same-day papers move speculative decoding past the "build a better draft architecture" era (EAGLE3, DFlash) into better training objectives and better system scheduling.

Training axis: on-policy distillation for the drafter. Draft-OPD (arxiv 2605.29343) diagnoses that SFT-built draft models plateau because of an offline-to-inference mismatch: the drafter trains on fixed target trajectories but is judged on the blocks it proposes under its own policy. Naive on-policy distillation fails because draft models cannot roll out reliably alone, and target-assisted rollout destroys the on-policy signal. Draft-OPD's fix: target-assisted rollout for stable continuations, but replay drafting from the verification-exposed error positions, so the drafter learns from target feedback on both accepted and rejected proposals. Over 5x lossless acceleration for thinking models, +23% over EAGLE-3, +13% over DFlash. This is the same covariate-shift lesson the wiki logs in TA-OPD (06-01), DRIFT (06-01), and DAgger-for-LLM-agents (05-14), now applied to the drafter — see knowledge-distillation.md.

System axis: pipeline-parallel, zero-bubble speculation. SPD (arxiv 2605.30852) replaces multi-token prediction (whose difficulty escalates with depth and adds serial drafting latency) with pipeline parallelism: partition the target into n stages, process n tokens in parallel, and aggregate intermediate features across pipeline depths to predict the next token in parallel with the target's pipeline step. Result: bounded prediction difficulty, higher acceptance, and zero latency bubbles (no idle stages) in single-sequence decode. Draft-OPD improves draft quality; SPD removes draft latency. Together they show the field's gains are migrating from architecture to objective and scheduling.

Current State (as of 2026-04-30)

Speculative decoding has crossed three axes of generalization in April 2026:

The pattern is now: wherever a generator has a slow target and a credible cheap proposer plus a verification signal that preserves target behavior, speculation applies. The draft does not need to be exactly compatible with the target's architecture or even produce the same kind of output (e.g., SDVG's drafter produces video blocks, not tokens) — it only needs to produce candidates that the verification signal can accept or reject.

Key Papers

Nemotron 3 Super (2026-04-21) — Embedded Multi-Token Prediction heads as the speculative drafter. The target model is its own drafter: an MTP head proposes future tokens during the same forward pass that produces the next token, eliminating the external draft model. → summary

SDVG (2026-04-22) — Speculative decoding for autoregressive video generation. Token-level rejection replaced by ImageReward-based quality routing. 1.3B drafter proposes blocks; accepted blocks enter the 14B target's KV cache directly. 1.59–2.09× speedup at 95.7–98.1% quality. → summary

Speculative Decoding for RL Rollouts (2026-04-30, NVIDIA) — Lossless integration of speculative decoding into NeMo-RL + vLLM for RL post-training. Target policy is the verifier; log-probs and policy gradients are computed against target. Draft alignment to rollout distribution is the dominant variable; sweet spot at k=3. 1.77× generation, 1.41× per-step at 8B; 2.5× end-to-end projection at 235B on 2048 GB200s. → summary

Key Concepts

• Lossless acceleration: target's output distribution is preserved; no quality regression.
• Draft alignment: the draft must approximate the target's distribution well; misaligned drafts (e.g., chat draft for a math rollout) lose most of the speedup. The draft initialization matters more than online adaptation once aligned.
• k = speculation depth: number of tokens the draft proposes per round. Higher k = higher upside per accept, but also more wasted draft work on rejection. Empirically k=3 is the sweet spot at 8B reasoning workloads; k=5–7 can be net-negative.
• Verification signal: exact rejection sampling (text), quality router with worst-frame aggregation (video), policy-target log-probs (RL rollouts). The signal must be cheap relative to the target generation cost.
• MTP head as embedded drafter: a small head on the target model that produces multi-token proposals. Eliminates the external draft model and stays automatically aligned because it is part of the target.
• Speculation under policy drift: in RL training, the target policy moves with each gradient step. Weight synchronization between target and draft is required to keep alignment.

Open Questions

• Content-adaptive k: empirically the optimal k varies with rollout phase (predictable math rollouts vs branchy reasoning). A learned k-schedule would beat the fixed-k regime.
• Biased speculation: current methods preserve losslessness. A drafter that deliberately proposes near high-reward regions (rather than approximating the current policy) could trade losslessness for sample efficiency in RL training.
• Cross-modal speculation: SDVG showed the verification signal does not have to be probability matching. The next test is whether speculation works for audio diffusion (worst-window quality routing?) or 3D synthesis.
• Composition with consumer-GPU pipeline parallelism (RoundPipe, 05-01): NeMo-RL spec dec is a generation-during-training optimization; RoundPipe is a training-loop optimization for consumer hardware. Combining them on a consumer cluster could shrink small-lab post-training cost dramatically.
• Composition with token-level value modeling (LenVM, 05-01): LenVM tells the model when to stop generating; speculative decoding makes each step cheaper. They should multiply — but no paper has measured the combined effect.

Related Pages

• KV Cache — speculation populates the target's KV cache cheaply
• Knowledge Distillation — drafts can be distilled, MTP heads can be co-trained
• RL for LLMs — rollout cost dominates, speculation is the answer