AI research wiki
inference-efficiency · 2026-05-27 · Tier 1

RT-Lynx: Activation Sparsity for Diffusion Transformers

RT-Lynx: Activation Sparsity for Diffusion Transformers

Source: HuggingFace daily papers (2026-05-27, 1 upvote) · arxiv 2605.26632 arxiv: 2605.26632 Date: 2026-05-27 Raw: raw/huggingface/2026-05-27-rt-lynx-putting-the-gemm-sparsity-in-a-right-way-for-diffusi.md Tier: 1 (GPU optimization, sparsity, custom kernels)

TL;DR

Diffusion Transformers (DiT, the architecture behind modern image generation) are expensive at inference. Semi-structured N:M sparsity (keep N of every M values, which maps to hardware sparse-GEMM units and nearly halves FLOPs) has been underused here because prior work sparsified weights, and pruning 50% of weights removes capacity and wrecks image quality. RT-Lynx's key observation flips the target: DiT activations are intrinsically sparse and far more robust to N:M sparsification than weights are. So it applies N:M sparsity to activations, adds error-compensation to claw back the small accuracy loss, and ships hand-optimized CUDA kernels for the activation-sparse setting, reaching up to 1.55x average speedup in linear layers while preserving generation quality.

Weight sparsification (prior work):        RT-Lynx (activation sparsification):
  prune 50% of W ──► removes capacity        keep weights dense
  ⇒ quality degrades                         sparsify activations X (N:M)
                                             + error compensation
  GEMM:  X · sparse(W)                        GEMM:  sparse(X) · W
                                             ⇒ ~half the FLOPs, quality preserved
                                             ⇒ 1.55x avg speedup (custom CUDA kernels)

Key findings

  • Activations are the right sparsity target in DiT. Empirically, DiT activations tolerate N:M semi-structured sparsification much better than weights, because the model capacity lives in the weights and pruning activations discards low-magnitude per-token signal instead.
  • Error compensation closes the gap. A correction step mitigates the accuracy loss from activation pruning, keeping generation quality at the original model's level.
  • Custom CUDA kernels make it real. N:M activation sparsity needs kernels tailored to sparsify the dynamic operand; RT-Lynx's deliver up to 1.55x average speedup in linear layers.

Relation to prior wiki state

RT-Lynx fits the wiki's "the iteration unit has heterogeneous information density, allocate compute proportionally" pattern, applied to the GEMM operand. The KV cache concept page tracked the same insight on the token axis (Make Each Token Count: not all cached tokens deserve attention budget) and the head axis (MISA: route on indexer heads). RT-Lynx puts it on the activation axis: not all activation entries carry signal, so structured-sparsify them and keep weights intact. It pairs with the broader sparsity-and-kernels hardware thread (the AgentKernelArena / KernelBench work on 05-09/05-19 about LLMs writing GPU kernels): RT-Lynx is a hand-written-kernel result, and the obvious next question is whether kernel-generation agents can produce the activation-sparse kernels automatically. It is also the diffusion-side complement to weight-quantization efficiency work; RT-Lynx argues sparsity and quantization are orthogonal levers (you can do both).

Why it matters

Image/video generation inference cost is a real production line item, and N:M sparsity maps directly to sparse-tensor-core hardware on recent NVIDIA GPUs, so a 1.55x linear-layer speedup with no quality loss is immediately bankable for any DiT-serving stack. The conceptual contribution (sparsify the activation, not the weight) likely transfers beyond diffusion to any transformer where activations are sparser than weights.

Research angle

  1. Transfer to LLM decode. If LLM activations are also more N:M-robust than weights, the same paradigm could accelerate language-model linear layers; untested here.
  2. Sparsity + activation quantization jointly. Error-compensated N:M activation sparsity composed with low-bit activation quantization is the natural stacking experiment.
  3. Auto-generated kernels. Whether a kernel-optimization agent can match RT-Lynx's hand-tuned CUDA for the activation-sparse GEMM is a clean benchmark for the kernel-agent thread.

Links