hardware · 2026-11-09 · Tier 1

CHARM+: ring attention that respects the slow link

CHARM+

TL;DR. Ring Attention splits a long sequence across GPUs and passes KV blocks around a ring, overlapping each hop with compute. Two things break that overlap: hops depend on each other in sequence, and in mixed topologies (two NUMA sockets over PCIe, or two nodes over a slower link than NVLink inside a node) the slowest link sets the pace for every synchronization. CHARM+ (arXiv 2611.05906) reorganizes the sequential ring into levels with parallel communication inside each level, and decouples cross-hardware traffic from intra-hardware traffic, re-merging partial attention results afterward. Against Ring Attention: forward/backward 1.3x/1.7x on dual-NUMA PCIe, 2.8x/2.9x on dual-node NVLink.

Keep the slow link off the critical path
Fast local rings run freely; cross-node traffic runs on its own stream and merges later.
flowchart LR
  Q["Local Q shard<br/><small>sequence slice</small>"] --> L["Intra-node ring<br/><small>NVLink, fast</small>"]
  Q --> X["Cross-node stream<br/><small>slow link, async</small>"]
  L --> P1["Partial attention<br/><small>local KV blocks</small>"]
  X --> P2["Partial attention<br/><small>remote KV blocks</small>"]
  P1 --> M["Re-merge<br/><small>combine softmax stats</small>"]
  P2 --> M
  M --> O["Output<br/><small>2.8x fwd vs Ring</small>"]
  classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
  classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
  classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
  classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
  class Q input
  class L,X loop
  class P1,P2,M core
  class O exit
Blue is the input shard, amber is communication, purple is attention compute, green is the result.

Relation to prior wiki

  • Same theme as TPUv7's DCP (decode context parallelism that shards KV by position) in SemiAnalysis's AgentX post (11-06): long-context performance is now a communication-topology problem.
  • Training-side counterpart to FLAMIA (11-04), which attacked attention cost inside one Blackwell GPU with microscaling formats.

Gaps

No comparison to newer topology-aware schemes (USP/Ulysses hybrids, DeepSpeed's tiled variants). No inference (decode) numbers; gains are forward/backward training kernels. Tested on two-domain topologies; behavior at 8+ nodes is not shown.

Concept pages: GPU kernels · memory hierarchy