NVIDIA Reshard collective
TL;DR. In RL post-training, every step ends with a refit: the trainer's updated weights must reach the inference engine that generates the next rollouts. Today that is an all-gather plus broadcast, moving about 2x the model per step (≈11.2TB for a 2.8T-parameter model in BF16). Reshard treats refit as converting one sharded tensor layout into another (DTensor to DTensor). It sends only the exact shard intersections each receiver needs, and groups replicas by NVLink domain so each byte crosses the slow network once and then fans out over NVLink. In NeMo RL on DGX H100 and GB200 (Qwen3-235B, DeepSeek-V3 671B): 2.7-5.7x faster refit. Hybrid in practice: Reshard for big FFN projections, gather-broadcast for small weights. Open-sourced as NCCL M2N Reshard in NVIDIA/nccl-extensions.
flowchart LR
T["Trainer shards<br/><small>FSDP/TP layout</small>"] --> I["Intersect layouts<br/><small>who needs which slice</small>"]
I --> N["Cross-node send<br/><small>once per NVLink domain</small>"]
N --> V["NVLink fan-out<br/><small>replicas share locally</small>"]
V --> G["Inference shards<br/><small>vLLM/TP layout</small>"]
T -.->|old way| X["All-gather + broadcast<br/><small>~2x model per step</small>"]
classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
classDef err fill:#ffe3e3,stroke:#e03131,color:#1b1b1b,stroke-width:2px
class T input
class I loop
class N,V core
class G exit
class X err
Relation to prior wiki
- Same move as CHARM+ (summary) the same window: put the slow link off the critical path by separating inter-domain from intra-domain traffic.
- RL post-training at trillion-parameter scale is becoming communication-bound, not FLOP-bound; see compute economics.
Concept page: GPU kernels