AI-Generated CUDA Kernels Silently Break Training and Inference

Date ingested: 2026-05-28 Source: r/MachineLearning [R] flair Links: r/ML post · raw

TL;DR

The author took several top-ranked submissions to NVIDIA's SOL-ExecBench (the same benchmark that doubleAI claimed speed-of-light wins on today) and tried using them in production workloads. Many broke. One concrete case: a fused embedding-gradient + RMSNorm backward pass that passes the benchmark verifier with margin caused training loss to diverge in a real transformer. Replacing the dataset with uniform random tokens fixed the divergence. So did swapping SGD for AdamW. The actual bug: the embedding-gradient half accumulates in bf16 instead of fp32. With uniform random tokens, the per-token contributions spread evenly and bf16 is enough; in real text a handful of high-frequency tokens get many gradient contributions summed into the same row, and bf16 loses precision exactly there.

The silent-failure pattern:

  benchmark verifier:        kernel passes ✓  (uniform tokens, simple optimizer)
  real training workload:    loss diverges ✗  (real text token distribution)
                                       ▲
                                       │
  root cause: embedding-grad accumulates in bf16 instead of fp32
              high-freq tokens accumulate many small grads
              → bf16 saturates exactly in the rows that matter most

Key findings

• Several top SOL-ExecBench submissions failed in production despite passing the benchmark's verifier.
• The fused embedding-gradient + RMSNorm backward kernel is the most-used training-time kernel, and the top submission silently broke training.
• The bug only triggers on real-text token distributions; uniform random tokens hide it.
• Mitigations like switching optimizer also hide the bug, making it look like an algorithmic problem rather than a kernel-precision one.
• This is the worst class of bug for research, because the symptoms ("the idea didn't work") match the masks ("the kernel was wrong").

How this fits prior wiki state

Paired with the doubleAI speed-of-light result (also today, also from r/CUDA), this is a clean two-paper picture: AI kernel writing now beats baselines at the benchmark, and verifier-passed kernels silently break real workloads. The same week the SemiAnalysis Miscompiles post reports AI agents finding hundreds of LLVM and ptxas bugs in days. Three sources, same week: AI is reshaping the low-level compute stack from both sides, generating optimized kernels AND finding the latent bugs in human and AI-written compilers and kernels. The risk is that the verifier gap on the generation side and the bug-discovery rate on the analysis side are now both faster than human review.

Connects also to [[2026-05-09-kernelbench-x-llm-gpu-kernel-benchmark]] and [[2026-05-19-agentkernelarena-gpu-kernel-optimization-agents-benchmark]], both of which evaluate kernel agents but rely on their own verifiers.

Related pages

• [[2026-05-28-doubleai-blackwell-sol-execbench]], paired result on the same benchmark
• [[gpu-kernels]], concept page
• [[2026-05-19-agentkernelarena-gpu-kernel-optimization-agents-benchmark]], kernel agent benchmark

Research angle

A "training-workload-canary" stage between benchmark verifier and adoption is the obvious operational fix. The deeper research question is whether kernel verifiers can be made distribution-aware so that they detect the embedding-gradient-class of precision-summation bugs at synthesis time. That requires the verifier to model the downstream gradient distribution, not just numerical equivalence on a fixed test input.