GPU Kernels and Accelerator Optimization

How AI computations are mapped to the specific instruction sets and memory hierarchies of GPU and accelerator hardware. Kernel optimization determines how efficiently a model actually runs — the gap between theoretical peak FLOPS and real throughput.

2026-08-31: the fourth kernel-agent result varies memory, which is the axis that reprices the moat

This page has recorded three kernel-agent results and each varied a different thing. Beyond Scaling (08-31) (Kurate cs.LG #4) varies the one none of them touched: what survives between runs.

The mechanism is an experience graph: optimization attempts, their profiled results, and the structural context they applied to, retrieved when a new kernel presents a matching structure. This page's reading of JAXBench left the door open for exactly this. That 6.4x correctness jump was judged here to be a retrieval result, not a capability result. An experience graph is the same insight pushed one step further, retrieving from what the agent measured rather than from what a vendor documented, and it should be worth more, because the docs never contained the answers in the first place.

Kernel work is the friendliest possible domain for experiential memory, and this is worth stating because the agent-memory literature had to fight for it elsewhere. Recuris (08-26) found that skill retrieval keyed on either the initial instruction or the full history degrades exactly as a task gets hard, and fixed it by keying retrieval on externally verified state, since self-reported state is vulnerable to omission and hallucination. In kernel optimization the state is verified for free: a profiler measurement is not a self-report. The obvious next design, which this paper does not appear to do, is to key experience retrieval on measured hardware counters rather than on source-text similarity.

The commercial consequence is a change in the shape of the moat, not its size. Every argument this page and compute-economics have recorded, from the 07-25 SemiAnalysis CUDA-moat analysis through Jalapeño, prices the moat as the cost of generating kernels for a new target, and all of them treat that as a level to be lowered. An experience graph makes it a decaying cost instead: the first target costs what it always did, every subsequent one is cheaper, because structural knowledge about tiling, occupancy and memory access ports better than any individual kernel does. A decaying moat is a worse position for an incumbent than a merely smaller one.

Two caveats, and the second is the one that decides the paragraph above. This entered via Kurate rather than HuggingFace, and all 40 entries on both Kurate boards still show score=1200 and win_rate=0.0% for a fourth week, so the tournament has not run and the ranking is recency, not quality. Treat the mechanism as the contribution and the magnitude as unverified. And cross-target transfer is unreported. Within-target experience reuse is a useful engineering result; cross-target transfer is what turns the moat into a decaying cost, and it is much harder to demonstrate. Open problem for this page: any paper reporting kernel-agent performance on target B after accumulating experience on target A, with a from-scratch control.

2026-08-26: a model wrote the kernels for a new ISA in three months

OpenAI's Jalapeño is the industrial escalation of this page's agent-written-kernel thread, and it is a much larger instance than anything here so far. AccelOpt (04-20) had an LLM agent optimizing AWS Trainium kernels on a mature stack, raising peak throughput utilization from 49% to 61% while matching Claude Sonnet 4 at 26x lower cost. Jalapeño is the version where the model creates the stack: OpenAI brought up DeepSeek R1, Kimi K2.5 and GPT-OSS on a brand-new ISA within three months of first silicon, starting from zero on software, and Codex wrote the working MLA kernels with no intervention from the kernel engineering team (OpenAI had no MLA implementation at all until it benchmarked DeepSeek). AI-assisted design also cut SIMD area 8% and matrix-engine area 10%.

The programming model is worth recording in its own right. Kernels are written near-assembly, some running ~3,000 lines, in Gluon — OpenAI's kernel language built on Triton, preserving Triton's SPMD model while exposing low-level abstractions. Its distinguishing feature is Linear Layouts, a layout algebra that formalizes the mapping between a hardware resource (say, register 5 of warp 9) and a tensor element, and thereby supports provably correct layout conversions and optimal memory swizzling. TensorInfo explicitly encodes layouts. Each Gluon program maps to a persistent thread, which suggests the persistent-kernel pattern where the programmer rather than the hardware scheduler assigns tiles. The internal serving engine is called Teacup.

The architectural claim is a memory-hierarchy refusal, and it is directly a KV-cache result. Jalapeño's stated primary design goal is eliminating memory movement of KV cache and weights, plus eliminating fixed latencies. It does this by dividing cores and HBM into slices where each core slice has a low-latency local view of its own HBM slice, pushing cross-slice synchronization onto a dedicated high-bandwidth collective network. It also uses an out-of-order core with an L1 cache, diverging from the software-managed scratchpad plus async DMA that TPU, Trainium and GPUs all use, on the argument that barrier and launch latencies are fixed overheads other designs must hide behind more work per core. The trade is a hard dependence on good prefetching, and SemiAnalysis's answer to that is telling: Codex in a good harness with detailed tracing finds the optimal prefetch per shape. That makes the hardware decision contingent on the code-generation capability, which is a genuinely new kind of architectural dependency and nobody has characterized the failure mode when the model gets it wrong.

The matrix engine uses MXFP formats and a weight-stationary systolic array like a TPU, but supports smaller shapes and dimensions, so it avoids the performance cliffs that awkwardly shaped matmuls expose on TPU, Trainium and Etched's large systolic arrays, where large batches or exactly-divisible model dimensions are needed to dodge tiling inefficiency. Plus 64-bit scalar cores, FP32/INT32 vector cores, tray-level redundancy and yield harvesting at core and channel level.

This bears directly on open problem 1 (whether FP4 kernel technique transfers across vendors' differing FP4 implementations). Jalapeño is MXFP4-native, which is the OCP standard AMD built its four-bit path around, not Nvidia's NVFP4. The relevant update is that the transfer question may matter less than assumed if per-target kernels become cheap to generate. SemiAnalysis puts the consequence bluntly: if Jalapeño succeeds, "the industry's obsession over programming models and perfect, universal compilers" is invalidated by frontier models — you do not need a great universal compiler if you can afford a near-optimal generated kernel per shape per target.

Also today, at the opposite end of the scale. TorchMorph closes a small but real gap: scipy.ndimage is the reference implementation for morphological transforms and is CPU-only and single-array, so it cannot appear inside a GPU training loop without a device-to-host round trip. TorchMorph exposes 22 operators (binary and greyscale morphology, exact and approximate distance transforms, entropy-regularized optimal transport) as fused CUDA kernels operating directly on (B, C, Spatial...) tensors with up to eight spatial dimensions, mirroring the scipy API argument-for-argument so pipelines port by changing an import. Batched throughput reaches up to 1.1e3x scipy on greyscale morphology and 350x on exact Euclidean distance transforms, with the Sinkhorn solver 42x faster than POT, binary and chamfer operators exact against SciPy and float operators agreeing to 1.8e-6. MIT licensed. The generalizable point is that a four-order-of-magnitude speedup here comes not from a clever algorithm but from deleting a host round-trip, which is the same diagnosis as every other entry on this page.

Current State (as of 2026-08-11)

This page has said for months that the wall is memory bandwidth rather than FLOPs. At the interactivity frontier it is neither, and the new wall gets worse with every GPU generation.

SemiAnalysis on TileRT (08-11) does the arithmetic that makes the point unavoidable. An 8-GPU HGX B200 has 64 TB/s aggregate HBM bandwidth; GLM-5 at NVFP4 needs about 21 GB of active-parameter traffic per generated token at batch size 1; the bandwidth roofline therefore permits roughly 3,047 tokens/s/user without speculative decoding. Real GPUs come nowhere close. The gap is kernel launch and synchronization overhead, which is invisible at conventional serving speeds and dominant as time-per-output-token approaches the sub-millisecond range, even with CUDA graphs. And the structural part: HBM bandwidth improves roughly 2 to 3x per GPU generation while memory latency has not improved at all, so the interactivity gap widens with every new part.

The fix is a compiler, which reframes "GPUs cannot do low latency" from a hardware fact into a software fact. TileRT statically compiles the entire decode graph into a single persistent kernel, removing launch boundaries and overlapping compute, memory movement and collective communication inside one launch. On the InferenceX GLM5 FP8 744B benchmark, one B200 decode server reaches up to 500 tokens/s/user, about 3x a GB300 NVL72 running conventional engines, and up to 2x better interactivity at iso-cost-per-output-token. It composes rather than replaces: under prefill-decode disaggregation TileRT takes latency-sensitive decode while vLLM and SGLang keep throughput-optimized prefill. Already in production at Xiaomi (MiMo V2.5 Pro UltraSpeed) and ZAI (GLM 5.1 HighSpeed). It comes from the same maintainer organization as the TileLang DSL, which puts it in the same lineage as the kernel-DSL work this page has tracked since AccelOpt (04-20) and JAXBench's Pallas kernels (08-03).

The commercial consequence SemiAnalysis raises, and it is the interesting one. Every argument for buying a specialist ultra-low-latency accelerator (Cerebras, Sambanova, NVIDIA's Groq-derived LPUs) rests on the claim that GPUs structurally cannot reach sub-millisecond TPOT. If a software layer closes most of that gap on standard silicon, the specialists' addressable market shrinks to the residual. SemiAnalysis explicitly weighs whether TileRT-on-GPUs disrupts their TAM, backed by quarter-by-quarter shipment estimates for LPU30, LPU40, WSE-3 and WSE-4.

Two limits this page should carry. First, static compilation is the whole trick and the whole constraint: a persistent kernel is compiled for a shape, and variable batch composition, dynamic sequence lengths and per-token MoE expert selection are all shape variance. The article does not report how much specialization production deployments needed. Second, it composes badly with the tiering direction landing the same day: OasisKV (08-11) keeps the full KV cache outside HBM and prefetches predicted blocks, and an unpredicted fetch is exactly the stall a fully overlapped static schedule handles worst. The two best serving results of the day pull in opposite directions on the throughput-interactivity curve, and no scheduler can currently express which one a given request wants, which is the per-request-deadline gap AAPT (08-04) exposed.

Benchmark note. InferenceX now has committed submissions from NVIDIA (Vera Rubin), Google (TPUv7) and AMD (MI455X UALoE72), which makes it the closest thing to a neutral inference scoreboard this wiki tracks.

Prior State (as of 2026-08-03)

The kernel-agent race becomes a three-vendor race, and Google publishes evidence that a chunk of its own programmability gap is documentation rather than silicon. JAXBench (08-03) (2607.20466, Google with Harvard and UC Berkeley) is the TPU-native counterpart to KernelBench: 50 JAX workloads, of which 17 are production operators extracted from the public MaxText library (Llama-3.1, DeepSeek-V3, Mixtral, Mamba-2, AlphaFold2) and 33 are KernelBench operators translated to Pallas and resized for high TPU v6e MXU utilization. Eight of the 17 ship with hand-optimized, block-size-tuned Pallas kernels from the public Tokamax library, which gives the suite an expert upper bound rather than a naive baseline, and every other agent benchmark should copy that.

The result splits the problem in two, and the split is the transferable part. Correctness is a documentation problem. Conditioning Gemini 3 Flash on curated TPU documentation raises per-sample correctness from 5.8% to 37.3% and solves 48 of 50 benchmarks at 1.28x geomean over XLA. Speed is a search problem. Autocomp's beam search takes the geomean to 1.36x, and on the eight expert-tuned kernels to 1.60x against XLA, recovering most of Tokamax's 2.08x while trailing specifically on paged and ragged attention, which are the irregular-memory kernels where hand tuning has always earned its keep. A 6.4x correctness jump from a documentation dump is a retrieval result, not a capability result, and it generalizes to any agent working against an interface underrepresented in pretraining.

Two connections to the entries below. The 07-25 SemiAnalysis analysis argued the CUDA moat is substantially an engineering-headcount advantage that agents erode; JAXBench applies that to the vendor whose moat was never CUDA, and if agents close most of the expert gap on Pallas from public docs, the software-maturity penalty for a non-NVIDIA target shrinks to the size of its documentation. And it echoes MXAttention (08-01) at a different level: that paper found a format's apparent inferiority was two fixable numerical bugs, this one finds an accelerator's apparent unprogrammability is partly missing docs. What is missing is a cross-agent comparison: only Gemini 3 Flash is reported, and an AgentKernelArena-style head-to-head on JAXBench is the point at which "which agent writes the best kernels" becomes portable across vendors. → summary

Current State (as of 2026-08-01)

MXFP4's accuracy deficit against NVFP4 may have been two fixable numerical mistakes rather than a property of the format. MXAttention (08-01) is a data-free post-training quantization framework for MXFP4 attention with exactly two components. Universal Optimal Scaling exploits the periodic structure of power-of-two microscaling to derive a distribution-independent optimal scaling boundary, Qmax = 7.25, with no calibration set and no search, which removes the clipping-versus-underflow trade that the E8M0 power-of-two scale forces. Pre-Normalization Quantization quantizes the unnormalized softmax exponentials before the row-wise sum, so normalization is preserved by construction rather than approximately, removing the row-wise normalization error the standard quantize-after-normalize ordering introduces. On Wan2.2 and HunyuanVideo it closes at least 95% of the VBench Imaging Quality gap between OCP MXFP4 and FP16, holds under 0.01 absolute degradation on every reported VBench metric, and is competitive with strong NVFP4 baselines with negligible fused overhead. It is shipped in MindIE-SD, so this is vendor inference-stack code.

This lands directly on the 07-25 ISA entry below. That entry recorded AMD's gfx1250/MI455X gaining native NVFP4 with a runtime discriminator, and read the strategic situation as NVFP4's richer scale format (16-element blocks, FP8 E4M3 per-block scale, FP32 global scale) buying accuracy that MXFP4's 32-element E8M0 blocks could not match, with gfx950/MI355X stuck on MXFP4 only. MXAttention argues the gap was never intrinsic. If the result generalizes past video diffusion, MXFP4-only hardware stops being second-tier for four-bit attention and the case for NVFP4 as the universal default weakens. The open question is whether Qmax = 7.25 survives a different tensor distribution, since the derivation is about the geometry of power-of-two scaling rather than about attention scores, which on its face should make it apply to weights too. Nobody has tested that. Second open question: PNQ's principle (quantize before the normalizing operation, not after) applies unchanged to RMSNorm and LayerNorm, and has not been tried there. → summary

Practitioner counterpoint on what the serving stack now composes. tinygrad published a full profile for DeepSeek-V4-Flash-0731 on two RTX Pro 6000 Blackwell GPUs: ~245 tok/s sustained single-user (median TPOT ~2.8 ms, TTFT ~0.4 s), beating the model's own runbook validated figure of 217 to 220, with W4A8 kernels, an fp8 KV cache, DSpark K5 fixed-depth speculative decode and 131k max context all composed in one configuration. Four separate research lines from eighteen months ago now run together on two workstation GPUs. The honest footnote is the transferable finding: long-context speculative acceptance of 90.5% is inflated by repetitive synthetic prompts and real code is approximately 64%, a 27-point workload gap that makes any speculative-decoding speedup quoted without its prompt distribution unfalsifiable. → summary

Current State (as of 2026-07-25)

A vendor productizes the kernel-agent loop, and the hardest engineering in it is anti-cheat. SemiAnalysis's AMD Advancing AI 2026 analysis documents ROCm.ai, AMD's public suite for agentic kernel work, and it answers the open thread this page has carried since 05-27 (whether the kernel-generation agents it tracks can move from research to production) with a shipped vendor stack rather than a paper. The pieces map one-to-one onto the loop: GEAK ("Generating Efficient AI-Centric Kernels," a mini-SWE-agent that writes and tunes Triton/HIP/FlyDSL kernels), Hyperloom (the orchestrator that profiles a serving workload, finds the bottleneck kernels, throws GEAK and GEMM-tuning agents at them, and gates every candidate on an end-to-end A/B before it counts), Magpie for evaluation, TraceLens for traces, Apex for exporting agent trajectories into an RL training pipeline, and AgentKernelArena — the same head-to-head harness this page has named since 05-27, now pitting Claude Code, Codex, Cursor, and GEAK against each other on identical kernel tasks under identical scoring. Hyperloom's optimizer scrapes SemiAnalysis's own InferenceX leaderboard and writes a competitor target the agent must clear, which makes a public benchmark into an agent reward signal.

Two findings matter more than the tooling. First, a verified number: GEAK logs a ~+21.8% end-to-end win from an MXFP8 decode-bound dense-linear rewrite on MI355X, with an honest caveat that grouped-MoE GEMM stalls around a 1.1x ceiling. Second, and more transferable, a large share of the engineering is anti-cheat. GEAK had to be stopped from silently scoring the unpatched baseline kernel instead of the agent's actual patch, and gained a GEAK_PROTECT_TEST_FILES mode that strips the agent's edits to the test harness so it cannot fake correctness by rewriting the reference. Apex ships a tamper detector for hardcoded print("PASS") plus a banned-library list so an agent cannot "optimize" a Triton kernel by quietly routing to a pre-tuned MIOpen or hipBLASLt call. SemiAnalysis's own framing: the models will reward-hack a benchmark the instant you let them, and the win is grit rather than intelligence. This is the mundane end of the same phenomenon ExploitGym (07-22) documented at the safety end, and it is the practical reason kernel-agent results need an end-to-end A/B gate rather than a self-reported speedup.

The strategic claim underneath: the CUDA moat was substantially an engineering-headcount advantage, and agents erode headcount advantages. SemiAnalysis's own two-and-a-half-person team used Claude Code and Codex agents to land day-0 fixes upstream (a DeepSeek-V4 fused-MHC kernel fix in TRT, a NixlConnector handshake fix for GQA-replicated KV heads, EAGLE3 speculative decoding for AMD MiniMax M3, FP8 KV-cache dtype support for MI300X/MI325X) and says this would not have been possible 3-6 months earlier. The second-order cost lands right back on this page's subject matter: each agent needs GPUs to test against, so agentic kernel work is now itself a major driver of internal cluster demand, and AMD's shortage of stable internal clusters is the top risk SemiAnalysis names. → summary

Also new on the ISA side (2026-07-25): AMD's gfx1250 (MI455X) speaks NVFP4 natively, shipping a compiled NVFP4 GEMM inside AITER with a runtime discriminator that dispatches NVFP4 versus MXFP4. NVFP4 (16-element blocks, FP8 E4M3 per-block scale, FP32 per-tensor global scale) is NVIDIA's Blackwell format and is becoming the default for FP4-quantized checkpoints; MXFP4 (32-element blocks, E8M0 power-of-two scale) is the OCP standard AMD built its four-bit path around. gfx950/MI355X does MXFP4 only. CDNA5 also adds an unsigned E5M3 scale format that repurposes the sign bit to drop the minimum representable magnitude from 2⁻⁹ to 2⁻¹⁷, offered as an alternative to NVFP4's extra global scale. The kernel-writer-facing change: CDNA5 drops to 32-thread waves (matching NVIDIA's warp), collapses Infinity Cache plus small L2 into a single 96MB L2 per Fabric-and-Cache Die (converging on the global→L2→shared hierarchy), and adds a Tensor Data Mover functionally equivalent to NVIDIA's TMA. Convergence on Hopper's programming model should reduce the porting tax, but it also means AMD's wave64 CDNA kernel corpus does not carry forward: gfx1250 wave32 kernels are largely unwritten.

Current State (as of 2026-05-27)

Activation sparsity gets its own kernels (2026-05-27): RT-Lynx. RT-Lynx flips diffusion-transformer sparsification from weights to activations, on the empirical finding that DiT activations are intrinsically sparse and far more robust to N:M semi-structured sparsification (keep N of every M values, maps to sparse-tensor-core hardware) than weights, which lose capacity when pruned. With error compensation and hand-optimized CUDA kernels for the activation-sparse GEMM (sparsify the dynamic operand, not the static weight), it reaches up to 1.55x average linear-layer speedup at preserved generation quality. The open thread: whether the kernel-generation agents tracked here (AccelOpt, AgentKernelArena, KernelBench-X) can auto-produce activation-sparse kernels rather than hand-tuning them.

Current State (as of 2026-04-21)

Kernel optimization is one of the hardest bottlenecks in production AI: hardware-specific, expert-intensive, and time-consuming. The dominant paradigm has been hand-tuned kernels by specialists (CUDA experts for NVIDIA, NKI experts for AWS Trainium). AccelOpt demonstrated LLM-agent-based automation; Nemotron 3 Super now shows the architectural path forward — hybrid SSM+MoE with native FP4 training and built-in speculative decoding, compressing the throughput gains from architectural decisions rather than kernel tuning alone. The SemiAnalysis goodput framework adds the economic layer: even optimal kernel performance gets negated by poor cluster reliability.

Key Papers

AccelOpt (2026-04-20) — Self-improving LLM agent for AWS Trainium kernel optimization. Maintains a memory of slow-fast kernel pairs from past iterations, uses it to guide generation of improved variants. Raises peak throughput from 49% → 61% on Trainium 1 (45% → 59% on Trainium 2). Matches Claude Sonnet 4 performance using open-source models at 26x lower cost. Introduces NKIBench, the first kernel benchmark for Trainium. → summary

Nemotron 3 Super (2026-04-21) — NVIDIA's hybrid Mamba-Attention MoE: 120B total / 12B active parameters, first Nemotron pretrained in NVFP4 (25T tokens), 1M context length. 2.2x throughput over GPT-OSS-120B, 7.5x over Qwen3.5-122B. MTP layers enable native speculative decoding with no external draft model. All artifacts open-sourced. Demonstrates convergence of SSM + sparse MoE + FP4 pretraining + native speculative decoding in a single architecture. → summary

SemiAnalysis GPU Cluster TCO / Goodput (2026-04-21) — Comprehensive framework based on 80+ neoclouds and 150+ customer interviews. Key finding: two providers at identical GPU-hour pricing can differ by 6–21% in useful work delivered (goodput). Three-tier recovery model (checkpoint-cold, checkpoint-hot, fault-tolerant). Gold-tier neoclouds (Nebius, Fluidstack, Crusoe) 5–15% cheaper than silver tier for large pretraining at the same nominal price. No open-source zero-overhead fault-tolerant training framework exists. → summary

Key Concepts

• Peak throughput utilization: fraction of the hardware's theoretical maximum FLOPS actually achieved. A kernel at 61% throughput leaves 39% of the hardware idle or stalled.
• NKI (Neuron Kernel Interface): AWS's programming model for Trainium accelerators, analogous to CUDA for NVIDIA GPUs. Writing efficient NKI code requires understanding Trainium's specific memory hierarchy (HBM, SBUF, PSUM) and compute engines.
• Memory hierarchy: modern AI accelerators have multiple memory tiers (high-bandwidth memory, on-chip SRAM, register files). Efficient kernels minimize data movement across tiers. This is the primary source of throughput gaps.
• Kernel memory: AccelOpt's key innovation — rather than trying to solve each kernel from scratch, accumulate (slow, fast) pairs and use them as few-shot examples. The quality of memory curation determines the quality of future improvements.
• Operator fusion: combining multiple operations (e.g., matmul + activation + normalization) into a single kernel pass to avoid intermediate memory round-trips. Expert-designed kernels like FlashAttention exploit this extensively.

Key Concepts

• Peak throughput utilization: fraction of the hardware's theoretical maximum FLOPS actually achieved. A kernel at 61% throughput leaves 39% of the hardware idle or stalled.
• NKI (Neuron Kernel Interface): AWS's programming model for Trainium accelerators, analogous to CUDA for NVIDIA GPUs.
• Memory hierarchy: modern AI accelerators have multiple memory tiers (HBM, on-chip SRAM, register files). Efficient kernels minimize data movement across tiers — the primary source of throughput gaps.
• NVFP4: NVIDIA's 4-bit floating point format for H100/H200 hardware. First demonstrated at pretraining scale in Nemotron 3 Super. Larger quantization error than FP8 requires careful loss scaling and gradient clipping.
• Native speculative decoding (MTP): embedding Multi-Token Prediction layers into the main model so it generates draft tokens internally, no separate draft model required. Removes deployment friction at the cost of some parameter budget per MTP head.
• Goodput: useful GPU work completed per dollar, accounting for downtime, fault recovery, and debugging time. The key cluster-level efficiency metric that nominal GPU-hour pricing obscures.
• Fault-tolerant training: job continues through a node failure without stopping and restarting. Currently: TorchFT (open source, 10%+ overhead), HyperPod Checkpointless (AWS-locked), TorchPass (licensed, zero overhead).

Open Problems

1. FP4 pretraining generalization: NVFP4 is hardware-locked to NVIDIA Hopper/Blackwell tensor cores. Can FP4 pretraining techniques transfer to AMD MI300X or AWS Trainium with different hardware FP4 implementations?
2. AccelOpt memory curation: optimal policy for which slow-fast pairs to retain, summarize, or discard as memory grows — analogy to KV cache eviction.
3. Open-source zero-overhead fault tolerance: TorchFT's 10%+ overhead comes from GLOO all-reduce. Theoretical floor should be much lower. No solution currently exists.
4. Goodput-aware routing: if goodput varies 3–15x across cluster tiers, an inference routing framework that factors in per-provider goodput loss could change the effective cost comparison for batch workloads.
5. SSM + MoE retrieval robustness: Mamba has known limitations on precise long-distance retrieval (needle-in-haystack). How does Nemotron's Mamba+LatentMoE combination perform on retrieval-heavy tasks vs pure-attention baselines at the same active parameter count?

Related Pages

• KV Cache
• AccelOpt summary
• Nemotron 3 Super: Hybrid Mamba-Attention MoE
• SemiAnalysis GPU Cluster Goodput

Current State (as of 2026-08-28): the loop closes, AI designs the silicon it runs on

Two things landed on the same day, one research and one industrial, and read together they change what this page is about.

Research: Beyond Scaling: Self-Evolving LLM Agents for Hardware Kernel Optimization (arXiv 2608.25570, Kurate cs.LG #5, not on HuggingFace). An experience-driven workflow (propose, compile, profile, diagnose, revise, against a real compiler and profiler, which is the only honest reward signal in this domain) plus an Experience Graph Memory that stores optimization episodes as linked nodes with typed edges, so a later kernel problem retrieves the transformation that worked on a structurally similar one. The title is the thesis: scaling the model is the wrong lever, scaling the retrievable memory of what worked is the right one.

This answers open problem 2 on this page directly. That problem read: "AccelOpt memory curation: optimal policy for which slow-fast pairs to retain, summarize, or discard as memory grows, analogy to KV cache eviction." An experience graph reframes it: you do not decide what to evict, you decide what to link, and retrieval prunes itself by following the edges relevant to the current kernel. Mark it partly answered, with the caveat that no eviction policy is free of a growth problem and the paper publishes no search cost.

It is also the fourth arrival at one structural idea from four subfields in one week. Recuris (08-26) split agent memory into verified Working Memory and Experiential Memory retrieved by that state. CaSKG (08-28) calibrates skill-graph edges by counterfactual intervention because unreliable edges are what make graph retrieval fail, and reports fewer environment steps as a result. Meta-Harness (08-25) got failure attribution by brute-force grepping up to 10M tokens of raw trace. The shared claim: an agent that improves over time needs structured, retrievable, attributable experience, and the structure of the memory is the design problem, not the model. CaSKG is the most relevant neighbour for this page, because "calibrate the edges before you trust the graph" is precisely the discipline a kernel experience graph will need once it is large.

Counter-reading worth keeping. Prime Agent (08-25) already reports matching or beating popular harnesses on GPU-kernel generation among its four task families, with no domain-specific kernel memory. Whether a kernel-specific experience graph beats a strong general harness on kernel work is an open comparison and neither paper runs it.

Industry: AI is designing the silicon (Hot Chips 2026, via The Information). OpenAI said Sol and Astra helped design Jalapeño, and this page already records the two concrete instances from the 08-26 Jalapeño summary: Codex wrote the working MLA kernels unaided, and AI-assisted design cut matrix-engine area 10%. Google's TPU team credited DeepMind with making TPU v8 6% more power efficient and 6% more powerful. Nvidia applies AI from stencil layout through on-die data movement. Design startup Agentrys raised $25M, led by Mark Ren, who ran Nvidia's design-automation effort for a decade and expects "very powerful agentic systems that can do chip design on their own nearly from start to finish."

The direction of travel is the story. AccelOpt (04-20) automated kernel writing for existing hardware, taking Trainium peak throughput utilization from 49% to 61% while matching Claude Sonnet 4 with open models at 26x lower cost. Hot Chips 2026 reports AI one level up, designing the hardware, with OpenAI's nine-month path to a competitive inference ASIC as the existence proof. The stack is now model → designs hardware → serves the model, and each turn is faster than the last.

Alignment note, and it is unusual. Research is proposing the memory architecture for kernel agents in the same week three silicon vendors put kernel and design agents in the shipping path. This wiki's Global View normally traces research running ahead of adoption or industry moving where research has not caught up. Here industry is ahead on deployment and research is ahead on mechanism: nobody at Hot Chips described an experience graph, they described "AI helped."

The measurement problem is real. Two 6% figures on TPU v8 and a 10% area reduction are the only quantified claims, none of them ablated, and Jalapeño's numbers come from OpenAI. "Our models helped design this chip and the chip is good" is not causal. This is the epistemic position harness optimization was in three months ago before DarwinX and AutoSaddler ran ablations, and here the ablations will be far harder, because you cannot cheaply re-tape-out a chip to isolate a variable.

Format politics now have an owner. This page records that NVFP4 is NVIDIA's Blackwell format and is becoming the default for FP4-quantized checkpoints, MXFP4 is the OCP standard AMD built its four-bit path around, and AMD's gfx1250 speaks NVFP4 natively with a runtime discriminator. As of 08-27, Nvidia has agreed to buy Hugging Face for $12.9 billion, roughly 80x forward revenue (summary). Which four-bit format the hub's default quantization tooling emits is a competitive lever, and it is now a defaults decision inside the company that owns one of the formats. That is the single most consequential thing that has happened to this page's format section, and it happened for commercial reasons that have nothing to do with kernels.