Memory Hierarchy for AI (concept)

The tiered memory system that modern AI — and especially agentic AI — runs on. The 2026 thesis: the binding constraint on AI infrastructure has shifted from compute (FLOPS) to memory (where state lives and how fast it moves). This page synthesizes what the wiki knows about each tier and how the research it tracks maps onto the hardware.

Current State (as of 2026-09-08): a vendor has budgeted on-chip SRAM around the KV cache, and the first third-party TPU inference numbers land

TPU inference externalization (09-08) (SemiAnalysis, InferenceX Official Preview) supplies this page with two things it did not have: an external measurement of TPU inference economics, and a hardware roadmap that treats the KV cache as the primary resident object rather than a consequence of serving.

The measurement. On apples-to-apples FP8 aggregated serving of Qwen3.5-397B, TPUv7 Ironwood costs $0.181 per million total tokens at 100 tokens/s/user, against $0.222 for B200 and $0.276 for B300, a 19% and 34% advantage. At concurrency 256 it delivers 50.4% more tokens per dollar than B200 and 96.0% more than B300, rising to 76.7% and 130.2% on Google's internal TCO of $1.03/chip-hour. The advantage is a TCO advantage, not a performance one: across most of the raw-performance curve TPU loses, and the concurrency-256 datapoint carries a 5.41s mean TTFT against B300's 2.40s. Two honest caveats from SemiAnalysis: in disagg-vs-disagg, GB200/GB300 NVL72 currently leads because the external TPU stack has no optimized prefill-decode disaggregation path yet (roughly 30% perf-per-dollar at mid latency against TPUv7 aggregated), and Ironwood has no native FP4, so NVIDIA leads outright there and the FP8-vs-FP8 framing is chosen to avoid comparing across a quality difference.

This is the memory wall priced by a third party. The 09-02 roofline chapter established that on a 70B model 99.66% of every decode step is spent moving bytes, which made this page's organizing claim that the only savings that count are the ones that move fewer bytes or skip the trip. Ironwood wins on bytes-per-dollar and loses on FLOPs, which is exactly the shape that claim predicts should win at inference.

The roadmap item that matters most for this page: TPUv8 splits training and inference silicon for the first time. TPU 8t keeps the 3D torus. TPU 8i replaces it with "Boardfly," a flatter dragonfly-style high-radix fabric that cuts network diameter more than half, roughly 16 hops to about 7 at 1,024-1,152 chips, carries 19.2 Tb/s of ICI bandwidth (double the prior generation), and 384 MB of on-chip SRAM, 3x the previous generation, sized specifically to hold the KV cache of reasoning and agentic models on-chip rather than round-tripping to HBM. That last clause is the first public instance of a vendor sizing an SRAM budget around the KV cache as the resident working set. The memory hierarchy is being redesigned around a data structure that did not exist as a design constraint two generations ago.

Tile geometry is now a hyperparameter tax, and this belongs on this page because it is a memory-layout fact. Every TPU through v5 used a 128x128 MXU; v6e onward is 256x256, 65,536 MACs/cycle. XLA pads any dimension below the array side length, and every padded cell burns a MAC multiplying by zero. So Llama 3 8B's head dimension of 128 caps its two attention matmuls at 50% MXU utilization, and gpt-oss's head dim of 64 caps them at 25%, before any kernel work. DeepSeek's MLA splits query/key into 128+64=192, awkward against any power-of-two array. On a GPU, small matrix-core tiles make head dim 64 nearly free, so GPU architecture let researchers pick head dimensions for eval performance without an inference penalty, and TPUs convert that free choice into a tradeoff. The consequence SemiAnalysis draws is an adoption one: bring-up cost varies enormously across models and correlates poorly with popularity.

Two kernel findings generalize past TPUs. First, KV cache lane layout: the batched attention kernel packed keys and values along the head dimension (packing factor four for FP8), so a model with one KV head per device wasted half of every 128-lane tile on padding. Putting the page's tokens on the lane axis and heads on the sublane axis doubled usable KV pages from 5,141 to 10,283, lifted throughput 16.5% at concurrency 128, and cut median TTFT 95% because requests stopped queueing for KV space, at a ~3% low-concurrency latency cost. Second, prefetch depth: the ragged-paged-attention heuristic set the KV compute block equal to the fetch block at ~16k tokens, starving the double-buffer. Splitting them (fetch 16k, compute 4k) took decode throughput from 64.9k to 96.3k tokens/s, a 49% gain reproduced across four runs, and the tuned-parameter table could not express the fix because it stored one block size per shape. Both are cases where the memory layout, not the arithmetic, was the whole bottleneck.

And the tiered-storage story arrives from both directions on the same day. Google is externalizing TPU-Sync, its disaggregated KV-cache transfer library (zero-copy via native PJRTBuffer descriptors), with native DRAM offloading and support for Mooncake Store including DRAM P2P pooling, which aggregates every host's KV storage into one logical pool so any chip can read any other server's cache, plus pooled NVMe on top. KVMem (09-08) independently pages agent KV state across GPU, host RAM and NVMe to virtualize a 1M-token workspace on a 24GB laptop GPU, beating summarize-and-discard compaction 43.8% → 48.4% on DeepSWE. Research and industry concluded the same day that KV state belongs on a storage tier rather than being summarized away, at opposite ends of the scale range, with no cross-reference.

Current State (as of 2026-09-02)

This page's founding thesis, that the binding constraint moved from compute to memory, now has its cleanest derivation and a trend claim that makes it worse over time. The Physics of LLM Inference, Chapter 1 (09-02) (Ken Huang, via starred Gmail) works the roofline model through language-model serving. Attainable throughput is min(P_peak, I × BW_mem) where operational intensity I is total arithmetic divided by bytes moved across the memory bus, and the ridge point I_ridge = P_peak / BW_mem is where the compute and bandwidth ceilings meet.

The numbers this page should carry.

• H100 SXM5 in FP8: 1,979 TFLOPS peak against 3.35 TB/s HBM3 puts the ridge at roughly 591 FLOP/byte. Single-stream decode sits at about 1.5 FLOP/byte, so it delivers under 0.3% of peak Tensor Core throughput. Not a software defect, the arithmetic of matrix-vector work at scale.
• The 70B decode proof, one line: 70 GB of FP8 weights stream in 20.9 ms; the roughly 140 GFLOPs they enable take 0.07 ms. Memory is 99.66% of step time.
• The two phases sit on opposite sides of the ridge. Prefill is matrix-matrix at 150 to 450 FLOP/byte and can saturate Tensor Cores, governing time-to-first-token. Decode is matrix-vector at 1 to 2 and streams every weight per token, governing inter-token latency. A fleet sized for the average of the two is wrong for both.
• Batching is the only escape and it is capped by this page's own subject matter. Intensity scales as I(B) ≈ (2 / S_weight) × B, so reaching the H100 ridge in FP8 would need about B = 296 concurrent decode streams. Production never runs that hot because KV cache memory and latency SLOs bind first.

The trend claim is the important one, and it is an argument this page has not made. Compute has scaled faster than HBM bandwidth across every accelerator generation, which pushes the ridge point rightward. So the batch size required to escape memory-bound decode rises with each new GPU. That inverts the intuition that better hardware makes serving optimization a transitional concern: each generation makes KV-cache capacity work more valuable, not less, because KV memory is precisely what prevents reaching the batch size the new silicon demands. That is the strongest available argument for why the scarce resource in 2027 is the same one it is today, and it belongs next to the HBM-allocation and packaging threads below.

It also supplies a criterion for grading everything on the efficiency side of the wiki. Any optimization that does not reduce bytes moved per token is fighting a 99.66% ratio uphill. Quantization, batching, speculative decoding with acceptance, and KV compression pass. FLOP-only reductions do not. Read that way, today's cross-model KV sharing (09-02), which cuts a Llama-70B-to-Qwen-7B handoff from 899ms to 138ms, is a different and better class of saving than it appears: it eliminates work in the compute-bound phase, the one place the accelerator runs near its ceiling, without touching the memory-bound phase that dominates steady state.

And a precision collision worth recording, which neither source notes. The compute-outpacing-bandwidth trend is the main reason the field keeps pushing to lower-precision training and serving. Lower precision means fewer exponent bits. TrainSDC (09-02) finds that backward-pass vulnerability to silent data corruption is governed by gradient exponent distributions rather than by computation location, at a mitigation cost of 1.65 to 6.76% runtime overhead. The hardware trend that makes quantization attractive is the same one that makes gradient-exponent corruption more dangerous, and no source in this wiki has priced that tradeoff jointly.

Caveat on sourcing: the free tier of this chapter covers the roofline, the prefill/decode split and the latency taxonomy. VRAM accounting, chunked prefill, continuous batching and PagedAttention, interconnect costs and cluster sizing are paywalled, so the ridge-point table for Blackwell-class parts cannot be verified against measured numbers from the public text.

Prior State (as of 2026-07-30)

A constraint that binds earlier than memory: the building. This page has argued since June that the binding constraint on AI infrastructure moved from compute to memory. SemiAnalysis's modular-construction analysis (07-30) adds a constraint that binds before either, and it has nothing to do with semiconductors. Trade labor cannot be produced on a two-year cycle. Electricians alone are 30 to 40% of total construction man-hours on a datacenter project, and SemiAnalysis's new Labor Model projects a shortage emerging in 2027, concentrated exactly where the buildout concentrates, in Texas and Ohio. The evidence that it already binds is a price rather than a forecast: Crusoe raised wages 30% to staff Abilene, which peaked above 9,000 workers. The analytically important move is treating reachable labor supply as a pool shared across states, so a project in one state reduces the labor available to its neighbors and site capacity stops being additive. That is a different failure mode from HBM allocation, which is a queue: labor is a substitution game between simultaneous projects.

The industry response is to move work into factories, and it works: modular compresses the build window by ~36%, or 7 to 9 months, at ~8% lower capex/MW, with 61GW+ of modular capacity tracked across 1,000+ sites and penetration projected past 30% of live capacity by end of 2028. The site layer cannot be modularized, since foundations are poured on the parcel, so this mitigates rather than removes the constraint. The supply-chain consequence belongs on this page: when on-site labor migrates into factory-built units, it reappears as vendor equipment content, and Vertiv's content per megawatt roughly doubles from ~$3.5M/MW to ~$7M/MW. That is the same substitution the labor model predicts, expressed as revenue, and it makes the modular OEM layer a structurally better position than the EPC layer.

Two financial datapoints landed the same day and read as this constraint priced. Meta's capex nearly doubled to $30 billion in one quarter, about half its revenue, while free cash flow fell 91% to just under $800 million and operating income fell 8%. Microsoft spent more in absolute terms at $35.8 billion, but that was 40% of revenue with opex up only 10%, so operating income rose 18%. Same category of spend, opposite cash discipline, and the market split accordingly. Alongside it, The Information reports "spend now, lease later" bridge financing is among the fastest-growing asks in infrastructure finance, with developers borrowing against projects that have no signed tenant to level land, run power studies, and queue for scarce gas turbines. Read together: developers racing a labor curve must commit capital before the lease exists, so the financing layer is absorbing the timing risk that modular construction is trying to remove physically. The forecasting correction for this page is concrete: 2027 US capacity should be modeled against electrician-hours in the concentrating states, not only against wafer starts and packaging capacity. Both constraints are real; this one binds first.

Current State (as of 2026-07-26)

The memory shortage crossed from a procurement problem into a national-security fight, and the person who escalated it is Tim Cook. Apple is lobbying the Trump administration for clearance to buy memory from China's CXMT and YMTC for devices sold outside the United States, arguing it eases the shortage and keeps iPhones cheaper (@MarioNawfal relay, 07-25, slot synthesis). Both firms are designated Chinese military companies by the Pentagon and YMTC is on the Entity List. These are the same suppliers Apple walked away from in 2022. Cook is simultaneously accusing Micron, the only large US memory maker, of gouging at 80% margins while calling the shortage a "100-year flood."

Those two claims cannot both be complaints. If the shortage genuinely is a hundred-year flood, then 80% margins are the price signal doing its job, not an abuse, and Cook's own framing is the strongest outside confirmation this page has of the structural-shortage thesis it has carried since June (Micron's ~3:1 HBM-to-DDR5 wafer trade ratio rising per generation, packaging as the bottleneck, top-end HBM allocation-driven into 2030). The wiki now has the same constraint visible from four independent vantage points inside two weeks: an accelerator vendor deleting a memory tier from a shipping module (AMD's MI455X LPDDR), a second signing a half-trillion-dollar arrangement to co-develop the next generation (NVIDIA and SK), a memory maker's stock up 213% year to date against NVIDIA's 10%, and now the largest consumer-hardware buyer on earth asking Washington to let it buy from an entity-listed supplier.

The forward-looking point for inference economics: DRAM and HBM contention are not separate markets in the ways that matter, because they compete for the same wafer starts and the same advanced packaging. Consumer device demand bidding against datacenter HBM is a mechanism by which phone volumes set a floor under inference cost, which is not how anyone models it. Watch whether the administration grants any carve-out, because a denial keeps that contention fully priced into HBM.

Current State (as of 2026-07-25)

Memory stops being a component and becomes the axis of competition, on the same day, in three forms. The 07-22 section below set a falsifiable watch on whether Meta would take the standard MI450 or the half-strength variant. Today's SemiAnalysis Advancing AI 2026 analysis resolves it: most of Meta's MI455 orders are the cut-down part (compute halved 8 XCDs → 4, HBM halved 12 stacks → 6, and downgraded 12-Hi → 8-Hi). The decision was made by RecSys infrastructure teams before Meta's TBD Lab existed, TBD has no interest in the resulting part, and SemiAnalysis says external customers will not want it either. AMD's Meta GenAI volume is the casualty exactly as the 07-22 prediction framed it, with one hedge: Zuckerberg has begun rapidly reworking Meta's infra org culture since that article.

Three concrete memory facts arrived with it. (1) The standard MI455X is a memory part first: 12 HBM4 stacks for 432GB at 23.3 TB/s, against Rubin's 8 stacks for 288GB. Fitting 3 cubes per base-die edge required growing the HBM-facing edge to 32mm (the MI300X AID's 29mm could not fit three). (2) NVIDIA answered with pin speed, not capacity, and moved a JEDEC spec to do it. It raised its HBM4 target to 10.7 Gbps, 40% above AMD's 7.6 Gbps and well past the original JEDEC HBM4 specification, purely to erase AMD's bandwidth lead. That lands Rubin at 22 TB/s from a 33% narrower bus, forces NVIDIA onto a much higher-quality bin, made suppliers rework HBM4, and delayed Rubin's own ramp. Competitive pressure between two accelerator vendors propagated backwards into a memory standard. (3) The shortage is now deleting product features. The up-to-1TB of direct-attached LPDDR per MI455X EAM module, present on earlier roadmaps as a second memory tier, has quietly disappeared, which SemiAnalysis reads as a consequence of tight supply. The second tier of accelerator-attached memory is a casualty of allocation.

The commercial counterpart landed the same 24 hours: NVIDIA announced a $500B partnership with SK Group, owner of SK hynix, explicitly to access more HBM and fill more datacenters, including joint HBM co-development plus a 2GW Vera Rubin DSX factory in Korea. One vendor signs a half-trillion-dollar supply arrangement with a memory maker; the other deletes a memory tier. That is what this page's structural-shortage thesis (Micron's ~3:1 HBM-to-DDR5 wafer trade ratio rising per generation, packaging bottleneck, top-end HBM allocation-driven into 2030) looks like when it reaches the balance sheet. Markets are pricing it accordingly: Micron is up 213% year to date and AMD 142%, against NVIDIA's 10% despite 83% expected revenue growth.

Rack-level memory movement is now a named cost line too. Helios connects 72 MI455X through 12 merchant Broadcom Tomahawk 6 switches (only 432 of each switch's 512 lanes usable, because 102.4T does not divide evenly into 72 GPUs, where NVIDIA sized the 28.8T NVSwitch for exactly 72 with zero waste). AMD's 200G SerDes cannot hold the copper backplane, so ~85% of Meta's scale-up links need Broadcom retimers, 550+ per rack. Backplane plus compute-tray content: $68,928 per rack, 10,368 differential copper pairs. The lesson the page has been accumulating since Vera Rubin: at rack scale, moving bytes is the cost, and co-design is the only lever on it.

Current State (as of 2026-07-22)

Two same-day hardware signals frame the co-design divide: NVIDIA ships a rack-co-designed Vera Rubin at 10x tokens/MW while Meta is documented wasting billions going off-menu. NVIDIA declared Vera Rubin in gigascale production at SIGGRAPH week (summary): CoreWeave's first measured silicon shows 10x more tokens per megawatt than Grace Blackwell NVL72 on DeepSeek-R1, framed explicitly as performance-per-watt and lowest token cost, plus Spectrum-6 (102.4 Tb/s Ethernet, 2x prior) and the Vera CPU (>2x faster, more concurrent agents). The headline is now efficiency, exactly the tokens-per-joule metric the energy-to-token position paper argued for, and it comes from extreme co-design across seven chips and five rack trays. The counterpoint, from SemiAnalysis the same day (summary), is Meta's infra org shipping the opposite: a $2.5B+ Rivos acquisition effectively dismantled, a custom GB200 (Ariel, one B200 per Grace in NVL36x2) with 14% higher TCO than the standard SKU that Meta's whole GB200 fleet ran on, and an upcoming plan to take a half MI450 (halved compute silicon and HBM stacks, HBM4 downgraded 12-Hi → 8-Hi) to chase a higher CPU:GPU ratio for RecSys embeddings. The MI450 matters for this page's memory thesis: the standard part has 12 HBM stacks and the largest CoWoS reticle on the market; gimping it to 8-Hi trades exactly the HBM capacity/bandwidth that GenAI training and inference are bottlenecked on. Falsifiable watch: whether Meta takes the standard MI450 (AMD gets its first Rubin-competitive GenAI volume) or the half-chip (AMD's Meta volume collapses, NVIDIA Rubin lock tightens). The meta-lesson both pieces teach: at the rack level, hardware/software co-design is the dividing line, and custom silicon without that discipline is a money furnace.

The tiers (fast/scarce → slow/abundant)

Core facts the wiki keeps returning to

• KV cache, not weights, is the dominant memory traffic at long context. Generation is memory-bandwidth-bound; as context grows the traffic shifts from model weights to KV cache. This is the hardware fact underneath every KV-cache software paper the wiki tracks. (Ken Huang memory survey, 06-07; UC Berkeley 2026 report.)
• Agentic workloads consume ~15x more tokens than traditional AI apps (NVIDIA), so they stress every tier at once, not just HBM.
• The memory shortage is structural. Micron's ~3:1 HBM-to-DDR5 wafer trade ratio (rising per HBM generation) + advanced-packaging bottleneck → tight/allocation-driven through 2026, selective relief 2027, broader 2028-2029, top-end HBM allocation-driven into 2030.
• Latency over throughput for agents. Interactive agents are judged on time-to-first-token and tail latency, which favors SRAM-first designs, disaggregated prefill/decode, and KV-aware tiering over raw batch throughput.

How the research maps onto the hardware

• KV-cache eviction / quantization / low-rank latent caches (VASE 06-03, VideoMLA/StateKV 06-01, LongAttnComp 06-02) are the software answer to "KV cache is the bottleneck tier" — they shrink what must live in HBM so cheaper tiers (LPDDR SOCAMM2, CXL, SSD) suffice. See kv-cache.
• Parametric context internalization (Code2LoRA, Video2LoRA 06-06) is the most aggressive move: bake context into weights so it never enters the KV cache at all. See parametric-context-internalization.
• Input-side compression (AdaCodec 06-06, SEAOTTER 06-07) cuts bytes/tokens before they reach the model — relief upstream of every memory tier.
• Compute rationing (CLEAR 06-05) is the demand-side dual: when HBM is scarce for five years, the serving stack must ration per-query compute by marginal utility, and (the open direction) ration KV-cache placement across tiers per-request.

Open problems / what to watch

• KV-aware tiering policies — deciding per-request which KV blocks live in HBM vs LPDDR/CXL/SSD. The hardware counterpart to CLEAR's per-query compute rationing; not yet a shipped serving-stack feature.
• Will PIM/CIM cross from research to product for attention/embedding/search, or stay niche behind analog-precision and compiler-maturity barriers?
• Does the SRAM-first bet (Groq) capture agentic inference as the market tilts toward low-latency, high-value tokens, or does ecosystem/software depth keep HBM-GPU dominant?

Sources

• Memory Technology for Agentic AI Workloads (Ken Huang, 06-07)
• Related hardware pages: gpu-kernels, SemiAnalysis 800VDC / cluster-goodput primers.
• Related efficiency: kv-cache, parametric-context-internalization

Supply answers, on the wrong side of the export-control line (2026-07-27)

CXMT listed in Shanghai and rose 472% on debut, opening at roughly 3.3 trillion yuan (about $487 billion) (summary, The Information). Discount the number heavily: a 472% first-day move in a market with capital controls, where the Semiconductor Newsletter's Week 30 issue reports state-affiliated capital supplying more than 90% of China private equity funding, is a policy signal rather than a price.

What survives the discount is that this is the first supply-side datapoint in a shortage this page has tracked entirely from the demand side. The prior four vantage points were all buyers scrambling for allocation: AMD deleting the LPDDR tier from MI455X, NVIDIA committing $500B with SK hynix's parent (07-25) which is locking supply through equity rather than purchase orders, Micron up 213% year to date against NVIDIA's 10%, and Tim Cook asking Washington for an entity-list carve-out to buy from CXMT and YMTC while simultaneously accusing Micron of gouging at 80% margins and calling the shortage a "100-year flood." An IPO of this size is capacity being funded.

The correction this forces to the page's model. The 07-26 digest predicted a denial of Apple's carve-out would keep memory contention fully priced into HBM, treating the carve-out as the channel by which CXMT capacity reaches the market. That conflated two separable things. CXMT can fund fabs domestically at a $487B valuation whether or not Apple is permitted to buy. The carve-out determines who CXMT sells to; it does not determine whether CXMT builds. Access and capacity have come apart, and the page should track them separately from here.

What actually matters next, in order: announced capex and fab timelines against the IPO proceeds, because wafer starts with dates are the only number that moves the constraint; whether CXMT is pursuing HBM or only conventional DRAM, since they compete for wafer starts but serve different markets and the reporting does not say; and the BIS determination on Apple, still the cleanest test of the allocation regime, now with the understanding that a denial slows distribution rather than construction.

2026-09-10: three physical inputs bind at once

This page's organizing thesis is that the binding constraint moved from FLOPs to memory. On 2026-09-10 that thesis got its strongest confirmation and its first serious complication, on the same day.

Confirmation. DeepSeek V4.1 Flash responds to HBM scarcity by architecture rather than by procurement: nearly half its parameters are Engram embeddings living on host LPDDR5 instead of HBM, the remaining backbone is mostly 4-bit, and the reported result is a quarter of the previous generation's HBM and an eighth of its SSD. Chapter 4 of Ken Huang's inference series supplies the numerics manual for the same move: Blackwell NVFP4 native 4-bit Tensor Core math, DeepSeek-V4's 128x128 tile-wise FP8 GEMM, and KIVI 2-bit streaming KV caches with per-channel keys and per-token values. Both are demand-side answers to a memory shortage, and they arrived within hours of each other.

Complication. Two other inputs turned out to be binding the same day, and neither is memory. Behind-the-meter power (09-10) records 75 GW of firm, binding orders for onsite generation, 20 GW of them placed in Q2 2026 alone, with over $150B of OpenAI-Oracle contracted spend depending on it. The Pulse #191 (09-10) reports a CPU shortage, caused not by training but by agents running tools: file reads, builds, test suites and container starts are conventional x86 work generated at machine speed.

The synthesis this page should carry forward: memory, electricity and general-purpose compute are simultaneously constrained, and the industry is attacking them with opposite strategies. Memory gets demand reduction (quantize, relocate to LPDDR, share across layers). Electricity gets supply expansion (buy 75 GW of turbines). CPU gets nothing yet, because nobody was measuring it. The asymmetry is informative: the research community optimizes the input it can address in software, procurement handles the one it can address with capital, and the one that falls between the two goes unmanaged until it breaks.

A second-order effect nobody has priced. If Engram-style architectures spread, and LongCat 2 and Qwen-3.8-Flash-Next have already adopted the approach, the relief valve for HBM scarcity routes demand into LPDDR, which the 09-10 robot-inference analysis notes is already competing for a shrinking pool of non-HBM wafers as incremental DRAM capacity is absorbed by HBM. Solving an HBM shortage by consuming more LPDDR works only while LPDDR is slack.

2026-09-15: the workload that wants exactly what yesterday's argument said to stop buying

Yesterday this page recorded the 4-hi HBM argument (09-14): an HBM4 cube exposes 2,048 data I/Os regardless of stack height, one die can drive at most 512, so 4-hi is the shortest stack that harvests full bandwidth while price tracks gigabytes, and 12-hi costs 26.3% more system-wide for 10% more throughput. Today the same publisher defined the workload that will set the next generation's requirements, and it wants capacity.

The Vera Rubin agentic-inference benchmark (09-15) characterizes agentic traffic by four properties, and two of them are memory demands. High prefix reuse: because turn n concatenates turn n-1's output, most context can be served from cache rather than recomputed, and as the session grows the cached-to-uncached ratio tends toward 1, with the explicit qualifier that this depends on how much storage is available to hold KV tensors. Sub-agent bursts: a session spawns short-lived sub-agents with fresh context, producing bursty and therefore unpredictable allocation. Read as a hardware specification, that says the binding constraint on agentic serving is KV capacity and the speed of moving KV tensors, not FLOPs, which is what makes the six-product co-design (Rubin GPU, Vera CPU, NVLink 6, ConnectX-9, BlueField-4, Spectrum-6) a memory-and-fabric answer rather than a compute one.

The tension is real and SemiAnalysis hold both positions in two days without reconciling them. The resolution is that capacity and bandwidth become separately purchasable once KV lives in a hierarchy: Ken Huang's Chapter 6 (09-15) describes Moonshot's Mooncake engine, deployed in the 2.8T Kimi K3 stack, spanning VRAM, host DRAM, local SSD and a distributed pool with zero-copy GPUDirect RDMA between tiers. Buy capacity at the cheapest tier meeting the latency budget and size HBM for bandwidth alone, which is exactly what the 4-hi argument asks for. Neither SemiAnalysis piece names this.

One detail in the Rubin piece sits oddly against all of the above and deserves flagging. The benchmarked rack is the production SKU at 2300W TDP with 1.5TB of CPU LPDDR5X per compute tray, half the Vera memory originally planned. Host DRAM is the second tier in exactly the hierarchy that resolves the capacity argument. Halving it, in the same article that defines the workload as KV-capacity-bound, is a decision the piece notes and does not analyse.

The gap that matters for the edge. A Brain Too Big to Carry (09-15) reports that Jetson Thor, the best robot compute purchasable today, delivers about 1/10th the FLOPs of a GB200 and roughly 1/30th of its memory bandwidth. The bandwidth gap is three times the compute gap, so edge inference is more bandwidth-starved than datacenter inference, not less. Every quantization, eviction and cache-layout result in this wiki was tuned against datacenter compute-to-bandwidth ratios. The techniques that win at a 30x worse ratio are plausibly a different set, and nobody has published that comparison.

Supply, for context. The Semiconductor Newsletter's Week 37 reports DRAM revenue up 59.5% as AI memory demand outpaces supply expansion, HBM shortage pushing Chinese AI accelerator prices up as much as 50%, Samsung targeting High-NA EUV for DRAM high-volume manufacturing by 2028, and SK hynix positioning 3D memory and full-stack AI memory as core directions. Memory equities nonetheless sold off hard on a safety essay this week (SK Hynix -6.4%, Kioxia -6.4%, Samsung -4.1%), which is the market pricing expectations while the revenue prices deliveries. See the pacing debate.

2026-09-14: the HBM content trend breaks, and short stacks become the optimum

This page's model of the memory shortage has been demand-side throughout. Buyers scrambling for allocation, architectures cutting capacity demand, and the 09-10 entry summarizing the response as "quantize, relocate to LPDDR, share across layers." SemiAnalysis's 4-hi HBM analysis (09-14) supplies the supply-side half, and it argues the shortage is partly self-inflicted by a packaging convention that stopped matching the workload.

The fact that drives everything: bandwidth per HBM cube does not depend on stack height, but price does. An HBM4/4E cube exposes 2,048 data I/Os regardless of how many core dies it contains, and a single die can drive at most 512 of them. Four dies times 512 equals 2,048, so 4-hi is the shortest stack that harvests full bandwidth. Cost tracks gigabytes. Every die above the fourth adds capacity and BOM and adds no bandwidth at all.

The reversal is already shipping. Nvidia's Rubin Ultra carries 192GB per GPU, down from 288GB on standard Rubin and Blackwell Ultra, moving the supply chain to 8-hi as standard less than a year after the industry was planning 16-hi and beyond. SemiAnalysis's claim is that 8-hi is a waypoint: frontier-lab hardware teams want 4-hi in their own ASIC programs from HBM4 onward.

The workload argument is the part this page should carry forward. Inference decode is bandwidth-bound, because every generated token reads all active parameters plus that user's KV cache. Pretraining is capacity-bound, holding weights, gradients and activations at once. RL behaves like inference but needs more capacity for large batches. Pretraining's share of total compute has collapsed as token demand grew and post-training became the main scaling vector, so the majority of allocated compute is now bandwidth-sensitive rather than capacity-sensitive. Capacity is thresholded, not continuous: enough to hold weights plus resident KV, then diminishing returns at constant cost. Cerebras and Groq are named as the failure case on the other side, enormous bandwidth with insufficient capacity.

Aggregate capacity has outrun model size, which is why the cut is affordable. An H100 HGX gave 640GB aggregate and Llama 3.1 405B ate 63% of it. GB300 NVL72 delivers a 9x larger scale-up world plus doubled per-GPU HBM, 18x the aggregate HBM of an H200 HGX, against roughly 7x growth in the largest open model (Kimi K3, 2.8T). One MXFP4 replica of Kimi K3 is 1,561GB, under 8% of the ~21TB on a GB300 NVL72. Rubin Ultra goes to NVL576, another 8x on world size, more than offsetting the one-third per-GPU cut.

The numbers that settle it. HBM4E at 13 Gbps across 2,048 pins gives 3,328 GB/s per stack; 32-Gb core dies give 16/32/48GB per cube and 128/256/384GB per Rubin Ultra GPU at 4/8/12-hi. Serving Kimi K3 on NVL576, each GPU holds only 16.8GB of weights. At 105 tokens/s/user, 12-hi buys nothing over 8-hi. At 213 tokens/s/user, nothing above 4-hi buys anything. On peak throughput 8-hi is +8% and 12-hi is +10% against 4-hi, while all-in system cost rises 12.1% and 26.3% respectively. Both premiums exceed both gains, so 4-hi delivers the lowest cost per token. Under a hypothetical 3x-larger model with KV scaling to match, 12-hi gains 47% and 8-hi 36%, and 8-hi then beats its premium below 180 tokens/s/user, but 12-hi over 8-hi is never worth it anywhere on the curve.

The metric this page should adopt: tokens per HBM wafer. The industry already optimizes tokens/watt because power is scarce and tokens/TCO because money is. HBM wafers are the third scarce input, and on that axis a wafer spent on 4-hi harvests roughly triple the bandwidth of the same wafer spent on 12-hi and double against 8-hi, better still after packaging yield. Going 4-hi more than doubles harvestable cubes, which moves the bottleneck onto logic wafers, base dies, substrates, PCBs, integration capacity and power, all of which SemiAnalysis judges able to ramp faster than DRAM. Whatever the AI supply chain cannot absorb frees wafers for conventional DRAM, where server DRAM per socket has been cut hard by HBM's cannibalization. That closes a loop this page opened on 09-10: the CPU shortage caused by agents running tools is partly a server-DRAM problem, and 4-hi is the first mechanism anyone has proposed that relieves it.

It reconciles with, rather than contradicts, the architecture thread. DeepSeek V4.1 Flash (09-10) parks nearly half its parameters on host LPDDR5 and runs a mostly-4-bit backbone to reach a quarter of the prior generation's HBM. SemiAnalysis notes it also cuts active KV cache roughly 75% against v4 Flash, and that at those levels a workload saturating HBM capacity becomes compute- and network-bound instead. The architectures are cutting capacity demand at exactly the moment the packaging is cutting capacity supply, and the two together are a stronger argument than either alone. The 09-10 warning still stands and gets sharper: solving an HBM shortage by consuming more LPDDR works only while LPDDR is slack, and 4-hi is the first proposal that adds LPDDR-class supply instead of consuming it.

It partly resolves this page's standing open problem. "KV-aware tiering policies, deciding per-request which KV blocks live in HBM vs LPDDR/CXL/SSD" was listed as not yet a shipped serving-stack feature. SemiAnalysis reports KV offload as routine in AgentX benchmarking, with many workloads' Pareto-optimal results now obtained with offload above certain concurrencies. The policy ships; the per-request version does not. And their restricted-HBM experiment supplies the failure boundary the whole placement thread lacked: offload absorbs a 36-44% KV budget cut invisibly, then loses ~30% of throughput in a cliff when occupancy saturates, with 10x the DRAM reads. kimi-k3-in-c's NVMe expert streaming (09-12) is the extreme version of the same bet and has never been characterized near its own cliff.

A new read-through this page has never had: architecture choices now have direct DRAM-procurement consequences. SemiAnalysis states it is "effectively confirmed" that GPT-6 Astra uses looped transformers, with speculation that Anthropic's largest models do too, and uses it as a rebuttal to the future-proofing objection: a looped model buys effective depth by reusing layers, so it scales through depth rather than width. Depth costs bandwidth and compute; width costs capacity. A frontier lab choosing depth is choosing the axis on which short HBM stacks are fine, and that is the first time an architecture decision on this wiki reads through to a memory purchase order. The claim is unsourced in the free portion of the article and should be tracked rather than believed.

Caveats. Every quantitative result here is SemiAnalysis's own roofline or its own benchmark, with supplier impact behind a paywall. The 3x-larger-model sensitivity is a single scenario, not a sweep, and it is precisely the scenario the 5-year-server-lifespan objection turns on.

2026-09-18: Rubin Ultra loses 80% of its HBM, and "offloadable by design" gets a test

The largest hardware fact of the day is stated in passing in a SemiAnalysis piece about something else: NVIDIA despec'd Rubin Ultra from 1024 GB to roughly 200 GB of HBM per chip. (Engrams Embedding Entendre, 09-18.) This page's founding thesis is that the binding constraint moved from FLOPs to memory, derived from the roofline result that on a 70B model 99.66% of every decode step is spent moving bytes. A five-fold cut in HBM per chip on the flagship roadmap does not challenge that thesis, it detonates the planning assumptions built on top of it. Every serving plan that assumed HBM capacity would keep growing on schedule needs rebuilding, and architectural memory reduction stops being an optimization and becomes an admission criterion.

The same study delivers the measurement the 09-10 V4.1-Flash page named as its single biggest gap. That page asked for "wall-clock serving numbers for the Engram host-memory path under concurrency, which is the regime where a PCIe round trip to LPDDR either amortizes or does not." SemiAnalysis replicated Engram, replaced the roughly 189 GiB table in the DeepSeek-V4.1-Flash configuration with a memory-mapped file, and benchmarked serving with it on SSD across six NVIDIA SKUs plus MI355X. It amortizes. More than that: even on high-capacity HBM parts, offloading the table to DRAM beats keeping it in HBM across most of the pareto, because the freed HBM buys larger batches and more concurrent sessions and that throughput gain outweighs the added read latency. Eight days from open question to published answer, resolving in the architecture's favour.

The mechanism behind why it works is the transferable part, and this page did not have it stated as a criterion. Engram's row addresses depend on token IDs, not on hidden states. The runtime therefore knows which rows it will need before computing anything and can prefetch them from host DRAM while earlier layers run. That is the property that separates "offloadable by design" from "offloaded as a fallback," and it is a crisp architecture-review question: are the addresses derivable from the input tokens or from the hidden state? Token-derived means prefetchable means the module may live on a cheaper tier for the rest of its life. Hidden-state-derived means it may not. Recommendation systems have run exactly this playbook for years, hot rows in fast memory and cold rows backed by SSD, and it is now arriving in LLM serving.

Two findings complicate the tidy version and both should be carried. First, gate scores do not identify cache-hot rows, for a mechanical reason: computing the gate requires the retrieved key, so the read is already paid for by the time the gate says it was not needed. Skipping reads would need a separate usefulness predictor running before retrieval, which does not exist. Second, Engram is not a detachable dictionary bolted onto an unchanged MoE. Suppressing it worsens token likelihood nearly everywhere, though GSM8K accuracy stays inside run-to-run variation, so grade-school math apparently does not route through it. On CRUXEval, removing Engram raised answer loss from 0.2848 to 0.3093 bits per token, and forcing the ablated model to reuse the original Engram-on expert routing made it worse still at 0.3375. The experts reroute to partially compensate. There is no clean "memory stores facts, experts reason" division; the two are co-adapted. Placement policy does get one actionable asymmetry: keeping Engram for prefill produces more correct answers than keeping it for decode, so if you must degrade, degrade the decode-side path.

The most important unmeasured interaction on this page now has a name. The 09-14 4-hi HBM study found KV offload to second-tier DRAM absorbs a large capacity cut almost invisibly and then fails as a cliff past roughly 70 concurrent sessions. This study finds Engram offload is a net win across most of the pareto. Both are the same bet on the host DRAM tier, made twice inside one model, and nobody has measured what happens when both are made at once. If KV state and Engram rows contend for the same DRAM bandwidth, the 09-14 cliff should arrive earlier. That experiment is cheap and decisive and nobody has run it.

It also enlarges a tension this page flagged on 09-10 and could not price. DeepSeek's answer to HBM scarcity is to consume more LPDDR, while the same week's supply analysis found commodity and LPDDR supply already competing for a shrinking pool of non-HBM wafers. A five-fold HBM cut on NVIDIA's flagship pushes far more of the industry down that same relief valve simultaneously. The second-order effect is still unpriced and now has a much larger first-order cause.

Materials read-through, from the same week's starred reading. The Semiconductor Newsletter's 2027-2033 materials roadmap (09-17) puts numbers under the supply story: a record $73.2 billion 2025 materials market, with packaging materials growing 9.3% against 5.4% for wafer-fab materials, a divergence the report attributes directly to AI accelerators, HBM and larger multi-die packages. It names HBM as the clearest near-term materials multiplier (more die per stack, denser TSV arrays, thinner wafers, stricter warpage control) and cites SK hynix expecting roughly 30% annual HBM market growth through 2030, flagged as a company forecast rather than consensus. It also names thermal materials as a first-order scaling constraint and concentrates supply risk in a small set of upstream nodes: Japan in silicon wafers, photoresists and CMP; China in primary low-purity gallium; southern Africa and Russia in primary ruthenium. The line to carry: qualification cycles run in quarters to years, so a nominally available substitute is not a production-qualified one. Against a five-fold HBM despec, that is the constraint that decides how fast anyone can respond.

2026-09-23: memory becomes the majority of the industry, and a vendor puts the KV cache on a roadmap

Semiconductor Week 38 (09-23) supplies the number that reframes this whole page: global semiconductor revenue reached $425 billion in Q2 2026, and memory exceeded 50% of the market for the first time. For most of the industry's history memory was the commodity half and logic carried the margin. This page has been tracking the inversion through its symptoms for a month: the 09-14 4-hi HBM study showing vendors choosing bandwidth over capacity, the 09-18 Engram study reporting NVIDIA despec'ing Rubin Ultra from 1024 GB to roughly 200 GB of HBM per chip, and the 09-22 four-regime decomposition showing decode attention is context-bound and decode experts are weight-bound, neither compute-bound. The revenue split is the market pricing that claim in dollars rather than another paper arguing it.

The item to carry forward is SK hynix announcing an AI memory architecture spanning HBF (high-bandwidth flash, aimed at capacity sitting behind HBM), PIM (processing-in-memory, compute placed inside the array so some operations never cross the bus), and explicitly "tiered KV cache management." Every entry on the KV cache page is a software argument about how to spend a memory budget under a fixed hierarchy: compress it, evict it, quantize it, share it across depth, offload it to a cheaper tier. KVMEM (09-23) built exactly that hierarchy in software, GPU to host RAM to NVMe, on a laptop, and got a 1M-token agent workspace at roughly 50 tokens/s out of a single 24 GB RTX 5090. A memory vendor naming the same tier as a product feature means the hierarchy those systems improvise is becoming something you buy rather than something you build.

The open question this page should hold, and it has no answer yet: if processing-in-memory can execute attention-adjacent operations inside the array, which of the cache-compression techniques on the KV page are solving a problem that moves? A compression result is an argument about a bus crossing. A technique that eliminates the crossing changes the denominator rather than the numerator. Nobody has published a comparison, and SK hynix supplied no capacity, latency or ship-date figure, so the software hierarchy remains the only measured version.

Two supporting figures from the same week. NVIDIA's Vera Rubin NVL72 delivered up to 3.7x GB300 throughput in an MLPerf preview, which is a vendor number and should be discounted as one, but the direction is consistent with the despec story: the gain is coming from the rack and the interconnect rather than from per-chip HBM capacity, which was cut roughly fivefold. That is only coherent if the workload is being reorganised around smaller resident caches. And Micron demonstrated a 512 GB DDR5 RDIMM at 9,200 MT/s while CXMT moved an 11.95 nm G5 DRAM platform into mass production, so the second tier that makes offloading viable is getting denser and faster at once, on both sides of the export-control line.

One constraint named across multiple items and worth recording on this page rather than only on compute-economics. McKinsey identified custom silicon, power and talent as the core AI-infrastructure constraints, with a separate item reporting parallel talent gaps in fabs and data centres, and an AI Energy Management Alliance launching to target grid-responsive data centres. Power has moved into the same sentence as compute in industry analysis rather than into a footnote, and none of the memory-hierarchy results on this page price energy at all.

2026-09-24: flash becomes an inference tier

Three results in one day move inference bytes below DRAM on purpose: LM-CXD holds prefix KV caches on a chunk-aware CXL-SSD within 1.5x of DRAM (a stock CXL-SSD is no faster than NVMe; the block interface is the bottleneck), Disaggregated Quantization streams prefill weights from SSD (1.78x TTFT at 8K), and Memory Attention makes value construction an offloadable token-indexed table. A Google paper the same day found model loading is 55-70% of cold-start latency for small quantized models on serverless CPUs. Context: SemiAnalysis's ClusterMAX 3.0 opens with "GPU supply has gone to zero," and memory passed 50% of semiconductor revenue last quarter (09-23).

2026-09-25: host DRAM is not a free tier

The CPU shortage essay adds a constraint to the offload results of 09-24 (Memory Attention's CPU-resident value tables, LM-CXD's CXL-flash prefix cache). CPU servers need DRAM, and DRAM wafers are being reallocated to HBM, so host memory is getting scarcer and pricier at the same time papers treat it as the cheap tier. Offload designs should be costed against current DRAM prices, not historical ones.

2026-09-28: consumer-machine MoE serving as a bandwidth-split problem

FreeToken treats a desktop as a two-path memory system: an expert missing from GPU memory can be copied over PCIe or computed on the CPU where it lives, but both paths read the same host-memory bandwidth. It measures the split per machine (a 5090 desktop wants nearly all PCIe; an 8GB laptop wants mostly CPU) and reports DeepSeek-V4-Flash 284B at 22 tok/s on a 32GB GPU. On the datacenter side, Dylan Patel's unexplained "Rubin HBM despec" note (cluster page) is the other end of the same constraint: HBM supply, not FLOPs, is shaping hardware.

2026-09-29: bandwidth, not capacity, from both the model and the memory stack

Two independent sources make the same point on the same day. On the model side, SemiAnalysis on GLM-5.3 sparse attention shows that top-k sparse attention (each query reads only the most relevant 2,048 past tokens) cuts the bytes read per decode step but not the bytes that must be stored, because the lightning indexer that picks the top-k scans the full context. On the hardware side, a SemiAnalysis chart ("Stack height buys capacity, not bandwidth") shows Rubin Ultra moving from Rubin's 12-Hi HBM4 (36 GB per stack, 288 GB per GPU) to 8-Hi (24 GB per stack, 192 GB per GPU) with the same 2,048-bit interface and pin speed: 33% fewer DRAM dies, bandwidth unchanged, 50% more bandwidth per GB. Decode is bandwidth-bound, so taller stacks add cost without speed. This confirms 4-hi HBM wins (09-14) and gives a mechanism for the "Rubin HBM despec" remark logged on 09-28.

The capacity that leaves HBM lands in host DRAM, which is now a working KV tier. SGLang's HiSparse runs HBM as an LRU cache over host memory, and on B200 going from 8 to 16 concurrent requests moves prompt-cache reuse from 90.3% GPU / 6.0% host to 54.8% GPU / 40.3% host, with total hit rate above 95%. The standing caveat from 09-25 applies: host DRAM is getting scarcer as DRAM wafers shift to HBM, so the cheap tier is not getting cheaper.

2026-10-01: Micron's quarter, and NAND as the KV tier

Micron (10-01): revenue $54.2B, nearly 5x, gross margin 86.8%; 2027 HBM mostly locked at higher prices; NVHBM, a custom HBM4E for Nvidia. NAND revenue rose about 8x to $14B in 18 months, data-center SSDs about 71%, credited to agent context and KV demand. This is the demand-side echo of the 09-29 HiSparse entry (KV spilling to host DRAM). Same day: CoreWeave put Vera Rubin NVL72 in production, Cognition reporting 3.8x output tokens per rack vs GB200 NVL72.

2026-10-03: memory sets the price of compute

• JPMorgan sees HBM industry revenue up ~2.5x in 2027 from ~63% bit growth and ~54% higher prices; HBM rises from 19% to 31% of DRAM capacity by 2028, pulling supply from conventional DRAM. Micron closed FY2026 at $133.19B revenue and guided Q1 to $61.5B (SEC filing via AI Breakfast).

• Tesla cut AI5 RAM to 72GB of LPDDR5 and AI6 to 144GB to secure Optimus volume, arguing bandwidth, not capacity, limits inference. A practitioner estimate of 200GB per humanoid times billions of robots exceeds annual DRAM output many times over.

• Nvidia's 64GB DGX Spark at $4,999 is half the memory of the original for 25% more money.

• Volantis ($88M) claims optical interconnect that attaches 220 memory chips to one GPU against eight today; the memory-wall answer moving into photonics.

• 2026-10-04: a forecast puts memory revenue near $1.5T in 2027 and only ~$1.6T in 2028 while Micron says supply tightens through 2028; Cerebras argues on-wafer SRAM is a supply strategy, not just a bandwidth one (page).

• 2026-10-04: Ken Huang's essay frames inference memory as six markets with different cycles: HBM structurally needed but capex- and packaging-exposed, DRAM and NAND growing but cyclical, SRAM tied to logic design, CXL derived demand, processing-in-memory not yet a substitute. Decode's bandwidth bound plus KV placement policy decides which tier captures value (page).

2026-10-11: kilolayer NAND and Hyper-NA

• Process monitor (10-11). A kilolayer 3D NAND perspective argues scaling becomes a coupled charge-storage problem (lateral migration, cell coupling, read disturb, channel resistance). That cuts against KV-offload designs that treat NAND as a cheap reliable tier. ZEISS and ASML outline a >=0.75-NA EUV concept (~5 nm half-pitch). Apple cut iPhone 18 Pro component orders 15-20% citing AI-driven memory prices.

2026-10-16: HBM hybrid bonding pushed to 2029+

SemiAnalysis on ASML's D2W bonder: ASML patents describe a track-based die-to-wafer hybrid bonder (independent die holders, stations move to the die, ORION-style alignment). Logic adopts first; SemiAnalysis expects HBM not to adopt D2W hybrid bonding before 2029, if at all. Same day, COUNT (host-RAM n-gram tables) and KV4-RL/JSSA (fewer HBM reads) push the software side of the HBM squeeze.

2026-10-19: Nvidia standardizes the HBM base die

• Nvidia reportedly asked Samsung, SK hynix and Micron to build the base die of its next custom HBM on TSMC 3nm (The Information via Seoul Economic Daily). A common logic layer makes the DRAM stack the swappable part and TSMC wins every order (industry).
• HBM pricing: an investor forecast puts HBM near $35/GB by 2030, more than double today, with HBM4/4E at 48% of demand by 2027. Unconfirmed by primary sources.
• Capacity as a design lever: Meta's MI350X work reshapes models to use 1.6x the HBM of B200 for up to 1.5x training throughput (summary).

2026-10-24: the H200 premium and Micron's split

• Memory sets the rent: H200 (141 GB) rental up ~160% this year vs H100 (80 GB) ~40% on the same compute die (secondhand index). Clean evidence for the 10-03 thesis.
• Micron roadmap: HBM4 2.8 TB/s per stack; vertical HBM on top of the GPU by 2028 (target); 4F² DRAM abandoned for 3D DRAM, diverging from SK hynix and Samsung; G11 NAND 400+ layers in H2 2027; "AI-class NAND" for weights and KV at 25M-100M IOPS.
• 3D DRAM review (Nature Reviews EE): cell-stacked 3D DRAM inherits 3D NAND's HAR etch and conformal deposition problems with much tighter leakage and retention limits.