Memory Hierarchy for AI (concept)

The tiered memory system that modern AI — and especially agentic AI — runs on. The 2026 thesis: the binding constraint on AI infrastructure has shifted from compute (FLOPS) to memory (where state lives and how fast it moves). This page synthesizes what the wiki knows about each tier and how the research it tracks maps onto the hardware.

Current State (as of 2026-09-02)

This page's founding thesis, that the binding constraint moved from compute to memory, now has its cleanest derivation and a trend claim that makes it worse over time. The Physics of LLM Inference, Chapter 1 (09-02) (Ken Huang, via starred Gmail) works the roofline model through language-model serving. Attainable throughput is min(P_peak, I × BW_mem) where operational intensity I is total arithmetic divided by bytes moved across the memory bus, and the ridge point I_ridge = P_peak / BW_mem is where the compute and bandwidth ceilings meet.

The numbers this page should carry.

• H100 SXM5 in FP8: 1,979 TFLOPS peak against 3.35 TB/s HBM3 puts the ridge at roughly 591 FLOP/byte. Single-stream decode sits at about 1.5 FLOP/byte, so it delivers under 0.3% of peak Tensor Core throughput. Not a software defect, the arithmetic of matrix-vector work at scale.
• The 70B decode proof, one line: 70 GB of FP8 weights stream in 20.9 ms; the roughly 140 GFLOPs they enable take 0.07 ms. Memory is 99.66% of step time.
• The two phases sit on opposite sides of the ridge. Prefill is matrix-matrix at 150 to 450 FLOP/byte and can saturate Tensor Cores, governing time-to-first-token. Decode is matrix-vector at 1 to 2 and streams every weight per token, governing inter-token latency. A fleet sized for the average of the two is wrong for both.
• Batching is the only escape and it is capped by this page's own subject matter. Intensity scales as I(B) ≈ (2 / S_weight) × B, so reaching the H100 ridge in FP8 would need about B = 296 concurrent decode streams. Production never runs that hot because KV cache memory and latency SLOs bind first.

The trend claim is the important one, and it is an argument this page has not made. Compute has scaled faster than HBM bandwidth across every accelerator generation, which pushes the ridge point rightward. So the batch size required to escape memory-bound decode rises with each new GPU. That inverts the intuition that better hardware makes serving optimization a transitional concern: each generation makes KV-cache capacity work more valuable, not less, because KV memory is precisely what prevents reaching the batch size the new silicon demands. That is the strongest available argument for why the scarce resource in 2027 is the same one it is today, and it belongs next to the HBM-allocation and packaging threads below.

It also supplies a criterion for grading everything on the efficiency side of the wiki. Any optimization that does not reduce bytes moved per token is fighting a 99.66% ratio uphill. Quantization, batching, speculative decoding with acceptance, and KV compression pass. FLOP-only reductions do not. Read that way, today's cross-model KV sharing (09-02), which cuts a Llama-70B-to-Qwen-7B handoff from 899ms to 138ms, is a different and better class of saving than it appears: it eliminates work in the compute-bound phase, the one place the accelerator runs near its ceiling, without touching the memory-bound phase that dominates steady state.

And a precision collision worth recording, which neither source notes. The compute-outpacing-bandwidth trend is the main reason the field keeps pushing to lower-precision training and serving. Lower precision means fewer exponent bits. TrainSDC (09-02) finds that backward-pass vulnerability to silent data corruption is governed by gradient exponent distributions rather than by computation location, at a mitigation cost of 1.65 to 6.76% runtime overhead. The hardware trend that makes quantization attractive is the same one that makes gradient-exponent corruption more dangerous, and no source in this wiki has priced that tradeoff jointly.

Caveat on sourcing: the free tier of this chapter covers the roofline, the prefill/decode split and the latency taxonomy. VRAM accounting, chunked prefill, continuous batching and PagedAttention, interconnect costs and cluster sizing are paywalled, so the ridge-point table for Blackwell-class parts cannot be verified against measured numbers from the public text.

Prior State (as of 2026-07-30)

A constraint that binds earlier than memory: the building. This page has argued since June that the binding constraint on AI infrastructure moved from compute to memory. SemiAnalysis's modular-construction analysis (07-30) adds a constraint that binds before either, and it has nothing to do with semiconductors. Trade labor cannot be produced on a two-year cycle. Electricians alone are 30 to 40% of total construction man-hours on a datacenter project, and SemiAnalysis's new Labor Model projects a shortage emerging in 2027, concentrated exactly where the buildout concentrates, in Texas and Ohio. The evidence that it already binds is a price rather than a forecast: Crusoe raised wages 30% to staff Abilene, which peaked above 9,000 workers. The analytically important move is treating reachable labor supply as a pool shared across states, so a project in one state reduces the labor available to its neighbors and site capacity stops being additive. That is a different failure mode from HBM allocation, which is a queue: labor is a substitution game between simultaneous projects.

The industry response is to move work into factories, and it works: modular compresses the build window by ~36%, or 7 to 9 months, at ~8% lower capex/MW, with 61GW+ of modular capacity tracked across 1,000+ sites and penetration projected past 30% of live capacity by end of 2028. The site layer cannot be modularized, since foundations are poured on the parcel, so this mitigates rather than removes the constraint. The supply-chain consequence belongs on this page: when on-site labor migrates into factory-built units, it reappears as vendor equipment content, and Vertiv's content per megawatt roughly doubles from ~$3.5M/MW to ~$7M/MW. That is the same substitution the labor model predicts, expressed as revenue, and it makes the modular OEM layer a structurally better position than the EPC layer.

Two financial datapoints landed the same day and read as this constraint priced. Meta's capex nearly doubled to $30 billion in one quarter, about half its revenue, while free cash flow fell 91% to just under $800 million and operating income fell 8%. Microsoft spent more in absolute terms at $35.8 billion, but that was 40% of revenue with opex up only 10%, so operating income rose 18%. Same category of spend, opposite cash discipline, and the market split accordingly. Alongside it, The Information reports "spend now, lease later" bridge financing is among the fastest-growing asks in infrastructure finance, with developers borrowing against projects that have no signed tenant to level land, run power studies, and queue for scarce gas turbines. Read together: developers racing a labor curve must commit capital before the lease exists, so the financing layer is absorbing the timing risk that modular construction is trying to remove physically. The forecasting correction for this page is concrete: 2027 US capacity should be modeled against electrician-hours in the concentrating states, not only against wafer starts and packaging capacity. Both constraints are real; this one binds first.

Current State (as of 2026-07-26)

The memory shortage crossed from a procurement problem into a national-security fight, and the person who escalated it is Tim Cook. Apple is lobbying the Trump administration for clearance to buy memory from China's CXMT and YMTC for devices sold outside the United States, arguing it eases the shortage and keeps iPhones cheaper (@MarioNawfal relay, 07-25, slot synthesis). Both firms are designated Chinese military companies by the Pentagon and YMTC is on the Entity List. These are the same suppliers Apple walked away from in 2022. Cook is simultaneously accusing Micron, the only large US memory maker, of gouging at 80% margins while calling the shortage a "100-year flood."

Those two claims cannot both be complaints. If the shortage genuinely is a hundred-year flood, then 80% margins are the price signal doing its job, not an abuse, and Cook's own framing is the strongest outside confirmation this page has of the structural-shortage thesis it has carried since June (Micron's ~3:1 HBM-to-DDR5 wafer trade ratio rising per generation, packaging as the bottleneck, top-end HBM allocation-driven into 2030). The wiki now has the same constraint visible from four independent vantage points inside two weeks: an accelerator vendor deleting a memory tier from a shipping module (AMD's MI455X LPDDR), a second signing a half-trillion-dollar arrangement to co-develop the next generation (NVIDIA and SK), a memory maker's stock up 213% year to date against NVIDIA's 10%, and now the largest consumer-hardware buyer on earth asking Washington to let it buy from an entity-listed supplier.

The forward-looking point for inference economics: DRAM and HBM contention are not separate markets in the ways that matter, because they compete for the same wafer starts and the same advanced packaging. Consumer device demand bidding against datacenter HBM is a mechanism by which phone volumes set a floor under inference cost, which is not how anyone models it. Watch whether the administration grants any carve-out, because a denial keeps that contention fully priced into HBM.

Current State (as of 2026-07-25)

Memory stops being a component and becomes the axis of competition, on the same day, in three forms. The 07-22 section below set a falsifiable watch on whether Meta would take the standard MI450 or the half-strength variant. Today's SemiAnalysis Advancing AI 2026 analysis resolves it: most of Meta's MI455 orders are the cut-down part (compute halved 8 XCDs → 4, HBM halved 12 stacks → 6, and downgraded 12-Hi → 8-Hi). The decision was made by RecSys infrastructure teams before Meta's TBD Lab existed, TBD has no interest in the resulting part, and SemiAnalysis says external customers will not want it either. AMD's Meta GenAI volume is the casualty exactly as the 07-22 prediction framed it, with one hedge: Zuckerberg has begun rapidly reworking Meta's infra org culture since that article.

Three concrete memory facts arrived with it. (1) The standard MI455X is a memory part first: 12 HBM4 stacks for 432GB at 23.3 TB/s, against Rubin's 8 stacks for 288GB. Fitting 3 cubes per base-die edge required growing the HBM-facing edge to 32mm (the MI300X AID's 29mm could not fit three). (2) NVIDIA answered with pin speed, not capacity, and moved a JEDEC spec to do it. It raised its HBM4 target to 10.7 Gbps, 40% above AMD's 7.6 Gbps and well past the original JEDEC HBM4 specification, purely to erase AMD's bandwidth lead. That lands Rubin at 22 TB/s from a 33% narrower bus, forces NVIDIA onto a much higher-quality bin, made suppliers rework HBM4, and delayed Rubin's own ramp. Competitive pressure between two accelerator vendors propagated backwards into a memory standard. (3) The shortage is now deleting product features. The up-to-1TB of direct-attached LPDDR per MI455X EAM module, present on earlier roadmaps as a second memory tier, has quietly disappeared, which SemiAnalysis reads as a consequence of tight supply. The second tier of accelerator-attached memory is a casualty of allocation.

The commercial counterpart landed the same 24 hours: NVIDIA announced a $500B partnership with SK Group, owner of SK hynix, explicitly to access more HBM and fill more datacenters, including joint HBM co-development plus a 2GW Vera Rubin DSX factory in Korea. One vendor signs a half-trillion-dollar supply arrangement with a memory maker; the other deletes a memory tier. That is what this page's structural-shortage thesis (Micron's ~3:1 HBM-to-DDR5 wafer trade ratio rising per generation, packaging bottleneck, top-end HBM allocation-driven into 2030) looks like when it reaches the balance sheet. Markets are pricing it accordingly: Micron is up 213% year to date and AMD 142%, against NVIDIA's 10% despite 83% expected revenue growth.

Rack-level memory movement is now a named cost line too. Helios connects 72 MI455X through 12 merchant Broadcom Tomahawk 6 switches (only 432 of each switch's 512 lanes usable, because 102.4T does not divide evenly into 72 GPUs, where NVIDIA sized the 28.8T NVSwitch for exactly 72 with zero waste). AMD's 200G SerDes cannot hold the copper backplane, so ~85% of Meta's scale-up links need Broadcom retimers, 550+ per rack. Backplane plus compute-tray content: $68,928 per rack, 10,368 differential copper pairs. The lesson the page has been accumulating since Vera Rubin: at rack scale, moving bytes is the cost, and co-design is the only lever on it.

Current State (as of 2026-07-22)

Two same-day hardware signals frame the co-design divide: NVIDIA ships a rack-co-designed Vera Rubin at 10x tokens/MW while Meta is documented wasting billions going off-menu. NVIDIA declared Vera Rubin in gigascale production at SIGGRAPH week (summary): CoreWeave's first measured silicon shows 10x more tokens per megawatt than Grace Blackwell NVL72 on DeepSeek-R1, framed explicitly as performance-per-watt and lowest token cost, plus Spectrum-6 (102.4 Tb/s Ethernet, 2x prior) and the Vera CPU (>2x faster, more concurrent agents). The headline is now efficiency, exactly the tokens-per-joule metric the energy-to-token position paper argued for, and it comes from extreme co-design across seven chips and five rack trays. The counterpoint, from SemiAnalysis the same day (summary), is Meta's infra org shipping the opposite: a $2.5B+ Rivos acquisition effectively dismantled, a custom GB200 (Ariel, one B200 per Grace in NVL36x2) with 14% higher TCO than the standard SKU that Meta's whole GB200 fleet ran on, and an upcoming plan to take a half MI450 (halved compute silicon and HBM stacks, HBM4 downgraded 12-Hi → 8-Hi) to chase a higher CPU:GPU ratio for RecSys embeddings. The MI450 matters for this page's memory thesis: the standard part has 12 HBM stacks and the largest CoWoS reticle on the market; gimping it to 8-Hi trades exactly the HBM capacity/bandwidth that GenAI training and inference are bottlenecked on. Falsifiable watch: whether Meta takes the standard MI450 (AMD gets its first Rubin-competitive GenAI volume) or the half-chip (AMD's Meta volume collapses, NVIDIA Rubin lock tightens). The meta-lesson both pieces teach: at the rack level, hardware/software co-design is the dividing line, and custom silicon without that discipline is a money furnace.

The tiers (fast/scarce → slow/abundant)

Core facts the wiki keeps returning to

• KV cache, not weights, is the dominant memory traffic at long context. Generation is memory-bandwidth-bound; as context grows the traffic shifts from model weights to KV cache. This is the hardware fact underneath every KV-cache software paper the wiki tracks. (Ken Huang memory survey, 06-07; UC Berkeley 2026 report.)
• Agentic workloads consume ~15x more tokens than traditional AI apps (NVIDIA), so they stress every tier at once, not just HBM.
• The memory shortage is structural. Micron's ~3:1 HBM-to-DDR5 wafer trade ratio (rising per HBM generation) + advanced-packaging bottleneck → tight/allocation-driven through 2026, selective relief 2027, broader 2028-2029, top-end HBM allocation-driven into 2030.
• Latency over throughput for agents. Interactive agents are judged on time-to-first-token and tail latency, which favors SRAM-first designs, disaggregated prefill/decode, and KV-aware tiering over raw batch throughput.

How the research maps onto the hardware

• KV-cache eviction / quantization / low-rank latent caches (VASE 06-03, VideoMLA/StateKV 06-01, LongAttnComp 06-02) are the software answer to "KV cache is the bottleneck tier" — they shrink what must live in HBM so cheaper tiers (LPDDR SOCAMM2, CXL, SSD) suffice. See kv-cache.
• Parametric context internalization (Code2LoRA, Video2LoRA 06-06) is the most aggressive move: bake context into weights so it never enters the KV cache at all. See parametric-context-internalization.
• Input-side compression (AdaCodec 06-06, SEAOTTER 06-07) cuts bytes/tokens before they reach the model — relief upstream of every memory tier.
• Compute rationing (CLEAR 06-05) is the demand-side dual: when HBM is scarce for five years, the serving stack must ration per-query compute by marginal utility, and (the open direction) ration KV-cache placement across tiers per-request.

Open problems / what to watch

• KV-aware tiering policies — deciding per-request which KV blocks live in HBM vs LPDDR/CXL/SSD. The hardware counterpart to CLEAR's per-query compute rationing; not yet a shipped serving-stack feature.
• Will PIM/CIM cross from research to product for attention/embedding/search, or stay niche behind analog-precision and compiler-maturity barriers?
• Does the SRAM-first bet (Groq) capture agentic inference as the market tilts toward low-latency, high-value tokens, or does ecosystem/software depth keep HBM-GPU dominant?

Sources

• Memory Technology for Agentic AI Workloads (Ken Huang, 06-07)
• Related hardware pages: gpu-kernels, SemiAnalysis 800VDC / cluster-goodput primers.
• Related efficiency: kv-cache, parametric-context-internalization

Supply answers, on the wrong side of the export-control line (2026-07-27)

CXMT listed in Shanghai and rose 472% on debut, opening at roughly 3.3 trillion yuan (about $487 billion) (summary, The Information). Discount the number heavily: a 472% first-day move in a market with capital controls, where the Semiconductor Newsletter's Week 30 issue reports state-affiliated capital supplying more than 90% of China private equity funding, is a policy signal rather than a price.

What survives the discount is that this is the first supply-side datapoint in a shortage this page has tracked entirely from the demand side. The prior four vantage points were all buyers scrambling for allocation: AMD deleting the LPDDR tier from MI455X, NVIDIA committing $500B with SK hynix's parent (07-25) which is locking supply through equity rather than purchase orders, Micron up 213% year to date against NVIDIA's 10%, and Tim Cook asking Washington for an entity-list carve-out to buy from CXMT and YMTC while simultaneously accusing Micron of gouging at 80% margins and calling the shortage a "100-year flood." An IPO of this size is capacity being funded.

The correction this forces to the page's model. The 07-26 digest predicted a denial of Apple's carve-out would keep memory contention fully priced into HBM, treating the carve-out as the channel by which CXMT capacity reaches the market. That conflated two separable things. CXMT can fund fabs domestically at a $487B valuation whether or not Apple is permitted to buy. The carve-out determines who CXMT sells to; it does not determine whether CXMT builds. Access and capacity have come apart, and the page should track them separately from here.

What actually matters next, in order: announced capex and fab timelines against the IPO proceeds, because wafer starts with dates are the only number that moves the constraint; whether CXMT is pursuing HBM or only conventional DRAM, since they compete for wafer starts but serve different markets and the reporting does not say; and the BIS determination on Apple, still the cleanest test of the allocation regime, now with the understanding that a denial slows distribution rather than construction.