Context Memorization: Attention-State Memory for Efficient Long-Context Generation
Source: HuggingFace Daily Papers 2026-05-20 · arXiv 2605.18226 · raw
TL;DR
Modern LLM applications increasingly use long conditioning prefixes to steer behavior at inference time. Prefix-augmented inference is effective but has two structural problems: the prefix's influence fades as generation continues, and attention computation over the prefix scales linearly with its length. Existing methods either keep the prefix in attention while compressing it, which still pays attention cost, or internalize it into model parameters via gradient training, which is expensive and ill-suited to prefix updates. The paper proposes attention-state memory, a training-free approach that externalizes the prefix as a lightweight lookup-based memory of precomputed attention states between prefix and query tokens. On ManyICLBench with LLaMA-3.1-8B, the method improves accuracy over in-context learning at 1K-8K memory budgets while reducing attention latency by 1.36x at 8K. On the NBA benchmark, it surpasses full-attention RAG using only 20% of its memory footprint.
Why it matters
Most existing prefix-compression methods (gisting, summary tokens, KV-cache eviction over the prefix) compress what is in the KV cache. This paper externalizes the prefix's contribution entirely as a precomputed lookup table over attention states between the prefix and a representative set of query tokens. The model never attends to the prefix at inference. The structural move is: the prefix has been replaced by its precomputed interactions, indexed for retrieval, kept as a tiny lookup-based memory rather than a tensor in the GPU.
Mechanism
For a fixed long prefix (a knowledge base, a long system prompt, a manual), precompute the attention states (Q-K-V interactions) between the prefix tokens and a representative query set. Store the result as a small lookup-based external memory. At inference, the model's queries hit the lookup memory, retrieve the precomputed attention states relevant to the query, and incorporate them without ever attending to the original prefix tokens. The result is constant-time prefix access (after precompute) instead of linear-time prefix attention.
Open questions and gaps
The "representative query set" used during precompute is the load-bearing design choice. How that set is selected, and how brittle the method is to query distributions outside what was anticipated, is unclear from the abstract. The 20% memory footprint on NBA is a single benchmark point; whether the discount holds on knowledge bases with high diversity or low redundancy is untested. Whether the method generalizes from 8B to frontier-scale models is the open scaling question.
Industrial implication
Long system prompts on agentic platforms (Cursor's rules, Claude Code's CLAUDE.md, RAG-over-corporate-knowledge) are exactly the regime this targets. The prefix is repeatedly identical across thousands of requests, the queries vary. Externalizing the prefix as a precomputed attention-state lookup table is the cleanest fit for that pattern. Expect a vLLM operator within 90 days if the precompute step is amortizable across the request volume.
Connections
- PEEK (today) carries an orientation prefix that adapts over time. Context Memorization carries a static prefix that gets queried efficiently. Different points on the same dimension: how to make a long context contribute without paying attention cost on every token.
- Make Each Token Count (2026-05-12) evicts cache entries to reduce attention dilution. Context Memorization avoids the cache for the prefix entirely. Both reach the conclusion that the full prefix-in-cache regime is suboptimal.
- EndPrompt (2026-05-19) extends context window via short-sequence training. Context Memorization keeps the context external and looks it up. The two are complementary, EndPrompt makes the model's intrinsic context cheap to extend, Context Memorization avoids needing the intrinsic context to grow at all for fixed-prefix workloads.