KVServe: Service-Aware KV Cache Compression for Disaggregated LLM Serving

Source: HuggingFace daily papers, 2026-05-22. arxiv: 2605.13734 Authors: Zedong Liu, Xinyang Ma (co-first), Dejun Luo, Hairui Zhao, Bing Lu, Wenjing Huang, Yida Gu, Xingchen Liu, Zheng Wei, Jinyang Liu, Dingwen Tao, Guangming Tan. Institute of Computing Technology, Chinese Academy of Sciences (ICT-CAS), University of Chinese Academy of Sciences, Shanghai Jiao Tong University.

TL;DR

Disaggregated LLM serving (PD separation, KV state disaggregation) turns the KV cache from internal GPU state into a network payload. For a Llama-3.1-70B model at 128K tokens, the KV cache is 39 GB and must be shipped across the network. Existing KV compression schemes (4-bit, 2-bit, mixed-precision quantization, lossless coding, Hadamard / affine transforms) are static configurations at runtime, but production workload mix, bandwidth, and SLO budgets vary over time. KVServe is the first service-aware adaptive KV-communication-compression framework. It (a) unifies KV compression into a modular strategy space, (b) introduces a Bayesian Profiling Engine that searches that space and distills a 3D Pareto candidate set, reducing offline search overhead by 50x, and (c) deploys a Service-Aware Online Controller that combines an analytical latency model with a lightweight bandit to select profiles under live constraints and correct offline-to-online mismatch. Integrated into vLLM. Up to 9.13x JCT (Job Completion Time) speedup in PD-separated serving and up to 32.8x TTFT (Time To First Token) reduction in KV-disaggregated serving.

Why this is Tier 1 core

KV cache is the user's top focus area. KVServe is the first paper this quarter to treat KV compression as a service-level (production-context-aware) optimization rather than a model-level fixed configuration. Three load-bearing ideas:

KV cache is now a network payload. Once disaggregated serving became common (PD separation between prefill and decode, or KV state offloaded to remote memory tiers for longer-context support), the KV cache moved from being a per-GPU resource to being communication infrastructure. KVServe centers this and frames KV compression as a network optimization problem, not just a memory-pressure optimization problem.
Bayesian Profiling Engine reduces offline search 50x. The configuration space (quantization scheme × precision × transform × layer × head) is too large for grid search. Bayesian profiling distills the Pareto frontier across compression ratio, quality degradation, and compression overhead.
Service-Aware Online Controller bridges offline-to-online mismatch. Offline profiling cannot predict every live workload. The bandit-augmented analytical latency model picks profiles under real-time constraints. This is the production-engineering glue that most academic KV-compression papers skip.

The architecture

                    ┌──────────────────────────────────┐
                    │   Service-Aware Online Controller │
                    │   • Analytical latency model      │
                    │   • Lightweight contextual bandit │
                    └──────────────────────────────────┘
                                    │
                                    │ profile selection
                                    ▼
┌──────────────────────────────────┐
│   3D Pareto Candidate Set         │
│   • Quality degradation           │
│   • Compression ratio             │
│   • Compression overhead          │
└──────────────────────────────────┘
                                    ▲
                                    │ Bayesian profiling
                                    │ (50x fewer trials)
                                    │
                                    ▼
┌──────────────────────────────────┐
│   Modular Strategy Space          │
│   • Quantization (4-bit, 2-bit,   │
│     mixed-precision)              │
│   • Lossless coding               │
│   • Hadamard / Affine transforms  │
│   • Cross-method recomposition    │
└──────────────────────────────────┘
                                    │
                                    ▼
                          ┌──────────────────┐
                          │      vLLM        │
                          │  PD-separated    │
                          │  KV-disaggregated│
                          └──────────────────┘

Numbers

Up to 9.13x JCT speedup in PD-separated serving.
Up to 32.8x TTFT reduction in KV-disaggregated serving.
50x reduction in offline search overhead via Bayesian profiling.
Llama-3.1-70B at 128K context produces a 39.06 GB KV cache, which sets the scale of the bandwidth problem.

Why this matters now

The disaggregated-serving wave is real. Anthropic's reported coding-tool revenue (Anthropic profitability post) depends on long-context Claude calls. NVIDIA's tokenomics push (today's social-stream morning) explicitly markets agent sandboxes as 50% faster on Vera vs CPUs and frames AI infrastructure spending at $3-4T by 2030 with token consumption growing 3,400%. KVServe's 9.13x JCT speedup is a direct lever on the unit economics that both stories depend on.

Gaps and open questions

The 9.13x JCT and 32.8x TTFT numbers are workload-conditional. The paper would benefit from a breakdown by model size, context length, and bandwidth budget.
KV-disaggregated TTFT improvement of 32.8x is enormous; the explanation likely lies in the prefill-side KV transfer being a dominant fraction of TTFT for long-context queries.
Whether the Service-Aware Online Controller's bandit converges under workload distribution shift is an open production concern.

Cross-references

RTPurbo / Full Attention Strikes Back (2026-05-22). orthogonal sparse-attention attack on the same long-context bottleneck.
Gated DeltaNet-2 (2026-05-22). replaces the KV cache entirely with linear recurrent state.
WorldKV (2026-05-22). training-free KV chunk retrieval and compression for autoregressive video.
SemiAnalysis GPU cluster goodput (2026-04-21). disaggregated-serving infrastructure context.

Source

Raw: raw/huggingface/2026-05-22-kvserve-service-aware-kv-cache-compression-for-communication.md.