QDA: Closed-Loop LLM Quantization Design with Agents (2611.23388)
TL;DR. MIT and NVIDIA hand post-training quantization (PTQ, compressing a trained model's weights and activations to fewer bits without retraining) to an LLM agent. The agent proposes a recipe, builds the quantized model, scores it, and revises the recipe over many rounds. Two tricks make the loop affordable. First, each candidate is scored in minutes by its output KL divergence from the full-precision model on held-out reasoning traces the agent never sees, and only promising ones get long generations to catch repetition loops. Second, a library of public PTQ methods is made mutually compatible and architecture-portable, so the agent can combine them without hand-porting. Across four reasoning models QDA beats the strongest public recipe by 0.9, 1.4 and 2.2 points at W4A16, W4A8 and W3A16, using 64 GPU-hours per search. On GLM-5.3-Flash (a 320B MoE) at NVFP4 W4A4 it recovers 98.3% of FP8 accuracy versus 96.9% for the best vendor checkpoint. All checkpoints run on existing serving kernels.
flowchart LR
M["BF16 model<br/><small>target to compress</small>"] --> A["Agent<br/><small>proposes a PTQ recipe</small>"]
L["PTQ library<br/><small>composable public methods</small>"] --> A
A --> B["Build<br/><small>quantize candidate</small>"]
B --> K{"KL proxy<br/><small>minutes, unseen traces</small>"}
K -->|promising| G["Long generations<br/><small>catch repetition loops</small>"]
K -->|weak| A
G --> O["Checkpoint<br/><small>runs on stock kernels</small>"]
classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
class M,L input
class A core
class B,K,G loop
class O exit
linkStyle 5 stroke:#f08c00,stroke-width:2px
linkStyle 6 stroke:#2f9e44,stroke-width:2px
Key findings
- +0.9 / +1.4 / +2.2 points over the best public recipe at W4A16 / W4A8 / W3A16, averaged over four reasoning models, 64 GPU-hours per search.
- 98.3% of FP8 accuracy for GLM-5.3-Flash at NVFP4 W4A4, against 96.9% for the best vendor checkpoint.
- No custom kernels: every recipe is constrained to formats existing serving stacks already run.
- The library, evaluator and harness are promised for release.
Relation to prior wiki pages
- Extends CompressionBench (11-16), which showed agents can ship compressed models under a budget (29 of 36 submissions passed). QDA is the first result where the agent beats expert recipes rather than just passing.
- Tension with the quantization noise floor (11-19): Cloudflare showed that serving-stack changes alone flip up to 0.92% of decisions. QDA's gains of 0.9 to 2.2 points are averaged accuracy; whether the smallest (W4A16) gain clears a measured floor is not shown.
- Same loop shape as self-improving harness work (gated self-edits, 11-19): a proposer plus a cheap, held-out grader. Here the grader (KL on unseen traces) is designed to avoid the overfitting the Resolution Hypothesis paper blames for stalled agent evolution.
Gaps
- KL as the selection signal is validated against final accuracy only on these four models; no correlation study across architectures.
- No decision-flip or noise-floor analysis.
Sources
- Paper · raw:
raw/huggingface/2026-11-19-qda-closed-loop-llm-quantization-design-with-agents.md - Social: @omarsar0; DAIR Academy pick (Nov 17).
Concept pages: quantization · agent harness engineering