AI research wiki
inference-efficiency · 2026-11-13 · Tier 1

SteerDistill: quantization-aware distillation for agents via closed-loop teacher steering

SteerDistill: quantizing LLM agents with a steering teacher

TL;DR

A 4-bit agent rarely fails on a single token. It fails by missing a task requirement or an environment condition, and then the whole trajectory drifts away from what the full-precision model would have done. Offline distillation data never visits those failure states, and the quantized student's own rollouts rarely contain a correct answer in them. SteerDistill fixes this with a loop: the full-precision teacher reads the student's failed turn, writes guidance, the student retries with that guidance, and the teacher refines the guidance from the execution feedback until the turn is verified correct. The student is then trained, without guidance, to match the steered student at corrected turns and the teacher everywhere else. Across five agent benchmarks and three models it beats the strongest quantized baseline by 1.67 to 2.18 points under W4A16 (4-bit weights, 16-bit activations), lands within one point of BF16, and beats QeRL on tau-squared-Bench by 1.84 points with 67% fewer training GPU hours.

Fix the turn where the 4-bit agent goes wrong, then forget the hint
The teacher never writes the answer. It steers the student until the student's own retry works.
flowchart LR
  S["W4A16 student<br/><small>rolls out task</small>"] -->|fails| T["BF16 teacher<br/><small>writes guidance</small>"]
  T --> R["Steered retry<br/><small>student plus hint</small>"]
  R --> V{"Verified?<br/><small>env feedback</small>"}
  V -->|no| T
  V -->|yes| D["Distill<br/><small>match steered turn</small>"]
  D --> F["Unassisted student<br/><small>within 1 pt of BF16</small>"]
  classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
  classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
  classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
  classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
  class S input
  class T,R core
  class V loop
  class D,F exit
  linkStyle 3 stroke:#f08c00,stroke-width:2px
Blue is the quantized student, purple is teacher and retry, amber is the verification loop, green is the distilled result.

Key findings

  • Failure diagnosis: quantized agents diverge in environment transitions (they overlook requirements), not in fluency.
  • Corrections are locally verified by execution, so the distillation target is a turn known to work in context.
  • +1.67 to +2.18 points over the best quantized baseline, within 1 point of BF16, shorter successful trajectories.
  • 67% fewer training GPU hours than QeRL (quantized RL) for a better tau-squared-Bench score.

Gaps

  • W4A16 only; no W4A4 or 3-bit result, where drift should be worse.
  • Requires a verifiable environment and a live teacher during training.
  • Three models, sizes not given in the abstract.

How it relates to prior wiki pages

  • Extends quantization: MACaQ (11-12, layer-local W4A4 at 2.3% runtime overhead) shrinks the model; SteerDistill repairs the behavior that shrinking breaks in multi-turn use. Together they suggest the quantization stack for agents is "compress, then behavior-distill".
  • Confirms the knowledge distillation pattern that on-policy, failure-state targets beat offline teacher traces (the TIP and OPRD line of work: most teacher tokens carry no signal; the useful ones sit where the student goes wrong).