From Raw Experience to Skill Consumption: a lifecycle study of model-generated agent skills

arXiv: 2605.23899 · HF Daily Papers: page · Date: 2026-05-25 Raw: farmer file

TL;DR

A utility-grounded evaluation framework that traces the full agent-skill lifecycle (experience generation, skill extraction, skill consumption) across 5 task domains. Two main findings: model-generated skills are useful on average but exhibit non-trivial negative transfer; and a model can be a strong extractor yet a weak consumer, or vice versa, with skill utility independent of model scale or baseline task strength. Translates the analysis into a concrete meta-skill that guides extraction toward features tied to actual utility, which consistently improves skill quality and reduces negative transfer.

The three-stage skill lifecycle (each stage is an independent design surface):

  ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐
  │ 1. Experience    │───►│ 2. Skill         │───►│ 3. Skill         │
  │    generation   │    │    extraction    │    │    consumption   │
  │ (which traj?)   │    │ (which model?)   │    │ (which model?)   │
  └──────────────────┘    └──────────────────┘    └──────────────────┘
                                    │                       │
                                    ▼                       ▼
                          extractor strength ───────decouples──── consumer strength
                          (scale does not predict either role)

  meta-skill guides extraction ──► features tied to actual utility
                              ──► reduces negative transfer

Key claims

The skill lifecycle has three stages: experience generation (what trajectories to run), skill extraction (how to distill a skill document), skill consumption (which agent runs the skill at inference). Each is independently variable.
Average effect is positive but variance is large; some skills hurt on some target models. Negative transfer is a real and frequent phenomenon, not a rare edge case.
Extractor strength and consumer strength decouple. Strong extractor + weak consumer is real. Weak extractor + strong consumer is real. Model scale alone does not predict either role.
The meta-skill identifies the features of a good skill (specificity of procedure, breadth of coverage, etc.) and guides extraction toward those features. Applied across domains, the meta-skill reduces negative transfer and lifts utility.

Relation to prior wiki content

This paper is a companion to SkillOpt released the same day; SkillOpt asks how to optimize a single skill; this paper asks whether skills generated by any current method are actually doing what's claimed. The two papers together close a loop: SkillOpt provides the optimizer, this paper provides the diagnostic framework that says when the optimizer's outputs will and won't transfer.

It extends the agent-skills thread that ran through May: Ctx2Skill (the 05-05 paper on context-driven self-evolving skills) made the case that skills should be self-evolving. MMSkills (the 05-18 paper on multimodal skill packages for visual agents) showed skills generalize across modality. This paper says: the field has been over-claiming. Skills do work, but negative transfer is real, and the lifecycle stage that matters most for utility is extraction, not generation or consumption.

The decoupling of extractor and consumer strength is the most actionable finding. It says the optimal pipeline is not "use the strongest model for everything"; it's "use the model that's strong as an extractor for extraction, and a different model that's strong as a consumer at runtime." This is a routing problem (which model for which stage of the skill lifecycle), and it connects directly to the wiki's LLM routing thread.

Research angle

The meta-skill is the practical contribution and the obvious lever for follow-up work. The current meta-skill is heuristic; replacing it with a learned policy over (extractor features, consumer features, domain features) is the next move. This would make skill extraction itself a routing problem.

The negative-transfer rate deserves a closer look. The paper reports it exists but doesn't characterize when. If there's a predictable subspace of skill-domain pairs where negative transfer happens, a pre-deployment filter could screen those out, which would dramatically improve the operational utility of model-generated skills.