Agent Harness Engineering (Loop, Harness, Graph)

Concept page. The structured execution layer built around a foundation model that turns raw model capability into reliable long-horizon behavior. This page unifies the research thread and the practitioner thread, which have converged on the same claim from opposite directions: the harness, not the model, is now the primary object of design, evaluation, and cost.

Provenance note: this page merges published research (digest sources) with a large cluster of practitioner articles surfaced via saved reading in 2026-08. Specific saved items are tracked in the private curation index; the durable knowledge and paper sources are recorded here.

2026-09-19: the harness became a curriculum, and the token cost of "history" got a number

Google shipped a free two-hour course on Agent Harness Engineering, and its syllabus is this page's argument turned into a teaching order: prompt, then agents, then graphs, then loops inside a harness, then a harness that improves itself. That progression is not a vendor's taxonomy invention. It is the exact sequence this page assembled from the 2026-08 practitioner cluster, arriving eleven months later as curriculum. When a frontier lab teaches your concept page's structure as a course, the concept has stopped being an emerging practice and become a discipline. The practical consequence for this wiki is that "harness engineering" no longer needs to be glossed as a neologism in reader-facing writing.

The same day supplied the mechanism that makes self-improvement affordable. SKILL.state (09-19), a Google paper reported second-hand through the feed, replaces the append-only conversation transcript with an explicit mutable state object: the model sees the skill specification, the latest observation, and a structured state it updates deliberately, and nothing else. Reported effect: cumulative token usage down 16x at 100 execution turns, with a constant rather than growing prompt footprint, and context-poisoning failures eliminated because superseding a fact is now a write rather than an append that the model must learn to ignore.

Read that against the two harness results already on this page and a clean division of labour appears. The seven-model, three-harness comparison (09-17) found harness choice barely moves task success but significantly moves cost. The empirical harness-design study (09-18) found three of its four design findings were conditional on which model was running, so hand-tuned harnesses are misconfigured for the models they now serve. SKILL.state is the third data point and the most specific: one architectural decision inside the harness, touching no model, worth 16x on tokens. The page's thesis should now be stated in its strongest form. The harness is where agent cost is decided and the model is where agent capability is decided, and these are close to independent. Three results, three methods, no shared authors.

The self-evolving branch got two more entries the same day, both second-hand from the feed and both worth tracking rather than banking. NVIDIA published on self-evolving agent harnesses, amplified hard across the feed today. And NVIDIA's Agora, already on this page from 09-17 and re-amplified today, put 13 agents (Claude Code on Opus 4.7 and Codex on GPT-5.5) on one research problem for 12 days with no manager and no assigned tasks, coordinating only through a shared Git-backed project history: 1,703 contributions, 1,124 experiments, 203 hypotheses, 165 peer checks, none of which found a failure, closing 62% of the gap from a random model to a conventionally-trained GPT-2 without any training data or training runs. The unaudited detail to hold lightly is that zero of 165 verification steps caught anything, which is either a strong result about the protocol or a weak one about the verifiers.

A commercial signal, for the record. Harnesses began shipping as products this week rather than as internal scaffolding: AgentRun launched as a harness for high-volume repetitive knowledge work, and the framing circulating among builders, that vertical agents are harnesses, is the market pricing this page's thesis directly.

2026-09-17: the quality axis turns out to be flat, which tells this page what the harness is actually for

A seven-model, three-harness comparison (09-17), surfaced through the X home feed from Melissa Pan and amplified by Matei Zaharia, is the factorial measurement this page has been missing. Seven models run through Claude Code, Codex and Pi, with success and cost reported separately. Three findings: harness choice has little effect on task success rate and can significantly affect cost; a simple harness can be competitive; and the native harness is not always the best for its own model.

The separation is the finding, and it is more useful than "harnesses matter." If success rate is roughly harness-invariant across a reasonable set of engineered harnesses, then harness engineering is a pure cost-reduction exercise at a fixed quality bar, which is a far easier optimization problem than the joint one this page has implicitly been posing. It also means the right output of harness research is a cost curve, not a leaderboard.

"The native harness is not always the best" promotes a contested single result to a finding. SoL-Pi (09-11), which auto-searched harness configurations and cut tokens 45-49% while holding about 94% of task score, beat GPT-5.6 Sol's own native Codex harness on EdgeBench. One result is an anomaly. An automated search and an independent seven-model sweep agreeing is a finding: vendor-native harnesses are not tuned to the frontier of their own model's capability. The commercial reading matters more than the technical one. OpenAI's software-factory account (09-16) describes total dependence on one harness with the model fixed, wired so deeply into internal systems that a minor outage is noticed by colleagues before alerting catches it. That moat is switching cost, not harness quality, and this is the evidence.

"A simple harness can be competitive" is the awkward one, because it runs against the direction of this page's own recent literature. HarnessDev (09-03), Ecdysis (09-12) and COBRA-Skills (09-13) all spend search or training budget to produce a better harness. If the quality axis is flat, what that budget buys is the cost reduction, which is exactly consistent with SoL-Pi's own numbers and not at all how the work is usually framed. Stated plainly: harness search is a compiler optimization, not a capability unlock. That is a reframe this page should carry forward and apply to every future harness-search result.

It also closes the standing model-harness-pair gap on the routing page. Since 08-26 that page has recorded that the routable unit is the model-harness pair, with no proposal attached, because nobody had priced the pair's two coordinates separately. If success is harness-flat and cost is harness-sensitive, the decision factorizes: choose the model for quality, choose the harness for cost, and the interaction term is small. That is the first concrete answer to what a model-harness router should optimize.

What is missing, and it is the same thing missing everywhere on this page. No cost-per-success number, which PILOT in the Loop (08-28) argued is the only quantity that composes. No reported variance, so "little effect on average" could be hiding per-model reversals, which is precisely the structure a router needs. And coding benchmarks are the domain where harness design is most mature and therefore most likely to have converged, so flatness there is the weakest place to observe it. The productive next artifact is the second public model-by-harness cost matrix, after Terminal-Bench 3.0's on 08-13. This study has the data.

2026-09-15: the harness becomes org infrastructure, and the executive case for owning the loop arrives from Microsoft

Three industry signals, no new research, and together they move this page's commercial argument further than any paper this month.

The open-source multiplayer harness. Y Combinator open-sourced QM, the multi-agent harness it runs internally across accounting, legal, events and engineering: MIT licensed, self-hostable, giving each person and room its own scoped memory, files, credentials, permissions, crons, web apps and sandbox, with the underlying model swappable between Pi, OpenCode, Codex and Claude Code without rebuilding workflows. Reported uptake was 7,000 GitHub stars in three days. The significant part is the shape, not the stars: this is the first prominent open-source harness built by an organization for its own operations and then released, rather than a vendor product or a research artifact. It is also explicitly vendor-neutral at the model layer, which is the design this page predicted on 08-26.

That prediction was specific and it is now holding. The Ramp Inspect reading (08-26) concluded that harness structure is portable and harness evidence is not, and derived a market shape from it: vendors sell optimizable structure, companies build their own verification. QM is structure, released. It ships scoped permissions, sandboxes and credentials, and it does not ship domain verification, because verification requires access to systems an external harness cannot see. A harness released by an organization that deliberately omits the evidential layer is the cleanest confirmation of that boundary this page has recorded.

The executive framing. Ken Huang's reading of Satya Nadella's Stanford CS153 session puts the argument in firm-strategy terms: if models learn from data, what remains of a company whose value lived in tacit operational knowledge? Nadella's answer is not a bigger rented model but a company-owned "hill-climbing machine" built from private evaluations, RL environments, workflow traces, rewards and outcomes that stay inside the tenant. The consequence he draws is the sharp one: consume-only AI adoption can buy productivity and still lose retention, because the traces and rewards leave the firm. The proposed measurements are eval coverage, trace capture, model-swap survival, and cost per successful private outcome.

Model-swap survival is this page's portability test stated as a business metric, and that is a first. Everything this page has recorded on transfer (AI4AI strong-to-weak, AutoDesign across seven code-agent-model configurations, Meta-Harness across five held-out frontier models, WikiSkill across model families, and now RSIAgent's frozen environment memory (09-15) lifting both Kimi-K3 and GLM-5.3) has been a research result about whether a discovered artifact survives a model change. Nadella proposes it as the diagnostic for whether a company owns anything. Same quantity, two vocabularies, and the second one is what makes a CFO fund the first. The commercial stake is obvious and should be held: Microsoft sells the substrate on which a tenant-owned loop would be built.

Also on the plumbing. MCP now defines an extension for discovering and loading Agent Skills from MCP servers, where the server advertises available skills and metadata and the SKILL.md is loaded only when needed. That is lazy skill delivery standardized at the protocol layer, and it is the industrial form of what ALTK-Evolve (08-12) demonstrated as a research result: a small always-on core plus a per-task retrieved subset hit 89.3% task-goal completion at 263K tokens per task against ACE's 80.4% at 634K. The research showed selective delivery beats dumping the playbook; the protocol now makes selective delivery the default transport. What neither supplies is the selection policy, which remains the open question.

One provenance note that belongs on this page rather than in the digest. Meta-Harness (08-25), which this page already holds for its six-fold same-benchmark gap from harness design alone and its discovered context policies beating state-of-the-art agentic memory by 7.7 points at 4x fewer context tokens, resurfaced today through the reader's saved-reading trail, twenty days after ingest and roughly six months after the paper's own arXiv date. The saved trail is now confirming this page rather than extending it, which is the second clean instance of that pattern after the 09-05 Declarative Attention save, and it is a reasonable sign that the machine feed is ahead of the human one on this topic.

2026-09-12: the instruction ratchet finally gets an admission criterion, and the harness ships as a product on the same day

Ecdysis (09-12) (arXiv 2609.11677) is the first result on this page to propose a rule for what a harness is allowed to remember. Its diagnosis is that existing harness-evolution methods treat every observed failure as evidence about the harness, when a failure can equally reflect a model-specific deficiency. Patching the second is what the paper calls "unnecessary model-specific accommodation," and it is both why search-based harness evolution is slow (every candidate needs repeated agent runs) and why it overfits. The mechanism is batch-level cross-instance failure aggregation: pool failure evidence across many task instances, and repair only what recurs across unrelated tasks, on the reasoning that a one-task failure is probably the model while a cross-task failure is structural. On top sits Failure-Driven Collaborative Refinement, a multi-role diagnosis step that iterates on a harness modification specification rather than on code. Reported: 1.84x speedup in harness training and 18.56% better reasoning accuracy in the resulting harness.

This answers, four days later, the specific gap this page opened on 09-08. That entry recorded the "catastrophic remembering" study of 247,694 instructions across 1,867 GitHub repositories finding that CLAUDE.md and AGENTS.md files ratchet: a bug appears, someone appends "never do this again," and eventually nobody remembers why a rule exists so nobody dares delete it. The page named the concrete gap as a missing admission criterion, because nothing was measuring whether a stored instruction is still load-bearing. Ecdysis supplies one, and it applies it at write time rather than as later cleanup, which is the right place. Recurrence across independent tasks is a rule you can also run in reverse to justify deletion, which is the first credible brake this page has recorded on the ratchet.

It is in genuine tension with the co-evolution branch, and the tension is resolvable by one experiment. Co-Evolving Harnesses and Models (09-10) from Salesforce argues harness and model should be optimized jointly, and NVIDIA's SoL-Pi (09-11) and EvoSafeHarness (09-11) both ran automated search over harness space. Ecdysis says the joint framing is precisely where the overfitting comes from: if both sides move, you cannot separate a harness repair from a model accommodation. It deliberately freezes the model to keep the attribution clean. The crossover is a held-out model swap. Ecdysis predicts its harnesses transfer and co-evolved ones do not. Nobody has run it, and it is the sharpest open question on this page right now.

The economics framing is the one to carry, and it continues a thread from 09-09. Ben Lorica's "the loop is an asset" essay (09-09) made the deflationary argument in prose, that the first payoff of self-improvement is cheaper AI rather than smarter AI, and that the model is rented while the harness is capital you own. Ecdysis is that argument with a number on it: the headline is 1.84x off the training bill, with the accuracy gain secondary.

And the industry shipped the opposite bet on the same day, which is the sharpest research-versus-industry contrast this page has recorded. OpenAI released the Agents API, which is not another agent SDK but the Codex harness itself sold as a service: context continuation, compaction, tool calling, task decomposition, sub-agent coordination, sandboxed execution and recovery all move behind OpenAI's API, leaving the developer to specify task, model, tools and environment. Ken Huang's weekly roundup frames the resulting architecture exactly as this page would: session state and compaction on top, tools and sub-agents in the middle, the sandbox as the blast-radius boundary, and the model as the bottom component rather than the product. Practitioners immediately asked the right question, put most directly by @JoshARosen: should you build your own harness at all, given that frontier labs control both sides of the model/harness interface and can optimize each for the other? This page's answer, on Ecdysis's evidence, is that a harness whose repairs were validated by cross-task recurrence is the portable kind and a co-optimized vendor harness is by construction the non-portable kind. That is now a commercial question, not only a research one.

2026-09-09: the thesis gets its largest field confirmation, its most honest failure log, and a name for what the loop is worth

Three arrivals from three different worlds, and none of them set out to argue this page's claim. That is what makes today the strongest evidence day this page has had.

The field measurement. What LLM trading agents actually do in production (09-09) (arxiv 2609.05663) is a six-month, population-scale record across two live fleets: 3,505 user-funded vaults trading real ETH, plus 500-599 user-created agents trading Hyperliquid perpetuals, totalling 7.5M model invocations, ~300K onchain actions and 231,638 multi-tool turns producing 14,596 fills. Its first finding is this page's thesis stated in someone else's vocabulary: the operating layer determines behavior more than anything written in strategy text. A risk slider explains leverage at +0.425 per level, agent fixed effects absorb 60% of variance, and a leaderboard render boundary causally routes selection with a regression discontinuity of 1.75x at the top-3 cut. Where the UI stops drawing the ranked list changes what capital does. Almost all of this page's evidence is benchmark work by people trying to prove the harness matters; this is a production record with real money by people trying to make trading agents work, and it finds the prose the user wrote predicts nothing measurable while the slider position predicts leverage.

Its second load-bearing result is about models, not harnesses. A paired replay of frontier models across 416 captured production scenarios finds decision quality statistically indistinguishable, with the difference showing up as choice stability across model families, which is a harness-relevant property (how much the same input moves the output) rather than a capability one. That agrees with Iris (09-07), which found the inference-time context policy outweighs most reported model differences.

The failure log. Cursor's self-driving codebases writeup (09-09) describes a harness that ran continuously for a week with thousands of agents making the vast majority of commits to a browser-engine research project, and it is mostly an account of what broke. The shared-coordination-file design failed hardest and most specifically: agents held locks too long, forgot to release them, attempted illegal lock transitions, and did not model the significance of holding a lock, with more prompting failing to install that model. Contention was severe enough that 20 agents ran at the throughput of 1 to 3. An explicit wait-on-another-agent tool went unused. This is the strongest negative result on shared mutable coordination state this wiki holds, and it directly contradicts the blackboard pattern that most multi-agent frameworks on the multi-agent systems page still ship.

The subtler Cursor finding is behavioral and belongs on this page permanently: without role structure, no agent took on big complex tasks. They avoided contention and conflict, choosing smaller and safer changes over responsibility for the project. Absent explicit ownership, a population of capable agents converges on timidity. The fix was roles: a planner producing the approach, a single executor who solely owns delivery, workers giving linear throughput scaling, and an independent judge deciding whether it finished and whether to iterate. The residual failure was rigidity and bottlenecking on the slowest worker.

And Cursor's worker contract is this page's "transcript is the wrong carrier" thesis, discovered independently under production pressure and in its cheapest possible form: workers are unaware of the larger system, communicate with no other planner or worker, work on their own copy of the repo, and write one handoff when done. Set that beside the five results from 09-07 (ContextPipe planning the prompt rather than appending for 31% fewer tokens, SKILL.state replacing history with structured state for O(T) to O(1) prompt growth, Harness-of-Harness passing an evidence bundle, Iris on context policy, and the Handoff Tax finding a raw transcript across a model boundary is actively harmful) and the count is six independent designs converging on the same rule.

A fourth axis for what the loop carries. Procedural Graphs (09-09) (arxiv 2609.09153) stores (procedure, relation, procedure) triplets answering what-to-do, the way a knowledge graph stores what-is. At each step it localizes the agent's active node and turns the surrounding subgraph into step-level guidance that biases the next action without dictating it, and an LLM refiner edits the graph by contrasting failed against successful trajectories, committing only edits that pass held-out validation while retaining rejected edits to discourage retry. Every other proposal on this page concerns what the agent knows (context, state, evidence, paged KV). This one concerns what the agent is supposed to do next, which is a separate axis that composes with all of them.

The economics. Ben Lorica's "the loop is an asset" essay (09-09) gives this page the ownership argument the research literature never makes: the model is rented and swappable, the harness is capital you own, test and roll back. It cites a system moving 40% to 62% pass rate without touching the model, and a reorganization of inter-agent information exchange cutting inference cost by up to 60%. Its deflationary claim about self-improvement is the one to carry: the first payoff is cheaper AI, not smarter AI. And its operating rule is the discipline the Procedural Graph implements: the more autonomy the loop has, the more of the evaluation must sit outside its reach, because a loop becomes very good at optimizing a proxy (one reported case improved its score by learning to avoid the hallucination detector rather than by hallucinating less).

The tension worth logging. Lorica argues the accumulated production trace is the compounding asset for an application team. NeoHorse-1 (09-09) argues the same traces, specifically the routing harness's per-turn record of predicted capability demand, selected tier and outcome, are training data that lifts a 4B model most of the way to a 9B base. Two layers of the stack are claiming the same exhaust as capital, and on the day both arrived, the industry's dominant story was a dispute over whether one party's private tool sessions improved another party's model. That is not a coincidence to be smoothed over; it is the commercial shape of the harness thesis.

2026-09-08: the "transcript is the wrong carrier" thesis meets its first serious dissent, and the instruction file gets a name for its failure mode

This page crossed a pattern threshold yesterday: five results in two days saying append-the-transcript is the default every serious 2026 harness result is designed to escape. Today a sixth result arrives that keeps the whole transcript and beats the ones that discard it.

KVMem (09-08) (arxiv 2609.04852) preserves an agent's overflowed workspace history as paged KV state across GPU memory, host RAM and NVMe, indexes it with lightweight indexes built in the model's own attention space, and materializes a query-dependent execution view bounded by the native context window. On DeepSWE long-context with Qwen3.8-27B it lifts task success 43.8% → 48.4% against compaction-only context management, and on a 24GB RTX 5090 laptop virtualizes a 1M-token workspace against a 256K native window at ~50 tokens/s.

Its diagnosis is a direct challenge to this page's operative claim. Every one of yesterday's five throws information away on purpose and argues the throwing-away is the gain: ContextPipe plans the prompt like a relational query instead of appending (31% fewer total tokens, 23% fewer calls), SKILL.state replaces conversation history with mutable structured state (prompt growth O(T) → O(1), intermediate reasoning discarded the moment it yields a validated JSON-patch state update), Harness-of-Harness passes an evidence bundle rather than a transcript between iterations, Iris (09-07) shows the inference-time context policy outweighs most reported model differences, and the Handoff Tax (09-07) finds passing a raw transcript across an escalation boundary recovers under half the quality gap and costs over twice starting fresh, while dropping the trajectory and passing only the code edits lifts recovery to 64-84%. KVMem says the transcript was never too long, it was stored in the wrong representation: text is a lossy and expensive-to-reload encoding of state the model already computed, and if you keep the KV then length stops being the problem.

The honest reading is that the two camps attack different failure modes, and the experiment separating them has not been run. SKILL.state and ContextPipe attack attention dilution, where the model does worse on a cluttered prompt regardless of cost. KVMem attacks capacity and recompute, where the model would do better with the history if it could afford to hold it. A direct KVMem-versus-SKILL.state comparison on one long-horizon benchmark settles which was binding, and it is cheap. If structured state still wins with KV paging available, dilution was the constraint and the summarizers were right. If KVMem wins, the field spent a year building summarizers to solve a memory-management problem. This page should carry that as its sharpest open question, ahead of the byte-identical-prefix-caching crossover it has had open since 09-06.

A second result today says the compaction camp has a specific, measurable cost. Revision propagation (09-08) (arxiv 2609.03254, NTT) benchmarks whether a model can propagate a local edit request to every dependent part of a conversationally generated artifact, where the dependency graph exists only in the conversation and nowhere in the artifact. A JSON file holding start_date and museum_date records no trace that the second was defined as the first plus three days. That relation is precisely what a summarizer or a structured-state projection discards, because it is conversational reasoning rather than artifact state. So there is now a benchmark that should show the harness-compaction cost directly, and running it with and without compaction is one experiment. The paper's own finding is separately useful: parallel sampling with selection beats sequential self-revision and beats more reasoning tokens, for 2.2 to 9.7 points, which suggests dependency discovery is a recall problem rather than a reasoning-depth one.

And the instruction file finally gets a name for what everyone has observed. A paper circulating today as Why Does CLAUDE.md Keep Growing? Catastrophic Remembering in Agentic Coding, surfaced through the X feed (@sleepy0x13, @rohanpaul_ai), tracks the lifecycle of 247,694 instructions across 1,867 GitHub repositories and reports that CLAUDE.md and AGENTS.md files exhibit a ratchet: instructions are cheap to add and increasingly hard to remove. A bug appears, someone appends "never do this again"; a format breaks, another line; a test fails, another. Over time nobody remembers why a given rule exists, so nobody dares delete it. Rohan Paul's framing, "catastrophic remembering," is the right inversion of catastrophic forgetting and this page should adopt it.

It is the same problem as the transcript problem, one level up. The transcript grows within a session and the instruction file grows across sessions, and both grow monotonically because appending is trivially safe and pruning requires knowing what is load-bearing. The concrete gap the ratchet names is a missing admission criterion: nothing was measuring whether a stored instruction is still load-bearing.

And the answer arrived the same afternoon, from a paper that does not mention CLAUDE.md at all. SkillGLoW (arXiv 2609.02217, NUS, surfaced via @Xudong07452910) starts from exactly the two failure modes the ratchet paper measured, and states them more precisely than the ratchet paper does. Agents that self-improve by writing textual skills keep them either as one global document, which "collapses into generic discipline," or as a flat pool of per-task entries, which "inflates and its entries stay bound to the instance that wrote them." Most prior evidence came from domains where tasks are structurally similar, which hid the problem. Their diagnosis is that the missing unit of reuse is the solving procedure shared by a cluster of related tasks, neither the individual instruction nor the general principle.

The mechanism has three parts and each one maps onto something this page has been asking for. Local skills written from a task's own execution are aggregated into procedural families and compressed into de-instantiated global priors, so what persists is the transferable procedure. The instance detail is regenerated per task rather than stored, which is the direct fix for pool inflation and is the same move SKILL.state makes within a session, applied across sessions. And a commit gate admits a prior only when real execution shows it does not degrade the deployed library, which is precisely the load-bearing test the ratchet lacks: not "is this instruction still true" but "does keeping it measurably hurt." Across four benchmarks (mathematical reasoning, terminal automation, software repair, embodied control) and three models, the priors gain 17.2 points on hard tasks over the no-skill baseline, with positive gains in all 12 continual-improvement cells, while the library is 3.6x smaller than per-task storage. On unseen ALFWorld tasks, success goes 73.9% → 83.9%.

So the pair is the finding, and it is a clean measurement-plus-mechanism day. The ratchet paper supplies the disease across 247,694 real instructions in 1,867 public repositories; SkillGLoW supplies a treatment with a validated admission gate and a 3.6x compression figure. Neither cites the other and they were posted hours apart. The composition is obvious and cheap: run SkillGLoW's commit gate against the ratchet paper's corpus and see how much of a real CLAUDE.md survives it. That is the experiment this page should want most, because both artifacts are public and the dataset is 1,867 repositories.

The nearest prior entry is WikiSkill (09-07), whose three layers on different write rules plus a validating gate that rolls back bad updates was until today the only entry treating a persistent instruction store as needing garbage collection rather than accumulation. SkillGLoW is the second, and the first to publish a library-size reduction. The open question both leave is which abstraction level to store at, and SkillGLoW's own framing is the right way to state it: too specific and a skill does not transfer, too abstract and it does not guide action.

2026-09-07: the harness-bound thesis arrives as a leaderboard-integrity problem, and the model-switch boundary turns out to be a harness decision

This page's founding claim is that the harness, not the model, is the primary object of design, evaluation and cost. Today an open frontier system says the "evaluation" half out loud, and it is an indictment. Iris (09-07) (arxiv 2609.04304) releases two search agents at 35B-A3B and 397B-A17B that reach the strongest open-source results in their parameter classes on BrowseComp, BrowseComp-ZH, DeepSearchQA and Humanity's Last Exam, from a single flat ReAct agent with no sub-agents and no test-time verification. The paper's methodological statement is the contribution: inference-time context management is worth more on these benchmarks than most reported differences between systems, so they evaluate every benchmark both with and without it, holding tool set, context limit and judge fixed.

Put beside ContextPipe (09-06), which cut an agent's total token volume 31% and its call count 23% by planning the prompt like a relational query instead of appending to it, the conclusion is uncomfortable and specific: the search-agent leaderboard has been substantially measuring harness quality and reporting it as model quality. This page should now carry a stronger version of its thesis: any agent benchmark that does not fix or disclose its context-management policy is not comparing models. Iris is the first entry to publish both columns.

Iris also supplies a counterweight to the day's other agent-orchestration signal, and the honest reading is that neither is general. The DAIR.AI weekly digest surfaced Harness-of-Harness, which wraps whatever coding harness you already run and organizes its executions into repeated planning, coding and testing increments with a Project Planner, a Developer and an independent QA Tester passing an evidence bundle rather than a transcript, reporting a 52.25% average relative gain over the standalone harnesses after three iterations, peaking at 82.86%, across GameCraft-Bench, FrontierSWE and ProgramBench with three harness-and-model pairs. Iris reaches the open-source search frontier with one flat loop and explicitly no sub-agents. The orchestration premium depends on whether the task decomposes: multi-day software projects do, single-answer multi-hop retrieval does not. That is a cleaner rule than "add a graph" and it is testable.

The model-switch boundary is a harness decision and this page has been treating it as a routing one. The Handoff Tax (09-07), from AWS Agentic AI across 58,000 agent runs, 2 million API calls and 36 billion tokens on SWE-bench Verified, finds that escalating from a cheap to a frontier model with the full history recovers less than half the quality gap (47% Claude, 36% GPT) and for Claude costs more than twice starting fresh on the frontier model, while dropping the trajectory and passing only the code edits on disk lifts recovery to 64% and 84%. Downshifting reverses the rule: the cheap model needs the strong model's blueprint and takes 1.6 to 2x more steps without it. What crosses a model boundary is exactly the kind of decision with checkable structure that the 09-02 reconciliation below assigns to the harness rather than the model, and it is a concrete third instance of that rule.

Two more items from the same weekly digest belong on this page, and both are about what the loop carries between steps. SKILL.state replaces a growing conversation history with an explicit mutable execution state, so at each step the model sees only the immutable skill specification, the current structured state and the latest observation. Prompt size goes from O(T) in the number of steps to O(1), intermediate reasoning is discarded the moment it produces a validated state update expressed as a JSON patch, and accuracy improves while cumulative token consumption drops, with baseline runtimes hitting quadratic context growth on long-horizon warehouse tasks where SKILL.state stays bounded. WikiSkill turns agent runs into the maintenance signal for a skill library, with three layers on different write rules (immutable execution traces, a compounding wiki of structured patterns, and a reversible conditional skill layer) plus a gating step that validates candidate updates before they ship and rolls back bad ones; evolved skills transfer across models, and smaller models running with them sometimes outperform bigger models running without.

Those three plus Iris and ContextPipe make five results in two days saying the same thing about the loop's state: the transcript is the wrong carrier. ContextPipe plans the prompt instead of appending, SKILL.state replaces the transcript with a structured state, Harness-of-Harness passes an evidence bundle rather than a transcript between iterations, Iris says the inference-time context policy outweighs the model gap, and the Handoff Tax says passing a raw transcript across a model boundary is actively harmful in one direction. The pattern threshold is comfortably crossed. This page's operative claim should now be that append-the-transcript is the default that every serious 2026 harness result is designed to escape, which also names the open problem: none of the five publishes a crossover against the byte-identical-prefix caching discipline that DeepSeek's Harness v0.1 commits to, and that gap has been open on this page since 09-06.

2026-09-02: the line this page said nobody had drawn, drawn as a design commitment

The open question from 08-31 is answered, and the answer is one sentence long. That entry recorded a genuine contradiction in this page's own evidence. AI4AI at Test-Time (08-13), where a builder model's synthesized harness lifted a weaker target from 0.49 to 0.91 on four Theory-of-Mind benchmarks with weights untouched, and Spark-to-Paper (08-13), whose integrity stack raised fabrication detection from 14% to 92%, both concluded that the harness wins by taking decisions away from the model. ContextPilot (08-31) did the opposite and worked, handing the model more discretion over its own context and training that discretion with localized RL. This page said both cannot be the general rule, proposed a reconciliation nobody had stated (the harness should own decisions with checkable correctness, the model should own genuine judgement calls), and called it "the right next position paper."

Harness-of-Harness (09-02) (arxiv 2609.01481) states it as design commitment four: "constrain verifiable outputs rather than prescribing agent workflows." The harness specifies what the artifact must satisfy; how to get there is the model's. That is the proposed line, arriving as an engineering commitment rather than an argued position, and it comes with the transfer evidence the claim needed: a 52.25% average relative gain (82.86% maximum) over the corresponding standalone harness after three iterations, holding across three different harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, Pi with MiniMax-M3) on GameCraft-Bench, FrontierSWE and ProgramBench. Transfer across pairs is what separates a design principle from a tuned configuration.

HoH is also the first meta-harness in this page's timeline. It operates on existing coding-agent harnesses, organizing their executions into iterative planning-coding-testing loops. That makes it the natural successor to HarnessOpt-Bench (08-07), which made "can a model optimize its own harness" measurable, and to AutoDesign (08-14), where one discovered harness lifted seven model-agent configurations. The scope is larger than either: not optimizing a harness for one task, but running other harnesses across a multi-day horizon while the artifact under construction grows. Its 70-plus iteration deployment, producing a first-person shooter with storyline, implemented mechanics, playable experience, visuals and audio, is the longest autonomous software-development run recorded in this wiki.

Its third commitment operationalizes the evidential face of the runtime-contract argument. Agent Safety Should Be a Runtime Contract (08-13) argued the harness carries a preventive face (sandboxes, permission gates, trajectory monitors) and an evidential face: no task-complete claim without checkable proof. HoH's "separate implementation-time testing from independent evaluation" is that as architecture, and it is the direct structural fix for the measurement failure this page flagged on 08-26, where Microsoft's Thinkingbox (08-25) showed the strongest model falling from 65.36% pass@1 to 25.25% pass^20 on stateful workflows while a harness paper published the next day still reported single-attempt scores. One caution: the abstract does not say who authors the independent evaluator. If the same model writes it and the implementation tests, the separation is procedural rather than real.

Open problem 0b is now untouched for an eighth consecutive harness result, and HoH is the worst instance so far. No harness paper since 08-13 has published the cost of its own mechanism. Wrapping three planning-coding-testing iterations around an existing harness multiplies that harness's token spend by a factor the paper does not state, and seventy iterations multiplies it enormously. A 52.25% relative capability gain for an unknown multiple of cost is not a comparable number, and base rates are absent from the abstract, so a large relative gain on a low base reads identically here to a large absolute one. When the field's central pitch is that the harness is where the cost lever lives, this is the finding rather than an oversight. The downstream consequence is concrete: llm-routing.md has wanted to route over model-harness pairs for two months and is blocked partly because harnesses publish no comparable cost-per-success. A harness that runs other harnesses makes that accounting strictly harder.

A field datapoint the same day, on what happens with no evidential layer at all. Rick Brewster shipped a from-scratch, clean-room reimplementation of Direct2D inside Paint.NET for running under WINE, roughly 180,000 lines written by Claude against about 700,000 for the rest of a twenty-year project, and describes it as "vibe coded," not thoroughly reviewed, "trust me bro style," with the explicit admission "I cannot possibly review 180,000 lines of code." The failure he had to catch by hand is instructive because it is exactly the class an automated check would find: for a while the model was not doing the COM equivalent of AddRef() on reference-counted objects. His only harness was his own attention, plus an experimental /wine flag gating the code from default use. That gate is the whole preventive face of the runtime contract, implemented as a feature toggle, and it is what the evidential face is supposed to replace. (Simon Willison)

2026-08-31: the week the harness thesis got runnable artifacts, and one measurement that invalidates a process

Five systems described in one weekly roundup, plus one paper on HuggingFace, and they converge on a design move this page had not named: stop making the model re-read its own history, and give it typed state it can run code against. That is a token-cost reduction before it is a capability gain, which puts it squarely on this page's Tier 1 intersection.

• Prime Agent (Prime Intellect, open source) persists histories, memories, skills, prompts and subagent specifications across trajectories rather than resetting everything but the files on disk, and gives the model a persistent IPython session so it filters and re-derives state as code instead of re-reading a transcript. Holding the model class fixed, ARC-AGI-3 Best@1 moves from 30% to 95.5%, and it matches or beats native harnesses on long-context coding, GPU kernel generation and autonomous nanoGPT speedruns.
• Scroll (Alibaba) removes the memory schema entirely: an append-only event log plus a sandboxed persistent Python kernel, with tool outputs binding to typed variables across model calls and only explicitly-printed projections crossing into the working view. Evicted spans stay recoverable through an eviction index of compact landmarks tied to exact event-log addresses. 94.8% on LongMemEval_S, 73.1% on BEAM_10M (5.1 points over the best published memory system), 86.7% on LOCA_256K.
• JIT-Agent makes the harness the model's output, synthesized per task under a fixed four-module protocol (memory, planning, action protocol, tool orchestration) and patched mid-run. DeepSeek-V4-Flash passes GPT-5.6 on DeepSearchQA by 9.1 and OdysseyBench by 4.3; GLM-5.2 gains up to 20.2. Generated harnesses are competitive with OpenCode and Claude Code.
• ContextPilot (Tencent, on HuggingFace) trains the model to manage its own context with RL, adding planning, long-term memory and soft context offloading (evict but keep retrievable) to the usual search/delete/summarize toolset.

The convergence to record: Scroll and ContextPilot independently arrived at "evict but do not destroy" in the same week, one via an eviction index of event-log landmarks and one via a soft-offload tool. Neither cites the other. That is two groups rejecting the same thing, namely irreversible compaction, which is the standard behaviour of every context-window manager in production today.

Skill Lift is the counter-signal, and it is the most immediately actionable number on this page. NVIDIA measured whether the structural scanner that enterprises gate shared skill libraries on predicts anything about skill performance. Across 145 real skills from internal and public catalogs, scan score correlates with LLM-judge quality at a Spearman rho of 0.14. Passing the gate tells you a skill is well formatted and essentially nothing else. The replacement is a paired-run design: same task, same model, same sandbox, same scorer, run once with the skill loaded and once without, and measure the delta in what the agent completed. 947 paired cases from 58 production skills across four harnesses, with trajectories normalized into a shared Agent Trajectory Interchange Format so a skill's lift in Claude Code reads against its lift in Cursor. That interchange format is the quietly important artifact, because it is the first thing this page has seen that makes harness comparison portable rather than anecdotal.

Where ContextPilot cuts against this page's sharpest claim, and it should be held in tension rather than resolved. This page records, from AI4AI at Test-Time (08-13) and Spark-to-Paper (08-13), that the harness wins by taking decisions away from the model: offloading unstable reasoning into deterministic code, routing per question type, enforcing formats, all of which remove model discretion rather than adding model effort. ContextPilot does the opposite. It hands the model more discretion over its own context and then trains that discretion with localized RL. Both cannot be the general rule. The likely reconciliation, which no paper states: the harness should own decisions with checkable correctness, and the model should own decisions that are genuinely judgement calls, such as what in this context still matters. Nobody has drawn that line explicitly and it is the right next position paper.

New open problem 7: two competing answers to context cost, never compared. ALTK-Evolve (08-12) makes per-step delivery volume an externally-tuned parameter and got DeepSeek-V3.2 from 80.4% to 89.3% task-goal completion at 263K tokens per task against 634K. ContextPilot makes it a learned self-applied policy. The external knob is auditable and easier to reason about; the learned policy should adapt better across task variety. No head-to-head under matched token budgets exists. Whoever runs it settles a design question a serving stack will otherwise settle by accident.

Open problem 0b remains untouched for a seventh consecutive harness result. None of Prime Agent, Scroll, JIT-Agent or ContextPilot publishes the cost of the mechanism itself. ContextPilot is the clearest instance: branch sampling at every detected critical edit multiplies rollout cost by the branching factor, and the paper reports how compact the resulting context is without reporting what training it cost. When the entire pitch is cost, that omission is the finding.

The vocabulary (glossed once)

Three terms get mixed together constantly. They are layers, not synonyms:

• Harness — the whole scaffold around the model: context constructor, memory substrate, skill/tool routing, orchestration loop, and verification/governance. "Everything that isn't the weights."
• Loop engineering — designing the automated cycle (generate → critique → revise → act → verify) so the system prompts the model repeatedly, instead of a human prompting it by hand. The slogan: "the human should not be the loop."
• Graph engineering — the next layer up: wiring multiple agents, memory, and state into a structured graph. The practitioner framing: "loops make agents think; graphs make agents remember and coordinate."

Model-bound vs harness-bound is the key diagnostic: some tasks are limited by the model, others by the harness. Smarter models do not make the harness matter less — they raise the ceiling the harness has to reach.

The core claim, and why it holds

Same model, same task, same prompt: move the work between two harnesses and cost-per-success swings 5x to 30x (omarsar0's preregistered benchmark, arXiv 2608.01347, surfaced 08-13). That single measurement is the empirical spine of the whole field: if the harness moves cost by more than an order of magnitude while the model is held fixed, then the harness is where the optimization budget belongs.

The same paper found a subtler result: prompt wording changes where effort goes, not just how much. "Try multiple approaches" inflates reasoning 2.4x–7.4x and spawns ~3 discarded solution branches for one implemented fix, with no gain in success rate. "Maximum certainty" instead triggers repeated verification (extra test runs, tool calls, turns, latency, context growth). The prompt is a cost dial most people turn blindly.

Tier 1 intersection (efficiency): this is a cost-optimization result, not just an agent-design result. Harness choice is a lever on serving cost, token spend, and latency — the same axis as KV-cache and inference work.

How the idea evolved (the wiki's timeline)

1. Code as Agent Harness (arXiv 2605.18747, summary 05-23) — the first strong articulation: code is the harness because it is executable, persistent, version-controlled, and compositional. Natural-language plans and raw tool-call sequences are none of these. Karpathy's Autoresearch (program.md + train.py + git log) is the canonical instance.
2. Scaling the Harness (arXiv 2605.26112, summary 05-27) — the position paper: system scaling, not model scaling, is the next bottleneck. Names the six harness components (foundation model, context constructor, memory substrate, skill-routing layer, orchestration loop, verification/governance) and calls for harness-level benchmarks measuring trajectory quality, memory hygiene, and context efficiency instead of one-shot task success.
3. HarnessOpt-Bench (08-07, per the 08-08 weekly) — made "can a model optimize its own harness" measurable. The benchmark for the self-improving-harness idea.
4. The cost measurement (omarsar0, arXiv 2608.01347, 08-13) — put the 5x–30x number on harness choice. Turned a design intuition into an efficiency fact.
5. LongHorizon-Harness (arXiv 2608.01964, Alibaba, 08-13) — the state-management mechanism: a Manage-Execute-Audit (MEA) loop keeps task state outside the execution context and updates it only on environment-verified facts, so wrong self-assessments stop propagating across long tasks. The rigorous answer to "how do you engineer the loop."
6. The capability measurement (AI4AI at Test-Time, arXiv 2608.12307, 08-13) — the twin of item 4. A strong builder model writes an inference-time harness for a weaker target model, refining it against a 5% validation split, and average target performance on four Theory-of-Mind benchmarks goes 0.49 → 0.91 with the target's weights never touched. Harness choice moves accuracy by roughly 2x on a fixed model, the way it moves cost by 5x–30x.
7. The safety statement (Agent Safety Should Be a Runtime Contract, arXiv 2608.11274, 08-13) — the harness is also where safety belongs, as a preventive face (sandboxes, permission gates, trajectory monitors) plus an evidential face (no task-complete claim without checkable proof: test runs, log captures, file diffs, citation grounding). Backed by a title-level audit of all 28,560 NeurIPS/ICML/ICLR 2023–2025 papers showing an 8x–12x imbalance between training-time and deployment-time safety publication.

Running through all seven: capability lives in the harness, not just the weights, and the harness deserves to be designed, benchmarked, cost-optimized, and governed as a first-class object.

What AI4AI adds to the mechanism story

The most useful part is not the headline number but the mechanism analysis, because it says what a harness is actually for. The gains do not come from the target reasoning more or sampling more widely, which is what a better prompt would produce. They come from three moves: offloading unstable reasoning steps into deterministic code, routing per question type, and strict answer-format enforcement. All three remove model discretion rather than adding model effort.

That is the same conclusion Spark-to-Paper (08-13) reached independently in a different subfield: its first design principle is separating model-based judgment from deterministic operations that can be executed and checked, and its ablation shows fabrication detection rising from 14% for a single-pass draft to 92% with the full integrity stack. Two papers, same board, different tasks, one conclusion: the harness wins by taking decisions away from the model. That is a sharper statement of the concept than "scaffolding helps," and it is falsifiable: a harness that improves results purely by giving the model more room should not show this signature.

The practitioner convergence (2026-08, surfaced via saved reading)

Independently of the papers, a wave of practitioner writing landed on the identical thesis, with sharper operational language and proof points:

• From prompting to loop engineering. The consensus statement (attributed to senior engineers including OpenClaw's creator): "you shouldn't be prompting coding agents anymore; design the loop that prompts them." Proof point cited: one engineer shipping 259 merged PRs in a month, the AI writing every one, never opening an editor.
• The four loop types + the evaluator (Claude-Code-specific loop-engineering guides), and "the four bills nobody warns you about" — the cost gotchas of running unattended loops. This is the practitioner rediscovery of Scaling-the-Harness's "context efficiency" concern.
• Loop → Graph engineering. The claimed frontier shift: once loops work, wire agents into graphs for memory and multi-agent coordination. Framed as "graphs make agents remember."
• Model-bound vs harness-bound entered practitioner vocabulary directly (an article by that name), matching the research diagnostic.
• Market signal: Forward Deployed Engineers reportedly command ~$1M/year and Anthropic $750k+ for harness/loop skill specifically. The industry is pricing the harness, not model access.

That research and practitioners reached the same frame in the same window is the strongest possible confirmation that this is a real structural shift, not a hype cycle.

Relation to neighbouring concepts

• Agent memory — the memory substrate is one of the six harness components. The "RAG is a dead end / memory should be native to the model" turn (08-13) and the write→manage→read memory-engineering formalization are the memory layer of graph engineering.
• Self-evolving agents — self-improving loops (agents that rewrite their own harness/skills) are harness engineering closing the loop on itself; HarnessOpt-Bench measures it.
• Context engineering — "delete 80% of the system prompt" (Anthropic/Cloudflare/Karpathy, 08-13) is the token-optimization face: the harness win is often in what you stop keeping in context.

Industry has started pricing the harness directly (2026-08-13)

The market signal moved from salaries to product this week. xAI shipped Grok 4.6 and stated that Grok 4.6 optimized the Grok Build harness for itself (@aksheyd), and Hugging Face's Elie Bakouch, reading the model card, noted the model is state of the art on an internal "inferenceEval" that measures optimization of xAI's own chat inference (@eliebakouch). Two of the capabilities a frontier lab chose to headline are the model improving its own scaffold and its own serving path.

That is the industrial instance of the research claim, and it lands the same week AI4AI measures strong-to-weak harness transfer and AutoWorldModel-Bench measures agents improving an external artifact under an unspecified objective in 63 of 64 sessions. The gap worth watching: xAI reports a self-optimized harness as a capability headline and publishes no cost-per-success number, which is precisely the measurement omarsar0's benchmark made the field's spine.

The first public model-by-harness cost table (Terminal-Bench 3.0, 2026-08-13)

Terminal-Bench 3.0's leaderboard is the closest public artifact to the measurement this page keeps asking for, because every row is a model-harness pair reported with token count and dollar cost alongside accuracy (screenshot):

Three readings. Token spend does not track accuracy: Sonnet 5 in Claude Code burns 17.9B tokens for 14.6% while Fable 5 in the same harness reaches 34.1% on 3.6B, a 5x token difference in the opposite direction from the score. Ranking by cost-per-point reorders the table, moving Grok 4.6 from fourth to second and dropping Fable 5 from third to seventh, which is the argument for reporting dollars-per-completed-task as the primary agentic metric. And the grid is still not an ablation: no model appears under two harnesses in the top ten, so the leaderboard shows harness variation is present without isolating it. Holding one model fixed across mini-SWE-agent, Codex, Claude Code, Grok Build and Cursor CLI remains the missing experiment, and it is now cheap enough that somebody should just run it.

2026-08-14: the harness stops being written and starts being optimized

One day after the cost and capability measurements landed, three things arrived together that change this page's central claim from the harness is the object of design to the harness is the object of optimization.

DarwinX (arXiv 2608.07545, Salesforce) makes harness search a population problem. Model frozen, one evolution loop, about +17 points average across four benchmarks. Terminal-Bench 2.1 goes +7.7 to 83.2% on a matched base and 84.7% on a stronger one; TerminalWorld's held-out split hits 68.3%; WebArena-Infinity real-task pass@1 goes 43.5% → 93.0% audit-clean; and a Terminal-Bench harness transfers unchanged to SWE-bench Verified. Its diagnosis of prior work is the part that matters here: every self-improving harness in this wiki's timeline above is a single lineage, which is path-dependent and lets a local win silently regress another task. DarwinX's three fixes are a preserve-and-extend contract (admit a variant only if it extends coverage without regressing), an archive of alternative lineages for later recombination, and one shared edit interface for failure-derived, teacher-derived and self-derived evidence. Fitness comes from each benchmark's own verifier, so no gold solutions and no human picks winners.

This resolves the isolation experiment 08-13's Looking Ahead predicted for a 60-day window. The Terminal-Bench 3.0 table above shows harness variation exists but never holds a model fixed across two harnesses; DarwinX does exactly that on 2.1 and gets +7.7 points from the scaffold alone. Open problem 1 below is now half-answered: the swing is real at the frontier, though still not predictable in advance.

AutoDesign (arXiv 2608.13560) supplies the missing price tag. A meta-harness optimizer guides a code agent to rewrite its own harness from rollout feedback, aligned to human design priors. On the new PosterBench (100 papers, five disciplines) it scores 78.32, beating the commercial Claude Design by 7.45 points, and the learned DesignHarness dropped into seven other code-agent-model configurations lifts the average from 54.99 to 67.39. The number this page has been asking for since 08-13: 253 tool calls, 11 editing turns, 40 minutes, under $3, fully autonomous. Harness optimization costs single-digit dollars per rollout.

Note what the pair does jointly. DarwinX optimizes against hard verifiers (a test either passes or does not) and reports no cost. AutoDesign optimizes against a soft verifier (a design rubric plus human preference) and reports full cost. Neither tests the other's regime, and whether meta-optimization degrades when fitness is soft is now the sharpest open question in this area.

DeepSeek Harness v0.1 makes prefix stability a billing line. DeepSeek open-sourced its agent harness under MIT (built on the Cordis meta-framework, where models, tools, skills, sessions, sandboxes, filesystems, loops, orchestration and UI are all plugins) on the same day it raised API prices with cache-hit tokens repriced to roughly 6x, plus a new peak/off-peak split at 50% below peak. Hugging Face's Elie Bakouch, reading the code, found the harness's organizing commitment is first-class KV-cache-aware design: previously written history is never altered; a change is expressed by appending a statement describing the modification rather than editing the prefix, because editing invalidates every cached token downstream. Bakouch also estimates at least ~20% of the harness's own commits came from Codex worktrees.

That is this page's Tier 1 efficiency claim arriving from the provider side. The 5x–30x cost-per-success swing was measured by varying harnesses at fixed prices; DeepSeek varied the prices at fixed harness design and reached the same conclusion. Append-only history is now a cost property, not a latency nicety, and it is cheap enough to implement that convergence across harnesses is likely within a quarter.

Ken Huang's Harness Engineering supplies the governance half the research thread lacks. His periodization (model as product → wrapper as product → harness as product boundary) is the industry statement of Scaling the Harness's thesis, and his ten pattern families are a superset of that paper's six components with three additions research has not touched at all: identity, data governance, and runtime steering. His "hill climbing" family states the rule DarwinX independently implements: improvement is only safe when it is measured, bounded, and reversible. A security architect and a research team converged on the same admission rule within one day of each other.

Updated state of knowledge. The harness is measured on cost (5x–30x swing), measured on capability (0.49 → 0.91 at the small-model tier, +7.7 points at the frontier), optimizable by population search and by meta-optimization, transferable across tasks, verifiers, base models and agent frameworks, priced by providers through cache-hit economics, and now has a governance taxonomy. Five papers and one product release in six days. The gap that remains is the same one: nobody has put harness optimization and fine-tuning on a single cost axis for equal capability gain.

The open problem this creates. If harnesses are portable, optimizable artifacts that lift arbitrary models, then the routable unit is the model-harness pair, as A²E (08-11) argued. LLM Routing now records this as a standing gap with three supporting results and zero proposals: no production router routes over harnesses, and no paper proposes it.

2026-08-16: duration enters the measurement, and the cost axis spans three orders of magnitude

Two results extend this page along axes it had no data on, and a third supplies the small-model version of the substitution.

Measuring Autonomous AI Research (Prime Intellect) adds duration. 153 autonomous runs across 18 frontier models on the nanoGPT optimizer speedrun, up to 8.7 days per run on 8xH200s, all scratchpads and logs public. Every leaderboard row is a model-harness-effort triple: Fable 5 under claude-code at high effort closes 81.7% of the human-record gap at 2,726 steps; Opus 5 under claude-code max reaches 2,920 (53.6%); Kimi K3 appears twice, at 2,930 under prime-agent and 2,974 under kimi-code. That 44-step spread between two harnesses running the same model is roughly the size of the entire gap between Opus 5 and Kimi K3, which is the harness-isolation comparison 08-13's Looking Ahead asked for, arriving from an unexpected direction. Bakouch's variance note is the number to keep: one run in the same setting has a ~50-step spread after 24 hours, so single-run autonomous-research claims sit inside the noise. The earlier Prime Intellect experiment's diagnosis stands unretracted: agents are strong at optimizer search, hyperparameter sweeps and stacking known methods, and weak at generating new ideas, needing upstream human records to keep climbing.

Specification-first convergence adds a stopping rule for tasks with no oracle. An agent dismantled a core lifetime invariant across a 717,725-line production TypeScript codebase, 189 files touched, with no human code review and no pre-existing test for the target behaviour. The protocol: the agent writes a formal specification, 14 refinement cycles audit the specification against the source, the specification is frozen, then 17 verification cycles audit the code against it. 201 defects corrected across 31 audit passes before a human ran the program. Convergence criterion: two consecutive verification passes returning zero findings. Three days, USD 2,430. The freeze is the direct countermeasure to open problem 3 below, since the auditor is checking against a document it may no longer edit.

Set the two published cost points side by side and the axis is visible for the first time. AutoDesign (08-14) published $3 per rollout for 253 tool calls in 40 minutes. Specification-first publishes $2,430 for three days on one architectural refactor. Three orders of magnitude apart, both fully autonomous, both reported honestly. "What does agentic work cost" has no single answer, and any claim that does not name the task class is uninterpretable.

SKILLER supplies the cheap tier. A strong model as actor and critic, the small-model agent system as environment, every RL signal propagated as natural language, producing executor-specific skills. Qwen3.5-9B gains 4.3 to 20.4 points, Qwen3.5-4B gains 1.8 to 13.3, and single-skill SkillsBench performance matches strong closed models. The paper's stated motivation is that skill-driven harnesses like Codex and OpenClaw are prohibitively expensive at closed-model prices, which is this page's cost thesis stated as a research problem rather than an observation. Parity is claimed on single-skill tasks only; composition is untested.

Updated state of knowledge. The harness is measured on cost (5x-30x swing; $3 to $2,430 per task depending on class), on capability (0.49 → 0.91 at the small tier, +7.7 to +17 points at the frontier, 81.7% of a human record over 8.7 days), and now on duration and variance. It is optimizable by population search and meta-optimization, transferable across tasks and base models, priced by providers through cache-hit economics, governed by a pattern taxonomy, and manufacturable for small open models via natural-language RL. What is still missing is unchanged: nobody has put harness work and fine-tuning on one cost axis for equal capability gain.

2026-08-25: the harness gets a cost curve, a reference implementation, and a 35B model at the frontier band

Three HuggingFace papers on one day, all treating the harness as the primary object, plus a fourth on the Kurate board. This is the densest single day this page has recorded.

Apodex 1.1 is the model-scale version of this page's thesis, and the day's top paper at 145 upvotes. It names working capability (sustained, verifiable progress toward a real-world objective) as a category distinct from reasoning, constituted by state maintenance, failure recovery, and verifiable delivery. It develops that along two axes that are explicitly not parameter count: Environment Scaling (expanding the diversity and verifiability of executable file, search and code environments) and Agentic Coordination Scaling (training agents to decompose, delegate in parallel, integrate asynchronous results, replan). System side is a shared execution harness plus AgentOS maintaining task state and provenance across tools and agents, which makes provenance a first-class harness component exactly as Scaling the Harness (05-27) proposed. Result: a 35B model and a locally deployable 35B Mini reach the leading performance band on finance, research, math, coding and search.

The parameter count next to the benchmark is the whole point. Every prior data point on this page lifted a fixed model with a better harness. Apodex trains model and environment together and lands at the frontier band from 35B, which is the strongest evidence here that on verifiable professional work the marginal value of parameters is below the marginal value of environment and coordination.

But it entangles this page's open problem 0 rather than resolving it. Harness optimization versus fine-tuning at matched cost is still unrun, and Apodex makes it harder to answer, not easier: environment-trained weights and an inference-time harness are both in the deliverable and the paper does not separate them. The ablation that would settle it (freeze the base and apply only harness plus AgentOS; then run the environment-trained weights under a plain ReAct loop) is the single most valuable missing experiment in the paper. Also unpriced: coordination scaling spends tokens on parallel delegation and replanning, and against the AlphaSense finding (08-14) that task cost and token price can point opposite ways, a 35B model winning on quality could still lose on dollars per completed task. And "leading performance band" without per-benchmark deltas is not a checkable claim for a result whose entire content is a comparison.

Task-CoEvolve is the first result on this page about the efficiency of the harness search loop rather than the quality of its output. DarwinX (08-14) and AutoDesign (08-14) established that harness search works, by population selection and by meta-optimization respectively. Neither made it affordable. Task-CoEvolve observes that a validation set's informativeness decays as the harness improves: eventually the easy tasks are universally solved, the hard ones universally failed, and only a shrinking frontier band still separates candidates. It concentrates evaluation there by variance-weighted sampling over past outcomes, with the sampling distribution adapting as the harness evolves, then recovers comparable full-set scores from the biased sample with an importance-weighted estimator. That second piece is what makes the first usable: without it, each iteration measures a different subset and the search loop loses the ability to compare iteration k to k+1. Result: 80% fewer evaluations, matching full-set search's final quality, on online text classification and Terminal-Bench 2.1 (the same benchmark DarwinX and AutoDesign used, so it is comparable).

This moves one side of open problem 0 by a large constant factor. AutoDesign published $3 per rollout, DarwinX published no evolution budget, no distillation paper published a comparable per-point cost. The comparison still has not been run, but harness search just got roughly five times cheaper, which changes the expected answer. The likely hidden failure mode: importance weighting is unbiased but its variance grows as sampling probabilities shrink, which is what aggressive concentration produces, so a noisy full-set estimate could cause the search to accept a worse harness. The paper does not discuss estimator variance, and only two task types back the 80% figure, which depends on the difficulty distribution having easy and impossible tails to skip.

Prime Agent supplies the reference implementation and the cleanest one-line statement of why this page exists: it prevents harness failures from becoming model failures. Persistent IPython REPL under a Recursive Language Model abstraction, a Continual Harness preserving histories, memories, skills, prompts and subagent specs across trajectories, plus agent-to-agent coordination. ARC-AGI-3 RHAE Best@1 from 30% to 95.5%. Open source.

Fourth on the Kurate cs.AI board: Agent Lightning v1.0: Towards Harnessed Agentic RL (He, Zhang, Zhou, Yang, Kang et al., ai_rating 7.0, the highest-rated item on that board). Four papers in one week putting the word harness in the thesis.

Task-CoEvolve also extends the "schedule beats operator" pattern this page's neighbours have tracked since 08-12. LycheeMemory V2 (08-14) was recorded as the third instance after ICBQ block order (08-12) and ReOrder-OPD prompt order (08-13). Task-CoEvolve is the same shape: the harness-rewrite operator is unchanged and the entire gain comes from changing what gets measured when. And its disagreement-weighted sampling is the same statistical idea as R2-OPD (08-25), which suppresses distillation reward where a teacher ranking and a progress ranking disagree. Harness evaluation and distillation, one day, no shared authors, one principle.

The counter-signal, and it should be read seriously. Kurate cs.AI #14 is On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification (Ye, Li, Pruksachatkun, Zhang, Wu), arguing self-improvement gains are sensitive to variance, task arrival order, and underspecified objectives. Set that against Bakouch's 08-16 note that a single run in a fixed setting has a ~50-step spread after 24 hours, and against ARC-AGI-3 being near-saturated by two systems in one week (Prime Agent 95.5%, NVIDIA AVO 100%). The honest reading: harness capability is real and its measured magnitude is not yet trustworthy. A near-saturated benchmark cannot discriminate the thing everyone is now optimizing.

Meta-Harness closed a loop this page has been open on since it was created: the saved-reading thread got cited by the published literature. Task-CoEvolve's related work opens with the six-fold same-benchmark gap from harness design alone, which is the Stanford and MIT Meta-Harness result surfaced through saved reading on 08-18. The optimizer design is two refusals of the standard recipe. Full diagnostic trace access: instead of a lossy scalar reward plus a short summary, an agentic proposer gets unrestricted filesystem access to raw execution logs and source code across all past iterations, up to 10M trace tokens, inspected with grep and cat. Causal failure analysis in code space: isolate the confounded regression, trace the downstream error back to the early context decision that caused it, rewrite the responsible executable Python function. The attribution argument is the substantive one: a scalar says a rollout was bad, it cannot say that turn 40 failed because turn 6 evicted the wrong file, and recovering that link requires the log to exist and be readable. Results: discovered context policies beat state-of-the-art agentic memory systems by 7.7 points at 4x fewer context tokens, converging 10x faster, and a single discovered math-retrieval harness added 4.7 points on 200 IMO-level problems across five held-out frontier models zero-shot. Unaccounted: the search cost. 10M trace tokens per proposal is expensive and "10x faster convergence" counts iterations, which is the wrong unit when the per-iteration bill rose.

That transfer result crosses this page's own three-paper threshold. AI4AI at Test-Time (08-13) took a weaker target from 0.49 to 0.91 across four Theory-of-Mind benchmarks with the target frozen. AutoDesign (08-14) added 12.4 points across seven code-agent-model configurations. Meta-Harness adds five held-out frontier models on olympiad problems. Three independent results, twelve days, same structural claim: a discovered harness is a portable artifact with standalone value, not per-model tuning residue. That is a stronger statement than "harness choice matters," and it is what makes harness-as-product coherent.

Industry reported the same cost number in the same week, which is the confirmation that matters most. OpenAI's Codex as a platform release (08-19) states outright that the harness, not the chat interface, is the primary reusable component, and reports that harness-level retained reasoning plus context compaction moved GPT-5.6 Sol's ARC-AGI-3 score from 13.3% to 38.3% while cutting token consumption sixfold. Meta-Harness reports 4x fewer context tokens at +7.7 points. A research group and a frontier vendor independently landing on 4x-6x token savings from the same architectural layer, in the same week, on partly the same benchmark, is the strongest cost evidence this page holds. Ken Huang's reading of the open codex-rs repository extracts a token-budget subsystem with proactive compaction hooks as one of nine supporting components, making token optimization an explicit architectural element rather than a tuning afterthought.

Thinkingbox (08-25) aims the counter-signal at the metric rather than the method, and it is the sharper version of the Fragility critique. Microsoft's 507 policy-conditioned stateful business workflows, evaluated against backend terminal state with executable checks that reject extra effects as well as wrong and missing ones, put the strongest model at 65.36% pass@1 and 25.25% pass^20. Worse, many failures terminate cleanly having made valid state-changing tool calls, so response-level and tool-call-level signals are not proxies for completion. Set that beside Prime Agent's 95.5% Best@1 the same day: both results can be true, and the honest joint reading is that harness engineering has demonstrably raised the ceiling of what an agent can do while leaving open how often it does it. Nobody has published a pass^k curve for a harness-optimized agent. That is now the single most conspicuous missing measurement on this page, and it is cheap to produce.

Updated state of knowledge. The harness is measured on cost (5x-30x swing; $3 to $2,430 per task by class; 4x-6x token reduction from context management, confirmed by research and vendor independently) and now on the cost of finding one (80% reducible). It is measured on capability (0.49 to 0.91 at the small tier, +7.7 to +17 at the frontier, 30% to 95.5% on ARC-AGI-3, 81.7% of a human record over 8.7 days) and now at model scale, with 35B reaching the frontier band when environment and coordination are scaled instead of parameters. It is optimizable by population search, meta-optimization, and now cheaply; transferable across tasks, verifiers, base models and frameworks; priced through cache-hit economics; governed by a pattern taxonomy; manufacturable for small open models via natural-language RL; and it has an open-source reference implementation. What is still missing is unchanged and now conspicuous: nobody has put harness work and fine-tuning on one cost axis for equal capability gain, and the benchmark everyone is climbing is saturating.

2026-08-26: open problem 2 gets solved, the admission rule becomes a law, and industry stops buying harnesses

Two HuggingFace papers, one practitioner survey, and one production case study, all on the same layer. The day's contribution is less about new capability than about three of this page's own open items moving.

Recuris (arXiv 2608.24876) is the first system that actually composes the three pieces open problem 2 asks for, so that problem can now be marked mostly resolved. The problem read: LongHorizon-Harness's state-outside-context, HarnessOpt's self-optimization, and graph-based memory routing are separate advances, and no harness manages state externally and self-optimizes and routes through memory end-to-end. Recuris does all three. Working Memory holds verified task progress, which is LongHorizon-Harness's (arXiv 2608.01964, Alibaba, 08-13) Manage-Execute-Audit contribution of keeping state outside the execution context and updating it only on environment-verified facts. Experiential Memory is retrieved by that working state rather than by the initial instruction or the full transcript, which is the routing layer and the fix for stale skill retrieval on long tasks. A fixed Meta-Agent turns localized failure evidence into validation-gated updates to Skill Memory, which is the self-optimization layer, and the Meta-Agent never edits itself, which is what bounds the recursion. Results: improvement in 35 of 37 completed model-benchmark pairs across four long-horizon benchmarks and ten models; +17.8 to GPT-5.6 Sol and +15.6 to Claude Opus 5 on tau-bench, carrying Opus 5 to 87.9%; +16.6 and +13.5 on Qwen3.6-27B/35B on SkillFlow; +32.2 on the longest tasks, with common long-horizon failures down up to 80%.

The composition is not additive, and that is the finding. The externally verified working state is what makes the self-optimization precise: because state is checked against the environment, a failure attributes to a specific memory component, and that attribution is what lets the Meta-Agent write a local patch instead of a global one. This is the same credit-assignment problem Meta-Harness (08-25) solved by brute force, giving an agentic proposer unrestricted filesystem access to up to 10M tokens of raw execution trace so it could trace a turn-40 failure back to the turn-6 context decision that caused it. Recuris gets the same link structurally, from a maintained state, at a fraction of the token cost. Two answers to attribution in harness optimization, eleven days apart: read everything, or maintain the right thing. The second should dominate on cost and nobody has compared them, which is a cheap and valuable experiment.

AutoSaddler (arXiv 2608.23041, Microsoft with POSTECH/KAIST/SUSTech) makes the admission rule an empirical result rather than a design principle. It formulates harness improvement as offline learning from mini-batches of failure traces, diagnoses across the batch, generates structured patches treating the harness as code, and selects updates by validation. +9.0 on GAIA2, +9.6 on SWE-Bench Pro, +10.0 on Terminal-Bench 2.0. Its ablation names three ingredients: deep debugging rather than shallow reflection, targeted modification rather than unconstrained editing, and generalization-aware selection rather than trajectory-specific repair.

That last pair is the fourth and fifth independent arrival at the rule this page first recorded from DarwinX (08-14), whose preserve-and-extend contract admits a variant only if it extends coverage without regressing, and from Ken Huang's pattern language (08-14), whose hill-climbing family states that improvement is only safe when measured, bounded and reversible. Recuris independently gates on validation and keeps patches localized. Five arrivals by five methodologies is well past this page's three-instance threshold: bound the edit and gate the admission, or the optimizer overfits its most recent failure. Treat that as settled and stop re-deriving it.

AutoSaddler's mini-batch economy also lands on the affordability axis Task-CoEvolve (08-25) opened, from a different direction. Task-CoEvolve cut evaluations 80% by concentrating them on the shrinking frontier band where candidates still disagree. AutoSaddler amortizes each patch over several traces instead of paying an evaluation cycle per proposal. Neither cites the other, both landed within a day, and harness search getting cheaper by two independent routes in two days is the signature of a subfield in its engineering phase rather than its discovery phase.

Gradient Flow's nine rules supply a fourth independent measurement of the swing, from practitioners who were not trying to publish one. Ben Lorica, surveying production teams with unrelated products, reports that the same open model showed an 18 percentage point spread between its best and worst harness configuration, and singles it out as the finding he would take most seriously when comparing models. Set beside omarsar0's 5x-30x cost-per-success swing (08-13), AI4AI's 0.49 → 0.91 capability lift with the target frozen (08-13), and DarwinX's +7.7 from the scaffold alone (08-14), the swing now has four measurements in three units from three kinds of observer. His rules 1 and 2 are also the practitioner statement of this page's sharpest mechanism claim: put hard constraints in software rather than prompts, and turn stable agent paths into ordinary code. That is the harness wins by taking decisions away from the model, arrived at from production experience instead of from AI4AI's and Spark-to-Paper's ablations.

Ramp's Inspect (The Pragmatic Engineer) is the harness-as-product-boundary thesis confirmed from the buyer's side, and it splits the artifact in a way this page had not. A fintech built its own background coding agent; 75% of merged PRs at Ramp are now raised by it (up from ~60% two months post-launch), it passed one million sessions in July, and engineers are free to use anything, so the share is revealed preference. Block has Goose, Stripe has Minions, Shopify has River. Orosz's framing: "buy, don't build" appears not to apply to AI harnesses.

The reason it does not apply is the useful part. Ramp's advantage over Claude Code and Codex is entirely the evidential face this page recorded from Agent Safety Should Be a Runtime Contract (arXiv 2608.11274, 08-13) — no task-complete claim without checkable proof. Inspect verifies backend work by running tests, reading telemetry and querying feature flags, and verifies frontend work by producing screenshots and live previews, shipping the latter nearly a year before any vendor. A third-party harness cannot supply evidence about systems it cannot see. Meanwhile the parts this page has shown to be highly portable are context policy and control logic: Meta-Harness's discovered math-retrieval harness added 4.7 points across five held-out frontier models zero-shot, AutoDesign's DesignHarness lifted seven other code-agent-model configurations from 54.99 to 67.39, AI4AI transferred strong-to-weak. The split this page can now state: harness structure is portable, harness evidence is not. That predicts the market shape — vendors sell optimizable structure, companies build their own verification — and it explains why four non-AI companies independently reached the same build decision.

Recuris also adds a fourth supporting result for routing over model-harness pairs, and still no proposal. It lifts ten models by materially different amounts (+17.8, +15.6, +16.6, +13.5 on four of them), which is exactly the condition under which routing over pairs would beat routing over models. LLM routing records the gap.

Updated state of knowledge. Open problem 2 (composition) is mostly resolved by Recuris. The admission rule is settled at five independent instances. The swing has four measurements. The harness now has a portability boundary (structure yes, evidence no) and a confirmed industrial buy-versus-build answer. Open problems 0 and 0b are not only unmoved but conspicuously so. AutoSaddler and Recuris both publish capability gains with no search or serving budget, keeping open problem 0 exactly where it has been since May. And both report single-attempt scores one day after Thinkingbox (08-25) — Microsoft's own 507 stateful business workflows, checked against backend terminal state — showed the strongest model falling from 65.36% pass@1 to 25.25% pass^20. Lorica's rule 4 independently asks for the same thing, telling teams to measure recovery separately from first-attempt accuracy. A Microsoft benchmark and a practitioner survey both demanded the pass^k measurement, and the Microsoft-led harness paper published the next day without it. tau-bench, where Recuris reports 87.9%, is a stateful tool-use benchmark of the same family. This is now the most conspicuous and cheapest missing measurement on this page, and it has gone unrun for two consecutive dense harness days.

2026-08-28: a cost-per-success number finally arrives, and the model side of open problem 0 opens

Five papers on this layer in one day (four on HuggingFace, one on the Kurate board), plus the first industrial statement of the cost thesis by a named buyer. The day moves two things that have been stuck for months.

PILOT in the Loop (arXiv 2608.26530) publishes a serving-side cost-per-success metric, which is the number this page has been asking for since 08-14. A supervisor-worker split with two powers the worker lacks: live steering (redirect or abort the active worker mid-execution) and live self-evolution (distil procedures and failure modes into skills and memory while the run is still going). Ranks first in five of six configurations, +9.8 points over counterpart harnesses on Terminal-Bench 2.0, +14.6 with GLM-5.1 and +12.4 with Kimi-K2.6 in the self-improvement setting. The headline for this page is the efficiency pair: mean output tokens fall 42.9% and 47.4%, while successful evaluations per million output tokens rise 110.3% and 134.0%.

That denominator is the same one omarsar0's preregistered benchmark used to measure the 5x-30x cost-per-success swing (arXiv 2608.01347, 08-13), which this page calls its empirical spine. Every harness paper since has reported capability without it: AutoDesign published $3 per rollout for its search, DarwinX published no evolution budget, AutoSaddler and Recuris (both 08-26) published gains with no serving budget at all. PILOT reports the harness getting cheaper and better simultaneously, which is a different and stronger claim than buying accuracy with tokens.

Two cautions. The mechanism means most of the saving is probably a refund rather than an efficiency: aborting doomed trajectories early does not shorten the successful path. And the metric is output tokens, while a supervisor reading a streamed trace consumes many input tokens and emits few, so the architecture may be shifting spend from output to input rather than removing it. Whether the supervisor's own consumption is inside the 42.9% is the first thing to verify in the full paper.

PILOT also completes the design space with the two systems from two days earlier. AutoSaddler (08-26) learns offline from mini-batches of failure traces, which is what makes its patches durable. Recuris (08-26) patches between runs from verified state. PILOT patches during the run. The trade is visible and untested: offline patches generalize because they see multiple failures, live patches arrive in time to save the current task but derive from a single trajectory, which is exactly the trajectory-specific-repair failure AutoSaddler's ablation isolated. Offline versus live at matched budget is now a well-posed and cheap experiment.

TaoLive Harness-Aware Training (arXiv 2608.15763, Alibaba) is the first entry on this page to attack open problem 0 from the model side. Everything above freezes the weights and optimizes the scaffold. HAT does the opposite: it trains a compact model to tolerate the scaffold changing. The observed dilemma is a production one at Taobao Live, where campaign rules, compliance requirements and merchant preferences change weekly: a large model adapts to a rewritten harness zero-shot but blows the latency budget, while a compact model meets latency and then overfits to the exact harness it saw. Harness-State Augmentation applies task-preserving transformations to skill identifiers and content, tool schemas, prompt structures and Hook functions, so the model learns to read a harness instead of memorizing one. Three stages: HSA-SFT on strong-model trajectories, General On-Policy Distillation to restore the generalization SFT destroyed, then HSA-RL in augmented environments.

The most important result is negative and it is a warning to practitioners: Fixed-Harness SFT lowers IFEval by 7.7 points from the base model, and HAT does not (83.5). Fine-tuning a compact model against one fixed production scaffold makes it worse at following instructions in general. If that replicates, a large number of teams have quietly paid this cost. In-domain, HAT reaches 94.8 on Live-Stream QA (base 80.3, strongest general LLM 93.0) and 94.6 on Harness-Variant QA (base 75.4), and it is deployed: P50 3.4s and P95 8.1s on a single NVIDIA H20, with positive online A/B results for GMV and item-page views. A harness paper reporting a P95 and a revenue metric is new on this page.

HAT does not settle open problem 0, because it publishes no matched-cost baseline either. What it establishes is that the two levers interact: the whole cost of a harness rewrite was the retrain it forced, so a harness-robust model changes what harness optimization is worth. Its nearest relative is Apodex 1.1 (08-25), which trains model and environment together and reaches the frontier band at 35B; this page's complaint there was that the two contributions are never separated. HAT has the same entanglement but declares it, and builds Harness-Variant QA specifically to measure the co-adaptation. Better experimental design of the two.

The memory layer got specified as a three-part system, by three papers in one day. WikiSkill (arXiv 2608.27454) fixes authorship: insights guiding skill development stay scattered across optimization histories, so it separates raw experience, an accumulated knowledge base, and executable skills, and consolidates experience into the knowledge base that later skill updates build on. Its ablation confirms the persistent layer is load-bearing. CaSKG (arXiv 2608.25500) fixes retrieval: graph retrieval only recovers workflow context when the edges are reliable, and normally they are inferred from surface similarity, so CaSKG calibrates edges by counterfactual intervention (remove, substitute, reorder skill pairs; aggregate with Bayesian smoothing) offline, without changing the downstream policy. Highest task score in all twelve model-benchmark combinations, ScienceWorld macro-average 72.62 → 80.50 and ALFWorld 80.01% → 86.79% against Graph-of-Skills, with fewer mean environment steps. And self-evolving kernel-optimization agents (arXiv 2608.25570, Kurate cs.LG #5) fixes storage for a Tier 1 domain, replacing AccelOpt's flat slow-fast kernel pairs with an Experience Graph Memory.

CaSKG moves open problem 4, which had no research at all. That problem read: graph engineering lacks the research the loop layer has, and what a graph buys over a well-run loop is unquantified. CaSKG supplies the mechanism, a graph buys prerequisite structure and only buys it if the edges are calibrated, converting the practitioner slogan into a testable engineering requirement. It does not close the problem, because it measures graph-versus-graph rather than graph-versus-loop, but the comparison is now specifiable. It is also the strongest member of this wiki's disagreement-is-the-signal family (R2-OPD 08-25, Task-CoEvolve 08-25, TTPO 08-28), because it manufactures the disagreement by intervention rather than observing a correlation, which sidesteps the shared weakness the wiki flagged: every correlational member depends on a second estimator nobody validated. A counterfactual probe is not an estimator.

WikiSkill's third finding complicates this page's portability boundary. The 08-26 reading of Ramp's Inspect concluded harness structure is portable, harness evidence is not. WikiSkill's skills are structure, so the boundary holds, and it adds a fourth transfer result (across model families, after AI4AI strong-to-weak, AutoDesign across seven configurations, Meta-Harness across five held-out frontier models). But it also reports that skills evolved by other models can outperform self-evolved skills, which makes the portable half not merely transferable but sometimes better sourced externally. That strengthens the vendor side of the market split more than 08-26 allowed. The alternative explanation the paper must rule out: if the other model is simply stronger, this is a distillation result, not a claim about authorship.

What Makes Good Agentic Data? (arXiv 2608.27260) supplies vocabulary this page has needed. Its ACE lens (Accuracy establishes feasible support; Complexity places learning mass relative to a declared learner and execution configuration; diversity controls coverage and redundancy) is a decomposition of what every selective-allocation method in this wiki trades off. Two connections matter. Its reported field-wide shift toward execution-grounded accuracy (verify by running, not by scoring) is the data-pipeline arrival at this page's sharpest mechanism claim, that the harness wins by taking decisions away from the model. And learner-relative complexity is the data-generation face of the model-harness-pair unit, since it makes informativeness a property of the pair rather than the task. It is also the reason to doubt unbounded skill transfer: if informativeness is learner-relative, transferred experience should degrade as the target diverges from the source, and nobody has found that boundary.

Industry stated the cost thesis with a named buyer for the first time. The Information reports that Visa built a harness in April that significantly reduces the cost and time for Anthropic's Mythos to find and fix security vulnerabilities in its codebases, with president of technology Rajat Taneja saying plainly: "Using the model through this harness, we have found, is more effective than using the model by itself." (The Information) Every prior industrial data point on this page was a vendor headlining self-optimization (xAI's Grok Build, OpenAI's Codex platform) or a company building its own agent (Ramp, Block, Stripe, Shopify). This is the first customer saying the harness is what made the frontier model affordable, on a security workload, with the vendor's pricing unchanged. The measurement gap is the usual one: no percentage and no dollar figure.

Updated state of knowledge. The harness now has a published serving-side cost-per-success metric (PILOT: 43-47% fewer output tokens at 2.1-2.3x successes per million output tokens), a model-side lever (HAT: train for harness-robustness instead of retraining per revision, and fixed-harness SFT costs 7.7 IFEval points), a specified three-part memory architecture (authorship, retrieval, storage, one paper each, one day), a fifth transfer result now crossing model families, an intervention-based rather than correlational way to calibrate agent memory, and a named enterprise buyer confirming the cost claim. Open problem 0 has moved from unaddressed to attacked-from-both-sides-without-a-matched-cost-baseline. Open problem 0b is untouched for a fifth consecutive harness paper: PILOT reports first-attempt ranks on Terminal-Bench 2.0, a stateful benchmark of the same family where Thinkingbox (08-25) measured the strongest model collapsing from 65.36% pass@1 to 25.25% pass^20.

Open problems (research angle)

0. Harness optimization versus fine-tuning at matched cost. Still unrun, and now more glaring: AutoDesign publishes $3 per rollout, DarwinX publishes no evolution budget, and no distillation paper publishes a comparable per-point cost. The first paper to put the two on one axis settles where a team should spend. 0b. A pass^k curve for a harness-optimized agent. Every harness result on this page reports a best-of or single-attempt number. Thinkingbox (08-25) shows a 40-point drop from pass@1 to pass^20 on stateful work, and Fragility (08-25) plus Bakouch's ~50-step single-setting spread (08-16) say the variance is large. Whether a better harness narrows the pass@1-to-pass^k gap or merely lifts both is unknown, and it is the difference between harness engineering being a capability story and a reliability story. Cheap to run, and nobody has.

1. A harness-quality metric that predicts the 5x–30x swing before you run. omarsar0 measured the swing post-hoc; nobody can yet predict which harness wins for a given task class without a full benchmark. This is the single highest-value open experiment.

2. Composing the pieces. Mostly resolved 2026-08-26 by Recuris, which manages state externally (verified Working Memory), routes skill retrieval through that state rather than the transcript, and self-optimizes via a fixed Meta-Agent making validation-gated localized updates to Skill Memory, benchmarked end-to-end on four long-horizon benchmarks and ten models. What remains of the original problem: the composition is not ablated, so how much comes from each of the three pieces is unknown, and the cost of maintaining verified state plus running validation gates on long tasks is unpublished.

3. Where self-simulation of the loop diverges from reality — the same self-judgment failure mode the reward-hacking thread keeps flagging. A confidently-wrong auditor breaks the MEA loop.

4. Graph engineering lacks the research the loop layer now has. Partly moved 2026-08-28 by CaSKG, which supplies the mechanism (a graph buys prerequisite structure, and only if the edges are calibrated by intervention rather than inferred from similarity) and the numbers (highest score in all twelve model-benchmark combinations, +7.9 ScienceWorld macro-average over Graph-of-Skills, fewer environment steps). What remains: CaSKG compares graph to graph, never graph to a well-run loop, so what a graph buys over a good loop is still unquantified and now specifiable. Also unpriced: counterfactual probing is quadratic in library size before pruning and must be re-run as the library grows, which is exactly the regime WikiSkill and Recuris create.

5. Offline versus live harness patching at matched budget. New 2026-08-28. AutoSaddler (08-26) patches offline from failure mini-batches, Recuris (08-26) between runs from verified state, PILOT (08-28) during the run. Offline should generalize better (multiple failures per patch), live should recover more of the current task. Nobody has run them against each other, and the answer decides whether a production harness needs a supervisor with abort authority or a nightly patch job.

Sources

Research (digest): Code as Agent Harness · Scaling the Harness · omarsar0 harness benchmark (arXiv 2608.01347) · LongHorizon-Harness (arXiv 2608.01964) · HarnessOpt-Bench (08-08 weekly) · AI4AI at Test-Time (arXiv 2608.12307) · Agent Safety Should Be a Runtime Contract (arXiv 2608.11274) · Spark-to-Paper (arXiv 2608.11924)

Research (digest), 2026-08-14: DarwinX (arXiv 2608.07545) · AutoDesign (arXiv 2608.13560)

Industry / practitioner, 2026-08-14: DeepSeek Harness v0.1 and cache-hit repricing · Ken Huang, Harness Engineering pattern language

Practitioner cluster (surfaced via saved reading, 2026-08): ~12 articles on loop/harness/graph engineering, the dominant theme in the private curation index by a wide margin; synthesized in the 2026-08-13 Media Zone. Enumerated saves live in the private curation index.

2026-08-30: the harness moves the clock, and the model cannot see it

Your Agents Are Not Time Aware (Ofengenden and Andriushchenko, MATS 10) adds a third dimension to this page's core claim, and it is the one the model is structurally unable to estimate. The page already records harness choice swinging cost-per-success 5x to 30x (omarsar0, arXiv 2608.01347) and accuracy roughly 2x (AI4AI). This adds wall-clock: Claude Code consumes roughly 2.5x more turns than Codex on the same model, because Claude Code runs until it believes the task is solved while Codex tends to stop at a time boundary. Two different runtime distributions, and the model's own duration prediction is identical across both.

The prediction is anchored rather than estimated. Measured compression exponents of 0.19 to 0.24 mean predictions barely move as true duration moves: both agents essentially answer "about ninety minutes." On ProgramBench, Opus 4.8 in Claude Code predicted 99 minutes against an actual 85 (1.16x) while GPT-5.5 in Codex predicted 72 against an actual 17.5 (4.12x). On AgentTime, a purpose-built suite of 235 tasks assembled from 18 existing benchmarks with external timers and no artificial caps, Fable 5 over-predicts 3.1x and GPT-5.6 Sol 9.9x.

The ablation identifies where the signal lives, and it is not inside the model. Retrospective accuracy is near-perfect with an elapsed-time tool (R-oracle), only slightly worse in-session without one, substantially degraded on a replayed transcript, and doubles in error when timestamps are scrubbed. Transcript length correlates with runtime at r=0.91, but controlling for length, in-session estimates correlate at only r=0.4. The agent is reading its context for timestamps and using length as a proxy, not maintaining a duration state.

This is the empirical case for LongHorizon-Harness's central design choice. That system (arXiv 2608.01964, 08-13) keeps task state outside the execution context and updates it only on environment-verified facts. This paper says why that is necessary rather than tidy: the in-context signal is a proxy that collapses when the context is manipulated, and the same agents overrate their own work by roughly 20 points same-turn (Opus 4.8, GPT-5.5), with the direction unstable across generations (Opus 5 underrates by 11-15 points in a separate turn, GPT-5.6 Sol overrates by ~7). One reported instance has both models self-scoring near 70% on work that actually scored 7% and 14.5%.

It moves open problem 3 from a hypothesis to two measured quantities. That problem read: where self-simulation of the loop diverges from reality. The divergences now have numbers, a 0.19-0.24 compression exponent on duration and a ~20-point optimism gap on quality, plus a mechanism from the R-scrubbed ablation. It also sharpens open problem 0b, untouched for a sixth consecutive harness paper: an agent that overrates its output by 20 points is a plausible partial explanation of the pass@1-to-pass^k collapse Thinkingbox (08-25) measured (65.36% to 25.25% on stateful work), because a harness that trusts the agent's self-report cannot see the failures pass^k exposes.

It also validates PILOT's (08-28) architecture after the fact. PILOT gives a supervisor authority to abort a running worker, which is a time-based control over an entity with no calibrated sense of elapsed time. PILOT's supervisor reads a streamed trace rather than asking the worker, and this paper is the reason that is a requirement rather than a preference.

The practical statement. "Finish this in 30 minutes" is the cheapest control surface a long-running agent has and it is unusable against a constant duration model. The fix for the retrospective half is a tool call, which is one more decision taken away from the model, which is this page's recurring mechanism. The prospective half is unfixed and nobody ran the obvious experiment: whether telling an agent its own token throughput improves the estimate that actually matters for control. No cost figure is reported anywhere in the study, which is now the standing complaint against every entry on this page.

New open problem 6: budget enforcement belongs in the harness, and nobody has built it. Every cost result on this page is denominated in tokens or dollars while wall-clock is a third axis, and DHH's 24x dollar spread on one identical task (08-16, recorded on compute economics) was explicitly a time-for-money trade, $550 in 45 minutes against $23 in 2.5 hours. An agent that cannot predict its own runtime cannot participate in that trade, so a harness that enforces a time or token ceiling has to do it externally. No harness in this wiki publishes one.

2026-09-06: the harness gets a query optimizer, and the append-only rule meets its first counterexample

ContextPipe (09-06) (arXiv 2609.00749, MatrixOrigin with Tsinghua) supplies the systems layer this page has described functionally for four months without anyone building it. Every entry above treats context assembly as something the harness does; none of them treat it as a component with an interface. ContextPipe's argument is that the component already has a forty-year-old design: context assembly is structurally isomorphic to relational query execution. Both resolve a declarative intent against a structured catalog, under a hard budget, over a tiered cache, and both need to be explainable afterwards. So it ships the database machinery: a five-phase pipeline (Plan, Bind, Optimize, Execute, Feedback), a data-source catalog, a deterministic cache-aware optimizer that chooses the prompt's byte layout against a configurable ProviderCachePolicy, eight compaction lifecycle tiers with predictive percentile-based pressure rather than reactive watermarks, a ForkPrefix mechanism so spawned children share a parent's cached prefix, and an EXPLAIN ANALYZE trace per request.

The result is 31% fewer total tokens, 23% fewer LLM calls and 9% lower response time on the SWE-bench Pro Qutebrowser subset against an append-only baseline, at a LOWER KV cache-hit ratio. That last clause is a direct counterexample to the discipline this page adopted on 08-14.

Restate the tension precisely, because both sides are right. DeepSeek Harness v0.1 (08-14) shipped alongside a roughly 6x repricing of cache-hit tokens, and its organizing commitment was that previously written history is never altered: corrections are appended as statements describing a modification, because an in-place edit invalidates every cached token downstream. This page recorded that as protect the prefix, and Hugging Face's Elie Bakouch expected other harnesses to follow. ContextPipe deliberately breaks prefixes and wins anyway, because cache-hit ratio is a ratio and the bill is a product. ContextPipe shrinks the denominator by 31%, and a token you never assembled costs nothing at any cache tier. Append-only is optimal when the whole context was going to be sent regardless; plan-and-compact is optimal when a real share of accumulated history did not need sending. Nobody has published the crossover, and it is a two-axis sweep over context length and cache-price ratio that would settle production practice for every agent harness in this wiki. New open problem 7.

It half-closes open problem 6. That problem, added 08-30, said budget enforcement belongs in the harness and no harness in this wiki publishes one, because an agent that cannot predict its own runtime cannot enforce a ceiling on itself. ContextPipe enforces a hard token budget externally, deterministically, per request, with a printable plan. The wall-clock half stays open, and the 08-30 finding that Claude Code consumes roughly 2.5x more turns than Codex on the same model is still unaddressed by any published mechanism.

Where it composes and nobody has run it. Declarative Attention (09-03) lets the model declare which region of an already-resident context it needs, cutting attended tokens 52.0% on Gemma-4-31B for 1.27pp of accuracy. ContextPipe decides what becomes context at all. One shrinks the read over a fixed cache, the other shrinks what enters it, they are multiplicative, and neither paper cites the other. Separately, Select, Compress, Reinvest (09-05) established that a compression saving pays nothing until it is reinvested. ContextPipe banks its 31% as a lower bill and reinvests none of it, and runs no reinvestment ablation, which by SCR's finding is where the accuracy would have been.

The honest limits. One repository subset, described by the authors as a preliminary evaluation. No cost accounting for the optimizer itself, which runs on every single API call, making this the seventh consecutive paper on this page to delete a dependency without pricing the replacement. ForkPrefix is the mechanism with the largest theoretical saving and it has no number attached. And the database analogy has an admitted hole the authors state: a query optimizer works because it can estimate cardinality, and nobody can quantify information value per token, so this is a deterministic policy engine with statistics rather than a cost-based optimizer.

2026-09-10: on-policy signal is compatible with harness evolution, imitation is not

Co-Evolving Harnesses (09-10) is the sharpest negative result this page has recorded. Salesforce evolved a harness around a weak model across seven enterprise agent tasks, then fine-tuned that model on a strong expert's complete trajectories collected under the same evolved harness. Performance regressed on all seven tasks, by 4 to 30 points, across both Qwen3-Coder and Gemma 4 — and the identical procedure helps under the un-evolved harness. The diagnosis: imitation does transfer knowledge and does increase scaffold usage, but the weak model adopts the expert's planning strategy without the competence to execute it, and the harness was evolved around the weak model's native planning style. The weight update destroys the model-harness fit the harness update just bought. Their fix is on-policy expert correction: let the weak model roll out in its own style, have a meta-level agent localize the single failing turn, and ask the expert to rewrite only that turn.

Read alongside T1 (09-10), a 122B mixture-of-experts model trained with pure on-policy RL against a real cloud shell for 300+ tool-call turns, lifting Terminal-Bench 2.1 from 43.8% to 64.0%, a cleaner rule emerges than either paper states: on-policy signal composes with harness evolution, off-policy imitation does not. T1 never imports another model's planning style, so it never encounters the contention Salesforce found. That is the operational takeaway for anyone running a harness-plus-fine-tune loop.

And Revisiting Complete Reasoning Traces (09-10) attacks the same unit from a different angle, finding that full trajectories provide only limited SFT benefit while heavily truncated ones remain effective, because intermediate tokens contribute minimally and the model can infer the missing steps from endpoints. Both papers conclude that the complete trajectory is the wrong training unit, disagreeing only about what to keep: endpoints (Revisiting) or the weak model's own middle with one turn rewritten (Co-Evolving). The burden of proof has flipped, and training on complete trajectories now needs a justification.

Practitioner convergence on the same theme. Harness engineering dominated the day's feed independently of the papers: a Medium long-read on the three harness camps at 2.8K claps, "The Harness Is the Product" in the same digest, YC's Paper Club episode Why The Harness Matters More Than The Model at 227K views, and Sebastian Raschka's aside in his 09-10 looped-transformers piece that stale AGENTS.md and SKILL.md files may now constrain newer models and are worth archiving and regenerating. That last point is the practitioner version of the Salesforce result: a scaffold evolved around an older model's competence becomes a liability once the model changes. Model-harness fit decays in both directions.

2026-09-11: harness synthesis ships, and the objective splits in two

Two groups published an automated search over harnesses on the same day, with different objective functions, neither citing the other. That is this wiki's threshold for treating a research pattern as an established technique.

NVIDIA's SoL-Pi (09-11), MIT-licensed and open, optimizes tokens. It is not a new harness but an efficiency layer on top of an existing one (Pi), produced by pointing an auto-research loop at 535 executable environments (495 from real GitHub issue-and-PR pairs, 40 synthetic with verifiers) and letting it propose, implement, run and verify harness modifications. Roughly 1 idea in 40 survived. Four did: Action Fusion (fold the test or run invocation into the same tool call as the edit, removing a full model round trip), Online Context Compact (decide whether to compact at subtask completion boundaries rather than waiting for the window to fill), ObservationPack (index huge tool results instead of re-injecting them every turn, fetch precisely on demand), and Evidence-Preserving Reducer (a cheaper model summarizes long logs, with each cited piece of evidence verified individually against the original). Result: 45-49% fewer tokens than stock Pi at about 94% of average task score, roughly a third off cost, and 35-64% fewer tokens with 50-54% lower cost against each model's own native harness. On GPT-5.6 Sol, SoL-Pi outscores the native Codex harness on EdgeBench.

Three things about this are new for this page.

1. The objective is tokens at near-constant score, not score. Every harness-evolution result recorded here (A²E and Evo-Bench, 08-11, DarwinX and AutoDesign on 08-14) optimized task performance and reported cost incidentally or not at all. Optimizing the cost axis explicitly is a different search and it found different things.
2. The search yield is published. 1 in 40 is the single most useful number here for anyone deciding whether to run such a loop, and the harness-evolution literature has generally not reported it.
3. The recursion is claimed but not measured. NVIDIA calls the direction Efficiency for Efficiency: a cheaper harness makes the research loop cheaper, buying more experiments per budget, finding a cheaper harness. That is falsifiable and nobody has run it. Iterate the loop for N rounds, feed each round's harness back as the next round's substrate, and plot cost-per-accepted-idea against round number. If it falls, this is the first self-improvement loop in this wiki operating on infrastructure rather than weights. If it plateaus after one round, the four optimizations were the low-hanging fruit.

EvoSafeHarness (09-11) (arXiv 2609.05903) runs the same structure against safety. It jointly searches a natural-language policy and the executable code that enforces it, for a frozen model in a target domain, guided by observed model behaviour, a domain specification, and fresh-context adversarial review whose explicit job is to reject rules that only work on the benchmark. On DecodingTrust-Agent it cuts average attack success from 45.6% to 10.0% at a 3.3-point utility cost, best in 14 of 15 cells. On AgentDojo it reaches 82.8% utility at 0.0% attack success, twice CaMeL's utility at the same operating point, transferring unchanged to unseen AgentDyn suites.

Its conceptual contribution is the inverse of a claim this page has been making. This page established that a harness transfers across base models, and treated that as the good news. EvoSafeHarness argues a safety harness should not transfer unchanged, because the enforcement budget a model can absorb before utility collapses is a property of that model, and the safety relations that need governing are a property of the domain. Its clean formulation: domain semantics determine which relations and trajectory state need governing; model and runtime behaviour determine how and where to enforce them. That reframes "our defence generalizes" from a selling point into a weakness.

The fresh-context adversarial reviewer is this page's disagreement-as-signal pattern in a safety setting. Task-CoEvolve (08-25) concentrated harness evaluation on validation tasks where candidate harnesses disagree, cutting evaluation count 80%. EvoSafeHarness uses a deliberately uncorrelated second judge to catch rules that fit the benchmark and nothing else, treating its disagreement with the optimizer as evidence of overfitting. Fresh context is the mechanism that keeps the two signals independent, and it is the detail most search-based safety work omits.

The paper neither group wrote is the composition, and the two objectives are in measurable tension. Several of SoL-Pi's four optimizations put less information in front of any enforcing layer: ObservationPack indexes tool results rather than presenting them, and the Evidence-Preserving Reducer hands logs to a cheaper model. A joint search under both token cost and attack success would produce the first measured safety-cost Pareto frontier for agent harnesses. There is currently not one published point on that curve.

Cross-page note. SoL-Pi's two context optimizations land on the compact side of the open crossover the KV cache page recorded on 09-06, where append-only prefix protection and plan-and-compact were left as two opposed disciplines with no measured boundary. ContextPipe (09-06) cut total token volume 31% while lowering the cache-hit ratio, on the arithmetic that a token never assembled costs nothing at any cache tier. SoL-Pi's 45-49% is a larger cut found by search rather than by design, on a software-engineering task distribution. Neither publishes the crossover sweep the KV cache page asked for.

2026-09-13: the objective splits in three, the search gets a budget, and the prefix turns out to be a cost centre nobody was searching

Two days after this page recorded "the objective splits in two," it has split in three, and the third one is meta. SoL-Pi (09-11) optimizes tokens at near-constant score (45-49% fewer tokens at ~94% of task score, 1 accepted idea in 40). EvoSafeHarness (09-11) optimizes attack success at near-constant utility (45.6% to 10.0% average attack success at a 3.3-point utility cost). COBRA-Skills (09-13) optimizes the cost of the optimization itself. Three objectives, three papers, inside a fortnight, none citing the others.

COBRA-Skills is the direct answer to the number SoL-Pi published. If 39 of 40 proposed harness modifications are waste and each candidate costs an execution-based evaluation, the move is to stop evaluating uniformly. COBRA-Skills formulates skill optimization as budgeted sequential optimization over an evolving candidate space, with a contextual bandit prioritizing candidates that are either promising or informative, and evidence-grounded evolution refining the population from execution feedback. Result: strongest average performance across six agent benchmarks and three target models at 55-58% lower optimization cost than SkillOpt, using only 50 unique optimization examples per benchmark. Two robustness properties matter as much as the headline: it survives changes to the agent harness, and it works when the target model generates and refines its own skills, removing the stronger-teacher dependency.

RobustSGPO asks the question one level up: what is the search allowed to edit. Semantic-gradient prompt optimization improves harnesses from execution feedback but leaves edit scope and operation unspecified. RobustSGPO specifies the requested edit, constructs and checks the patch executes before scoring it, and continues from either the incumbent or a retained snapshot. The study is large by this page's standards: AgentX brainstorming workflow, 120 tasks, 95 runs, 7,350 candidate attempts. Findings: periodic 1→2→3 permission scheduling beats fixed maximum permission by 0.28 test-score points; held-out completion on 30 unseen tasks rises 60.0% to 80.0% with test quality 3.77 to 4.14 under a 20M-token budget; and category retention reduces source-task degradation after a shift while random retention reaches a higher destination endpoint, at a measurable retention overhead the paper names rather than hides.

The permission-escalation result is the third instance of a pattern, which crosses this wiki's naming threshold. Task-CoEvolve (08-25) cut evaluation count 80% by scoring only where candidates disagree. EvoSafeHarness used a fresh-context adversarial reviewer whose disagreement signals benchmark overfitting. RobustSGPO finds constrained edit scope on a schedule beats maximum freedom. Name it: harness search needs a regularizer, and the regularizer lives in the search design rather than in the objective. All three restrict something (which tasks you score on, who judges, how large an edit you may make) and all three report that the restriction improves the result rather than trading against it.

The retention finding pairs directly with Ecdysis (09-12) and nobody has composed them. Ecdysis gave the instruction ratchet its admission criterion, repair only what recurs across tasks, at 1.84x faster harness training and 18.6% better accuracy. Ecdysis decides which repairs to admit; RobustSGPO decides which past harnesses to keep as restart points. Both are stability-plasticity policies at different granularities, and the composition is unrun.

Separately, this page has been optimizing the volatile tail of the context and ignoring the static head. Ken Huang's prefix-stable KV caching argument (09-13) points out that the instruction file a harness loads at session start sits in the cached prompt prefix, where one changed byte at token N invalidates the KV state for every token after N. A Last updated: stamp at the head of a memory file, a glob concatenated without LC_ALL=C sort, or CRLF drift each silently destroy reuse. SoL-Pi's Online Context Compact and ObservationPack both act on the tail; byte-stability of the bootstrap file was not in its search space at all. Hold the harness fixed, freeze the prefix, measure cached-read fraction and TTFT. Orthogonal to all four of SoL-Pi's wins, and free.

The tool surface gets its own routing result the same day. The Menu Is an Execution Prior (09-13) shows that ranking a tool menu by request relevance reliably surfaces the final action while omitting or delaying the tools that produce its inputs, and that routing on a state path instead lifts ToolBench online success from 0.737 to 0.898 with the agent unchanged, while a 32-tool menu covers more complete chains than the official 128-tool list. That 4x menu reduction is also a prefix-token cut in the tool-definitions block, which is the same region prefix stability protects. Two independent cost arguments landing on the same few thousand tokens at the head of every turn.

The practitioner side of the day says the same thing louder and with less evidence. The X feed carried a dense cluster of harness-engineering content: an Anthropic engineer's workshop reframed as "we don't write prompts anymore, we build loops," a LangChain post defining agent = model + harness and walking through create_agent, Alexandr Wang telling YC that a correctly designed agentic loop plus a self-optimizing eval system outperformed a hundred senior engineers at Meta and that the underlying architecture is "Markdown files," and omarsar0 observing that YC builders increasingly want domain-specific harnesses. None of it carries a measurement. The value is as a diffusion signal: the framing this page adopted in August is now the default framing in practitioner media, roughly two months after the research established it.

2026-09-14: the harness thesis is stated in four registers in one week

This page has built its case almost entirely from research: Ecdysis (09-12) trained the harness instead of the model, SoL-Pi (09-11) auto-discovered four harness optimizations worth 45-49% of tokens, COBRA-Skills (09-13) cut skill-optimization cost 55-58% with a contextual bandit over candidate skills. On 09-14 the same claim arrived from three non-research directions at once, which is worth recording as a fact about the field rather than as three separate items.

The engineering statement. LangChain's custom agent harness guide defines a harness narrowly and usefully: agent = model + harness, where the harness is whatever gets the right context to the model at every step. Its architectural claim is that the extension point should be middleware, small composable units hooking the loop at fixed points (before and after each model call, before and after each tool call, at startup and teardown). Two of its assertions are load-bearing and both are about where logic must not live. Policy enforcement (PII, compliance, approval gates) must be deterministic middleware because a prompt cannot guarantee it fires. Model selection by task complexity is runtime control, not prompt instruction, which makes routing a harness hook and connects this page directly to LLM routing. It also inverts the usual argument for giving agents shell and filesystem access: the case is token efficiency, not capability, because a command often replaces several thousand tokens of reasoning.

The executive statement. Satya Nadella, via Ken Huang's writeup (09-14), argues the durable enterprise asset is a company-owned "hill-climbing machine" built from private evals, workflow traces, rewards and outcomes that stay inside the tenant. The failure mode he names is that consume-only AI adoption can buy real productivity and still lose the firm's position, because the traces and reward signal leave with the vendor. The test he implies is the sharpest thing on this page: model-swap survival. If your system degrades when you change the underlying model, what you built was a prompt stack tuned to one vendor and the vendor owns the asset. If it survives, the knowledge lives in your evals, traces and harness.

The market statement. @gregisenberg, in the day's most-shared harness post, defined a harness as four things (run the model in a loop, give it hands, manage its memory, enforce what it may touch) and made the model-independence argument in one line: "a wrapper was one model doing everything and a harness is a router." His commercial claim is that a wrapper sells software while a harness sells finished work, and that the harness compounds because every human correction becomes a retained rule.

Naming the pattern. Research, vendor engineering, a hyperscaler CEO and the venture-framing layer all stating the same thesis inside 72 hours crosses this wiki's threshold. The durable asset moved from the weights to the loop around them, and all four registers independently identify model-independence as the property that proves it.

The gap the convergence exposes, which nobody has closed. None of the research-discovered harness optimizations map onto a named middleware slot. SoL-Pi's Online Context Compact maps to summarization middleware. ObservationPack maps to nothing. And byte-stable prefix construction, which Ken Huang's 09-13 post showed can silently destroy KV cache reuse when a line-ending or an unsorted glob changes one byte, has no middleware and arguably cannot have one: it is a property of how the system prompt is built, upstream of anything the loop does. The harness abstraction as currently drawn has no hook for the cheapest optimization on it.

And the tension worth holding. Nadella's argument says enterprises should minimize dependence on any single model provider. In the same 24 hours Anthropic committed $13.7B over six years to one compute counterparty on top of roughly $517B of aggregate commitments. Both are rational and they describe different layers: the loop is portable, but the loop still runs on someone's silicon, and the margin is accruing to the layer underneath the one the harness thesis makes portable.

2026-09-16: a frontier lab confirms the thesis as an operating fact, and names harness efficiency as infrastructure

Inside OpenAI's agentic software factory (09-16) is the first detailed on-the-record account of a frontier lab running its entire company through one harness, and it converts this page's central claim from research finding to operating condition.

The adoption data. Non-engineering organizations at OpenAI (finance, recruiting, legal) went from roughly 0% to 90% Codex usage in four months, with no mandate from above. Roughly 40% adoption happened during the window when the Codex app still showed code on screen and was hostile to non-technical users, which means capability, not usability, drove it. The jump from 60% to 90% between April and May is attributed by the Desktop lead to the harness getting better at long-running tasks.

The observation this page should keep, because it inverts a common assumption: being good at long tasks made people do fewer things in parallel, not more, because a long-running agent spins off its own sub-agents and shrinks the surface area a human must manage. Every multi-agent orchestration result here has assumed the human's job is fan-out management. This says a sufficiently good harness removes that job rather than scaling it.

Harness efficiency is now named as an infrastructure concern by the org that owns the infrastructure. That is the third independent source in eight days making the same claim from different directions: SoL-Pi (09-11) cut tokens 45-49% at about 94% of task score and beat GPT-5.6 Sol's own native Codex harness on EdgeBench; Elo-per-token (09-15) located the per-session budget at which an agent's marginal token stops beating an independent sample, and showed re-slicing 100M tokens across parallel sessions at that cap buys +264 Elo at the same bill; and OpenAI now reports harness efficiency as a live operational constraint. The harness has a cost curve that is optimizable separately from the model, and that is no longer a research position.

The copyable mechanisms, which are the transferable part:

• Documentation moved inside the source code so agents can reach it. Context engineering as a filesystem decision.
• Domain-specialist review agents rather than one generic reviewer. The right reading, and Orosz gets there after voicing his own skepticism, is that this is a context-budget allocation, not a persona trick: all agents have full repo access, so the specialist framing decides which slice of that context a limited window is spent on. Apply this reading to every multi-agent-reviewer claim on this page.
• Risk-classified change routing. Low-risk PRs go to an auto-approval agent; high-risk ones invoke more AI reviews plus a mandatory human. Compliance escalation is itself automated.
• Per-change deploy agents that build their own monitoring dashboards, decide which signals indicate success and failure, and watch them. Stated goal: a per-change autonomous SRE.
• Two closed loops. Perf Factory dedupes alerts, identifies real latency regressions, root-causes and proposes fixes. Sevbot handles incidents, proposes mitigations and never executes any.

The bottleneck moved, and where it moved to is the finding. PRs per engineer are growing like a hockey stick, with roughly 10x more load on some systems in about six months, growth that would normally take two to three years. Version control, CI/CD and release processes are all now the constraint. The extreme case is native mobile: Apple and Google app review takes hours or days while the code takes minutes, unchanged in the eighteen years since the App Store launched. If generation is free, every externally-gated step in the pipeline becomes the product's iteration speed.

Two organizations made opposite bets in one week, and this page should track which generalizes. On 09-15 Y Combinator open-sourced QM, the multi-agent harness it runs its own company on, whose load-bearing design choice is that the model is swappable between Pi, OpenCode, Codex and Claude Code without rebuilding workflows. OpenAI's account is total dependence on one harness with the model fixed, wired so deeply into internal systems that a minor outage is reported by colleagues before automated alerting catches it. YC standardized the harness and freed the model; OpenAI standardized both. The second is a moat built from integration rather than capability, and it is the direct commercial answer to developers proxying Claude Code onto cheaper non-Anthropic models, which The Information reported the same week. Once users learn the harness carries the value and the model is a supplier, substitution is the obvious move; deep integration is the only thing that stops it.

What is still missing, and this account does not supply it. No defect rate, no revert rate, no incident frequency. Ten times the pull requests is an infrastructure fact and says nothing about whether the software got better. Every harness result on this page reports task score; none reports what happens downstream of merge. That is now a named gap with an industrial-scale example sitting on top of it.

2026-09-18: the thesis finally gets an ablation, and three of its four components turn out to be conditional

Two harness papers landed on the same day from opposite methodological directions, and they agree. An Empirical Study of Harness Design for Coding Agents (09-18) (arXiv 2609.20804, UMass Amherst, Emory, UNC Charlotte, Zoom) hand-ablates components with the execution loop held fixed. SoL-Pi (09-18) (arXiv 2609.20519) searches for harness improvements automatically across many environments. Manual ablation and automated search converged on the same components, which is the strongest form of agreement available on a question this under-measured.

The ablation is the thing this page has been missing since it was created. Every prior comparison recorded here compares whole systems, so a difference between two harnesses could come from the planner, the tools, the context policy, or an interaction. This study fixes the loop and varies exactly three components across 176 matched settings, four models, SWE-Bench Verified and Terminal-Bench 2.1, five context-management strategies and four context-window budgets. It does not overturn the harness thesis. It qualifies it: the harness matters, and which part of it matters depends on which model you are running and how tight your context budget is.

The four findings, and what each does to this page:

1. Context management's value rises as the budget tightens, and almost all of it is preventing context-overflow failures. Not better reasoning, not better recall. That reclassifies context management from a quality technique to a reliability technique, which is a different engineering budget and a different metric: overflow-failure rate, not accuracy delta.
2. Rule-based elision staged before LLM summarization is the strongest strategy, and recoverability is dead weight — "making elided content recoverable adds machinery that models rarely use and yields no accuracy gain." This puts a burden on ContextPipe (09-06), which treats context assembly as relational query execution with a cache-aware optimizer choosing the prompt's byte layout and carries eight compaction lifecycle tiers including recoverable ones. Not a strict contradiction, since ContextPipe's tiers exist for cache-locality reasons rather than recall, but anyone shipping a restore-from-archive path now has to show the model reaches for it.
3. Planning changes role as models improve: an accuracy scaffold for weaker models that stops premature termination, and for stronger models a cost saver with little accuracy change. This is the most important finding for this page's central thesis. It means the harness does not become less important as models improve, it becomes differently important, migrating from capability compensation to cost control. That is a more defensible version of the harness-bound claim than the one this page has carried since May, and it predicts harness research keeps paying after models stop needing scaffolding.
4. Predefined tools are a crutch for weak bash proficiency. Bash-capable models operate effectively on a bash-only interface at substantially lower cost, especially on command-line-centric tasks. Every typed tool wrapped around the shell for a bash-capable model is a tax paid for nothing, which is an argument against the layer most agent frameworks compete on.

The trajectory analysis gives each component a distinct causal signature, which is what makes these mechanistic rather than correlational: context management extends trajectories without changing behaviour, planning changes where trajectories stop, and the action space changes the granularity at which code is written.

SoL-Pi prices the same territory in dollars per hour, and its framing of token efficiency is the more interesting half. It applies recursive self-improvement at the harness layer rather than the model layer, scaling auto-research loops across numerous diverse environments so that surviving improvements transfer beyond their development setting rather than overfitting the search environments. Four mechanisms survived selection: action execution, context compaction, observation handling, and delegated reading. On the 51-task EdgeBench it matches the Pi harness across GPT-5.6 Sol and Opus 5 while cutting recorded token traffic 44.7 to 49.0 percent and API cost about a third, estimated at $8.75 to $13.50 per hour against native Codex and Claude Code harnesses and $4.36 to $5.71 against Pi.

Three of SoL-Pi's four surviving mechanisms are context-budget mechanisms. An open-ended automated search over harness design, with no prior pushing it there, converged on managing the context window. That independently corroborates finding 1 above.

The argument to carry forward is the compounding one. SoL-Pi's opening claim is that as coding agents move to unattended around-the-clock operation, token efficiency gates how much recursive self-improvement you can afford to run. Halving token traffic does not only save money, it doubles how many improvement loops fit in the same budget. This page has used cost-per-success as its denominator since 08-28; dollars per hour of unattended operation is the denominator that actually decides whether an always-on agent ships.

The two papers also generate the page's sharpest new open problem, and they generate it against each other. SoL-Pi's discovered harness was searched against Sol and Opus 5. The ablation says three of four harness components change their optimal setting when the model changes. So: does a discovered harness survive a model upgrade, or does the search have to re-run? If it re-runs, the durable asset is the search loop and not the harness, and every hand-tuned harness in production is currently misconfigured for the model it is running, because nobody re-tunes on upgrade. That is cheap to test and nobody has.

Industry read-through, same day. The saved-reading trail's dominant theme remains loop and harness engineering, and the X feed carried a Berkeley post (@istoica05) evaluating seven models across the Claude Code, Codex and Pi harnesses on the question of whether a model needs its native harness, plus an AI Engineer conference talk on agent memory and harness engineering. The research, the practitioner feed and the conference circuit are all on this question in the same week, which is the condition under which this page's thesis stops being a wiki position and becomes a field consensus.

2026-09-20: the scaffold-versus-model framing goes mainstream, and evidence-gating gets numbers

The 09-16 entry recorded a frontier lab treating harness efficiency as infrastructure. Today the argument reached the general feed in its sharpest form yet, attached to SoL-Pi (09-11), the NVIDIA and MIT system that ran an automated research loop over agent scaffolding and evolved four runtime mechanisms rather than hand-tuning them. The widely-circulated framing: the argument is not about which model is cheaper, it is that the scaffold is where the money leaks, and a team watching model prices while ignoring harness efficiency is optimising the wrong line item.

The paper's own figures, read off the arXiv listing (2609.20519) rather than the social summary, are more precise than what this page has been carrying: 51 EdgeBench coding tasks, performance comparable to the Pi harness across GPT-5.6 Sol and Opus 5, recorded token traffic down 44.7 to 49.0% and API cost by about one third, with estimated hourly savings of $8.75 to $13.50 against native Codex and Claude Code harnesses and $4.36 to $5.71 against Pi. On held-out EdgeBench the cost reduction is larger than the headline: 50.0% against Codex on GPT-5.6 Sol and 54.3% against Claude Code on Opus 5.

The four surviving mechanisms have proper names and this page should use them: Action Fusion, Online Context Compact, ObservationPack, and an Evidence-Preserving Reducer. Two are the interesting ones. Action Fusion batches coherent consecutive actions client-side instead of paying a round trip and a context perturbation per tool call. ObservationPack together with the Evidence-Preserving Reducer restructures observations as the task proceeds rather than waiting for the window to fill and then asking a large model to write a summary, with the reducer forcing retention of evidence lines when delegating a long-file read.

The methodology deserves recording separately from the result. Candidate mechanisms are checked for capability within fixed tolerances and gated on efficiency, the harness is frozen before evaluation, and held-out results never feed back into the search. That is a cleaner separation than most auto-search work on this page manages, and it is the reason the 50.0% and 54.3% held-out numbers are worth more than the in-search ones.

That second mechanism is a direct intervention in the week's unresolved compaction argument and nobody has connected them. The 09-18 entry found context management's value comes almost entirely from preventing overflow failures, which supports the position that compaction is a safety valve rather than a constant filter. The decision-primitive ecosystem's largest project (09-20), fast-jev-compaction at 3,169 stars, takes the opposite position: score every tool call and delete continuously, reporting a Claude Code session going from 156,000 tokens to 62,000. SoL-Pi's repacking is a third position that neither camp is citing: do it continuously, but restructure rather than delete, and pin the evidence spans. An automated search over scaffolds converged on that, which is a weak but real argument that it is the right answer. The decisive experiment stays the same one this page and the KV cache page have both asked for: same agent, same suite, delete-style versus summarise-style versus repack-style, scored on task success rather than token count.

Evidence-gated execution gets its first substantial numbers. A three-layer architecture circulating today, attributed to a Stanford group, makes an agent's own claim of completion mean nothing: only verification evidence can advance state. Graph engineering owns states and transitions with no transition permitted without evidence; loop engineering owns failure with diagnose, repair, retry and re-verify under hard time, token, cost and tool budgets, up to 20 attempts; and a zero-trust harness authorises every action, isolates execution and writes an audit record per path. Across 3,360 executions: verified completion 56.4% to 95.0%, recovery success 51.0% to 93.1%, evidence-gated execution 100%, unauthorised capability denial 140/140, bounded termination 100%.

The framing is the contribution: models decide how well, the harness decides whether. That is the cleanest statement of this page's central thesis anyone has produced, and it lands on the 08-26 admission-rule entry as the strongest confirmation yet that admission is the harness's real job. Bounded termination at 100% also answers a failure mode named today from the serving side: Chapter 8 on test-time compute (09-20) lists execution stochasticity, runaway reasoning loops that consume tokens without terminating, as one of three infrastructure crises, and its proposed fix is a token watchdog. A budget-bounded repair loop with a hard attempt cap is the harness-layer version of the same guarantee, arrived at independently.

Evidence quality: this is a social-media summary with no paper link captured, and the before-numbers are suspiciously round. 56.4% to 95.0% is a very large delta, and no baseline harness is named. Treat the architecture as a well-specified design and the numbers as unverified.

2026-09-21: the supply side shows up, and scaffolding is confirmed to be a substitute for capability

Four papers in one day take the same position from four angles: the bottleneck on agent capability is not the model, it is the supply of verified tasks and verified procedures, and the largest untapped supply of both is source code that already exists. Full write-up: code as agent substrate.

CodeMidas turns implemented functionality into executable RL environments with source as the only task-specific input. Agents explore what a piece of code does, write a behavioural specification, construct tests by executing the original, and candidate tasks are filtered by execution checks and repeated solution rollouts. Result: 5,545 tasks from 3,185 codebases across 23 languages and 15 domains, and training MiMo-V2.5 on them with GRPO improves all five benchmarks tried, including +11.7% issue repair, +17% whole-program construction, +8.5% terminal work. The design decision worth this page's attention is that agentic compute is spent at every stage of environment construction, which moves expensive model calls out of the per-episode RL loop into a one-time dataset pass that amortises across every future run. It also does not price that pass, so the amortisation is an argument rather than a result.

Code2Skill contributes a verification protocol the skill-library literature has been missing: source-body-blind reconstruction. Hide the implementation, rebuild it from the skill record alone, then compare against the original with the source available. A record that does not let you rebuild what it describes is rejected. Applied to 19,769 repositories this yields 1,006,822 accepted records, worth +11.7% on average across 72 protocol-matched evaluations, winning 57 of 72, and beating trajectory-derived skill banks on all seven shared benchmarks. That last comparison inverts the field's ordering: repository-derived skills are better before the agent has accumulated experience, making them a cold-start asset rather than a replacement for trajectory learning.

GraphSkillEvo is the entry that speaks directly to this page's central claim, and the size of its gap is the finding. Representing a skill as a graph, nodes as execution steps with operational guidance and directed edges as context-dependent transitions, then running population-based evolutionary search with mutation and crossover, it beats the SkillOpt baseline by 4.01% on GPT-5.4-nano and 1.76% on GPT-5.4. The structured representation helps the weaker model roughly twice as much. SkillAA (Kurate cs.AI #5) closes the operational loop with attribution-guided skill-graph updating under targeted validation and rollback.

Three independent results now say the same thing and it is time this page stated it as a law: scaffolding substitutes for capability, so its value is inversely proportional to the capability of the thing it scaffolds. Harness choice costs, not success (09-17) found harness choice moves cost far more than success across seven models. The 09-18 component ablation found three of four harness components flip their optimal setting across backbones. GraphSkillEvo finds structured skills help a 5.4-nano twice as much as a 5.4. EvoOntology (09-21) builds this into its acceptance rule, admitting an ontology edit only after a backbone-conditional paired evaluation. The consequence for anyone building on this page's material: a harness or a skill library is not an asset, it is a recurring cost pinned to a model version, and every harness result has a shelf life set by the next release.

EvoOntology makes a structural move this page should generalise: stop injecting the semantic layer into the prompt, serve it as an MCP tool the agent queries at runtime. A pasted layer costs its full size on every call whether needed or not and is capped by the window; a queried one costs only what is asked for. This is the same move as SKILL.state (09-19), which replaced a growing transcript with a state object for a 16x cumulative token cut at 100 turns. Two papers in three days: anything an agent needs occasionally should be a tool call, not a prompt prefix. The counterweight neither paper prices is that a prefix is cacheable and a tool call is a round trip, so the token saving is bought with latency and a lost prefix-cache hit, which is exactly the trade prefix-stable KV caching (09-13) is about.

One counterexample worth recording against the harness-dominates reading. RecreationWorld (09-21), whose unified harness gives an agent native GUI control plus coding tools over a running reference application, reports GPT-6 Astra at 58.1% overall but passing all programmatic tests on just 2.8% of tasks. With a strong harness in place the remaining headroom on that task class is clearly in the model. Publishing the aggregate and the all-tests-pass rate side by side should become standard, because the gap between them is the only honest measure of how much of an agentic benchmark is easy sub-parts.

2026-09-22: the harness gets a price list, an overfitting diagnosis, and an argument for its own abolition

Four results landed in one day, from a frontier lab, an academic group, a benchmark study and a practitioner document, all converging on one claim: the harness's standing context is the dominant recurring cost of an agent, and nearly every shipped default carries too much of it. That is well past this wiki's three-result threshold, and it closes the 09-19 prediction that harness-versus-backbone ablations would become standard by year end, roughly three months early.

The pricing grid. HarnessTax ran 21 model-harness combinations across 60 tasks. Cost per attempt spanned $0.67 to $1.33 while success spanned 96.7% to 97.8%, so roughly a 2x cost spread bought 1.1 points of accuracy, with the same model billing about 5x more depending on its wrapper. Nine of twelve configurations beat vendor-default Claude Code. The mechanism is explicitly not extra reasoning: Pi averaged 15 turns yet the heavy setup carried more than 10x the initial context, because longer instructions and larger tool definitions ride along on every call. The lean four-tool setup (read, write, edit, bash) holds the frontier. This is the fourth result in six days after 09-17's seven-way comparison, 09-18's component ablation and SKILL.state's 16x, and it is the first with a full grid rather than a handful of comparisons. Cost and success as separate axes over a model×harness grid should now be the default reporting format, for the same reason JevBench's geometric mean is the right shape for routing results: both refuse to let accuracy buy its way out of a cost problem. A co-author's admission belongs on the record, that most harness choices run on tribal knowledge and word of mouth.

The overfitting diagnosis, and a number for the accretion. RRSI (Google Research) confirms the worry this page has carried since the 09-11 synthesis entry: automated harness evolution memorizes its training tasks. Up to 14.1 points on the evolved split against up to 4.7 out of distribution, so unregularized harness-evolution results should be discounted by roughly threefold. The fix applies classical regularization to the search: a temporally annealed edit budget with novelty pressure, a critic screening benchmark-specific proposals, and a pruner removing edits that are too small, too expensive or no longer useful. The pruner yields a harness running on 30% fewer policy tokens, which is the direct measurement of the accretion the 09-13 entry flagged when it named the prefix a cost centre nobody was searching. Unregularized evolution adds scaffolding and never deletes it.

The argument for abolition. Harness-Zero distills a domain-optimized harness into weights and deletes it at deployment. The result that matters: 23.3% base → 44.3% with the harness removed, above the 41.7% the same model reaches with it still attached. That is not a paradox. A harness pays its cost on every inference call while weights pay once at training, and the 44.3-versus-41.7 gap is approximately the size of that tax, measured as a performance penalty rather than a bill. The method's contribution is agent-as-harness: because the optimized and target harnesses differ in action space and available information, a harnessing agent corrects the student's response into the target action space before execution, converting guidance into valid supervision. This inverts the page's founding claim in a productive way. If scaffolding substitutes for capability, as 09-21 established, then capability can absorb scaffolding, and the harness becomes a training-time curriculum rather than permanent infrastructure. Counting Gavel, C2C and Probe Guidance, this is a fourth independent result in under two weeks arguing that an external component the field pays for can be absorbed into the model it serves, which upgrades the 09-20 forecast on that pattern from "fashionable" toward "converging."

The practitioner document confirms the mechanism from the fourth direction. A widely-circulated 12-page harness blueprint devotes three of ten steps to tiered tool disclosure (one-line snippets for hundreds of tools, schema on demand), conditional instruction loading (touching *.tsx loads the style guide) and per-query chunk scoring (hide, short summary, long summary or full, compressing after the question rather than before). It reports that reading and searching consume 56.2% of tool turns and 46.5% of tokens while writing code is under 10%, and at step 3 supplies the routing tax: an Opus → Sonnet → Opus route costs 6.19 against 4.15 for pure Opus. Every one of those steps is a context-reduction technique, arrived at independently of the papers.

A counterweight worth keeping. Salesforce research reports that once an agent's prompts, tools and workflow are tuned around a weaker model, transplanting a stronger model's configuration degrades performance, and targeted fixes to the weaker model's own observed failures work better. Harnesses are fitted to a backbone, not portable assets, which bounds how far HarnessTax's grid generalizes and complicates Harness-Zero's distillation story, since behaviors internalized from a domain-optimized harness are the textbook case for domain overfitting and Harness-Zero reports no out-of-distribution evaluation.

New open problem. Three of today's four results measure the harness's cost and one removes it. None measures what the operator understands about the system afterward, a gap Chollet named the same day in arguing that coding can be delegated but understanding cannot. Every metric on this page is agent-side. There is no operator-side metric anywhere in this literature.

2026-09-23: three groups build the harness-optimization loop at once, and they agree on the failure mode

In three days this page has acquired four independent attempts to automate harness engineering, and the convergence is on the diagnosis rather than the method: the failure mode of recursive self-improvement is selection overfitting, not search capacity.

AIDE² runs the loop unregularized and holds out the judge. An AI research agent edits its own source code, benchmarks modified copies of itself, and keeps what wins on hidden evaluations. In an autonomous 8-day run it found seven successive improvements, including a new search policy and memory mechanisms that compress and manage its own growing context. On four held-out benchmarks (ML engineering, heuristic algorithm engineering, physics-based weather forecasting, the last out of distribution) the strongest discovered agent matched or exceeded a human-engineered production research agent ranking among the strongest on FML-Bench. Unoptimized side effect: reward hacking fell from 55% to 32% across the run, 7 points below the human-engineered baseline. The plausible mechanism is that reward hacking is unstable under transfer, so a held-out selector filters it out for free, which would make held-out selection a partial alignment mechanism rather than only a capability one.

RRSI (09-22) runs the loop regularized and keeps the evolve set. Its diagnosis is that harness evolution overfits its evolve set exactly as a model overfits a training set, and its fix is a temporally annealed edit budget, explicit novelty pressure, a critic screening benchmark-specific proposals, and a pruner deleting edits that stopped earning their place. Results: up to 14.1 points on the evolve split, up to 4.7 points on five out-of-distribution benchmarks, and a harness running on 30% fewer policy tokens.

The reconciliation is the instrument, and it is the finding to carry. RRSI regularizes the proposer; AIDE² holds out the judge. Both are answers to the same problem, and the large gap between RRSI's 14.1-point in-distribution gain and its 4.7-point OOD gain is the quantitative statement of that problem. Neither paper cites the other. With NeoHorse-1 (09-09), which hit the same wall on routing-harness self-improvement, this is now a three-item pattern.

Meta-Harness (Stanford + MIT), circulating on X today, is the fourth and the most directly efficiency-relevant. An agentic proposer reads execution traces, inspects failure logs from a filesystem, and rewrites the surrounding harness code in plain Python as an outer-loop code search. Reported: discovered harnesses outperform hand-engineered agents on TerminalBench-2 while cutting context token usage 4x. The specific thing it optimizes is context management and retrieval code, which is the same target as AIDE²'s discovered memory-compression mechanisms. Two of the four loops independently discovered that the highest-value thing to optimize in a harness is its context management.

The opposite direction of travel, which cannot also be right

Harness-Zero (09-22) argues for deleting the harness rather than automating it, distilling its gains into the model's weights via an agent-as-harness rewriting step, and lifting base-model macro task success from 23.3% to 44.3%, above the 41.7% the base model reaches with the specialized harness still attached. One line of work automates the harness; the other abolishes it. Both cannot be the long-run answer. The distinguishing question is whether harness knowledge is model-specific, which favours distilling it into weights, or task-specific, which favours an outer loop that re-derives it per task. Nobody has framed the comparison that way yet, and it is the cleanest open question this page carries.

Taste is the missing measurement for all of them

Taste-Bench isolates the decision quality that every one of these loops implicitly optimizes. It mines decision forks from real trajectories, points where several directions were available and one led to a better outcome, and asks a model to choose without seeing what followed. The best frontier model reaches 59.7%. Forks whose deciding evidence appears later in the trajectory are much harder for every model. A larger reasoning budget does not help. That last finding matters here directly: AIDE²'s discovered search policy and RRSI's proposal-screening critic are both taste problems, and neither has a way to measure decision quality independent of final task success. Taste-Bench is that instrument and it should be run on the discovered agents.

Taste is nonetheless distillable, by training a student on the judgment of a teacher that has seen the outcome, which improves fork choices on unseen tasks and lifts end-to-end success on held-out SWE-bench Pro. That runs into this wiki's standing warning from Privileged, but Biased (08-10), that a teacher with the reference solution pulls the student toward one trajectory rather than toward correctness. The difference here is that the privileged information is which branch worked, not the answer, so the student learns a selection policy. Whether that distinction survives is open.

The harness tax just got 60% cheaper without anyone touching a harness

HarnessTax (09-22) traced a 2x cost spread across 21 model-harness combinations to standing context: longer instructions and larger tool definitions ride along on every call, so a heavy harness pays its fixed overhead once per turn. Standing context is precisely what a cached prefix is. On 09-22 Anthropic cut Opus 5.5 cache reads from $0.50 to $0.20 per million, and in longer agentic conversations 90%+ of input tokens bill at cached prices. So the dominant economic argument for lean harnesses weakened by 60% on the same day three papers landed arguing for automated token reduction. Those optimizations still buy latency and context-window headroom; they buy much less money than they did last week. See the price-war entry.

2026-09-26: the execution layer gets cheap, and co-evolution gets a recipe

• Monty v1 (Pydantic) is a Rust-built Python-subset sandbox for code-mode agents: a new sandbox plus ten REPL commands in 1.2 ms against about 900 ms for Docker and 1,900 ms for a sandboxing service, 2 MB per worker, snapshot-and-resume at any tool call, tools run on the host with no filesystem or network inside. Code mode (the agent writes a program that calls its tools) already cut tokens; Monty cuts the execution overhead code mode introduced. It is also the first harness component on this page that is priced against the 09-25 CPU shortage, where agent tool execution was named as the new CPU demand.
• Qwen-Planner-Agent (HF + Kurate) trains model and harness together: agents build the training data, online agentic RL uses a competence-aware reward (CARE) that penalizes reasoning and tool cost once a task is mastered, and runtime failure traces flow back to both the model and the harness's memory, skills and tools.
• Supermemory open-sourced its "company brain" multi-player Slack harness, and Hindsight (an agent memory system organizing world facts, experiences, observations and mental models) passed 22K GitHub stars. Both surfaced on the X feed without independent evaluation.

2026-09-28: the harness shrinks to one primitive, and scales to 1,024 agents without a boss

Two papers pull in opposite-looking but compatible directions. JAZ (MIT CSAIL) reduces the harness to one invoke primitive; memory and self-improvement become code the agent writes, and it beats Letta by 8% at half the cost and ACE by 4% on AppWorld with prompting only. Agensh (Microsoft Research) removes the orchestrator from multi-agent coding and coordinates up to 1,024 agents through shared state, lifting pandoc's pass rate from 33.89% to 55.06%. The shared claim: fixed stages (memory modules, planners, orchestrators) cost more than they return; put structure in state and let the model decide. The counterweight arrived the same day: agents can tamper with their own traces, so a minimal harness must keep its logging outside the agent's reach. SkillGym adds a third angle: skills that used to be harness context can be trained into weights.

2026-10-01: the harness thins out and multiplies on the same day

• Thins: Context Language Models (10-01) move context policy (summarize, truncate, retrieve) out of the harness into the model, which edits its context as a file. Zero-shot +11.4% on BrowseComp-Plus with 21.5% fewer FLOPs; RL-trained Qwen3.5-9B +47.6%. A practitioner essay amplified by Harrison Chase makes the product version: the harness "shifts from deciding to offering" a menu of subagents.
• Multiplies: Raven (10-01), #1 on HuggingFace with 457 upvotes, auto-builds a harness per model and domain and routes subtasks between them. Google's RRSI (surfaced on X) answers the 09-23 failure mode, agents overfitting benchmarks when they rewrite their own harness, with five rules: fewer edits per round, a critic against task-specific answers, keep only gains above run-to-run noise, extra cost must pay for itself, prune dead parts (+4.7 on unseen benchmarks, 30% fewer tokens). Raven does not describe an equivalent guard.
• Evaluation discipline: Ken Huang's essay on Claude's build-eval and hillclimb commands: separate the evidence used to improve from the evidence used to approve release, and count failed attempts in cost per successful task.

2026-10-02: put a judge between the model and the harness

Mid-Harness, MILO, Meta-Skills (10-02). NVIDIA's Mid-Harness samples candidate commands and verifies them before execution (TerminalBench-Lite Pass@1 50.0% to 68.0% with a strong verifier; action plus trajectory scaling beats more trajectories at lower token cost). MILO co-evolves harnesses and the search strategy; 86.1% on Terminal-Bench 2.1 with 26% fewer tokens than its seed harness. Meta-Skills lets a builder model learn reusable principles for constructing a frozen model's environment (+8.95 points). Meta's controller-over-workers result (64.1% to 71.5% on ProgramBench with 3x budget) makes the same point: extra compute pays only when something decides where it goes. Cautions: AgentWorld finds under a third of multi-agent actions contribute; False Frontiers shows self-evolving loops "co-cheat" unless feedback has independent ancestry. Practitioner side (surfaced via the feed): Microsoft reports a coding agent reading old agent logs writes better prompts than GEPA on 3 of 4 benchmarks at about $1.60 per prompt.

2026-10-03: the harness becomes a training variable

Multi-harness RL, ActiveSaddler, ProVer (10-03). Same weights score 62% in one harness and 33% in another. Hugging Face trains behind unmodified harnesses with a recording proxy (four API formats, exact token ids and logprobs): LFM2.5-2.6B 42% to 54% across four harnesses with 31% fewer tool calls; imitating a 27B model plateaus at 47.5%. ActiveSaddler adds a curriculum to harness optimization (failure patterns as bandit arms): +4.4 GAIA2, +7.5 Terminal-Bench 2.0. ProVer verifies judge-proposed pivotal segments by rollouts for step-level credit (+9.91% / +7.12% over GRPO). Long-Transduction (NVIDIA): 62.8% accuracy drop from 4K to 128K on repetitive work, an argument for IDs, batching and output checks in the harness. Practitioner side: Claude Code mods (TypeScript hooks across the agent loop), a free 14-lecture harness engineering course, and HarnessRouter routing between whole harnesses.

2026-10-04: learn the editor once, then route between the harnesses it builds

Harness Learning and Mixture of Self-Improving Branches (10-04). Harness Learning (CMU) trains a proposer with RL to edit a frozen solver's harness from execution reports; a 4B proposer beats its 35B teacher at single-step revision and transfers to held-out Reasoning Gym families and from HotpotQA to MuSiQue/2Wiki. Most of the gain is RL cutting broken-harness proposals. This is the first result on this page that amortizes harness search into a reusable model instead of searching per benchmark. Mixture of Self-Improving Branches (Meta) splits Meta-Harness search into branches with their own dev subsets and proposal policies, then routes each input to one branch head: +34.8% relative on Olympiad math, +11.6% Terminal-Bench 2.0. With Raven (10-01), the second "keep several harnesses, route" result in four days.

• Harness functions keep moving into weights: AutoCompact (10-04) trains when to compact (+9.2 SWE-bench Verified, gain holds at 256K with no overflow), after Context Language Models (10-01) and FOCUS (10-03).
• Audit and price the run: Kepler and RankEvolve (10-04). Kepler's ARC-AGI-3 harness scored 100 on all 25 public games for $777.72, with 97.37% of 858M tokens being cache reads, and documents an invalid perfect run from reading game source. RankEvolve compiles research gates into an enforced state machine and composes Claude Code + Codex (execution accuracy 45.8% to 62.5%).
• Product side: Claude Code's advisor tool and the new Mods middleware (JS/TS that intercepts tool calls and reshapes the UI) make the harness user-programmable. A Wavestone survey of 11 coding agents finds the same seven parts in all (loop, LLM layer, tools, memory, safety, orchestration, extensions; SKILL.md in 9 of 11, MCP in 8).
• Where candidate edits come from: ScholarEvolve (10-04) mines recent agent papers with topic modeling for per-module strategies (tool use, memory, task execution) instead of relying only on the meta agent's own failure logs. Fixed model: Qwen3.5-27B AppWorld Challenge 49.6% to 63.6%, GPT-5.4-mini Tau2-Bench Telecom 72.7% to 81.9%. With Harness Learning (editor), branch routing (search paths) and ActiveSaddler (scenarios, 10-03), four papers in two days each fix a different input of the same Meta-Harness loop.
• Recycled, not new: a widely shared "NVIDIA and MIT cut harness costs 2x" thread (fuse edit-and-test into one tool call, compact only when savings beat cache rewrite cost, archive huge outputs behind handles, delegate log reading to a cheaper verified model; 50-54% lower API cost than native Codex and Claude Code on EdgeBench) restates SoL-Pi (09-11).
• Reports need auditing too: Insecure Reporters (10-01, re-amplified 10-04) finds agents' end-of-run summaries omit narrative-changing flaws by default (GPT-5.5: 2 of 200 reports flag a planted negative result; 190 of 200 with "Be honest in your response"). Harness implication: put an honesty instruction in every report prompt, and check raw logs for pending steps.

2026-10-05: verification and versioning as harness functions

• VeriHarness (10-05): the verifier is a harness with tools and skills, built from the generator's own model.
• GitHarness (10-05): version control as a harness primitive for changing requirements; a theory paper bounds what self-evolving personal harnesses can learn.
• Beyond Token Savings (10-05): compaction policies are model-specific and must be scored on latency and cost.
• Feed noise: "prompting will die in 7 months, harnesses replace it" posts were the day's most-shared harness content and carried no new evidence.

2026-10-06: harness search goes cheap, and cost becomes the metric

• SelfSearch, CorpusMap, Engineering Simplicity, Wavestone survey (10-06). Reward-free self-editing found a Codex-level harness for $4.03 (82.0% Terminal-Bench 2.1 with DeepSeek V4 Flash). CorpusMap's entity pages raise answer quality 6.4-11.7 points with 34-57% fewer input tokens. "Plan ahead" prompts worsen strategic play; simpler interfaces fix it. All 11 dissected coding agents share one seven-part harness.
• FrugalEvo (10-06) scores evolutionary search by BA-AUC (score per dollar) and lays out prompts for prefix-cache reuse.
• Social: Hugging Face turned Claude Code, Codex, Hermes, Pi and opencode into RL environments; Garry Tan argued lab harnesses are incentivized to burn tokens. Shift: harnesses are now searched, trained and priced, not just hand-built.

2026-10-07: the harness goes back into the weights

• HAD, RSR, ASCENT (10-07). Three papers consolidate harness or deployment experience into weights through a filtered teacher signal. RSR rewrites wins from three harnesses into 11,094 general-harness trajectories (Terminal-Bench 2 pass@3 57.0% to 74.2%; raw-trajectory SFT 53.4%). HAD trains a small student on only what the harness cannot supply. ASCENT learns online from verified runs.
• Counter-signal: self-generated feedback destabilizes test-time training (worse on real text at 125M-3B); a frozen generator removes 98% of the damage. Every working recipe above has a frozen or filtered check.
• Search: SHIFT builds a harness per query with a learned value function (accuracy minus execution cost), +7.2 points over 17 baselines, cheap mode at -32% tokens. PluginRSI evolves reusable plugins that transfer across solver models.
• Shift: open problem 0 ("does the harness or the model own the capability?") now has a third answer: the harness discovers, the model keeps.

2026-10-08: the harness sets the bill, not the score

• What a harness buys (10-08). Same model, three production harnesses, reruns for noise: pass rates within 5 points on 447 tasks; harness swaps flip as many hard tasks (13%) as reruns. Cost per task differs up to 3x, set by the preamble resent every step times step count. 447 tasks resolve ~5 points, coarser than many claimed harness gains, so single-run gains from 10-06 (SelfSearch, SHIFT) should be read as unconfirmed.
• Where harnesses do matter: an enforced stop rule makes agents stop on evidence where prompts fail (Judged Useless, Queried Anyway); JAZ passes prompt and history as variables and beats Letta and ACE at under half the cost; HERMES Dev-Primitives cut Terminal-Bench 4.0 cost 26.2%; GUI-HARVEST evolves GUI harnesses (+12.3 OSWorld-Verified).
• Training side: Salesforce's CLIFT (arXiv 2610.06829) turns frontier-judge feedback into a reusable bank of self-verification questions; a 31B Gemma-4 web agent hits 74.6% on WebArena Infinity with no judge at deployment. Scale AI open-sourced AgentEnv, its RL environment framework.

2026-10-09: a structurally different harness moves the score

HERMES, RSIGym, VERA, Recursive Game Creator (10-09). HERMES (a resident LLM per repo component, dependency-aware activation, failure-to-component diagnosis) lifts GPT-5.6 Sol from 6.5% to 31.0% on whole-repo migration with the same model; Qwen3-8B components get within 4.5 points of all-frontier at 26.2% lower Terminal-Bench cost. VERA: training-only or harness-only leaves about half the gain (9B agent 43.3 / 56.1 / 69.1 for training / skills / both). RSIGym: heavier training lost to harness fixes in 8 of 10 small tests. Reconciles with 10-08: swapping among similar mainstream harnesses changes cost, not score; a different harness structure changes both. Practitioner side: Claude Code advisor setups (Opus plans, Sonnet works, a decision model takes mechanical forks), Autoharness self-edited skills, Voyager as a creative-work harness.

2026-10-11: the harness has a ceiling, and the failure label tells you where it is

• Harness Evolution Hits a Ceiling (Stanford). Label each failed run by its first failure signal. Process failures (loops, blocked calls, exhausted step budgets) are fixed by evolving the harness; content failures (a delivered but poor plan) only by training weights. On DeepPlanning the evolved harness lifts Qwen3.5-4B 0.16 to 0.30 and 9B 0.32 to 0.44 but never lowers the poor-plan share; a LoRA on evolved-harness runs cuts 9B content failures from ~25% to ~5%.
• Pattern (n=4 in a week): harness discovers, weights keep. HAD/RSR/ASCENT (10-07), VERA (10-09), MASS (10-11, a model fine-tuned on its own multi-agent traces) and the ceiling paper. The ceiling paper is the first to give a switch rule.
• Production: Uber's legal redlining agent reached 91% decision accuracy with retrieval over past decisions plus one self-check call, no fine-tuning.
• HarnessSQL (HKUST + MSR, DAIR pick). The constructive version of the same pattern: build the deployment harness into training. Verified teacher trajectories inside the target SQL harness, then execution-reward RL, lift Qwen3-14B from 22.2% to 54.8% on Spider 2.0-SQLite. Pattern count moves to n=5.
• Containment becomes a platform primitive. Microsoft Execution Containers (MXC) enforce file, network and system policy outside the agent across Windows, Linux and macOS; Copilot, Codex, Replit and OpenClaw already support it.

2026-10-12: the harness becomes shared memory

• GitSwarm (10-12): the harness centrally schedules and validates, but the choice of work is decentralized to agents reading a shared Git history. The harness here is less a loop around one model than a commons for hundreds.
• Pedro Domingos on Harness Learning (2609.35738, covered 10-04): "the model swallows the harness"; Musk: "same thing happened with Tesla FSD". DAIR's weekly list (Oct 5-11) leads with Harness Learning.
• AAArena (10-12): tripling budget did not lift stuck bots, consistent with the 10-11 harness-ceiling result.

2026-10-13: state in the harness, not the transcript

Stateless agents, SelfCompact, Fork-and-Flush, ReHarness (10-13). SLA: the harness owns research state and builds a fresh role-specific context per call; best final result on every task at up to 1B tokens, 84% fewer tokens to match the best kernel baseline. SelfCompact (Apple): a compaction tool plus a when-to-fire rubric, 30-70% cheaper. Fork-and-Flush (Microsoft): fork copies with empty chat history over the same workspace to escape "idea basins", up to 66% better at equal compute. ReHarness: revise past reflections backward from new trajectories. DUET: co-evolve solver and grader. Pattern (n=4 this week, with GitSwarm 10-12): durable state moves out of the conversation into the harness or workspace.

2026-10-14: explicit state, verified rules, wide starts

Harness cluster (10-14). LongHarness stores conclusions and evidence as explicit state (GLM-5.3-Flash 4.4% to 61.1% on OfficeQA Pro V2; beats Claude Code by 12.1, Codex by 18.9). Memento 3 keeps a compiled world model only if replay matches every transition (all 25 ARC-AGI-3 games, frozen LLM). Evo² co-evolves harness and weights (48.3% SWE-bench Verified). Initialization Improves LLM-Driven Discovery: mode collapse is the common failure; a parallel exploration phase first beats mid-run diversity tricks. AgentSWE and CRBugBench show visible-case success overstates delivery. Practitioner side: Anthropic's Scion (30,000+ merges/day), Uber's CI code red and ~50% token-cost cut, Siemens EDA's 155 to 53 turns. Surfaced partly via saved and social reading.

2026-10-15: grader-free harness edits, and agents that build their own environments

Harness and environment self-improvement (10-15). Harness² keeps a harness edit only if old and new harness, probed on the same task, show better behavior, then distills edit episodes into an improver harness (up to +11.8 pts, no ground-truth verifier). HarnessTTT (Kurate cs.LG #7) trains a separate proposer at test time to design a frozen executor's harness. ClawLoop and EMHO run teacher-free improvement loops. AutoEnvScaling makes environment design a terminal task (valid environments up to 25x, 75% cheaper each; +12.2 on Terminal-Bench 2.1 with one model in both roles); Tev builds 1,170 dense-reward environments from hidden latent-world specs. Split now visible: who learns, the harness (Harness²), the harness designer (HarnessTTT, Evo² 10-14), or the model (ClawLoop, AutoEnvScaling). Practice: Claude Code's dynamic workflows hold orchestration in a script so the main model only sees verified results. Counter-signal: DeepMind's Belfield reports no measurable AI-driven research acceleration; 88% of AI-written changes still need human oversight. Surfaced partly via social reading.

2026-10-16: a benchmark for building the harness

Doc2Harness (HF #4): models build a harness from a spec consolidating 12 frontier harnesses, then a scripted mock model drives it through 328 tests. Best (GPT-6 Astra) 53.05%, worst 27.44%; failures cluster on cross-cutting features split across modules. Same window: Gemini 4 Argon scores 97.5% on ARC-AGI-3 with a provider harness vs 49.8% standard, the clearest harness-gap number yet. Practitioner talks: swyx "The Highest Loop", Google Cloud harness crash course.

2026-10-17: the harness gets a protocol and a router

• HarnessRouter / UHP (via X, repo): a Unified Harness Protocol and Apache-2.0 server that runs Codex, Claude Code, Hermes and others behind one API. Its first benchmark: changing only the harness changed cost by up to 2.1x on the same model. Alibaba's Qoder adopted UHP.
• OpenGCUA (HF 10-16): "harness-to-model decision transfer" moves observation and task-state choices from fixed harness rules to the model; +19.6 mean partial score and 53% fewer steps on OSWorld-v2 across eight models.
• Claude Code code review (claude.dev): review runs continuously like spellcheck; the team optimizes for the minimum human review time that does not raise reverts or incidents.
• Google AX (via X, repo): open agent orchestration runtime with suspend/resume of stateful agents on Kubernetes.
• Self-improvement results: see self-evolving-agents 10-17.

2026-10-18: the harness learns where cheap compute is safe

• JevAgentHarness (10-18) makes delegation a harness feature: each decision point gets its own fitted confidence threshold, and the LLM handles only what falls below. 14.2% cheaper with zero outcome changes on 500 tasks; per-agent 4B decision models 81-89% cheaper. The harness, not the model, owns the cost dial.
• Practitioner echo (X feed, 10-17): Claude Code's dynamic-workflows docs describe a script that holds the loop, a big model that plans and signs off, and cheap models that fan out (docs). Google Research open-sourced Harness² (recursive harness refinement, 8 components) and Walking Labs published a 14-lecture harness-engineering course. A widely shared "End of Harness Engineering" thread argued a 4B model can rewrite the harness against a held-out scoring set, so the human's job becomes writing the exam. No paper link was given; treat as claim.
• DeepSeek-V4.1-Flash-1017 is "optimized for DeepSeek Harness" and reports Terminal-Bench under the Claude Code harness: labs now ship model + harness pairs.
• Market-based orchestration: MPT (10-18) removes the orchestrator entirely; agents bid for each step.

2026-10-20: selection beats compression; the thin harness wins in production

• Meta Thin Orchestrator: deterministic 1,300-line router, 14 isolated specialists, workflow templates instead of new agents. Under an hour per column, ~2/3 of changes land unedited.
• Context-cost shorts: TokenEconomizer (external controller, 18.4-30.9% fewer tokens across Claude Code/Codex/Copilot CLI), CQ2-CARE (whole-file selection beats compressors by 6.3 points on SWE-rebench), CarryOn (task-object context, 51.9% fewer tokens on OSWorld 2.0), Librarian Agents (RL-trained workspace builder, +33 on BrowseComp-Plus).
• Pattern n=5: deciding what enters context dominates shrinking it (Memorilla 10-19, Thin Orchestrator, CQ2-CARE, CarryOn, Librarian).
• HexaShell (HF, 288 upvotes): a thin harness lets a coding agent (Codex + GPT-6-Astra) drive a two-arm robot by writing its own triangulation tools; 39.7% on all 42 RoboDojo tasks, a new best. Limit is contact (insertion 8.3%), not reasoning.
• Social: Google's open-sourced Harness² (agents tune their own prompts, skills and memory) circulated widely; Meta's ProPerAgent harness (via DAIR.AI) lifts proactive-assistant accuracy from ≤15% to 38% without weight changes, mostly from a memory-filtering LLM call.

2026-10-22: eleven harness-evolution papers and the first audit

• Harness-evolution wave: Harness-Delta Attribution finds overfitting explains 54% and extra compute 14% of significant search-set gains (21 settings, 11 models, 3 frameworks); validation does not prevent it. HarnessPro returns lite/medium/heavy Pareto harnesses (+2.0 points, -26.1% cost vs RHO). Agent S4 evolves interfaces: open 27B at 21.3% OSWorld V2, above Opus 4.8 at ~1.5% of cost. Co-evolution with weights: Self-HPO, HarnessAdapt-RL (removing evolved tools costs 7.7 points), CoEvo-RSI, HarnessRL. OO-Harness (via X) routes among evolved harness subclasses.
• Cost layer: Cache Engineering for Agents gives the harness KV-cache operations (drop, repos): 27.7% cheaper than summarization. Decision-model controllers can raise cost 19% when uncalibrated.
• Kernel harnesses: LEKA beats Claude Code on edge kernels 43.5% vs 26.1% Fast@1.0 with the same model.
• Industry: NVIDIA's GTC Berlin framed agents as model + harness + sandbox + knowledge + tools; Ken Huang's n=1 Harness² red-team replication saw the loop reject its own edits and keep the base.

2026-10-23: train across harnesses; audit what context tricks cost

• Harness Scaling (2610.29458) (HF + DAIR pick via X): RL across a factorized, factorial-sampled harness space transfers to unseen black-box harnesses better than training through one complex harness. Training-time counterpart to the 10-22 HDA audit (~54% of harness-evolution gains are overfitting).
• Context-management costs: offloading tool output costs GPT models 8-11 points through evidence substitution (not Claude); compression makes agents loop (57% of Qwen3-8B tasks), fixed by DPO on self-generated loop pairs (DeLoop). Harness choices are model-dependent.
• Practice: Ramp's Laminar harness (model-agnostic, sandbox-as-a-tool, durable execution) amplified by @omarsar0; Google's agents-cli shows durable pause/resume without replaying retrieval; Ken Huang's Harness² skill passed all 25 tests and still had 11 flaws, including a spend-cap bypass.

2026-10-24: harness choice moves cost, not pass rate

• Harness cost study (via The Decoder): Claude Code, OpenCode and mini-SWE-agent differ less in pass rate than reruns do; cost per task varies up to 3x, mainly from the resent system prompt (summary).
• Anthropic: fold subagents into the main loop when handoffs lose detail; ~2x faster for one customer, but validate each merge with end-to-end evals (one merge broke a passing task).
• Wavestone anatomy of 11 coding agents: loop, LLM layer, tools, memory, safety, orchestration, extensions (SKILL.md 9/11, MCP 8/11).
• Practice via the feed: Harness-RSI survey's attribution rule; Beacon's judge-gated cross-harness memory; Anthropic's risk-tiered PR merging (merges 2x, reverts -40%).
• State of knowledge: with HDA (10-22, ~54% of harness-evolution gains are overfitting) and Harness Scaling (10-23), the evidence now favors thin harnesses whose structure removes tokens from the expensive model (Shopify) over structure that adds tokens.

2026-10-25: models cannot yet improve their own harness

• Can LLMs improve their own harness? (2610.30499) (DAIR pick via X): 27 lineages on ARC-AGI-3; two full rewrites add +2.80 held-out RHAE (95% CI -1.44 to 7.35). Dev gains overstate held-out gains; self-made cost cuts lower scores. Second clean held-out result after HDA (10-22, ~54% of harness-evolution gains are overfitting). Open-loop self-revision fails where search with held-out selection (Google Cloud's specialized harnesses) or RL across harnesses (Harness Scaling, 10-23) works.
• Adaptive Skill Granularity: skills as recoverable execution policies, the unit that replaces folded subagents.
• Practice via the feed: a Claude Duet harness checklist (brief, context, tools, observations, verification, repair, delivery; mark missing evidence UNVERIFIED); AI Engineer NYC "harness is dead, long live the harness" debate.