Agent Harness Engineering (Loop, Harness, Graph)

Concept page. The structured execution layer built around a foundation model that turns raw model capability into reliable long-horizon behavior. This page unifies the research thread and the practitioner thread, which have converged on the same claim from opposite directions: the harness, not the model, is now the primary object of design, evaluation, and cost.

Provenance note: this page merges published research (digest sources) with a large cluster of practitioner articles surfaced via saved reading in 2026-08. Specific saved items are tracked in the private curation index; the durable knowledge and paper sources are recorded here.

2026-09-02: the line this page said nobody had drawn, drawn as a design commitment

The open question from 08-31 is answered, and the answer is one sentence long. That entry recorded a genuine contradiction in this page's own evidence. AI4AI at Test-Time (08-13), where a builder model's synthesized harness lifted a weaker target from 0.49 to 0.91 on four Theory-of-Mind benchmarks with weights untouched, and Spark-to-Paper (08-13), whose integrity stack raised fabrication detection from 14% to 92%, both concluded that the harness wins by taking decisions away from the model. ContextPilot (08-31) did the opposite and worked, handing the model more discretion over its own context and training that discretion with localized RL. This page said both cannot be the general rule, proposed a reconciliation nobody had stated (the harness should own decisions with checkable correctness, the model should own genuine judgement calls), and called it "the right next position paper."

Harness-of-Harness (09-02) (arxiv 2609.01481) states it as design commitment four: "constrain verifiable outputs rather than prescribing agent workflows." The harness specifies what the artifact must satisfy; how to get there is the model's. That is the proposed line, arriving as an engineering commitment rather than an argued position, and it comes with the transfer evidence the claim needed: a 52.25% average relative gain (82.86% maximum) over the corresponding standalone harness after three iterations, holding across three different harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, Pi with MiniMax-M3) on GameCraft-Bench, FrontierSWE and ProgramBench. Transfer across pairs is what separates a design principle from a tuned configuration.

HoH is also the first meta-harness in this page's timeline. It operates on existing coding-agent harnesses, organizing their executions into iterative planning-coding-testing loops. That makes it the natural successor to HarnessOpt-Bench (08-07), which made "can a model optimize its own harness" measurable, and to AutoDesign (08-14), where one discovered harness lifted seven model-agent configurations. The scope is larger than either: not optimizing a harness for one task, but running other harnesses across a multi-day horizon while the artifact under construction grows. Its 70-plus iteration deployment, producing a first-person shooter with storyline, implemented mechanics, playable experience, visuals and audio, is the longest autonomous software-development run recorded in this wiki.

Its third commitment operationalizes the evidential face of the runtime-contract argument. Agent Safety Should Be a Runtime Contract (08-13) argued the harness carries a preventive face (sandboxes, permission gates, trajectory monitors) and an evidential face: no task-complete claim without checkable proof. HoH's "separate implementation-time testing from independent evaluation" is that as architecture, and it is the direct structural fix for the measurement failure this page flagged on 08-26, where Microsoft's Thinkingbox (08-25) showed the strongest model falling from 65.36% pass@1 to 25.25% pass^20 on stateful workflows while a harness paper published the next day still reported single-attempt scores. One caution: the abstract does not say who authors the independent evaluator. If the same model writes it and the implementation tests, the separation is procedural rather than real.

Open problem 0b is now untouched for an eighth consecutive harness result, and HoH is the worst instance so far. No harness paper since 08-13 has published the cost of its own mechanism. Wrapping three planning-coding-testing iterations around an existing harness multiplies that harness's token spend by a factor the paper does not state, and seventy iterations multiplies it enormously. A 52.25% relative capability gain for an unknown multiple of cost is not a comparable number, and base rates are absent from the abstract, so a large relative gain on a low base reads identically here to a large absolute one. When the field's central pitch is that the harness is where the cost lever lives, this is the finding rather than an oversight. The downstream consequence is concrete: llm-routing.md has wanted to route over model-harness pairs for two months and is blocked partly because harnesses publish no comparable cost-per-success. A harness that runs other harnesses makes that accounting strictly harder.

A field datapoint the same day, on what happens with no evidential layer at all. Rick Brewster shipped a from-scratch, clean-room reimplementation of Direct2D inside Paint.NET for running under WINE, roughly 180,000 lines written by Claude against about 700,000 for the rest of a twenty-year project, and describes it as "vibe coded," not thoroughly reviewed, "trust me bro style," with the explicit admission "I cannot possibly review 180,000 lines of code." The failure he had to catch by hand is instructive because it is exactly the class an automated check would find: for a while the model was not doing the COM equivalent of AddRef() on reference-counted objects. His only harness was his own attention, plus an experimental /wine flag gating the code from default use. That gate is the whole preventive face of the runtime contract, implemented as a feature toggle, and it is what the evidential face is supposed to replace. (Simon Willison)

2026-08-31: the week the harness thesis got runnable artifacts, and one measurement that invalidates a process

Five systems described in one weekly roundup, plus one paper on HuggingFace, and they converge on a design move this page had not named: stop making the model re-read its own history, and give it typed state it can run code against. That is a token-cost reduction before it is a capability gain, which puts it squarely on this page's Tier 1 intersection.

• Prime Agent (Prime Intellect, open source) persists histories, memories, skills, prompts and subagent specifications across trajectories rather than resetting everything but the files on disk, and gives the model a persistent IPython session so it filters and re-derives state as code instead of re-reading a transcript. Holding the model class fixed, ARC-AGI-3 Best@1 moves from 30% to 95.5%, and it matches or beats native harnesses on long-context coding, GPU kernel generation and autonomous nanoGPT speedruns.
• Scroll (Alibaba) removes the memory schema entirely: an append-only event log plus a sandboxed persistent Python kernel, with tool outputs binding to typed variables across model calls and only explicitly-printed projections crossing into the working view. Evicted spans stay recoverable through an eviction index of compact landmarks tied to exact event-log addresses. 94.8% on LongMemEval_S, 73.1% on BEAM_10M (5.1 points over the best published memory system), 86.7% on LOCA_256K.
• JIT-Agent makes the harness the model's output, synthesized per task under a fixed four-module protocol (memory, planning, action protocol, tool orchestration) and patched mid-run. DeepSeek-V4-Flash passes GPT-5.6 on DeepSearchQA by 9.1 and OdysseyBench by 4.3; GLM-5.2 gains up to 20.2. Generated harnesses are competitive with OpenCode and Claude Code.
• ContextPilot (Tencent, on HuggingFace) trains the model to manage its own context with RL, adding planning, long-term memory and soft context offloading (evict but keep retrievable) to the usual search/delete/summarize toolset.

The convergence to record: Scroll and ContextPilot independently arrived at "evict but do not destroy" in the same week, one via an eviction index of event-log landmarks and one via a soft-offload tool. Neither cites the other. That is two groups rejecting the same thing, namely irreversible compaction, which is the standard behaviour of every context-window manager in production today.

Skill Lift is the counter-signal, and it is the most immediately actionable number on this page. NVIDIA measured whether the structural scanner that enterprises gate shared skill libraries on predicts anything about skill performance. Across 145 real skills from internal and public catalogs, scan score correlates with LLM-judge quality at a Spearman rho of 0.14. Passing the gate tells you a skill is well formatted and essentially nothing else. The replacement is a paired-run design: same task, same model, same sandbox, same scorer, run once with the skill loaded and once without, and measure the delta in what the agent completed. 947 paired cases from 58 production skills across four harnesses, with trajectories normalized into a shared Agent Trajectory Interchange Format so a skill's lift in Claude Code reads against its lift in Cursor. That interchange format is the quietly important artifact, because it is the first thing this page has seen that makes harness comparison portable rather than anecdotal.

Where ContextPilot cuts against this page's sharpest claim, and it should be held in tension rather than resolved. This page records, from AI4AI at Test-Time (08-13) and Spark-to-Paper (08-13), that the harness wins by taking decisions away from the model: offloading unstable reasoning into deterministic code, routing per question type, enforcing formats, all of which remove model discretion rather than adding model effort. ContextPilot does the opposite. It hands the model more discretion over its own context and then trains that discretion with localized RL. Both cannot be the general rule. The likely reconciliation, which no paper states: the harness should own decisions with checkable correctness, and the model should own decisions that are genuinely judgement calls, such as what in this context still matters. Nobody has drawn that line explicitly and it is the right next position paper.

New open problem 7: two competing answers to context cost, never compared. ALTK-Evolve (08-12) makes per-step delivery volume an externally-tuned parameter and got DeepSeek-V3.2 from 80.4% to 89.3% task-goal completion at 263K tokens per task against 634K. ContextPilot makes it a learned self-applied policy. The external knob is auditable and easier to reason about; the learned policy should adapt better across task variety. No head-to-head under matched token budgets exists. Whoever runs it settles a design question a serving stack will otherwise settle by accident.

Open problem 0b remains untouched for a seventh consecutive harness result. None of Prime Agent, Scroll, JIT-Agent or ContextPilot publishes the cost of the mechanism itself. ContextPilot is the clearest instance: branch sampling at every detected critical edit multiplies rollout cost by the branching factor, and the paper reports how compact the resulting context is without reporting what training it cost. When the entire pitch is cost, that omission is the finding.

The vocabulary (glossed once)

Three terms get mixed together constantly. They are layers, not synonyms:

• Harness — the whole scaffold around the model: context constructor, memory substrate, skill/tool routing, orchestration loop, and verification/governance. "Everything that isn't the weights."
• Loop engineering — designing the automated cycle (generate → critique → revise → act → verify) so the system prompts the model repeatedly, instead of a human prompting it by hand. The slogan: "the human should not be the loop."
• Graph engineering — the next layer up: wiring multiple agents, memory, and state into a structured graph. The practitioner framing: "loops make agents think; graphs make agents remember and coordinate."

Model-bound vs harness-bound is the key diagnostic: some tasks are limited by the model, others by the harness. Smarter models do not make the harness matter less — they raise the ceiling the harness has to reach.

The core claim, and why it holds

Same model, same task, same prompt: move the work between two harnesses and cost-per-success swings 5x to 30x (omarsar0's preregistered benchmark, arXiv 2608.01347, surfaced 08-13). That single measurement is the empirical spine of the whole field: if the harness moves cost by more than an order of magnitude while the model is held fixed, then the harness is where the optimization budget belongs.

The same paper found a subtler result: prompt wording changes where effort goes, not just how much. "Try multiple approaches" inflates reasoning 2.4x–7.4x and spawns ~3 discarded solution branches for one implemented fix, with no gain in success rate. "Maximum certainty" instead triggers repeated verification (extra test runs, tool calls, turns, latency, context growth). The prompt is a cost dial most people turn blindly.

Tier 1 intersection (efficiency): this is a cost-optimization result, not just an agent-design result. Harness choice is a lever on serving cost, token spend, and latency — the same axis as KV-cache and inference work.

How the idea evolved (the wiki's timeline)

1. Code as Agent Harness (arXiv 2605.18747, summary 05-23) — the first strong articulation: code is the harness because it is executable, persistent, version-controlled, and compositional. Natural-language plans and raw tool-call sequences are none of these. Karpathy's Autoresearch (program.md + train.py + git log) is the canonical instance.
2. Scaling the Harness (arXiv 2605.26112, summary 05-27) — the position paper: system scaling, not model scaling, is the next bottleneck. Names the six harness components (foundation model, context constructor, memory substrate, skill-routing layer, orchestration loop, verification/governance) and calls for harness-level benchmarks measuring trajectory quality, memory hygiene, and context efficiency instead of one-shot task success.
3. HarnessOpt-Bench (08-07, per the 08-08 weekly) — made "can a model optimize its own harness" measurable. The benchmark for the self-improving-harness idea.
4. The cost measurement (omarsar0, arXiv 2608.01347, 08-13) — put the 5x–30x number on harness choice. Turned a design intuition into an efficiency fact.
5. LongHorizon-Harness (arXiv 2608.01964, Alibaba, 08-13) — the state-management mechanism: a Manage-Execute-Audit (MEA) loop keeps task state outside the execution context and updates it only on environment-verified facts, so wrong self-assessments stop propagating across long tasks. The rigorous answer to "how do you engineer the loop."
6. The capability measurement (AI4AI at Test-Time, arXiv 2608.12307, 08-13) — the twin of item 4. A strong builder model writes an inference-time harness for a weaker target model, refining it against a 5% validation split, and average target performance on four Theory-of-Mind benchmarks goes 0.49 → 0.91 with the target's weights never touched. Harness choice moves accuracy by roughly 2x on a fixed model, the way it moves cost by 5x–30x.
7. The safety statement (Agent Safety Should Be a Runtime Contract, arXiv 2608.11274, 08-13) — the harness is also where safety belongs, as a preventive face (sandboxes, permission gates, trajectory monitors) plus an evidential face (no task-complete claim without checkable proof: test runs, log captures, file diffs, citation grounding). Backed by a title-level audit of all 28,560 NeurIPS/ICML/ICLR 2023–2025 papers showing an 8x–12x imbalance between training-time and deployment-time safety publication.

Running through all seven: capability lives in the harness, not just the weights, and the harness deserves to be designed, benchmarked, cost-optimized, and governed as a first-class object.

What AI4AI adds to the mechanism story

The most useful part is not the headline number but the mechanism analysis, because it says what a harness is actually for. The gains do not come from the target reasoning more or sampling more widely, which is what a better prompt would produce. They come from three moves: offloading unstable reasoning steps into deterministic code, routing per question type, and strict answer-format enforcement. All three remove model discretion rather than adding model effort.

That is the same conclusion Spark-to-Paper (08-13) reached independently in a different subfield: its first design principle is separating model-based judgment from deterministic operations that can be executed and checked, and its ablation shows fabrication detection rising from 14% for a single-pass draft to 92% with the full integrity stack. Two papers, same board, different tasks, one conclusion: the harness wins by taking decisions away from the model. That is a sharper statement of the concept than "scaffolding helps," and it is falsifiable: a harness that improves results purely by giving the model more room should not show this signature.

The practitioner convergence (2026-08, surfaced via saved reading)

Independently of the papers, a wave of practitioner writing landed on the identical thesis, with sharper operational language and proof points:

• From prompting to loop engineering. The consensus statement (attributed to senior engineers including OpenClaw's creator): "you shouldn't be prompting coding agents anymore; design the loop that prompts them." Proof point cited: one engineer shipping 259 merged PRs in a month, the AI writing every one, never opening an editor.
• The four loop types + the evaluator (Claude-Code-specific loop-engineering guides), and "the four bills nobody warns you about" — the cost gotchas of running unattended loops. This is the practitioner rediscovery of Scaling-the-Harness's "context efficiency" concern.
• Loop → Graph engineering. The claimed frontier shift: once loops work, wire agents into graphs for memory and multi-agent coordination. Framed as "graphs make agents remember."
• Model-bound vs harness-bound entered practitioner vocabulary directly (an article by that name), matching the research diagnostic.
• Market signal: Forward Deployed Engineers reportedly command ~$1M/year and Anthropic $750k+ for harness/loop skill specifically. The industry is pricing the harness, not model access.

That research and practitioners reached the same frame in the same window is the strongest possible confirmation that this is a real structural shift, not a hype cycle.

Relation to neighbouring concepts

• Agent memory — the memory substrate is one of the six harness components. The "RAG is a dead end / memory should be native to the model" turn (08-13) and the write→manage→read memory-engineering formalization are the memory layer of graph engineering.
• Self-evolving agents — self-improving loops (agents that rewrite their own harness/skills) are harness engineering closing the loop on itself; HarnessOpt-Bench measures it.
• Context engineering — "delete 80% of the system prompt" (Anthropic/Cloudflare/Karpathy, 08-13) is the token-optimization face: the harness win is often in what you stop keeping in context.

Industry has started pricing the harness directly (2026-08-13)

The market signal moved from salaries to product this week. xAI shipped Grok 4.6 and stated that Grok 4.6 optimized the Grok Build harness for itself (@aksheyd), and Hugging Face's Elie Bakouch, reading the model card, noted the model is state of the art on an internal "inferenceEval" that measures optimization of xAI's own chat inference (@eliebakouch). Two of the capabilities a frontier lab chose to headline are the model improving its own scaffold and its own serving path.

That is the industrial instance of the research claim, and it lands the same week AI4AI measures strong-to-weak harness transfer and AutoWorldModel-Bench measures agents improving an external artifact under an unspecified objective in 63 of 64 sessions. The gap worth watching: xAI reports a self-optimized harness as a capability headline and publishes no cost-per-success number, which is precisely the measurement omarsar0's benchmark made the field's spine.

The first public model-by-harness cost table (Terminal-Bench 3.0, 2026-08-13)

Terminal-Bench 3.0's leaderboard is the closest public artifact to the measurement this page keeps asking for, because every row is a model-harness pair reported with token count and dollar cost alongside accuracy (screenshot):

Three readings. Token spend does not track accuracy: Sonnet 5 in Claude Code burns 17.9B tokens for 14.6% while Fable 5 in the same harness reaches 34.1% on 3.6B, a 5x token difference in the opposite direction from the score. Ranking by cost-per-point reorders the table, moving Grok 4.6 from fourth to second and dropping Fable 5 from third to seventh, which is the argument for reporting dollars-per-completed-task as the primary agentic metric. And the grid is still not an ablation: no model appears under two harnesses in the top ten, so the leaderboard shows harness variation is present without isolating it. Holding one model fixed across mini-SWE-agent, Codex, Claude Code, Grok Build and Cursor CLI remains the missing experiment, and it is now cheap enough that somebody should just run it.

2026-08-14: the harness stops being written and starts being optimized

One day after the cost and capability measurements landed, three things arrived together that change this page's central claim from the harness is the object of design to the harness is the object of optimization.

DarwinX (arXiv 2608.07545, Salesforce) makes harness search a population problem. Model frozen, one evolution loop, about +17 points average across four benchmarks. Terminal-Bench 2.1 goes +7.7 to 83.2% on a matched base and 84.7% on a stronger one; TerminalWorld's held-out split hits 68.3%; WebArena-Infinity real-task pass@1 goes 43.5% → 93.0% audit-clean; and a Terminal-Bench harness transfers unchanged to SWE-bench Verified. Its diagnosis of prior work is the part that matters here: every self-improving harness in this wiki's timeline above is a single lineage, which is path-dependent and lets a local win silently regress another task. DarwinX's three fixes are a preserve-and-extend contract (admit a variant only if it extends coverage without regressing), an archive of alternative lineages for later recombination, and one shared edit interface for failure-derived, teacher-derived and self-derived evidence. Fitness comes from each benchmark's own verifier, so no gold solutions and no human picks winners.

This resolves the isolation experiment 08-13's Looking Ahead predicted for a 60-day window. The Terminal-Bench 3.0 table above shows harness variation exists but never holds a model fixed across two harnesses; DarwinX does exactly that on 2.1 and gets +7.7 points from the scaffold alone. Open problem 1 below is now half-answered: the swing is real at the frontier, though still not predictable in advance.

AutoDesign (arXiv 2608.13560) supplies the missing price tag. A meta-harness optimizer guides a code agent to rewrite its own harness from rollout feedback, aligned to human design priors. On the new PosterBench (100 papers, five disciplines) it scores 78.32, beating the commercial Claude Design by 7.45 points, and the learned DesignHarness dropped into seven other code-agent-model configurations lifts the average from 54.99 to 67.39. The number this page has been asking for since 08-13: 253 tool calls, 11 editing turns, 40 minutes, under $3, fully autonomous. Harness optimization costs single-digit dollars per rollout.

Note what the pair does jointly. DarwinX optimizes against hard verifiers (a test either passes or does not) and reports no cost. AutoDesign optimizes against a soft verifier (a design rubric plus human preference) and reports full cost. Neither tests the other's regime, and whether meta-optimization degrades when fitness is soft is now the sharpest open question in this area.

DeepSeek Harness v0.1 makes prefix stability a billing line. DeepSeek open-sourced its agent harness under MIT (built on the Cordis meta-framework, where models, tools, skills, sessions, sandboxes, filesystems, loops, orchestration and UI are all plugins) on the same day it raised API prices with cache-hit tokens repriced to roughly 6x, plus a new peak/off-peak split at 50% below peak. Hugging Face's Elie Bakouch, reading the code, found the harness's organizing commitment is first-class KV-cache-aware design: previously written history is never altered; a change is expressed by appending a statement describing the modification rather than editing the prefix, because editing invalidates every cached token downstream. Bakouch also estimates at least ~20% of the harness's own commits came from Codex worktrees.

That is this page's Tier 1 efficiency claim arriving from the provider side. The 5x–30x cost-per-success swing was measured by varying harnesses at fixed prices; DeepSeek varied the prices at fixed harness design and reached the same conclusion. Append-only history is now a cost property, not a latency nicety, and it is cheap enough to implement that convergence across harnesses is likely within a quarter.

Ken Huang's Harness Engineering supplies the governance half the research thread lacks. His periodization (model as product → wrapper as product → harness as product boundary) is the industry statement of Scaling the Harness's thesis, and his ten pattern families are a superset of that paper's six components with three additions research has not touched at all: identity, data governance, and runtime steering. His "hill climbing" family states the rule DarwinX independently implements: improvement is only safe when it is measured, bounded, and reversible. A security architect and a research team converged on the same admission rule within one day of each other.

Updated state of knowledge. The harness is measured on cost (5x–30x swing), measured on capability (0.49 → 0.91 at the small-model tier, +7.7 points at the frontier), optimizable by population search and by meta-optimization, transferable across tasks, verifiers, base models and agent frameworks, priced by providers through cache-hit economics, and now has a governance taxonomy. Five papers and one product release in six days. The gap that remains is the same one: nobody has put harness optimization and fine-tuning on a single cost axis for equal capability gain.

The open problem this creates. If harnesses are portable, optimizable artifacts that lift arbitrary models, then the routable unit is the model-harness pair, as A²E (08-11) argued. LLM Routing now records this as a standing gap with three supporting results and zero proposals: no production router routes over harnesses, and no paper proposes it.

2026-08-16: duration enters the measurement, and the cost axis spans three orders of magnitude

Two results extend this page along axes it had no data on, and a third supplies the small-model version of the substitution.

Measuring Autonomous AI Research (Prime Intellect) adds duration. 153 autonomous runs across 18 frontier models on the nanoGPT optimizer speedrun, up to 8.7 days per run on 8xH200s, all scratchpads and logs public. Every leaderboard row is a model-harness-effort triple: Fable 5 under claude-code at high effort closes 81.7% of the human-record gap at 2,726 steps; Opus 5 under claude-code max reaches 2,920 (53.6%); Kimi K3 appears twice, at 2,930 under prime-agent and 2,974 under kimi-code. That 44-step spread between two harnesses running the same model is roughly the size of the entire gap between Opus 5 and Kimi K3, which is the harness-isolation comparison 08-13's Looking Ahead asked for, arriving from an unexpected direction. Bakouch's variance note is the number to keep: one run in the same setting has a ~50-step spread after 24 hours, so single-run autonomous-research claims sit inside the noise. The earlier Prime Intellect experiment's diagnosis stands unretracted: agents are strong at optimizer search, hyperparameter sweeps and stacking known methods, and weak at generating new ideas, needing upstream human records to keep climbing.

Specification-first convergence adds a stopping rule for tasks with no oracle. An agent dismantled a core lifetime invariant across a 717,725-line production TypeScript codebase, 189 files touched, with no human code review and no pre-existing test for the target behaviour. The protocol: the agent writes a formal specification, 14 refinement cycles audit the specification against the source, the specification is frozen, then 17 verification cycles audit the code against it. 201 defects corrected across 31 audit passes before a human ran the program. Convergence criterion: two consecutive verification passes returning zero findings. Three days, USD 2,430. The freeze is the direct countermeasure to open problem 3 below, since the auditor is checking against a document it may no longer edit.

Set the two published cost points side by side and the axis is visible for the first time. AutoDesign (08-14) published $3 per rollout for 253 tool calls in 40 minutes. Specification-first publishes $2,430 for three days on one architectural refactor. Three orders of magnitude apart, both fully autonomous, both reported honestly. "What does agentic work cost" has no single answer, and any claim that does not name the task class is uninterpretable.

SKILLER supplies the cheap tier. A strong model as actor and critic, the small-model agent system as environment, every RL signal propagated as natural language, producing executor-specific skills. Qwen3.5-9B gains 4.3 to 20.4 points, Qwen3.5-4B gains 1.8 to 13.3, and single-skill SkillsBench performance matches strong closed models. The paper's stated motivation is that skill-driven harnesses like Codex and OpenClaw are prohibitively expensive at closed-model prices, which is this page's cost thesis stated as a research problem rather than an observation. Parity is claimed on single-skill tasks only; composition is untested.

Updated state of knowledge. The harness is measured on cost (5x-30x swing; $3 to $2,430 per task depending on class), on capability (0.49 → 0.91 at the small tier, +7.7 to +17 points at the frontier, 81.7% of a human record over 8.7 days), and now on duration and variance. It is optimizable by population search and meta-optimization, transferable across tasks and base models, priced by providers through cache-hit economics, governed by a pattern taxonomy, and manufacturable for small open models via natural-language RL. What is still missing is unchanged: nobody has put harness work and fine-tuning on one cost axis for equal capability gain.

2026-08-25: the harness gets a cost curve, a reference implementation, and a 35B model at the frontier band

Three HuggingFace papers on one day, all treating the harness as the primary object, plus a fourth on the Kurate board. This is the densest single day this page has recorded.

Apodex 1.1 is the model-scale version of this page's thesis, and the day's top paper at 145 upvotes. It names working capability (sustained, verifiable progress toward a real-world objective) as a category distinct from reasoning, constituted by state maintenance, failure recovery, and verifiable delivery. It develops that along two axes that are explicitly not parameter count: Environment Scaling (expanding the diversity and verifiability of executable file, search and code environments) and Agentic Coordination Scaling (training agents to decompose, delegate in parallel, integrate asynchronous results, replan). System side is a shared execution harness plus AgentOS maintaining task state and provenance across tools and agents, which makes provenance a first-class harness component exactly as Scaling the Harness (05-27) proposed. Result: a 35B model and a locally deployable 35B Mini reach the leading performance band on finance, research, math, coding and search.

The parameter count next to the benchmark is the whole point. Every prior data point on this page lifted a fixed model with a better harness. Apodex trains model and environment together and lands at the frontier band from 35B, which is the strongest evidence here that on verifiable professional work the marginal value of parameters is below the marginal value of environment and coordination.

But it entangles this page's open problem 0 rather than resolving it. Harness optimization versus fine-tuning at matched cost is still unrun, and Apodex makes it harder to answer, not easier: environment-trained weights and an inference-time harness are both in the deliverable and the paper does not separate them. The ablation that would settle it (freeze the base and apply only harness plus AgentOS; then run the environment-trained weights under a plain ReAct loop) is the single most valuable missing experiment in the paper. Also unpriced: coordination scaling spends tokens on parallel delegation and replanning, and against the AlphaSense finding (08-14) that task cost and token price can point opposite ways, a 35B model winning on quality could still lose on dollars per completed task. And "leading performance band" without per-benchmark deltas is not a checkable claim for a result whose entire content is a comparison.

Task-CoEvolve is the first result on this page about the efficiency of the harness search loop rather than the quality of its output. DarwinX (08-14) and AutoDesign (08-14) established that harness search works, by population selection and by meta-optimization respectively. Neither made it affordable. Task-CoEvolve observes that a validation set's informativeness decays as the harness improves: eventually the easy tasks are universally solved, the hard ones universally failed, and only a shrinking frontier band still separates candidates. It concentrates evaluation there by variance-weighted sampling over past outcomes, with the sampling distribution adapting as the harness evolves, then recovers comparable full-set scores from the biased sample with an importance-weighted estimator. That second piece is what makes the first usable: without it, each iteration measures a different subset and the search loop loses the ability to compare iteration k to k+1. Result: 80% fewer evaluations, matching full-set search's final quality, on online text classification and Terminal-Bench 2.1 (the same benchmark DarwinX and AutoDesign used, so it is comparable).

This moves one side of open problem 0 by a large constant factor. AutoDesign published $3 per rollout, DarwinX published no evolution budget, no distillation paper published a comparable per-point cost. The comparison still has not been run, but harness search just got roughly five times cheaper, which changes the expected answer. The likely hidden failure mode: importance weighting is unbiased but its variance grows as sampling probabilities shrink, which is what aggressive concentration produces, so a noisy full-set estimate could cause the search to accept a worse harness. The paper does not discuss estimator variance, and only two task types back the 80% figure, which depends on the difficulty distribution having easy and impossible tails to skip.

Prime Agent supplies the reference implementation and the cleanest one-line statement of why this page exists: it prevents harness failures from becoming model failures. Persistent IPython REPL under a Recursive Language Model abstraction, a Continual Harness preserving histories, memories, skills, prompts and subagent specs across trajectories, plus agent-to-agent coordination. ARC-AGI-3 RHAE Best@1 from 30% to 95.5%. Open source.

Fourth on the Kurate cs.AI board: Agent Lightning v1.0: Towards Harnessed Agentic RL (He, Zhang, Zhou, Yang, Kang et al., ai_rating 7.0, the highest-rated item on that board). Four papers in one week putting the word harness in the thesis.

Task-CoEvolve also extends the "schedule beats operator" pattern this page's neighbours have tracked since 08-12. LycheeMemory V2 (08-14) was recorded as the third instance after ICBQ block order (08-12) and ReOrder-OPD prompt order (08-13). Task-CoEvolve is the same shape: the harness-rewrite operator is unchanged and the entire gain comes from changing what gets measured when. And its disagreement-weighted sampling is the same statistical idea as R2-OPD (08-25), which suppresses distillation reward where a teacher ranking and a progress ranking disagree. Harness evaluation and distillation, one day, no shared authors, one principle.

The counter-signal, and it should be read seriously. Kurate cs.AI #14 is On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification (Ye, Li, Pruksachatkun, Zhang, Wu), arguing self-improvement gains are sensitive to variance, task arrival order, and underspecified objectives. Set that against Bakouch's 08-16 note that a single run in a fixed setting has a ~50-step spread after 24 hours, and against ARC-AGI-3 being near-saturated by two systems in one week (Prime Agent 95.5%, NVIDIA AVO 100%). The honest reading: harness capability is real and its measured magnitude is not yet trustworthy. A near-saturated benchmark cannot discriminate the thing everyone is now optimizing.

Meta-Harness closed a loop this page has been open on since it was created: the saved-reading thread got cited by the published literature. Task-CoEvolve's related work opens with the six-fold same-benchmark gap from harness design alone, which is the Stanford and MIT Meta-Harness result surfaced through saved reading on 08-18. The optimizer design is two refusals of the standard recipe. Full diagnostic trace access: instead of a lossy scalar reward plus a short summary, an agentic proposer gets unrestricted filesystem access to raw execution logs and source code across all past iterations, up to 10M trace tokens, inspected with grep and cat. Causal failure analysis in code space: isolate the confounded regression, trace the downstream error back to the early context decision that caused it, rewrite the responsible executable Python function. The attribution argument is the substantive one: a scalar says a rollout was bad, it cannot say that turn 40 failed because turn 6 evicted the wrong file, and recovering that link requires the log to exist and be readable. Results: discovered context policies beat state-of-the-art agentic memory systems by 7.7 points at 4x fewer context tokens, converging 10x faster, and a single discovered math-retrieval harness added 4.7 points on 200 IMO-level problems across five held-out frontier models zero-shot. Unaccounted: the search cost. 10M trace tokens per proposal is expensive and "10x faster convergence" counts iterations, which is the wrong unit when the per-iteration bill rose.

That transfer result crosses this page's own three-paper threshold. AI4AI at Test-Time (08-13) took a weaker target from 0.49 to 0.91 across four Theory-of-Mind benchmarks with the target frozen. AutoDesign (08-14) added 12.4 points across seven code-agent-model configurations. Meta-Harness adds five held-out frontier models on olympiad problems. Three independent results, twelve days, same structural claim: a discovered harness is a portable artifact with standalone value, not per-model tuning residue. That is a stronger statement than "harness choice matters," and it is what makes harness-as-product coherent.

Industry reported the same cost number in the same week, which is the confirmation that matters most. OpenAI's Codex as a platform release (08-19) states outright that the harness, not the chat interface, is the primary reusable component, and reports that harness-level retained reasoning plus context compaction moved GPT-5.6 Sol's ARC-AGI-3 score from 13.3% to 38.3% while cutting token consumption sixfold. Meta-Harness reports 4x fewer context tokens at +7.7 points. A research group and a frontier vendor independently landing on 4x-6x token savings from the same architectural layer, in the same week, on partly the same benchmark, is the strongest cost evidence this page holds. Ken Huang's reading of the open codex-rs repository extracts a token-budget subsystem with proactive compaction hooks as one of nine supporting components, making token optimization an explicit architectural element rather than a tuning afterthought.

Thinkingbox (08-25) aims the counter-signal at the metric rather than the method, and it is the sharper version of the Fragility critique. Microsoft's 507 policy-conditioned stateful business workflows, evaluated against backend terminal state with executable checks that reject extra effects as well as wrong and missing ones, put the strongest model at 65.36% pass@1 and 25.25% pass^20. Worse, many failures terminate cleanly having made valid state-changing tool calls, so response-level and tool-call-level signals are not proxies for completion. Set that beside Prime Agent's 95.5% Best@1 the same day: both results can be true, and the honest joint reading is that harness engineering has demonstrably raised the ceiling of what an agent can do while leaving open how often it does it. Nobody has published a pass^k curve for a harness-optimized agent. That is now the single most conspicuous missing measurement on this page, and it is cheap to produce.

Updated state of knowledge. The harness is measured on cost (5x-30x swing; $3 to $2,430 per task by class; 4x-6x token reduction from context management, confirmed by research and vendor independently) and now on the cost of finding one (80% reducible). It is measured on capability (0.49 to 0.91 at the small tier, +7.7 to +17 at the frontier, 30% to 95.5% on ARC-AGI-3, 81.7% of a human record over 8.7 days) and now at model scale, with 35B reaching the frontier band when environment and coordination are scaled instead of parameters. It is optimizable by population search, meta-optimization, and now cheaply; transferable across tasks, verifiers, base models and frameworks; priced through cache-hit economics; governed by a pattern taxonomy; manufacturable for small open models via natural-language RL; and it has an open-source reference implementation. What is still missing is unchanged and now conspicuous: nobody has put harness work and fine-tuning on one cost axis for equal capability gain, and the benchmark everyone is climbing is saturating.

2026-08-26: open problem 2 gets solved, the admission rule becomes a law, and industry stops buying harnesses

Two HuggingFace papers, one practitioner survey, and one production case study, all on the same layer. The day's contribution is less about new capability than about three of this page's own open items moving.

Recuris (arXiv 2608.24876) is the first system that actually composes the three pieces open problem 2 asks for, so that problem can now be marked mostly resolved. The problem read: LongHorizon-Harness's state-outside-context, HarnessOpt's self-optimization, and graph-based memory routing are separate advances, and no harness manages state externally and self-optimizes and routes through memory end-to-end. Recuris does all three. Working Memory holds verified task progress, which is LongHorizon-Harness's (arXiv 2608.01964, Alibaba, 08-13) Manage-Execute-Audit contribution of keeping state outside the execution context and updating it only on environment-verified facts. Experiential Memory is retrieved by that working state rather than by the initial instruction or the full transcript, which is the routing layer and the fix for stale skill retrieval on long tasks. A fixed Meta-Agent turns localized failure evidence into validation-gated updates to Skill Memory, which is the self-optimization layer, and the Meta-Agent never edits itself, which is what bounds the recursion. Results: improvement in 35 of 37 completed model-benchmark pairs across four long-horizon benchmarks and ten models; +17.8 to GPT-5.6 Sol and +15.6 to Claude Opus 5 on tau-bench, carrying Opus 5 to 87.9%; +16.6 and +13.5 on Qwen3.6-27B/35B on SkillFlow; +32.2 on the longest tasks, with common long-horizon failures down up to 80%.

The composition is not additive, and that is the finding. The externally verified working state is what makes the self-optimization precise: because state is checked against the environment, a failure attributes to a specific memory component, and that attribution is what lets the Meta-Agent write a local patch instead of a global one. This is the same credit-assignment problem Meta-Harness (08-25) solved by brute force, giving an agentic proposer unrestricted filesystem access to up to 10M tokens of raw execution trace so it could trace a turn-40 failure back to the turn-6 context decision that caused it. Recuris gets the same link structurally, from a maintained state, at a fraction of the token cost. Two answers to attribution in harness optimization, eleven days apart: read everything, or maintain the right thing. The second should dominate on cost and nobody has compared them, which is a cheap and valuable experiment.

AutoSaddler (arXiv 2608.23041, Microsoft with POSTECH/KAIST/SUSTech) makes the admission rule an empirical result rather than a design principle. It formulates harness improvement as offline learning from mini-batches of failure traces, diagnoses across the batch, generates structured patches treating the harness as code, and selects updates by validation. +9.0 on GAIA2, +9.6 on SWE-Bench Pro, +10.0 on Terminal-Bench 2.0. Its ablation names three ingredients: deep debugging rather than shallow reflection, targeted modification rather than unconstrained editing, and generalization-aware selection rather than trajectory-specific repair.

That last pair is the fourth and fifth independent arrival at the rule this page first recorded from DarwinX (08-14), whose preserve-and-extend contract admits a variant only if it extends coverage without regressing, and from Ken Huang's pattern language (08-14), whose hill-climbing family states that improvement is only safe when measured, bounded and reversible. Recuris independently gates on validation and keeps patches localized. Five arrivals by five methodologies is well past this page's three-instance threshold: bound the edit and gate the admission, or the optimizer overfits its most recent failure. Treat that as settled and stop re-deriving it.

AutoSaddler's mini-batch economy also lands on the affordability axis Task-CoEvolve (08-25) opened, from a different direction. Task-CoEvolve cut evaluations 80% by concentrating them on the shrinking frontier band where candidates still disagree. AutoSaddler amortizes each patch over several traces instead of paying an evaluation cycle per proposal. Neither cites the other, both landed within a day, and harness search getting cheaper by two independent routes in two days is the signature of a subfield in its engineering phase rather than its discovery phase.

Gradient Flow's nine rules supply a fourth independent measurement of the swing, from practitioners who were not trying to publish one. Ben Lorica, surveying production teams with unrelated products, reports that the same open model showed an 18 percentage point spread between its best and worst harness configuration, and singles it out as the finding he would take most seriously when comparing models. Set beside omarsar0's 5x-30x cost-per-success swing (08-13), AI4AI's 0.49 → 0.91 capability lift with the target frozen (08-13), and DarwinX's +7.7 from the scaffold alone (08-14), the swing now has four measurements in three units from three kinds of observer. His rules 1 and 2 are also the practitioner statement of this page's sharpest mechanism claim: put hard constraints in software rather than prompts, and turn stable agent paths into ordinary code. That is the harness wins by taking decisions away from the model, arrived at from production experience instead of from AI4AI's and Spark-to-Paper's ablations.

Ramp's Inspect (The Pragmatic Engineer) is the harness-as-product-boundary thesis confirmed from the buyer's side, and it splits the artifact in a way this page had not. A fintech built its own background coding agent; 75% of merged PRs at Ramp are now raised by it (up from ~60% two months post-launch), it passed one million sessions in July, and engineers are free to use anything, so the share is revealed preference. Block has Goose, Stripe has Minions, Shopify has River. Orosz's framing: "buy, don't build" appears not to apply to AI harnesses.

The reason it does not apply is the useful part. Ramp's advantage over Claude Code and Codex is entirely the evidential face this page recorded from Agent Safety Should Be a Runtime Contract (arXiv 2608.11274, 08-13) — no task-complete claim without checkable proof. Inspect verifies backend work by running tests, reading telemetry and querying feature flags, and verifies frontend work by producing screenshots and live previews, shipping the latter nearly a year before any vendor. A third-party harness cannot supply evidence about systems it cannot see. Meanwhile the parts this page has shown to be highly portable are context policy and control logic: Meta-Harness's discovered math-retrieval harness added 4.7 points across five held-out frontier models zero-shot, AutoDesign's DesignHarness lifted seven other code-agent-model configurations from 54.99 to 67.39, AI4AI transferred strong-to-weak. The split this page can now state: harness structure is portable, harness evidence is not. That predicts the market shape — vendors sell optimizable structure, companies build their own verification — and it explains why four non-AI companies independently reached the same build decision.

Recuris also adds a fourth supporting result for routing over model-harness pairs, and still no proposal. It lifts ten models by materially different amounts (+17.8, +15.6, +16.6, +13.5 on four of them), which is exactly the condition under which routing over pairs would beat routing over models. LLM routing records the gap.

Updated state of knowledge. Open problem 2 (composition) is mostly resolved by Recuris. The admission rule is settled at five independent instances. The swing has four measurements. The harness now has a portability boundary (structure yes, evidence no) and a confirmed industrial buy-versus-build answer. Open problems 0 and 0b are not only unmoved but conspicuously so. AutoSaddler and Recuris both publish capability gains with no search or serving budget, keeping open problem 0 exactly where it has been since May. And both report single-attempt scores one day after Thinkingbox (08-25) — Microsoft's own 507 stateful business workflows, checked against backend terminal state — showed the strongest model falling from 65.36% pass@1 to 25.25% pass^20. Lorica's rule 4 independently asks for the same thing, telling teams to measure recovery separately from first-attempt accuracy. A Microsoft benchmark and a practitioner survey both demanded the pass^k measurement, and the Microsoft-led harness paper published the next day without it. tau-bench, where Recuris reports 87.9%, is a stateful tool-use benchmark of the same family. This is now the most conspicuous and cheapest missing measurement on this page, and it has gone unrun for two consecutive dense harness days.

2026-08-28: a cost-per-success number finally arrives, and the model side of open problem 0 opens

Five papers on this layer in one day (four on HuggingFace, one on the Kurate board), plus the first industrial statement of the cost thesis by a named buyer. The day moves two things that have been stuck for months.

PILOT in the Loop (arXiv 2608.26530) publishes a serving-side cost-per-success metric, which is the number this page has been asking for since 08-14. A supervisor-worker split with two powers the worker lacks: live steering (redirect or abort the active worker mid-execution) and live self-evolution (distil procedures and failure modes into skills and memory while the run is still going). Ranks first in five of six configurations, +9.8 points over counterpart harnesses on Terminal-Bench 2.0, +14.6 with GLM-5.1 and +12.4 with Kimi-K2.6 in the self-improvement setting. The headline for this page is the efficiency pair: mean output tokens fall 42.9% and 47.4%, while successful evaluations per million output tokens rise 110.3% and 134.0%.

That denominator is the same one omarsar0's preregistered benchmark used to measure the 5x-30x cost-per-success swing (arXiv 2608.01347, 08-13), which this page calls its empirical spine. Every harness paper since has reported capability without it: AutoDesign published $3 per rollout for its search, DarwinX published no evolution budget, AutoSaddler and Recuris (both 08-26) published gains with no serving budget at all. PILOT reports the harness getting cheaper and better simultaneously, which is a different and stronger claim than buying accuracy with tokens.

Two cautions. The mechanism means most of the saving is probably a refund rather than an efficiency: aborting doomed trajectories early does not shorten the successful path. And the metric is output tokens, while a supervisor reading a streamed trace consumes many input tokens and emits few, so the architecture may be shifting spend from output to input rather than removing it. Whether the supervisor's own consumption is inside the 42.9% is the first thing to verify in the full paper.

PILOT also completes the design space with the two systems from two days earlier. AutoSaddler (08-26) learns offline from mini-batches of failure traces, which is what makes its patches durable. Recuris (08-26) patches between runs from verified state. PILOT patches during the run. The trade is visible and untested: offline patches generalize because they see multiple failures, live patches arrive in time to save the current task but derive from a single trajectory, which is exactly the trajectory-specific-repair failure AutoSaddler's ablation isolated. Offline versus live at matched budget is now a well-posed and cheap experiment.

TaoLive Harness-Aware Training (arXiv 2608.15763, Alibaba) is the first entry on this page to attack open problem 0 from the model side. Everything above freezes the weights and optimizes the scaffold. HAT does the opposite: it trains a compact model to tolerate the scaffold changing. The observed dilemma is a production one at Taobao Live, where campaign rules, compliance requirements and merchant preferences change weekly: a large model adapts to a rewritten harness zero-shot but blows the latency budget, while a compact model meets latency and then overfits to the exact harness it saw. Harness-State Augmentation applies task-preserving transformations to skill identifiers and content, tool schemas, prompt structures and Hook functions, so the model learns to read a harness instead of memorizing one. Three stages: HSA-SFT on strong-model trajectories, General On-Policy Distillation to restore the generalization SFT destroyed, then HSA-RL in augmented environments.

The most important result is negative and it is a warning to practitioners: Fixed-Harness SFT lowers IFEval by 7.7 points from the base model, and HAT does not (83.5). Fine-tuning a compact model against one fixed production scaffold makes it worse at following instructions in general. If that replicates, a large number of teams have quietly paid this cost. In-domain, HAT reaches 94.8 on Live-Stream QA (base 80.3, strongest general LLM 93.0) and 94.6 on Harness-Variant QA (base 75.4), and it is deployed: P50 3.4s and P95 8.1s on a single NVIDIA H20, with positive online A/B results for GMV and item-page views. A harness paper reporting a P95 and a revenue metric is new on this page.

HAT does not settle open problem 0, because it publishes no matched-cost baseline either. What it establishes is that the two levers interact: the whole cost of a harness rewrite was the retrain it forced, so a harness-robust model changes what harness optimization is worth. Its nearest relative is Apodex 1.1 (08-25), which trains model and environment together and reaches the frontier band at 35B; this page's complaint there was that the two contributions are never separated. HAT has the same entanglement but declares it, and builds Harness-Variant QA specifically to measure the co-adaptation. Better experimental design of the two.

The memory layer got specified as a three-part system, by three papers in one day. WikiSkill (arXiv 2608.27454) fixes authorship: insights guiding skill development stay scattered across optimization histories, so it separates raw experience, an accumulated knowledge base, and executable skills, and consolidates experience into the knowledge base that later skill updates build on. Its ablation confirms the persistent layer is load-bearing. CaSKG (arXiv 2608.25500) fixes retrieval: graph retrieval only recovers workflow context when the edges are reliable, and normally they are inferred from surface similarity, so CaSKG calibrates edges by counterfactual intervention (remove, substitute, reorder skill pairs; aggregate with Bayesian smoothing) offline, without changing the downstream policy. Highest task score in all twelve model-benchmark combinations, ScienceWorld macro-average 72.62 → 80.50 and ALFWorld 80.01% → 86.79% against Graph-of-Skills, with fewer mean environment steps. And self-evolving kernel-optimization agents (arXiv 2608.25570, Kurate cs.LG #5) fixes storage for a Tier 1 domain, replacing AccelOpt's flat slow-fast kernel pairs with an Experience Graph Memory.

CaSKG moves open problem 4, which had no research at all. That problem read: graph engineering lacks the research the loop layer has, and what a graph buys over a well-run loop is unquantified. CaSKG supplies the mechanism, a graph buys prerequisite structure and only buys it if the edges are calibrated, converting the practitioner slogan into a testable engineering requirement. It does not close the problem, because it measures graph-versus-graph rather than graph-versus-loop, but the comparison is now specifiable. It is also the strongest member of this wiki's disagreement-is-the-signal family (R2-OPD 08-25, Task-CoEvolve 08-25, TTPO 08-28), because it manufactures the disagreement by intervention rather than observing a correlation, which sidesteps the shared weakness the wiki flagged: every correlational member depends on a second estimator nobody validated. A counterfactual probe is not an estimator.

WikiSkill's third finding complicates this page's portability boundary. The 08-26 reading of Ramp's Inspect concluded harness structure is portable, harness evidence is not. WikiSkill's skills are structure, so the boundary holds, and it adds a fourth transfer result (across model families, after AI4AI strong-to-weak, AutoDesign across seven configurations, Meta-Harness across five held-out frontier models). But it also reports that skills evolved by other models can outperform self-evolved skills, which makes the portable half not merely transferable but sometimes better sourced externally. That strengthens the vendor side of the market split more than 08-26 allowed. The alternative explanation the paper must rule out: if the other model is simply stronger, this is a distillation result, not a claim about authorship.

What Makes Good Agentic Data? (arXiv 2608.27260) supplies vocabulary this page has needed. Its ACE lens (Accuracy establishes feasible support; Complexity places learning mass relative to a declared learner and execution configuration; diversity controls coverage and redundancy) is a decomposition of what every selective-allocation method in this wiki trades off. Two connections matter. Its reported field-wide shift toward execution-grounded accuracy (verify by running, not by scoring) is the data-pipeline arrival at this page's sharpest mechanism claim, that the harness wins by taking decisions away from the model. And learner-relative complexity is the data-generation face of the model-harness-pair unit, since it makes informativeness a property of the pair rather than the task. It is also the reason to doubt unbounded skill transfer: if informativeness is learner-relative, transferred experience should degrade as the target diverges from the source, and nobody has found that boundary.

Industry stated the cost thesis with a named buyer for the first time. The Information reports that Visa built a harness in April that significantly reduces the cost and time for Anthropic's Mythos to find and fix security vulnerabilities in its codebases, with president of technology Rajat Taneja saying plainly: "Using the model through this harness, we have found, is more effective than using the model by itself." (The Information) Every prior industrial data point on this page was a vendor headlining self-optimization (xAI's Grok Build, OpenAI's Codex platform) or a company building its own agent (Ramp, Block, Stripe, Shopify). This is the first customer saying the harness is what made the frontier model affordable, on a security workload, with the vendor's pricing unchanged. The measurement gap is the usual one: no percentage and no dollar figure.

Updated state of knowledge. The harness now has a published serving-side cost-per-success metric (PILOT: 43-47% fewer output tokens at 2.1-2.3x successes per million output tokens), a model-side lever (HAT: train for harness-robustness instead of retraining per revision, and fixed-harness SFT costs 7.7 IFEval points), a specified three-part memory architecture (authorship, retrieval, storage, one paper each, one day), a fifth transfer result now crossing model families, an intervention-based rather than correlational way to calibrate agent memory, and a named enterprise buyer confirming the cost claim. Open problem 0 has moved from unaddressed to attacked-from-both-sides-without-a-matched-cost-baseline. Open problem 0b is untouched for a fifth consecutive harness paper: PILOT reports first-attempt ranks on Terminal-Bench 2.0, a stateful benchmark of the same family where Thinkingbox (08-25) measured the strongest model collapsing from 65.36% pass@1 to 25.25% pass^20.

Open problems (research angle)

0. Harness optimization versus fine-tuning at matched cost. Still unrun, and now more glaring: AutoDesign publishes $3 per rollout, DarwinX publishes no evolution budget, and no distillation paper publishes a comparable per-point cost. The first paper to put the two on one axis settles where a team should spend. 0b. A pass^k curve for a harness-optimized agent. Every harness result on this page reports a best-of or single-attempt number. Thinkingbox (08-25) shows a 40-point drop from pass@1 to pass^20 on stateful work, and Fragility (08-25) plus Bakouch's ~50-step single-setting spread (08-16) say the variance is large. Whether a better harness narrows the pass@1-to-pass^k gap or merely lifts both is unknown, and it is the difference between harness engineering being a capability story and a reliability story. Cheap to run, and nobody has.

1. A harness-quality metric that predicts the 5x–30x swing before you run. omarsar0 measured the swing post-hoc; nobody can yet predict which harness wins for a given task class without a full benchmark. This is the single highest-value open experiment.

2. Composing the pieces. Mostly resolved 2026-08-26 by Recuris, which manages state externally (verified Working Memory), routes skill retrieval through that state rather than the transcript, and self-optimizes via a fixed Meta-Agent making validation-gated localized updates to Skill Memory, benchmarked end-to-end on four long-horizon benchmarks and ten models. What remains of the original problem: the composition is not ablated, so how much comes from each of the three pieces is unknown, and the cost of maintaining verified state plus running validation gates on long tasks is unpublished.

3. Where self-simulation of the loop diverges from reality — the same self-judgment failure mode the reward-hacking thread keeps flagging. A confidently-wrong auditor breaks the MEA loop.

4. Graph engineering lacks the research the loop layer now has. Partly moved 2026-08-28 by CaSKG, which supplies the mechanism (a graph buys prerequisite structure, and only if the edges are calibrated by intervention rather than inferred from similarity) and the numbers (highest score in all twelve model-benchmark combinations, +7.9 ScienceWorld macro-average over Graph-of-Skills, fewer environment steps). What remains: CaSKG compares graph to graph, never graph to a well-run loop, so what a graph buys over a good loop is still unquantified and now specifiable. Also unpriced: counterfactual probing is quadratic in library size before pruning and must be re-run as the library grows, which is exactly the regime WikiSkill and Recuris create.

5. Offline versus live harness patching at matched budget. New 2026-08-28. AutoSaddler (08-26) patches offline from failure mini-batches, Recuris (08-26) between runs from verified state, PILOT (08-28) during the run. Offline should generalize better (multiple failures per patch), live should recover more of the current task. Nobody has run them against each other, and the answer decides whether a production harness needs a supervisor with abort authority or a nightly patch job.

Sources

Research (digest): Code as Agent Harness · Scaling the Harness · omarsar0 harness benchmark (arXiv 2608.01347) · LongHorizon-Harness (arXiv 2608.01964) · HarnessOpt-Bench (08-08 weekly) · AI4AI at Test-Time (arXiv 2608.12307) · Agent Safety Should Be a Runtime Contract (arXiv 2608.11274) · Spark-to-Paper (arXiv 2608.11924)

Research (digest), 2026-08-14: DarwinX (arXiv 2608.07545) · AutoDesign (arXiv 2608.13560)

Industry / practitioner, 2026-08-14: DeepSeek Harness v0.1 and cache-hit repricing · Ken Huang, Harness Engineering pattern language

Practitioner cluster (surfaced via saved reading, 2026-08): ~12 articles on loop/harness/graph engineering, the dominant theme in the private curation index by a wide margin; synthesized in the 2026-08-13 Media Zone. Enumerated saves live in the private curation index.

2026-08-30: the harness moves the clock, and the model cannot see it

Your Agents Are Not Time Aware (Ofengenden and Andriushchenko, MATS 10) adds a third dimension to this page's core claim, and it is the one the model is structurally unable to estimate. The page already records harness choice swinging cost-per-success 5x to 30x (omarsar0, arXiv 2608.01347) and accuracy roughly 2x (AI4AI). This adds wall-clock: Claude Code consumes roughly 2.5x more turns than Codex on the same model, because Claude Code runs until it believes the task is solved while Codex tends to stop at a time boundary. Two different runtime distributions, and the model's own duration prediction is identical across both.

The prediction is anchored rather than estimated. Measured compression exponents of 0.19 to 0.24 mean predictions barely move as true duration moves: both agents essentially answer "about ninety minutes." On ProgramBench, Opus 4.8 in Claude Code predicted 99 minutes against an actual 85 (1.16x) while GPT-5.5 in Codex predicted 72 against an actual 17.5 (4.12x). On AgentTime, a purpose-built suite of 235 tasks assembled from 18 existing benchmarks with external timers and no artificial caps, Fable 5 over-predicts 3.1x and GPT-5.6 Sol 9.9x.

The ablation identifies where the signal lives, and it is not inside the model. Retrospective accuracy is near-perfect with an elapsed-time tool (R-oracle), only slightly worse in-session without one, substantially degraded on a replayed transcript, and doubles in error when timestamps are scrubbed. Transcript length correlates with runtime at r=0.91, but controlling for length, in-session estimates correlate at only r=0.4. The agent is reading its context for timestamps and using length as a proxy, not maintaining a duration state.

This is the empirical case for LongHorizon-Harness's central design choice. That system (arXiv 2608.01964, 08-13) keeps task state outside the execution context and updates it only on environment-verified facts. This paper says why that is necessary rather than tidy: the in-context signal is a proxy that collapses when the context is manipulated, and the same agents overrate their own work by roughly 20 points same-turn (Opus 4.8, GPT-5.5), with the direction unstable across generations (Opus 5 underrates by 11-15 points in a separate turn, GPT-5.6 Sol overrates by ~7). One reported instance has both models self-scoring near 70% on work that actually scored 7% and 14.5%.

It moves open problem 3 from a hypothesis to two measured quantities. That problem read: where self-simulation of the loop diverges from reality. The divergences now have numbers, a 0.19-0.24 compression exponent on duration and a ~20-point optimism gap on quality, plus a mechanism from the R-scrubbed ablation. It also sharpens open problem 0b, untouched for a sixth consecutive harness paper: an agent that overrates its output by 20 points is a plausible partial explanation of the pass@1-to-pass^k collapse Thinkingbox (08-25) measured (65.36% to 25.25% on stateful work), because a harness that trusts the agent's self-report cannot see the failures pass^k exposes.

It also validates PILOT's (08-28) architecture after the fact. PILOT gives a supervisor authority to abort a running worker, which is a time-based control over an entity with no calibrated sense of elapsed time. PILOT's supervisor reads a streamed trace rather than asking the worker, and this paper is the reason that is a requirement rather than a preference.

The practical statement. "Finish this in 30 minutes" is the cheapest control surface a long-running agent has and it is unusable against a constant duration model. The fix for the retrospective half is a tool call, which is one more decision taken away from the model, which is this page's recurring mechanism. The prospective half is unfixed and nobody ran the obvious experiment: whether telling an agent its own token throughput improves the estimate that actually matters for control. No cost figure is reported anywhere in the study, which is now the standing complaint against every entry on this page.

New open problem 6: budget enforcement belongs in the harness, and nobody has built it. Every cost result on this page is denominated in tokens or dollars while wall-clock is a third axis, and DHH's 24x dollar spread on one identical task (08-16, recorded on compute economics) was explicitly a time-for-money trade, $550 in 45 minutes against $23 in 2.5 hours. An agent that cannot predict its own runtime cannot participate in that trade, so a harness that enforces a time or token ceiling has to do it externally. No harness in this wiki publishes one.