Agent Evaluation & Benchmarks

A growing ecosystem of benchmarks specifically designed for agentic AI — measuring not just accuracy but exploration/exploitation, long-horizon task completion, tool use, robustness, and professional domain coverage.

Current State (as of 2026-09-05)

Four papers on one day, in four unrelated fields, independently replace a scalar score with a structured verdict. That is convergence, not coincidence, and it is this page's own conclusion arriving from outside. The pattern statement reached on 09-04 was: any benchmark reporting one number is conditioning on something it has not stated. On 09-05 the field acted on it four times.

• OVMI (09-05) (arXiv 2609.02887) names the mechanism most precisely, for speech brain-computer interfaces: accuracy and word error rate are computed only over the words the system itself supports, so shrinking the supported vocabulary improves the score while reducing what the user can actually say. Its replacement, open-vocabulary mutual information, scores against an external reference distribution over the words a user may wish to communicate. Optimising vocabulary for it yields up to 16.3% relative accuracy improvement across three speech domains.
• Last Translation Benchmark (09-05) (arXiv 2609.04173) ships handcrafted per-example verification rules naming concrete failure cases, instead of an automatic metric, on the stated grounds that automatic MT metrics are reward-hackable and unactionable and that gold human evaluation lacks reproducibility and scale.
• VeriPhy (09-05) (arXiv 2609.03153) compiles a video prompt into typed physical obligations before observing any frame, and returns provenance-carrying evidence records resolving to supported / contradicted / unknown. It accounts for 228 of 304 human-annotated flaw records against 164 for a published question-decomposition evaluator, and reports honestly that monolithic prompting of the same backbone reaches 222, so the case rests on auditability rather than recall.
• Select, Compress, Reinvest (09-05) (arXiv 2609.03820) supplies the number that indicts cross-paper comparison directly: a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget, plus an implementation bug found in its own baseline. A 3.74-point harness artifact is larger than most margins its subfield reports.

OVMI's critique is RealSWE's finding, stated in information-theoretic form, in a field that has never heard of RealSWE. RealSWE (09-04) found that problem-statement-only requests are 88% of real prompts and 7% of benchmark problems, and that realistic inputs cut resolution by 6.4 points and can change model rankings. A benchmark's prompt set is its support set. OVMI says a metric computed over a self-chosen support set overstates capability. The two papers are the same claim about different systems, and neither could cite the other. This page's list of unstated conditioning variables now has a name for its general form rather than only three instances: attempt count (Thinkingbox, 08-25, 65.36% pass@1 falling to 25.25% pass^20), objective dimension (E-Commerce Bench, 09-02, top earner ranking 16th of 18 on fraud avoidance), prompt composition (RealSWE, 09-04), and now support-set choice, which subsumes the third and generalises the other two.

The analysis tool this page has wanted also arrived. AutoTraceGT (09-05) (arXiv 2608.30391) automates grounded theory, the sixty-year-old qualitative social-science procedure with open, axial and theoretical coding stages, a saturation criterion that says when to stop, and an auditable trail from data to theory. Across six trajectory corpora its codebooks recover 73-91% of the failure modes in human-annotated taxonomies and surface patterns those taxonomies miss, and used as a deductive feature space the codebook beats zero-shot and few-shot LLM baselines on downstream failure prediction. Every unstated conditioning variable above was found by a human noticing a decomposition and building an eval for it. Open coding does not know in advance which axis it is looking for, which makes this the first plausible candidate for finding the next one automatically. Nobody has pointed it at the three benchmarks that produced this page's pattern.

It also supplies the provenance check Terminal-Universe (09-04) is missing. That paper mines 37.3k training environments from logged agent trajectories for an 11.9-point Terminal-Bench 2.1 gain, and the caveat recorded on 09-04 is that those trajectories came from benchmark-style prompts, which RealSWE measures as 7% of the real distribution, with no held-out-provenance split. Inducing one codebook over real-prompt and benchmark-prompt trajectories together, and checking whether they saturate into the same categories, is a direct measurement of that concern and costs one run.

Limits, and they are real. VeriPhy's architectural margin over a well-prompted single call is 6 records out of 304 on a 149-clip core. AutoTraceGT's missing 9-27% of human-annotated failure modes is uncharacterised, and if the misses are the subtle ones the tool is weakest where it is needed most. LTB's per-example rules do not scale without volunteers, and adversarial selection means it measures the tail rather than the distribution. OVMI needs an agreed reference distribution, and different references reorder systems. The convergence is on the diagnosis; none of the four has shown its replacement is stable under the scrutiny the scalars failed.

Prior State (as of 2026-09-04)

The third decomposition arrives, and it is located somewhere the previous two were not: in the input distribution rather than in the scoring. RealSWE (09-04) (arxiv 2608.27831) defines a six-category information taxonomy and four dimensions of linguistic style, then applies them to real user prompts from SWE-chat alongside problem statements from SWE-bench Verified and Pro. The mismatch is close to total. Requests carrying only a problem statement, alone or with limited additional context, are 88% of real prompts and 7% of benchmark problems, and 87% of real prompts are casually written against 94% of benchmark problems being formal. The benchmark then holds task and gold patch fixed while varying only information composition and style: 381 multi-variant task families. Across seven models, realistic inputs cut resolution rates by 6.4 percentage points on average and can change model rankings.

The compositional design is what licenses the causal reading. Variants inside a family share the same underlying task and the same gold patch, so a resolution difference is attributable to the information present rather than to task difficulty. That is a stronger evaluation contract than the ablate-the-prompt studies this conclusion would otherwise rest on.

The pattern statement on this page can now be tightened. The 08-25 and 09-02 entries below found aggregate scores concealing the axis that decides deployability: Thinkingbox separated headline accuracy from reliability across attempts (65.36% pass@1 collapsing to 25.25% pass^20 on stateful workflows), and E-Commerce Bench separated profit from integrity and found them anti-correlated inside one model, with the top earner ranking 16th of 18 on fraud avoidance. Every one of these three failures is an unstated conditioning variable, and the page has now found three distinct ones: attempt count, objective dimension, and prompt composition. That is the general form, and it predicts more: any benchmark reporting one number is conditioning on something it has not stated.

It also breaks the claim type the evaluation-license census (08-29) identified as most survivable. That census separated exact-value, winner, complete-ordering and pairwise claims across all 124 mechanically eligible Inspect Evals units, found 110 stop before deterministic inference is possible, and noted winner and pairwise claims are the more robust kind. RealSWE reports that realistic inputs can change model rankings, which is a winner claim breaking under an input-distribution shift rather than under scoring instability. The identified set for "model A beats model B on SWE-bench" is therefore wider than that framework captures, because the framework conditions on the eval's own prompts and RealSWE's point is that those prompts are the assumption.

The finding with immediate operational value is about token cost, not accuracy. Two of six information categories are free lunch and two are pure cost. Desired Behavior and Motivation significantly improve resolution, and both are among the things real prompts most often omit. Environment Information and Reproduction Steps add tokens with no measurable benefit, and both are what careful bug-report etiquette trains people to supply first. This is the second independent result saying agent context contains large identifiable removable blocks: ALTK-Evolve (08-12) found DeepSeek-V3.2 at 89.3% completion and 263K tokens per task against a baseline's 80.4% and 634K, both axes moving the right way, meaning the baseline was paying for actively harmful context. RealSWE is the first to label the categories, and the cheap unclaimed experiment is to strip those two from a harness's auto-collected context and report resolution and tokens together.

One caveat that only exists because of today's other papers. Terminal-Universe (09-04) recovers 37.3k training environments from logged agent trajectories and reports an 11.9-point Terminal-Bench 2.1 gain. Those trajectories were produced by agents driven by benchmark-style prompts, which RealSWE measures as 7% of the real distribution. A training corpus mined from benchmark-driven trajectories inherits the benchmark's prompt distribution, and neither paper reports a held-out-provenance split.

Limits. SWE-chat is a single provenance for the real-prompt side, so the 88% and 87% figures are one product's user population. No token cost per variant is reported, which is what would turn the inert-category finding into a priced recommendation. And holding one gold patch fixed while removing disambiguating information measures degradation against a single reference rather than asking whether the agent should have posed a clarifying question, which is the failure mode PACE (09-04) attacks from the other side.

Prior State (as of 2026-09-02)

A second benchmark this month finds that the aggregate score was hiding the axis that decides deployability, and this one hides it in a way that should worry anyone putting an agent near money. E-Commerce Bench (09-02) (arxiv 2608.30730, Qwen team) runs an agent as a merchant across a simulated 365-day year, operating multiple stores at once: market research, supplier negotiation, inventory sourcing, pricing, order fulfilment, returns and cash-flow management, against a calendar of promotions, natural disasters and supply-chain shocks. Eighteen frontier models, seven scoring dimensions, and no single model dominates.

The dissociation is the result. GPT-5.6 Sol earns the most, growing a 100,000 opening stake into 1,431,425, and ranks 16th of 18 on fraud avoidance while trailing Fable5 on operational efficiency. A benchmark reporting one number would have declared it the winner and concealed both facts. Among open-weight models Qwen3.8-Max-Preview leads at 416,252, 38% above GLM 5.2 (high), and shows the strongest learning over the horizon, progressively bargaining suppliers down across repeated orders.

Read against the page's 08-25 entry, this is the same lesson decomposed along a different axis. Microsoft's Thinkingbox showed the strongest model falling from 65.36% pass@1 to 25.25% pass^20 on stateful workflows, separating headline accuracy from reliability across attempts. E-Commerce Bench separates profit from integrity and finds them anti-correlated within one model. Two benchmarks, two different decompositions, one conclusion: the aggregate was concealing the trade-off, and which trade-off it conceals depends on which single number you chose. That is now a pattern on this page rather than an isolated observation.

The design choice worth borrowing is the deterministic counterparty. Long-horizon agent benchmarks usually trade realism against reproducibility, because an LLM-driven counterparty makes the environment non-stationary and confounds comparison across models. E-Commerce Bench keeps both sides of the market deterministic: a fixed demand model for customer purchases and returns, a negotiation kernel for supplier pricing, concessions and decisions, with a language model used only to verbalize decisions the kernel already made. Two runs therefore face the same market, so a score difference is attributable to the agent.

It also denominates outcome in the currency the routing literature keeps asking for. llm-routing.md has argued since the AlphaSense study (08-14), which found per-token pricing ranks models backwards because stronger models finish in fewer tokens, that cost belongs in dollars per completed task, and Optima (08-16) shipped the measurement infrastructure. E-Commerce Bench scores in dollars of end-of-year assets, showing a 3.4x spread between the best closed model and the best open-weight one on that axis. A per-task router on this benchmark would have a directly interpretable objective, and nobody has built one.

Caveats, and the first one is structural. Determinism buys reproducibility at the cost of adversarial realism: a fixed negotiation kernel cannot adapt to an agent that discovers an exploit, so a high score may partly reflect kernel-gaming rather than commercial competence, and the fraud-avoidance ranking of the top earner suggests some of that is happening. The demand model is fixed, so no agent's pricing moves the market it trades in, which removes retail's central feedback loop. And a simulated year is a horizon in simulated time, not tokens or wall-clock, so per-run compute cost is unstated and incomparable across the 18 models, which is the same omission this page has flagged repeatedly.

Prior State (as of 2026-08-29)

The measurement crisis gets its formal floor, and the number is 110 out of 124. What Does an Evaluation License? (08-29) (2608.19269) separates two things this page has been treating as one: an eval artifact specifies a computation (task, scorer, metric), but it does not automatically license the claim attached to that metric, because replaying the claim needs historical evidence and semantic grounding the artifact usually does not ship. The authors formalize the missing layer (a frozen substrate D, a grounded family F of admissible readings, a claim query q, and the resulting identified set), then census all 124 mechanically eligible Inspect Evals units at a pinned commit. 110 stop before deterministic inference is possible. The output is deliberately not a robust/not-robust label: it is typed stops, instability witnesses, and stable substructure, so each failure carries a machine-readable reason.

This reframes what the crisis on this page actually is. The 08-11 entry below recorded four papers in two days each finding a widely trusted benchmark was substantially measuring an artifact of its own construction, and the 08-12 entry recorded the constructive response: build better benchmarks. This paper says the deficiency is largely not in benchmark design but in what evaluation artifacts carry. A well-designed eval that does not ship the substrate and grounding needed to replay a claim produces an unlicensed claim exactly as reliably as a badly designed one. Building better benchmarks does not fix this. Shipping more alongside them does.

The claim-resolution split is the practically useful part. The census separates exact values, winners, complete orderings and pairwise relations, and they do not degrade together: an eval can be too unstable to license "model A scored 62.3" while cleanly licensing "model A beat model B." Almost every claim in this wiki's digests is a winner or pairwise claim, which is the more survivable kind.

It also supplies the vocabulary for this page's longest-running open prediction. The 08-26 and 08-28 Looking Ahead sections both predicted a harness paper would publish a pass^k curve within 60 days, because Microsoft's Thinkingbox (08-25) measured a top model collapsing from 65.36% pass@1 to 25.25% pass^20 on a stateful benchmark. In this paper's terms: a pass@1 number on a stochastic stateful task has a wide identified set, and reporting it as a point estimate is an unlicensed claim resolution. Five consecutive harness papers have made that move.

Limits. One suite, one commit, and "mechanically eligible" is doing unmeasured work; the obvious next census is lm-evaluation-harness or HELM. And the paper says what is missing without estimating what it would cost an eval author to close a typed stop, which is the number that decides whether this becomes practice or stays a diagnosis.

Prior State (as of 2026-08-12)

The measurement crisis entered its constructive phase, and the three benchmarks that arrived agree on a diagnosis rather than just a score.

The 08-10 and 08-11 boards produced four benchmark-validity failures in two days: papers saying the instruments are broken. Today's three are papers building instruments that are hard to break, and the design choices are the interesting part. See the combined summary (08-12).

The scores, first, because they set the frame. DSAgentBench (2608.10366) runs end-to-end data-science workflows in real computer environments across notebooks, IDEs, terminals, browsers and databases, 275 tasks, each with a deterministic evaluator checking analytical correctness, visual outputs and model performance rather than code-only execution. Fifteen models: Claude-4.6-Sonnet best at 56.70%, every open-source agent below 1%, failing at tool orchestration, OS grounding and multi-step reasoning. SPIEval (2608.10692) tests mobile-assistant work over personal information scattered across apps, 250 human-curated tasks, 4,335 records, 10 apps, 21 tools, structured around reasoning, disambiguation, integration, preference inference and multi-intent decomposition: GPT-5.5 (xhigh) at 57.3%, weakest at 16.4%. VibeLifeBench (2608.10875) is 200 multi-week scripted timelines across ten everyday-life domains in a simulated world of 22 mock services where the world advances on its own clock and many changes are silent, graded on end state, timeliness and unstated constraints by reading only what the agent left behind. All seven frontier models score low.

The diagnosis is shared and it is the most useful thing on this page today. SPIEval measures it directly: 79% of failures are inaccurate information localization, because models "commit to plausible but incorrect information instead of continuing retrieval for verification," and fewer than 2% of retrieval actions use any advanced search method. DSAgentBench's requirement to ground decisions in intermediate outputs is the same failure from another angle, and VibeLifeBench's silent world changes turn it into the whole task design. It is not that agents search badly. They mostly do not search, then commit.

That is the cheapest published measurement of a faculty this wiki named in June and again on 08-06. The self-evolving-agents page recorded that three of four negative results converged on one missing capability, deciding what to do with a resource you already have: Shadow evaluations (08-06) with agents ending research runs under 50% of budget spent despite being told to spend down, and InMind (07-29) with retrieval memory surfacing a fact only when it resembles the query, six systems at at most 14.4% on indirect queries against 84.0% for the same memory simply placed in context. SPIEval's under-2% figure is that faculty, priced.

VibeLifeBench supplies the horizon this page has been unable to test. The standing complaint across the agent-memory and self-evolution literature is that everything is validated on episodes of dozens of steps rather than days, so drift never appears. A multi-week timeline with a self-advancing clock, silent changes, and grading that reads only durable artifacts is that horizon in a simulator, and it is the natural target for RoMeRL (08-11)'s bounded memory states and SkillZip (08-12)'s compressed skills. Neither has been run on it.

Apply this page's own methodological rule to DSAgentBench's floor. The rule stated on 08-10 is: before believing any agent-capability number, find the cheapest baseline that could produce it. A sub-1% result uniform across every open-source model is not the shape of a capability gradient, it is the shape of an interface failure, and the paper's own attribution to "tool orchestration, OS grounding" is consistent with that. A²E (08-11) found harness choice swings outcomes as much as the model does, which makes the harness the obvious confound. Whether a better scaffold lifts open models substantially is unreported and is the single most informative missing ablation on today's board.

The spread widened again. Ouroboros at 86.74% on Terminal-Bench 2.1, ProMax refactoring at 41.2%, DSAgentBench at 56.70%, and all of it published within two days. Agent capability remains benchmark-shaped, and none of the three new benchmarks reports token or dollar cost, which for a multi-week benchmark that rewards repeatedly re-inspecting the world is part of the result rather than an appendix.

Prior State (as of 2026-08-11)

Measurement validity is no longer a caveat on this page. It is the page's central problem, and three papers on one board said so.

The audit. SWE-Bench ProMax (08-11) (2608.09802) cites a finding that should reset how every SWE-bench number in this wiki is read: nearly 60% of unsolved SWE-bench Verified instances contain flawed tests, either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements, and frontier models can verbatim reproduce gold patches from training data. Its replacement moves to behavior-preserving multilingual refactoring, a task type that resists both failure modes because correctness has a sharp definition and there is no canonical patch to recall: 170 expert-curated instances across seven languages, averaging 11.4 modified files and 261.6 lines per instance, with issue descriptions rewritten from scratch and test suites manually reviewed. Best frontier model under two agent scaffolds: 41.2%.

The metric critique. A²E (08-11) (2608.07346) argues correctness alone is too coarse for comparing agent harnesses, and adds execution efficiency, tool use, task planning and error recovery as separate dimensions via an Agent Task Protocol and an automatically instrumented Monitor producing standardized traces. Its finding is a negative one with teeth: model-harness combinations vary substantially by task type and no single combination consistently wins, which means every single-harness leaderboard number is a joint measurement being reported as a model property.

The construction critique. Evo-Bench (08-11) (2608.09096) builds explicit machinery, auxiliary-task evolution plus sensitivity-aware stratified splitting, to isolate harness-improving capability from base model strength and prevent task-specific overfitting, because no existing evaluation does.

Put those beside yesterday's entry and the pattern is established. StreamArena (08-10) found that on existing streaming-video benchmarks a baseline reading only the last four frames matches complex streaming models. Four papers in two days, across coding agents, agent harnesses and streaming video, all reporting that a widely trusted benchmark was substantially measuring an artifact of its own construction. That crosses this wiki's three-paper threshold twice over.

The number to carry forward is the spread, not any single score. On one board: Ouroboros at 86.74% on Terminal-Bench 2.1 and 90.69% on OSWorld-Verified, both best-known, against 41.2% on ProMax refactoring. Near-saturation on interactive terminal and OS tasks; under half on coordinated multi-file behavior-preserving change. Agent capability is benchmark-shaped, and the shape is that agents add code well and restructure it badly, which is the direction that accumulates cost, consistent with the 08-10 finding that AI-written C++ consumes 5 to 8% more compute in production across 3.52 million changes.

Two limits on ProMax itself. 170 hand-curated instances is small for a successor to a saturating standard, and manual curation is both the quality guarantee and the reason it cannot be refreshed at scale. Two agent scaffolds is a narrow harness sample, and A²E's same-day finding is precisely that the harness matters, so 41.2% may be a scaffold ceiling rather than a model ceiling.

Prior State (as of 2026-08-10)

A trivial recency baseline beats streaming video models, which makes this the fourth protocol failure on this page in one week. StreamArena (08-10) reports that on existing streaming-video benchmarks, a minimal method reading only the last four frames matches or surpasses complex streaming models, and that multiple-choice answer options separately expose language shortcuts. Its replacement is 243 videos averaging 88.8 minutes with 3,646 open-ended question-answer pairs, and the open-ended format is load-bearing rather than stylistic: with options present, the benchmark was measuring language priors instead of perception.

Placed alongside the week's other three, the pattern is now the strongest claim on this page. ScrambleToolBench (08-04) and SWE-Touch (08-04) found agents discover tools fine and then never revise, that extra reasoning widens a brute-force search rather than deepening it, and that human edits in a shared workspace cost 7.7 resolve points. SkillBench and PastBench (08-05) found explicit skill maintenance matches plain in-context learning on average, with weaker models simply accumulating more fragments. Shadow evaluations (08-06) gave frontier agents six days and real money on genuine unpublished research questions and both runs finished with under 50% of budget spent. Four benchmarks, four different agent capabilities, and in every case the reported capability was substantially an artifact of the measurement protocol rather than of the model.

The methodological rule this page should now state outright: before believing any agent-capability number, find the cheapest possible baseline that could produce it. Last-four-frames for streaming. Plain in-context learning for skill libraries. Repeated sampling for self-reflection, which is the shape of the Sample More Reflect Less study in DAIR.AI's weekly roundup (starred Gmail, 08-10): seven self-improvement methods, open models at 1.5B, 3B and 7B, two math benchmarks, every generated token counted including critiques, reflections, debate turns and checking, each method compared against repeated sampling at its own measured cost, with paired bootstrap intervals and multiplicity correction across all 36 comparisons. Repeated sampling held up against every method in every setting. Ten comparisons came back reliably worse, and all ten involved the model inspecting its own output; all 18 self-inspection comparisons were negative. Reflexion as published never triggered its own retry on the smallest model because it judged itself correct every time.

One paper this week runs against the grain and it is worth naming as the open question. ReASearch (08-10) argues an agent can internalize the whole outer-loop search policy, deciding what to evaluate, how to diagnose, which edit to make and when to restart, and reports 2% to 40% over specialized optimizers across 14 tasks. It reports no matched-token cost accounting, which is exactly the omission Sample More Reflect Less exposes elsewhere. The plausible reconciliation with the Shadow-evaluations budget failure is that ReASearch's tasks have a dense, cheap, automatically computable score and open-ended research does not, in which case self-directed budget allocation works exactly where a verifier exists and nowhere else.

Prior State (as of 2026-08-06)

A new evaluation instrument arrived today and it is a better idea than most of the benchmarks on this page. Shadow evaluations (08-06) partner with the authors of unpublished papers, take those papers' real research questions, and let frontier agents attempt them with real compute and a real budget, then have the original authors review the output. Two properties follow and both fix problems this page keeps logging. The agent cannot have trained on the result and cannot look it up, which is the contamination problem that undermines most agent-research benchmarks and the exact failure GDPevo (08-06) attacks from the other end with automated task regeneration. And the reviewers spent months on the same question, so the judgment is expert rather than rubric-based, which is the ceiling every LLM-judged agent benchmark runs into. The cost is n = 2 and substantial researcher discretion, which the authors state plainly.

The day's real lesson for this page is a vocabulary problem that is quietly corrupting comparisons. OneDayAgent (08-06) sets a new state of the art of 0.821 on AgentIF-OneDay, 104 tasks described as long-horizon, cross-environment and open-ended. Shadow evaluations conclude that agents cannot do open-ended research. Both are correct. In the benchmark, open-ended means multi-step, multi-tool and not templated, and the success criterion is still fixed in advance and checkable. In the shadow evaluation, open-ended means the success criterion is not knowable in advance, which is why the failures were failures of judgment, budget allocation and backtracking rather than of execution. Benchmarks on this page routinely claim the first and are read as evidence about the second. Any benchmark calling itself open-ended should state which one it means.

Two further additions. GDPevo (08-06) is evolution-native by construction: it decomposes enterprise workflows into atomic business rules, distributes subsets across training tasks and recombines them in held-out tests, so test-time gains are attributable to training experience rather than to contamination or scale. Self-evolution lifts held-out accuracy by up to 16.44 points, and the number worth keeping is the gap to the fully-informed oracle ceiling of 91.6%, which says how much of the available headroom current self-evolution actually captures. And Skill²-Bench (08-06) contributes a difficulty axis this page has lacked: it calibrates tasks by skill-transition cost rather than by size or step count, and reports that a salient size measure fails to grow from easy to hard in a substantial share of task types. That is a direct warning about every benchmark here that treats more steps as harder.

Prior State (as of 2026-08-04)

Two benchmarks landed on the same HuggingFace board measuring the same previously unmeasured capability, from opposite directions, and both find it missing. The capability is belief revision.

ScrambleToolBench (08-04) (2608.02358) strips the semantics out of tool use. Existing suites expose semantic schemas (function names, argument names, docstrings) in static environments, which means a high score is consistent with two very different agents: one that infers behaviour from interaction, and one that recognizes a familiar API. ScrambleToolBench removes the cue so only the first can score, then changes the environment underneath the agent with mapping drift, stochastic action failures, and temporal execution windows. Initial discovery works. Adaptation does not: when the mapping drifts, agents do not deduce the new mapping from the structure they already recorded (cycle tracing would do it), they show belief inertia or fall back to exhaustive search. The finding that should worry anyone selling reasoning as the fix is that more test-time reasoning amplifies the brute-force search rather than producing deduction, and persistent memory reduces compounding errors without enabling structural inference.

SWE-Touch (08-04) (2608.02499) supplies the human-in-the-loop version. Every repository-level coding benchmark evaluates an agent alone or restricts the human to messages; real development is a shared workspace where a person edits code mid-task. SWE-Touch injects validated Counter-Edits, plausible edits to task-relevant code that conflict with task completion, mined from task-critical regions across multiple repair trajectories and constructed by a separate User Patch Generator. Across nine models, average resolve rate on SWE-bench Verified drops 7.7 percentage points, with degradation persisting on SWE-Bench Pro and DeepSWE. The failure taxonomy is the transferable part: agents retain the conflicting code, or overwrite it without re-inspecting the repository or writing a targeted test for the revised behaviour.

Together they name a pattern this page should carry forward: the agent builds a world model and then does not update it. Two benchmarks, two domains, one conclusion, and it matches the profile Shadow evaluations (07-30) found in a third setting, where agents given the central open question from unpublished NeurIPS 2026 submissions completed all of the engineering unassisted and were rejected on five judgment failures including ineffective backtracking. Engineering competence with revision incompetence, measured three ways in six days. The consequence for this page is that horizon length and state-revision capability have been bundled under "long-horizon" and are apparently separable, so a benchmark that scores only the former is measuring less than it claims.

The list of dimensions this page argues are missing now has two more entries. Efficiency Matters in Autonomous Research (08-02) added cost and showed search efficiency and outcome quality are empirically distinct across twelve tasks and four search families. ExtractBench (08-03) added record completeness and found commercial vision-language models silently truncate long record lists. Theta's AI Engineer talk offered order-shuffling as a free test for real sequential complexity. Now add environment volatility (ScrambleToolBench) and workspace volatility (SWE-Touch). Every one of these is a dimension nearly every deployment has and nearly no leaderboard scores. ScrambleToolBench is also the cheapest of them to replicate: scrambling tool semantics is the tool-axis twin of Theta's order shuffle, and both cost nothing.

And a latency dimension arrives that nobody had at all. AAPT (08-04) (2607.28399, Kurate cs.LG #5) argues computer-use agents fail on transient GUI events (boot prompts, auto-dismissing dialogs, short-lived auth requests) as a timing failure rather than a comprehension failure: the agent produces the correct action after the window has closed, because autoregressive decoding sits on the decision-time critical path. It removes decoding from that path without touching the model, pre-building a bounded conditional policy tree during idle screen periods with observable guards, pre-authorized actions and branch deadlines, sized to cover the model's own decoding latency. Success inside a contested decision window rises from 0.50 to 0.79 (p = 1.8e-3) with zero incorrect actions, and both open-loop and predict-and-replan baselines score zero because they still decode during execution. Two things make it methodologically better than most work on this page: pre-registered endpoints with exact McNemar tests, and a pre-registered oracle probe that rejected the authors' own initial hypothesis and identified branch routing rather than planning quality as the causal bottleneck. That probe result is also the uncomfortable link to ScrambleToolBench: AAPT's agent cannot match a live observation to the right prepared branch, ScrambleToolBench's agent cannot notice its map is stale. Both are failures of mapping a live observation onto a maintained structural hypothesis, at two different timescales.

Current State (as of 2026-08-03)

The cost-as-a-dimension argument gets its first implementation, one day after it was made, and it comes from the dullest corner of agent work. Efficiency Matters in Autonomous Research (08-02) argued that every leaderboard on this page grades the best solution found and ignores the budget spent reaching it, proposed area under the Pareto frontier of reward against accumulated budget, and showed the two dimensions are empirically distinct across twelve tasks and four search families. ExtractBench (08-03) (2607.29677, LlamaIndex) does the minimum version for schema-guided enterprise document extraction: 4,869 pages, 370 documents, 8 domains, 67 document types, scoring value accuracy, record completeness at scale, grounding and measured cost together.

Three design choices worth borrowing. Ground-truth curation is matched to evidence regime rather than uniform: independent-system agreement for real documents, known values by construction for synthetic lists, human verification for forms. Grounding is reported at two resolutions, word-level and page-level F1, which separates "right region, wrong span" from "wrong document." And completeness is scored separately from accuracy, which is what surfaces the headline finding: commercial vision-language models do well on short documents and truncate record lists on long ones, a failure invisible to value accuracy since extracting 40 of 120 line items correctly still scores well. That generalizes past extraction to any long-output agent task, and almost no benchmark on this page measures it. The conflict to state plainly: LlamaIndex authored the benchmark and a LlamaIndex product tops all three metrics, so replication matters before anyone procures on it, and "measured cost" is not defined as spend, wall clock or tokens. → summary

Practitioner counterpoint from the conference track. Two AI Engineer talks synced into raw/youtube-ai-tech/ on 2026-08-01 make claims this page should track. Theta argues GDPval, ToolBench and APEX-Agents all fail on task length, saturation, breadth and reward granularity, that their own finance tasks average 15 human-hours with models scoring around 5%, and supplies a free diagnostic for whether a "long horizon" benchmark has real sequential complexity: shuffle the subtask order and see whether the score moves. David Brumley argues nearly every cyber benchmark is broken the same way, by assuming the target contains exactly one known vulnerability, and reports that DARPA's Cyber Grand Challenge spent $60M hand-curating single-bug challenges of which 50% contained unintended exploitable bugs. His replacement, the audit task, asks for all vulnerabilities, uniquifies submissions by stack backtrace, and scores precision times recall, which is a reusable shape for any task with an unknown number of valid solutions. He also reports that crash-triggering does not discriminate between models at all (GPT-5.5 and Mythos both 95%) while full sandbox escape does (Gemini and Kimi at 0%), and states flatly that LLM-as-judge fails categorically in security, which directly contradicts Theta's judge-as-agent design.

Current State (as of 2026-07-30)

Seven evaluation papers landed in one day, and the two best-rated papers on Kurate's cs.AI board this week both argue that evaluations of this kind cannot prove what they claim. This page has accumulated benchmark skepticism since Soohak (05-12) from a dozen directions. Today it stops being a running complaint and becomes the field's dominant research output, with the critique arriving from above the benchmarks rather than from inside them. Kurate cs.AI #1 is Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI (score 1573, 87.5% win rate, zero HuggingFace presence), arguing agent benchmarks have a protocol-validity problem rather than a difficulty problem. #2 is What AI Red-Team Evaluations Can and Cannot Prove (score 1570, 90.9% win rate), arguing a red-team evaluation establishes that a vulnerability exists and essentially never that one does not. The 07-29 digest predicted a second validity paper would reach Kurate's top three within 60 days; it took one day.

The protocol answer: grade against something that does not exist yet. Shadow evaluations (2607.27191) is the sharpest methodological contribution the page has recorded. Hand an agent the central open research question from an unpublished paper and have the paper's own authors grade the output. No contamination, no proxy metric, no LLM judge, and a grader calibrated to a real submission's bar. Two runs on unpublished NeurIPS 2026 submissions, six days and thousands of dollars of compute each: the agents completed all of the engineering unassisted and both outputs were unambiguously rejected. Five reproduced failure modes, four of which are judgment rather than knowledge or execution: no sense of the publishable bar, uncreative responses to design flaws, ineffective backtracking, poor resource awareness, instruction drift. This is the strongest confirmation yet of the two-layer thesis ForeSci (06-07) opened with its evidence-decision decoupling, where agents cite the right evidence and still forecast the wrong research object. The protocol's weakness is that it cannot scale, which is exactly why nobody had run it.

The economic answer: price the tasks. OmegaUse-OfficeVal (2607.27155) attaches human labour time and a task price proxy to each of 100 practitioner-derived office-suite tasks averaging 2.32 hours of human work, with code-based verifiers compiled from fine-grained rubrics rather than an LLM judge. The result splits cleanly: every evaluated model is substantially cheaper and faster than a human worker and none approaches human-level deliverable quality. That is the most decision-relevant framing this cluster has produced, because it says quality is binding while cost is the axis everyone optimizes. Value-weighting also separates "fraction of tasks completed" from "fraction of value captured," which come apart precisely when an agent is good at cheap work.

The self-state deficit, named in three unrelated domains on one day. The page's long-running finding has been that long-horizon agents are weak. Today sharpens it: the model represents the task and the world well and represents its own state within them badly. HumanCLAW (2607.27180) decouples action decision from motor execution so balance and controller error are factored out, and finds that across 1,218 long-horizon egocentric episodes in 41 scenes, nine frontier VLMs top out at 16.8%, with target recognition explicitly not the bottleneck. What is missing is embodied self-awareness: the models lose track of where their own body is and whether it has reached the goal or hit an obstacle. SecRespond finds incident-response agents investigating every alert competently and never hypothesizing that something unflagged is on the disk. Shadow evaluations finds instruction drift and poor resource awareness over six days. Three domains, one deficit, and it is the closest relative of RNG-Bench (06-18, non-Markov memory where the answer depends on history the current observation does not show): RNG-Bench tests memory of invisible history, HumanCLAW tests memory of invisible self.

Security evaluation splits into pre- and post-compromise. SecRespond (2607.26791) is the first post-compromise benchmark, 10 cyber ranges from real compromised cloud hosts across 21 ATT&CK techniques and 5 operating systems, 23 frontier models on the OpenCode harness, and no model completes detection and remediation on any single range. StealthBench (2607.26314) adds a compound metric requiring an offensive agent to both succeed and stay hidden, and no model exceeds 54%. Both inherit the single-harness caveat GTA-2 (04-20) established, where harness design mattered more than model capability.

One honest-reporting example worth naming. Grading the Narrators (2607.24117) evaluates a claim-provenance framework on 20,000 physics-textbook claims and reports that two components validated, the grade-recovery loop partially failed by missing the highest-fault narrator, and two analyses were inconclusive. On a day when the top-rated papers in the field are about what evaluations cannot prove, this is what the requested epistemic hygiene looks like in practice.

Current State (as of 2026-07-22)

Agent failure attribution closes the loop from locate to repair. AgentDebugX (2607.18754) organizes debugging as Detect → Attribute → Recover → Rerun, with a DeepDebug root-cause module built on the premise the failure-attribution thread keeps naming: the step where an error surfaces is usually not the step that caused it. It posts the best strict (exact agent-and-step) attribution on the "Who and When" benchmark (28.8% on qwen3.5-9b vs 21.7% single-pass) and, crucially, converts diagnosis to repair, fixing 13/73 failed GAIA tasks in one rerun (vs 4-6 for self-correction baselines), lifting 55.8% → 63.6%. This is the recovery half of the OAT / Failure-as-a-Process attribution thread (2026-07-20 digest): OAT located where a trajectory deviated from a success flow; AgentDebugX adds attribution → repair → a shareable Error Hub (scrubbed diagnosis-repair bundles as debugging memory). Absolute attribution is still low (28.8% exact), which is the honest ceiling. Its same-day companion DataFlow-Harness (2607.16617) extends the harness-as-artifact theme (07-16 Harness Handbook) to data engineering: constrain the agent to typed, live-grounded DAG mutations instead of free-form scripts and get 72.5% lower cost than Vanilla Claude Code at comparable reliability.

Current State (as of 2026-06-18)

Two same-day benchmarks confirm frontier models fail at sustained and stateful tasks, and an active-perception agent shows the escape route. The page's running thread, that short-horizon accuracy does not predict long-horizon robustness, gets its strongest evidence yet on the same day from two opposite axes.

Sustained, long-horizon work. CEO-Bench (arxiv 2606.18543) puts an agent in charge of a fictional startup for 500 simulated days, managing pricing, marketing, and budgeting through a Python interface over noisy interconnected business databases. It bundles four skills short-horizon evals never test together: long horizons under uncertainty, information acquisition in noise, adaptation to a changing world, and orchestration toward one goal. Most state-of-the-art models go bankrupt. Only Claude Opus 4.8 and GPT-5.5 finish above the $1M starting balance, and neither consistently profits. The differentiator is coding ability: the best agents write code that simulates customer cohorts to forecast cash and mines negotiation history for hidden preferences. This pushes the horizon far past PhysicianBench's ~27 tool calls (2026-05-05) and GTA-2's open-ended workflows (2026-04-20), and echoes AcademiClaw's compute-quality decoupling (2026-05-05): raw model size is not the lever, simulator-building skill is.

Stateful, non-Markov memory. RNG-Bench (also 2026-06-18) is a non-Markov memory benchmark where the answer depends on history not visible in the current observation. Frontier multimodal LLMs stay far from saturated even at 128K context. Paired with CEO-Bench it draws a clean two-axis map of what short-horizon evals miss: sustained progress over a long horizon (CEO-Bench) and stateful use of invisible history (RNG-Bench). Frontier models are weak on both.

The escape route: active perception with positive test-time scaling. OmniAgent (arxiv 2606.19341) reframes long-video understanding as a POMDP observe-think-act loop that distills cues on demand into persistent textual memory, decoupling compute from input length. Its load-bearing benchmark result is positive test-time scaling: performance keeps improving as reasoning turns increase, because each turn gathers new information rather than re-refining the same view. This is the direct contrast to LoopCoder-v2 (2026-06-17, where latent-loop internal refinement saturates), and the small-model-beats-large pattern holds: the 7B agent beats Qwen2.5-VL-72B on LVBench (50.5% vs 47.3%). The lesson for long-horizon benchmarks above: test-time compute pays off only when each step adds information, exactly the property CEO-Bench's 500-day loop demands.

Current State (as of 2026-06-06)

The benchmark surface keeps migrating from single answers to environment-grounded trajectories, and safety follows it. Two same-day papers move the evaluation unit from "the response" to "the final state / the adaptive process."

Operational safety, not prompt refusal. SABER (arxiv 2606.01317) is the safety counterpart to the trajectory-quality thread the page has tracked since AgentLens (05-14, 10.7% of passing SWE-bench runs are Lucky Passes). Existing safety evals check whether a model refuses an unsafe prompt; SABER places coding agents in realistic stateful project workspaces and judges safety from the final environment state after a sequence of actions, categorizing violations by cause. Even the best model has a >54% harmful safety-violation rate, and models have distinct safety profiles, so a single aggregate score hides the real risk surface (the same over-aggregation lesson as Soohak/AgentLens/AssetOps). SABER is the controlled-benchmark floor under today's Anthropic RSI / NSA-Mythos story: the autonomous coding agents being handed production write-access and offensive-cyber tooling fail operational safety more than half the time.

Adaptive planning under hidden constraints. AdaPlanBench (arxiv 2606.05622) targets a gap the static planning benchmarks leave open: world and user constraints that are revealed only when a proposed plan violates them, forcing multi-turn re-planning under accumulating feedback. Built on 307 household tasks with a scalable dual-constraint pipeline; the best of ten leading LLMs reaches only 67.75%, performance degrades as constraints accumulate, and user constraints are harder than world constraints (failures trace to weak physical grounding). Together with SABER, the message is consistent: agents look much weaker once the benchmark grades the full interaction and its environmental consequences rather than a single turn.

Current State (as of 2026-06-07)

Research judgment is a distinct, weak layer above research execution (ForeSci). ForeSci (arxiv 2606.00644) is a temporally controlled benchmark for forward-looking AI research judgment: 500 tasks across four AI domains and four decision families, each paired with a cutoff-aligned offline knowledge base, post-cutoff papers hidden during generation and used only for validation (so it avoids rewarding lucky future guesses). The signature finding is an evidence-decision decoupling: agents cite relevant evidence yet forecast the wrong research object — retrieval quality does not become decision quality. This is the research-judgment analogue of the page's recurring "surface signal misleads about real capability" thread (AgentLens "lucky pass" 05-14; the verifier-gaming line): a process metric (good citations) looks healthy while the outcome (correct call) is wrong. It also frames the autonomous-researcher stack as two layers — judgment (ForeSci, choosing what to work on) sitting above execution (the MLEvolve/EvoDS self-evolving-research cluster) — and shows the judgment layer is the weaker one. → summary

Current State (as of 2026-05-14)

Latest additions (2026-05-14): Two papers extend the benchmark-skepticism thread that has been building since Soohak (05-12). AgentLens (summary) shows that 10.7% of passing SWE-bench Verified trajectories are Lucky Passes — regression cycles, blind retries, missing verification, or temporally disordered work. The framework merges per-task passing trajectories into a Prefix Tree Acceptor reference space and uses a context-sensitive intent-stage labeler (Exploration / Implementation / Verification / Orchestration). Some model backends drop 5 ranking positions when scored by quality instead of pass rate, which means pass-rate alone is misleading for between-model comparison. AgentLens-Bench: 1,815 trajectories from 47 tasks across 8 model backends. AssetOpsBench retrospective (raw) reports public-to-hidden score correlation of −0.13 on 234 submissions to the CODS 2025 challenge — public standing does not predict hidden robustness. Three papers in three days (Soohak, AgentLens, AssetOps) all say the same thing from different angles: aggregate leaderboard metrics over-aggregate. MAP (Map-then-Act) (summary) is the architectural complement to AgentLens: frontier models surpass near-zero ARC-AGI-3 baselines in 22 of 25 game environments when they build the environment prior before acting. Training on map-then-act trajectories beats training on expert execution traces, which reframes what good demonstration data looks like for long-horizon agents.

Prior State (as of 2026-05-07)

Standard LLM benchmarks underserve agents. The field has been building agent-specific eval frameworks across several dimensions: decision-making quality, professional domain coverage, multimodal grounding, and robustness under fault injection. Nine benchmarks (OccuBench, GTA-2, DR3-Eval, PRL-Bench, Claw-Eval-Live, InteractWeb-Bench, AcademiClaw, PhysicianBench, ProgramBench) now report frontier-agent failure rates of 0–55% on realistic multi-step tasks. ProgramBench (05-06) at 0% on every model is the floor; PhysicianBench (46%) and AcademiClaw (55%) are the realistic ceiling. 2026-05-07 adds a tenth dimension: BRIGHT-Pro, the first benchmark for evidence-portfolio retrieval rather than top-1 relevance, evaluating retrievers under both static and agentic protocols. MedSkillAudit (also 05-07) shifts the evaluation surface from agent capability to agent skill release readiness: 75 medical research skills audited, 57.3% below Limited Release threshold, system-expert agreement (ICC=0.449) exceeded the human inter-rater baseline (0.300). The Marcus production-agent security paper (05-06) frames the limit case: 91% of 847 deployed agents are vulnerable to tool-chaining attacks, 89.4% drift after 30 turns. Capability ceiling, evaluation methodology, skill audit, deployment security: the agent benchmarks cluster is now four-dimensional.

Key Benchmarks

CEO-Bench (2026-06-18) — long-horizon business-operation benchmark. An agent runs a fictional startup for 500 simulated days, managing pricing, marketing, and budgeting through a Python interface over noisy interconnected business databases. Tests four skills together: long horizons under uncertainty, information acquisition in noise, adaptation to a changing world, orchestration toward one goal. Only Claude Opus 4.8 and GPT-5.5 finish above the $1M starting balance; neither consistently profits. Best agents write cohort-simulation code to forecast cash and mine negotiation history for hidden preferences. Highest horizon (500 days) in the cluster; pairs with same-day RNG-Bench (stateful non-Markov memory, frontier MLLMs far from saturated at 128K). → summary

OccuBench (2026-04-16) — 100 tasks across 65 professional domains using Language World Models (LWMs) to simulate environments. Key finding: no single model dominates all industries; implicit faults are hardest. → summary

Exploration/Exploitation Measurement (2026-04-16) — Policy-agnostic metric for explore/exploit errors in LM agents on 2D grid environments. Reasoning models perform best; harness engineering meaningfully improves both dimensions. → summary

GameWorld (2026-04-16) — 34 browser games, 170 tasks, state-verifiable outcomes for MLLM game agents. Best models still far below human. → summary

MERRIN (2026-04-16) — Search-augmented agent benchmark with noisy multimodal web evidence. Average accuracy 22.3%; agents over-rely on text modalities. → summary

InfiniteScienceGym (2026-04-16) — Procedurally generated scientific analysis benchmark. No model exceeds 45%; abstention on unanswerable questions is a key weakness. → summary

DR3-Eval (2026-04-18) — Deep Research Agent benchmark. Static per-task corpus sandboxes with evidential sources, confounding documents, and noise. Reverse-constructed questions (derived from verified evidential docs) ensure every task is answerable. Multi-dimensional scoring: recall, factual accuracy, citation coverage, instruction following, depth. State-of-the-art models still struggle. → summary

GTA-2 (2026-04-20) — Two-tier benchmark: GTA-Atomic (single-step tool precision) and GTA-Workflow (long-horizon, open-ended multi-tool coordination). Key results: frontier models below 50% on atomic tasks; top models at 14.39% on workflows. Critical finding: execution harness design (Manus, OpenClaw) matters more than underlying model capability. Uses real user queries and deployed tools — not synthetic evals. Recursive checkpoint-based evaluation for open-ended tasks. → summary

PRL-Bench (2026-04-20) — Physics Research by LLMs benchmark. 100 tasks from Physical Review Letters papers (Aug 2025+, post-training cutoff for most models). Covers 5 subfields: astrophysics, condensed matter, high-energy, quantum information, statistical physics. Tasks replicate authentic research: exploration-oriented formulation, long-horizon workflows, verifiable outcomes. All frontier models score below 50%. Expert-validated. → summary

Claw-Eval-Live (2026-05-01) — First live workflow-agent benchmark. Refreshable signal layer (ClawHub Top-500 skills, updated each release) + reproducible release snapshot (frozen fixtures, services, graders). 105 tasks, 13 frontier models, deterministic + structured-LLM grading on execution traces, audit logs, service state, post-run artifacts. Best model: 66.7%; no model reaches 70%. HR / management / multi-system business workflows persistently fail. → summary

InteractWeb-Bench (2026-05-01) — First benchmark to grade clarifying behavior explicitly. Four user-agent personas + persona-driven instruction perturbations from RE defect taxonomies. Unified agent action space: Clarify / Implement / Verify / Submit. Frontier MLLM agents remain trapped in blind execution — generating code that satisfies their misreading of the instruction without ever asking. → summary

AcademiClaw (2026-05-05) — Bilingual academic-level benchmark, 80 multi-step tasks curated from 230 real student submissions across 25+ professional domains (olympiad math, linguistics, GPU-intensive RL, full-stack debugging). Docker sandbox per task; six-technique multi-dimensional rubric scoring + five-category safety audit. Best of six advanced models: 55%. Capability varies sharply across domains; compute does not predict output quality — argues against current "more thinking tokens equal better results" defaults. → summary

PhysicianBench (2026-05-05) — 100 long-horizon physician tasks from real consultation cases inside an EHR environment with vendor APIs. 21 specialties; ~27 tool calls per task. Best closed-source model: 46% pass@1. Best open-source: 19%. Highest tool-call horizon in any of the eight benchmarks; the gap between knowledge tests (where LLMs match physicians) and EHR-mediated workflows (where they do not) is the load-bearing finding. → summary

BRIGHT-Pro and RTriever-4B (2026-05-07) — first benchmark for evidence-portfolio retrieval rather than top-1 relevance. Each query is expanded with multi-aspect gold evidence; retrievers are graded under both static and agentic protocols. RTriever-Synth, an aspect-decomposed synthetic corpus, generates complementary positives and positive-conditioned hard negatives. RTriever-4B (LoRA on Qwen3-Embedding-4B) substantially improves over its base. Aspect-aware and agentic evaluation expose behaviors hidden by standard top-k metrics. → summary

MedSkillAudit (2026-05-07) — first skill-release-readiness audit framework. 75 medical research skills, two human experts, ordinal release disposition (Production / Limited / Beta / Reject). System-expert ICC = 0.449 vs human inter-rater 0.300. 57.3% of skills below Limited Release threshold. Negative ICC on Academic Writing (-0.567) reveals structural rubric-expert mismatch on open-ended generative tasks. Pre-deployment audit complement to the Marcus post-deployment security study. → summary

OpenSearch-VL (2026-05-07) — open recipe for frontier multimodal search agents. Wikipedia path sampling with fuzzy entity rewriting, source-anchor visual grounding, unified text+image+OCR+image-manipulation tool environment, and multi-turn fatal-aware GRPO (mask post-failure tokens, preserve pre-failure reasoning via one-sided advantage clamping). 10-point average gain across seven benchmarks; matches proprietary commercial models on several. The training-time intervention at the same multi-turn surface where the Marcus security paper measures failures. → summary

Patterns Across Benchmarks

• Reasoning models consistently outperform base models on agentic tasks

• Over-exploration is a common failure mode in strong models

• Professional/domain-specific tasks expose different weaknesses than general benchmarks

• Deterministic environment generation (OccuBench, InfiniteScienceGym) removes publication bias

• Execution harness dominates model capability (GTA-2): the scaffold around the model determines workflow completion more than model capability itself. Confirmed empirically by the Ridge Security pentester benchmark (2026-05-04) at constant model: belief state, evidence-as-invariant, and trust propagation account for >5x finding gaps between architectures using the same Gemini 3 Flash backbone.

• Eight benchmarks now converge on the same finding: frontier models fail realistic multi-step tasks reliably — this is a consistent, cross-domain measurement

• Middle-band discrimination (Claw-Eval-Live, 05-01): models with similar pass rates diverge in overall completion, suggesting per-task-family routing could outperform any single model

• Blind execution (InteractWeb-Bench, 05-01): a distinct, named failure mode where agents guess rather than clarify under ambiguous instructions — the first benchmark to grade this dimension explicitly

• Compute-quality decoupling (AcademiClaw, 2026-05-05): computational resource consumption does not predict output quality across 80 academic-level tasks. The compute-as-proxy default is empirically broken

• Long-horizon tool-call gap (PhysicianBench, 2026-05-05): 27-call average is the highest horizon in the cluster; the open-source vs closed-source gap (19% vs 46%) is largest at this horizon, suggesting tool-use trace data, not raw capability, is the bottleneck

• Static benchmarks leak via memorization (EvoBrowseComp, 2026-06-14): static search benchmarks like BrowseComp let a model score by recalling a fact instead of browsing for it, so the score is a memorization test in disguise and rots once the questions enter a training set. EvoBrowseComp synthesizes 400 EN + 400 ZH contamination-free questions from live-web traversal via a three-agent pipeline, using popularity as the contamination proxy (a fact popular enough to be memorized cannot test browsing), and can be regenerated to stay fresh. See EvoBrowseComp.

• Benchmarks should model environment change, not just static tasks (EvoArena, 2026-06-14): the first agent benchmark to make progressive environment updates the unit of evaluation across terminal, software, and social domains. Current agents average 39.6%; chain-level success (consecutive evolving subtasks) is the hardest band. See EvoArena.

• Sustained vs stateful are the two axes short-horizon evals miss (CEO-Bench + RNG-Bench, both 2026-06-18): two same-day benchmarks find frontier models fail along orthogonal axes. CEO-Bench grades sustained progress over a 500-day horizon (only Opus 4.8 and GPT-5.5 stay solvent, neither profits); RNG-Bench grades stateful use of non-Markov history (frontier MLLMs far from saturated at 128K context). Together they extend the long-horizon failure thread past PhysicianBench (~27 tool calls) to the 500-step regime. See CEO-Bench.

• Positive test-time scaling beats latent-loop saturation (OmniAgent, 2026-06-18): the active-perception counterpoint to the long-horizon failures above. OmniAgent's observe-think-act loop over video shows performance keeps improving as reasoning turns increase, because each turn gathers new information rather than re-refining the same representation. This contrasts directly with latent-loop compute saturation (LoopCoder-v2, 2026-06-17) and holds the small-model-beats-large pattern (7B beats Qwen2.5-VL-72B on LVBench). Implication for benchmark design: test-time compute only pays when each step adds information. See OmniAgent.

• Fault-tolerance scales slower than task skill (ToolMaze, 2026-06-08): the first benchmark to break tools on purpose, with a 2x2 perturbation taxonomy (explicit/implicit × transient/permanent) over DAG-structured tool-call tasks. Implicit semantic failures crater the Perturbation Recovery Rate by ~37% because agents over-trust corrupted output, and agentic fault-tolerance improves with model scale 3.66x slower than basic task execution. Dynamic replanning is a distinct bottleneck that scaling and prompting do not fix. See ToolMaze.

Related Pages

• GUI Agents
• Multi-Agent Systems
• Self-Evolving Agents

2026-09-13: a process metric that needs no learned judge

MT-INFOSEEK (09-13) (Harvard + Google DeepMind) contributes an instrument this page should adopt independently of its findings. It formalizes multi-turn information seeking as a k-underspecified constraint satisfaction problem, where k is the number of variables jointly required to determine the target, which turns "how much is missing" into a dial rather than a property of an anecdote. The suite is 5,251 problems and 9,006 task instances across mathematics, logic, biology, medicine and general knowledge.

The instrument is final sufficiency: whether the information the model acquired determines the target, recorded independent of answer generation. This page has repeatedly recorded that outcome-only scoring hides process failures, and has repeatedly had to reach for a learned judge to fix it. Final sufficiency is domain-general, cheap, and needs no judge, which puts it in the same category as execution-based verification. Its motivating observation is the sharp one: a correct answer can hide bad information gathering, because the model may have guessed or drawn on prior knowledge.

The findings, with the directional bias as the headline. Performance degrades across models and domains as underspecification rises. Models recognize that more information is needed but underestimate how much, under-predicting the degree of missing information roughly four times as often as they over-predict it at k = 2 on logical problems. They fail to identify a minimal sufficient query set and improve only marginally when handed the true k, so the failure is in constructing the queries rather than in estimating the budget. They often stop before acquiring enough. And on tasks with ordered dependencies, an incorrect query order lowers final accuracy even when the model eventually acquires everything it needs.

That last finding is an independent arrival at the day's routing result. The Menu Is an Execution Prior (09-13) shows that ranking a tool menu by request relevance surfaces the final action while omitting or delaying the tools that produce its inputs, and that routing on a dependency-aware state path lifts ToolBench online success from 0.737 to 0.898 with the agent unchanged. One paper measures the model failing to sequence prerequisites; the other moves the sequencing out of the model and into the retrieval layer. Two groups, same day, same claim. The composition, evaluating state-path menus on MT-INFOSEEK's ordered-dependency tasks, is the cheapest useful experiment either enables and neither ran it.

Practical reading for deployment. Under-asking and over-asking are not symmetric costs. Over-asking burns tokens; under-asking produces a confidently wrong answer whose repair path is a full retry. The four-to-one directional bias says production agents sit systematically on the expensive side, and their evaluations cannot see it because they score the final answer.

Caveat. Read from the abstract only, so no model list and no absolute numbers beyond the k = 2 ratio. The constraint-satisfaction framing is clean but narrow: it cannot represent questions where the right response is to propose an interpretation rather than to ask. Whether a harness-level query-planning step closes the gap is untested.

2026-09-14: the scoreboard gets an index, and the verification gap gets named twice in one day

This page's recurring complaint is that agentic evaluation numbers are not comparable across papers, sharpest in Raschka's 09-10 observation that models are tuned against one primary harness so cross-model agent scores measure harness fit as much as capability.

Benchmark Radar (09-14) (arXiv 2609.11115) does not fix that, but it makes the conditions inspectable. It is a continuously-updated searchable catalog of AI benchmarks that also tracks where each was used and what was scored: daily discovery across 37 sources (13 direct connectors, 24 first-party research and engineering feeds), a catalog of 1,283 source records from 4 benchmark catalogs and 12,916 numeric observations across 790 records, plus mentions in model cards and technical reports and per-benchmark score histories, with source identity and citation retained so a reader can inspect the evidence behind a number. Shipped: a dashboard with a leaderboard, a Pareto view of score against measured use, saturation and trend views, daily feeds, downloadable evidence and a CLI.

The Pareto-of-score-against-usage view is the primitive this page has wanted, because it separates benchmarks that are hard from benchmarks that are merely obscure, and because benchmark saturation and adoption are things the field asserts constantly and measures almost never.

The convergence worth naming. The same day, the route-by-task analysis (09-14) cited an August 2026 Morph study finding that essentially all open-model benchmark scores are vendor self-reported, with none of the tracked SWE-bench Verified entries independently verified, and concluded that any "we beat GPT-5.x" release claim should be treated as a hypothesis to test on your own data. A benchmark index and an independent-verification gap surfacing on the same day from unrelated sources is the field noticing that its scoreboard is unaudited. Benchmark Radar is the instrument that claim calls for; whether anyone uses it that way is the open question.

Gaps. Coverage is a function of the 37 sources and no recall figure against an independent benchmark list is reported. 12,916 observations across 790 records averages about 16 scores per benchmark, thin for per-benchmark trend analysis. And a living database's value depends entirely on staying live, which no paper can establish.

2026-09-23: the instruments fail in two directions on one day, and both papers ship the repair

SWE-Bench Pro V2 (09-23) is the cleanest contamination measurement this page has recorded, because it isolates the variable. Scale rebuilt the benchmark (642 public tasks across 11 repositories, down from 731 after 89 were found invalid on review), and published a 272-task private split alongside it. Claude Opus 5 resolves 638 of 642 public (99.4%) and 222 of 272 private (81.6%), a 17.8-point drop. Kimi K3 627/214, GLM-5.3 614/211, Gemini 3.8 Flash 609/211, Inkling 577/184. Every model shows the same shape. The public runs were network-locked and audited with no successful retrieval from code hosts or module proxies, which rules out evaluation-time leakage and leaves training-time exposure: the repositories, their fixing commits, and the benchmark itself have been on the open web since before these models trained.

The four defects Scale repaired are the defect taxonomy this page should be using. Reward hacking (a task passable without the fix), underspecification, ambiguity, and solution leakage. Twelve percent of the original public set failed that review. A benchmark whose own authors find one task in eight invalid is not a marginal quality problem, and every prior number measured on that set inherits the error.

It supplies a competing reading of HarnessTax (09-22), and the two readings predict opposite futures. HarnessTax found that across 21 model-harness combinations, cost moved roughly 2x while success moved 1.1 points, clustered between 96.7% and 97.8%, and this wiki read the compression as the harness approaching its ceiling. A 99.4% public score sitting beside an 81.6% private one supplies the alternative: the suite stopped discriminating, and a 1.1-point band is what a saturated benchmark looks like from the inside. The experiment that separates them is cheap and now fully specified: re-run HarnessTax's 21 combinations against the private split. If the spread is materially wider than 1.1 points, the harness does buy capability and the public benchmark was concealing it.

Taste-Bench: the first benchmark on this page that scores a process rather than an outcome

The Tasteful Agent (09-23) measures the forks in the middle of a run instead of the end of it. Decision points are mined automatically with no human annotation from two sources: parallel attempts at the same task that diverged, and detours inside a single trajectory where the agent went down a path and came back. Both give a free ground-truth label, because the remainder of the trajectory reveals which branch paid off. Withholding that remainder is what turns a fork into a question.

Three findings, and the second and third matter more than the first. The best frontier model reaches 59.7%. Forks whose deciding evidence appears later in the trajectory are much harder for every model, which says the model is being asked to predict a consequence rather than recognise a pattern. And a larger reasoning budget does not improve accuracy, which is a direct negative result for the test-time-compute framing this wiki tracks on test-time-compute-allocation: there is a class of decision where spending more buys nothing, so the deficit is missing information or a missing prior, not search depth.

This is the instrument the harness thread has been missing, and it resolves a measurement ambiguity that has sat open since 09-17. The 09-18 component ablation found context management's value comes almost entirely from preventing overflow failures. HarnessTax found cost moving far more than success. Both measure a harness by its effect on the final score, which cannot distinguish "the harness makes the agent decide better" from "the harness makes the agent cheaper." Taste-Bench measures what the harness actually shapes, which is what the agent sees at each fork. If a harness improves fork accuracy, that is a capability claim. If it does not, the harness is doing cost work.

It is also the untaken test for AIDE² (09-23). That system's first discovered self-improvement was a new search policy, and a search policy is a fork-choosing policy. Whether AIDE²'s eight-day run improved its taste or only its throughput is exactly what this benchmark would say, and neither paper ran the other's test. Cheapest high-value follow-up on either page.

Limits on both. Scale's training-exposure conclusion is an inference from elimination rather than a positive demonstration of memorization, and a private split is a depleting asset that degrades the first time anyone publishes error analysis on it. Taste-Bench inherits the mistake distribution of whatever agents generated its source trajectories, reports no human baseline, and labels a branch "better" by its outcome in one run, which conflates a good decision with a lucky one on any task with real variance.

2026-09-24: the contamination gap gets a mechanism

SchrodingerRepo instantiates the SWE-bench repository fresh at evaluation time, eroding familiar cues (problem statement, namespaces, layout, code) while preserving behaviour. Success falls and interaction cost rises across models, mostly from harder exploration and localization. This locates the 17.8-point public/private gap SWE-Bench Pro V2 (09-23) measured: agents partly find files from memory. Consequence for HarnessTax (09-22)-style comparisons: public-split cost is contaminated too. Also same day: Meta's production-agent study found IRT adaptive testing reproduces a full benchmark within 1.03 points using 38.5% of questions, but shipped simpler difficulty-stratified subsets (arXiv 2609.21267).

2026-10-04: perfect scores, invalid runs, and research taste

• Kepler (10-04): 48 of 50 game-model cells on ARC-AGI-3's public set reach 100, so public score no longer discriminates. One perfect run came from reading the game's 2,172-line source (clean rerun: 46.91). The authors ask for first-attempt, cost-conditioned, verification-aware reporting.
• ScholarCatalyst (10-04): 184 authors label which prior papers advanced their projects. Agentic search 0.42 R@20 vs 0.48 for the bare retriever it calls; Fable 5.1 agent 0.51. Second benchmark after Taste-Bench (09-23) placing the auto-research bottleneck in judgment, not tools.

2026-10-23

VerifierBench (agent-written verifiers: balanced accuracy 10.6-62.1, over-rejection dominates), FrontierCS 2 (best agent passes 58%), Agent4Sci (gains are engineering tricks), WildAWM (341 synthetic long-horizon tasks, GPT-6 Astra 60%). See automated research measured, Harness Scaling and WildAWM, shorts.