Agent Evaluation & Benchmarks
A growing ecosystem of benchmarks specifically designed for agentic AI — measuring not just accuracy but exploration/exploitation, long-horizon task completion, tool use, robustness, and professional domain coverage.
Current State (as of 2026-09-02)
A second benchmark this month finds that the aggregate score was hiding the axis that decides deployability, and this one hides it in a way that should worry anyone putting an agent near money. E-Commerce Bench (09-02) (arxiv 2608.30730, Qwen team) runs an agent as a merchant across a simulated 365-day year, operating multiple stores at once: market research, supplier negotiation, inventory sourcing, pricing, order fulfilment, returns and cash-flow management, against a calendar of promotions, natural disasters and supply-chain shocks. Eighteen frontier models, seven scoring dimensions, and no single model dominates.
The dissociation is the result. GPT-5.6 Sol earns the most, growing a 100,000 opening stake into 1,431,425, and ranks 16th of 18 on fraud avoidance while trailing Fable5 on operational efficiency. A benchmark reporting one number would have declared it the winner and concealed both facts. Among open-weight models Qwen3.8-Max-Preview leads at 416,252, 38% above GLM 5.2 (high), and shows the strongest learning over the horizon, progressively bargaining suppliers down across repeated orders.
Read against the page's 08-25 entry, this is the same lesson decomposed along a different axis. Microsoft's Thinkingbox showed the strongest model falling from 65.36% pass@1 to 25.25% pass^20 on stateful workflows, separating headline accuracy from reliability across attempts. E-Commerce Bench separates profit from integrity and finds them anti-correlated within one model. Two benchmarks, two different decompositions, one conclusion: the aggregate was concealing the trade-off, and which trade-off it conceals depends on which single number you chose. That is now a pattern on this page rather than an isolated observation.
The design choice worth borrowing is the deterministic counterparty. Long-horizon agent benchmarks usually trade realism against reproducibility, because an LLM-driven counterparty makes the environment non-stationary and confounds comparison across models. E-Commerce Bench keeps both sides of the market deterministic: a fixed demand model for customer purchases and returns, a negotiation kernel for supplier pricing, concessions and decisions, with a language model used only to verbalize decisions the kernel already made. Two runs therefore face the same market, so a score difference is attributable to the agent.
It also denominates outcome in the currency the routing literature keeps asking for. llm-routing.md has argued since the AlphaSense study (08-14), which found per-token pricing ranks models backwards because stronger models finish in fewer tokens, that cost belongs in dollars per completed task, and Optima (08-16) shipped the measurement infrastructure. E-Commerce Bench scores in dollars of end-of-year assets, showing a 3.4x spread between the best closed model and the best open-weight one on that axis. A per-task router on this benchmark would have a directly interpretable objective, and nobody has built one.
Caveats, and the first one is structural. Determinism buys reproducibility at the cost of adversarial realism: a fixed negotiation kernel cannot adapt to an agent that discovers an exploit, so a high score may partly reflect kernel-gaming rather than commercial competence, and the fraud-avoidance ranking of the top earner suggests some of that is happening. The demand model is fixed, so no agent's pricing moves the market it trades in, which removes retail's central feedback loop. And a simulated year is a horizon in simulated time, not tokens or wall-clock, so per-run compute cost is unstated and incomparable across the 18 models, which is the same omission this page has flagged repeatedly.
Prior State (as of 2026-08-29)
The measurement crisis gets its formal floor, and the number is 110 out of 124. What Does an Evaluation License? (08-29) (2608.19269) separates two things this page has been treating as one: an eval artifact specifies a computation (task, scorer, metric), but it does not automatically license the claim attached to that metric, because replaying the claim needs historical evidence and semantic grounding the artifact usually does not ship. The authors formalize the missing layer (a frozen substrate D, a grounded family F of admissible readings, a claim query q, and the resulting identified set), then census all 124 mechanically eligible Inspect Evals units at a pinned commit. 110 stop before deterministic inference is possible. The output is deliberately not a robust/not-robust label: it is typed stops, instability witnesses, and stable substructure, so each failure carries a machine-readable reason.
This reframes what the crisis on this page actually is. The 08-11 entry below recorded four papers in two days each finding a widely trusted benchmark was substantially measuring an artifact of its own construction, and the 08-12 entry recorded the constructive response: build better benchmarks. This paper says the deficiency is largely not in benchmark design but in what evaluation artifacts carry. A well-designed eval that does not ship the substrate and grounding needed to replay a claim produces an unlicensed claim exactly as reliably as a badly designed one. Building better benchmarks does not fix this. Shipping more alongside them does.
The claim-resolution split is the practically useful part. The census separates exact values, winners, complete orderings and pairwise relations, and they do not degrade together: an eval can be too unstable to license "model A scored 62.3" while cleanly licensing "model A beat model B." Almost every claim in this wiki's digests is a winner or pairwise claim, which is the more survivable kind.
It also supplies the vocabulary for this page's longest-running open prediction. The 08-26 and 08-28 Looking Ahead sections both predicted a harness paper would publish a pass^k curve within 60 days, because Microsoft's Thinkingbox (08-25) measured a top model collapsing from 65.36% pass@1 to 25.25% pass^20 on a stateful benchmark. In this paper's terms: a pass@1 number on a stochastic stateful task has a wide identified set, and reporting it as a point estimate is an unlicensed claim resolution. Five consecutive harness papers have made that move.
Limits. One suite, one commit, and "mechanically eligible" is doing unmeasured work; the obvious next census is lm-evaluation-harness or HELM. And the paper says what is missing without estimating what it would cost an eval author to close a typed stop, which is the number that decides whether this becomes practice or stays a diagnosis.
Prior State (as of 2026-08-12)
The measurement crisis entered its constructive phase, and the three benchmarks that arrived agree on a diagnosis rather than just a score.
The 08-10 and 08-11 boards produced four benchmark-validity failures in two days: papers saying the instruments are broken. Today's three are papers building instruments that are hard to break, and the design choices are the interesting part. See the combined summary (08-12).
The scores, first, because they set the frame. DSAgentBench (2608.10366) runs end-to-end data-science workflows in real computer environments across notebooks, IDEs, terminals, browsers and databases, 275 tasks, each with a deterministic evaluator checking analytical correctness, visual outputs and model performance rather than code-only execution. Fifteen models: Claude-4.6-Sonnet best at 56.70%, every open-source agent below 1%, failing at tool orchestration, OS grounding and multi-step reasoning. SPIEval (2608.10692) tests mobile-assistant work over personal information scattered across apps, 250 human-curated tasks, 4,335 records, 10 apps, 21 tools, structured around reasoning, disambiguation, integration, preference inference and multi-intent decomposition: GPT-5.5 (xhigh) at 57.3%, weakest at 16.4%. VibeLifeBench (2608.10875) is 200 multi-week scripted timelines across ten everyday-life domains in a simulated world of 22 mock services where the world advances on its own clock and many changes are silent, graded on end state, timeliness and unstated constraints by reading only what the agent left behind. All seven frontier models score low.
The diagnosis is shared and it is the most useful thing on this page today. SPIEval measures it directly: 79% of failures are inaccurate information localization, because models "commit to plausible but incorrect information instead of continuing retrieval for verification," and fewer than 2% of retrieval actions use any advanced search method. DSAgentBench's requirement to ground decisions in intermediate outputs is the same failure from another angle, and VibeLifeBench's silent world changes turn it into the whole task design. It is not that agents search badly. They mostly do not search, then commit.
That is the cheapest published measurement of a faculty this wiki named in June and again on 08-06. The self-evolving-agents page recorded that three of four negative results converged on one missing capability, deciding what to do with a resource you already have: Shadow evaluations (08-06) with agents ending research runs under 50% of budget spent despite being told to spend down, and InMind (07-29) with retrieval memory surfacing a fact only when it resembles the query, six systems at at most 14.4% on indirect queries against 84.0% for the same memory simply placed in context. SPIEval's under-2% figure is that faculty, priced.
VibeLifeBench supplies the horizon this page has been unable to test. The standing complaint across the agent-memory and self-evolution literature is that everything is validated on episodes of dozens of steps rather than days, so drift never appears. A multi-week timeline with a self-advancing clock, silent changes, and grading that reads only durable artifacts is that horizon in a simulator, and it is the natural target for RoMeRL (08-11)'s bounded memory states and SkillZip (08-12)'s compressed skills. Neither has been run on it.
Apply this page's own methodological rule to DSAgentBench's floor. The rule stated on 08-10 is: before believing any agent-capability number, find the cheapest baseline that could produce it. A sub-1% result uniform across every open-source model is not the shape of a capability gradient, it is the shape of an interface failure, and the paper's own attribution to "tool orchestration, OS grounding" is consistent with that. A²E (08-11) found harness choice swings outcomes as much as the model does, which makes the harness the obvious confound. Whether a better scaffold lifts open models substantially is unreported and is the single most informative missing ablation on today's board.
The spread widened again. Ouroboros at 86.74% on Terminal-Bench 2.1, ProMax refactoring at 41.2%, DSAgentBench at 56.70%, and all of it published within two days. Agent capability remains benchmark-shaped, and none of the three new benchmarks reports token or dollar cost, which for a multi-week benchmark that rewards repeatedly re-inspecting the world is part of the result rather than an appendix.
Prior State (as of 2026-08-11)
Measurement validity is no longer a caveat on this page. It is the page's central problem, and three papers on one board said so.
The audit. SWE-Bench ProMax (08-11) (2608.09802) cites a finding that should reset how every SWE-bench number in this wiki is read: nearly 60% of unsolved SWE-bench Verified instances contain flawed tests, either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements, and frontier models can verbatim reproduce gold patches from training data. Its replacement moves to behavior-preserving multilingual refactoring, a task type that resists both failure modes because correctness has a sharp definition and there is no canonical patch to recall: 170 expert-curated instances across seven languages, averaging 11.4 modified files and 261.6 lines per instance, with issue descriptions rewritten from scratch and test suites manually reviewed. Best frontier model under two agent scaffolds: 41.2%.
The metric critique. A²E (08-11) (2608.07346) argues correctness alone is too coarse for comparing agent harnesses, and adds execution efficiency, tool use, task planning and error recovery as separate dimensions via an Agent Task Protocol and an automatically instrumented Monitor producing standardized traces. Its finding is a negative one with teeth: model-harness combinations vary substantially by task type and no single combination consistently wins, which means every single-harness leaderboard number is a joint measurement being reported as a model property.
The construction critique. Evo-Bench (08-11) (2608.09096) builds explicit machinery, auxiliary-task evolution plus sensitivity-aware stratified splitting, to isolate harness-improving capability from base model strength and prevent task-specific overfitting, because no existing evaluation does.
Put those beside yesterday's entry and the pattern is established. StreamArena (08-10) found that on existing streaming-video benchmarks a baseline reading only the last four frames matches complex streaming models. Four papers in two days, across coding agents, agent harnesses and streaming video, all reporting that a widely trusted benchmark was substantially measuring an artifact of its own construction. That crosses this wiki's three-paper threshold twice over.
The number to carry forward is the spread, not any single score. On one board: Ouroboros at 86.74% on Terminal-Bench 2.1 and 90.69% on OSWorld-Verified, both best-known, against 41.2% on ProMax refactoring. Near-saturation on interactive terminal and OS tasks; under half on coordinated multi-file behavior-preserving change. Agent capability is benchmark-shaped, and the shape is that agents add code well and restructure it badly, which is the direction that accumulates cost, consistent with the 08-10 finding that AI-written C++ consumes 5 to 8% more compute in production across 3.52 million changes.
Two limits on ProMax itself. 170 hand-curated instances is small for a successor to a saturating standard, and manual curation is both the quality guarantee and the reason it cannot be refreshed at scale. Two agent scaffolds is a narrow harness sample, and A²E's same-day finding is precisely that the harness matters, so 41.2% may be a scaffold ceiling rather than a model ceiling.
Prior State (as of 2026-08-10)
A trivial recency baseline beats streaming video models, which makes this the fourth protocol failure on this page in one week. StreamArena (08-10) reports that on existing streaming-video benchmarks, a minimal method reading only the last four frames matches or surpasses complex streaming models, and that multiple-choice answer options separately expose language shortcuts. Its replacement is 243 videos averaging 88.8 minutes with 3,646 open-ended question-answer pairs, and the open-ended format is load-bearing rather than stylistic: with options present, the benchmark was measuring language priors instead of perception.
Placed alongside the week's other three, the pattern is now the strongest claim on this page. ScrambleToolBench (08-04) and SWE-Touch (08-04) found agents discover tools fine and then never revise, that extra reasoning widens a brute-force search rather than deepening it, and that human edits in a shared workspace cost 7.7 resolve points. SkillBench and PastBench (08-05) found explicit skill maintenance matches plain in-context learning on average, with weaker models simply accumulating more fragments. Shadow evaluations (08-06) gave frontier agents six days and real money on genuine unpublished research questions and both runs finished with under 50% of budget spent. Four benchmarks, four different agent capabilities, and in every case the reported capability was substantially an artifact of the measurement protocol rather than of the model.
The methodological rule this page should now state outright: before believing any agent-capability number, find the cheapest possible baseline that could produce it. Last-four-frames for streaming. Plain in-context learning for skill libraries. Repeated sampling for self-reflection, which is the shape of the Sample More Reflect Less study in DAIR.AI's weekly roundup (starred Gmail, 08-10): seven self-improvement methods, open models at 1.5B, 3B and 7B, two math benchmarks, every generated token counted including critiques, reflections, debate turns and checking, each method compared against repeated sampling at its own measured cost, with paired bootstrap intervals and multiplicity correction across all 36 comparisons. Repeated sampling held up against every method in every setting. Ten comparisons came back reliably worse, and all ten involved the model inspecting its own output; all 18 self-inspection comparisons were negative. Reflexion as published never triggered its own retry on the smallest model because it judged itself correct every time.
One paper this week runs against the grain and it is worth naming as the open question. ReASearch (08-10) argues an agent can internalize the whole outer-loop search policy, deciding what to evaluate, how to diagnose, which edit to make and when to restart, and reports 2% to 40% over specialized optimizers across 14 tasks. It reports no matched-token cost accounting, which is exactly the omission Sample More Reflect Less exposes elsewhere. The plausible reconciliation with the Shadow-evaluations budget failure is that ReASearch's tasks have a dense, cheap, automatically computable score and open-ended research does not, in which case self-directed budget allocation works exactly where a verifier exists and nowhere else.
Prior State (as of 2026-08-06)
A new evaluation instrument arrived today and it is a better idea than most of the benchmarks on this page. Shadow evaluations (08-06) partner with the authors of unpublished papers, take those papers' real research questions, and let frontier agents attempt them with real compute and a real budget, then have the original authors review the output. Two properties follow and both fix problems this page keeps logging. The agent cannot have trained on the result and cannot look it up, which is the contamination problem that undermines most agent-research benchmarks and the exact failure GDPevo (08-06) attacks from the other end with automated task regeneration. And the reviewers spent months on the same question, so the judgment is expert rather than rubric-based, which is the ceiling every LLM-judged agent benchmark runs into. The cost is n = 2 and substantial researcher discretion, which the authors state plainly.
The day's real lesson for this page is a vocabulary problem that is quietly corrupting comparisons. OneDayAgent (08-06) sets a new state of the art of 0.821 on AgentIF-OneDay, 104 tasks described as long-horizon, cross-environment and open-ended. Shadow evaluations conclude that agents cannot do open-ended research. Both are correct. In the benchmark, open-ended means multi-step, multi-tool and not templated, and the success criterion is still fixed in advance and checkable. In the shadow evaluation, open-ended means the success criterion is not knowable in advance, which is why the failures were failures of judgment, budget allocation and backtracking rather than of execution. Benchmarks on this page routinely claim the first and are read as evidence about the second. Any benchmark calling itself open-ended should state which one it means.
Two further additions. GDPevo (08-06) is evolution-native by construction: it decomposes enterprise workflows into atomic business rules, distributes subsets across training tasks and recombines them in held-out tests, so test-time gains are attributable to training experience rather than to contamination or scale. Self-evolution lifts held-out accuracy by up to 16.44 points, and the number worth keeping is the gap to the fully-informed oracle ceiling of 91.6%, which says how much of the available headroom current self-evolution actually captures. And Skill²-Bench (08-06) contributes a difficulty axis this page has lacked: it calibrates tasks by skill-transition cost rather than by size or step count, and reports that a salient size measure fails to grow from easy to hard in a substantial share of task types. That is a direct warning about every benchmark here that treats more steps as harder.
Prior State (as of 2026-08-04)
Two benchmarks landed on the same HuggingFace board measuring the same previously unmeasured capability, from opposite directions, and both find it missing. The capability is belief revision.
ScrambleToolBench (08-04) (2608.02358) strips the semantics out of tool use. Existing suites expose semantic schemas (function names, argument names, docstrings) in static environments, which means a high score is consistent with two very different agents: one that infers behaviour from interaction, and one that recognizes a familiar API. ScrambleToolBench removes the cue so only the first can score, then changes the environment underneath the agent with mapping drift, stochastic action failures, and temporal execution windows. Initial discovery works. Adaptation does not: when the mapping drifts, agents do not deduce the new mapping from the structure they already recorded (cycle tracing would do it), they show belief inertia or fall back to exhaustive search. The finding that should worry anyone selling reasoning as the fix is that more test-time reasoning amplifies the brute-force search rather than producing deduction, and persistent memory reduces compounding errors without enabling structural inference.
SWE-Touch (08-04) (2608.02499) supplies the human-in-the-loop version. Every repository-level coding benchmark evaluates an agent alone or restricts the human to messages; real development is a shared workspace where a person edits code mid-task. SWE-Touch injects validated Counter-Edits, plausible edits to task-relevant code that conflict with task completion, mined from task-critical regions across multiple repair trajectories and constructed by a separate User Patch Generator. Across nine models, average resolve rate on SWE-bench Verified drops 7.7 percentage points, with degradation persisting on SWE-Bench Pro and DeepSWE. The failure taxonomy is the transferable part: agents retain the conflicting code, or overwrite it without re-inspecting the repository or writing a targeted test for the revised behaviour.
Together they name a pattern this page should carry forward: the agent builds a world model and then does not update it. Two benchmarks, two domains, one conclusion, and it matches the profile Shadow evaluations (07-30) found in a third setting, where agents given the central open question from unpublished NeurIPS 2026 submissions completed all of the engineering unassisted and were rejected on five judgment failures including ineffective backtracking. Engineering competence with revision incompetence, measured three ways in six days. The consequence for this page is that horizon length and state-revision capability have been bundled under "long-horizon" and are apparently separable, so a benchmark that scores only the former is measuring less than it claims.
The list of dimensions this page argues are missing now has two more entries. Efficiency Matters in Autonomous Research (08-02) added cost and showed search efficiency and outcome quality are empirically distinct across twelve tasks and four search families. ExtractBench (08-03) added record completeness and found commercial vision-language models silently truncate long record lists. Theta's AI Engineer talk offered order-shuffling as a free test for real sequential complexity. Now add environment volatility (ScrambleToolBench) and workspace volatility (SWE-Touch). Every one of these is a dimension nearly every deployment has and nearly no leaderboard scores. ScrambleToolBench is also the cheapest of them to replicate: scrambling tool semantics is the tool-axis twin of Theta's order shuffle, and both cost nothing.
And a latency dimension arrives that nobody had at all. AAPT (08-04) (2607.28399, Kurate cs.LG #5) argues computer-use agents fail on transient GUI events (boot prompts, auto-dismissing dialogs, short-lived auth requests) as a timing failure rather than a comprehension failure: the agent produces the correct action after the window has closed, because autoregressive decoding sits on the decision-time critical path. It removes decoding from that path without touching the model, pre-building a bounded conditional policy tree during idle screen periods with observable guards, pre-authorized actions and branch deadlines, sized to cover the model's own decoding latency. Success inside a contested decision window rises from 0.50 to 0.79 (p = 1.8e-3) with zero incorrect actions, and both open-loop and predict-and-replan baselines score zero because they still decode during execution. Two things make it methodologically better than most work on this page: pre-registered endpoints with exact McNemar tests, and a pre-registered oracle probe that rejected the authors' own initial hypothesis and identified branch routing rather than planning quality as the causal bottleneck. That probe result is also the uncomfortable link to ScrambleToolBench: AAPT's agent cannot match a live observation to the right prepared branch, ScrambleToolBench's agent cannot notice its map is stale. Both are failures of mapping a live observation onto a maintained structural hypothesis, at two different timescales.
Current State (as of 2026-08-03)
The cost-as-a-dimension argument gets its first implementation, one day after it was made, and it comes from the dullest corner of agent work. Efficiency Matters in Autonomous Research (08-02) argued that every leaderboard on this page grades the best solution found and ignores the budget spent reaching it, proposed area under the Pareto frontier of reward against accumulated budget, and showed the two dimensions are empirically distinct across twelve tasks and four search families. ExtractBench (08-03) (2607.29677, LlamaIndex) does the minimum version for schema-guided enterprise document extraction: 4,869 pages, 370 documents, 8 domains, 67 document types, scoring value accuracy, record completeness at scale, grounding and measured cost together.
Three design choices worth borrowing. Ground-truth curation is matched to evidence regime rather than uniform: independent-system agreement for real documents, known values by construction for synthetic lists, human verification for forms. Grounding is reported at two resolutions, word-level and page-level F1, which separates "right region, wrong span" from "wrong document." And completeness is scored separately from accuracy, which is what surfaces the headline finding: commercial vision-language models do well on short documents and truncate record lists on long ones, a failure invisible to value accuracy since extracting 40 of 120 line items correctly still scores well. That generalizes past extraction to any long-output agent task, and almost no benchmark on this page measures it. The conflict to state plainly: LlamaIndex authored the benchmark and a LlamaIndex product tops all three metrics, so replication matters before anyone procures on it, and "measured cost" is not defined as spend, wall clock or tokens. → summary
Practitioner counterpoint from the conference track. Two AI Engineer talks synced into raw/youtube-ai-tech/ on 2026-08-01 make claims this page should track. Theta argues GDPval, ToolBench and APEX-Agents all fail on task length, saturation, breadth and reward granularity, that their own finance tasks average 15 human-hours with models scoring around 5%, and supplies a free diagnostic for whether a "long horizon" benchmark has real sequential complexity: shuffle the subtask order and see whether the score moves. David Brumley argues nearly every cyber benchmark is broken the same way, by assuming the target contains exactly one known vulnerability, and reports that DARPA's Cyber Grand Challenge spent $60M hand-curating single-bug challenges of which 50% contained unintended exploitable bugs. His replacement, the audit task, asks for all vulnerabilities, uniquifies submissions by stack backtrace, and scores precision times recall, which is a reusable shape for any task with an unknown number of valid solutions. He also reports that crash-triggering does not discriminate between models at all (GPT-5.5 and Mythos both 95%) while full sandbox escape does (Gemini and Kimi at 0%), and states flatly that LLM-as-judge fails categorically in security, which directly contradicts Theta's judge-as-agent design.
Current State (as of 2026-07-30)
Seven evaluation papers landed in one day, and the two best-rated papers on Kurate's cs.AI board this week both argue that evaluations of this kind cannot prove what they claim. This page has accumulated benchmark skepticism since Soohak (05-12) from a dozen directions. Today it stops being a running complaint and becomes the field's dominant research output, with the critique arriving from above the benchmarks rather than from inside them. Kurate cs.AI #1 is Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI (score 1573, 87.5% win rate, zero HuggingFace presence), arguing agent benchmarks have a protocol-validity problem rather than a difficulty problem. #2 is What AI Red-Team Evaluations Can and Cannot Prove (score 1570, 90.9% win rate), arguing a red-team evaluation establishes that a vulnerability exists and essentially never that one does not. The 07-29 digest predicted a second validity paper would reach Kurate's top three within 60 days; it took one day.
The protocol answer: grade against something that does not exist yet. Shadow evaluations (2607.27191) is the sharpest methodological contribution the page has recorded. Hand an agent the central open research question from an unpublished paper and have the paper's own authors grade the output. No contamination, no proxy metric, no LLM judge, and a grader calibrated to a real submission's bar. Two runs on unpublished NeurIPS 2026 submissions, six days and thousands of dollars of compute each: the agents completed all of the engineering unassisted and both outputs were unambiguously rejected. Five reproduced failure modes, four of which are judgment rather than knowledge or execution: no sense of the publishable bar, uncreative responses to design flaws, ineffective backtracking, poor resource awareness, instruction drift. This is the strongest confirmation yet of the two-layer thesis ForeSci (06-07) opened with its evidence-decision decoupling, where agents cite the right evidence and still forecast the wrong research object. The protocol's weakness is that it cannot scale, which is exactly why nobody had run it.
The economic answer: price the tasks. OmegaUse-OfficeVal (2607.27155) attaches human labour time and a task price proxy to each of 100 practitioner-derived office-suite tasks averaging 2.32 hours of human work, with code-based verifiers compiled from fine-grained rubrics rather than an LLM judge. The result splits cleanly: every evaluated model is substantially cheaper and faster than a human worker and none approaches human-level deliverable quality. That is the most decision-relevant framing this cluster has produced, because it says quality is binding while cost is the axis everyone optimizes. Value-weighting also separates "fraction of tasks completed" from "fraction of value captured," which come apart precisely when an agent is good at cheap work.
The self-state deficit, named in three unrelated domains on one day. The page's long-running finding has been that long-horizon agents are weak. Today sharpens it: the model represents the task and the world well and represents its own state within them badly. HumanCLAW (2607.27180) decouples action decision from motor execution so balance and controller error are factored out, and finds that across 1,218 long-horizon egocentric episodes in 41 scenes, nine frontier VLMs top out at 16.8%, with target recognition explicitly not the bottleneck. What is missing is embodied self-awareness: the models lose track of where their own body is and whether it has reached the goal or hit an obstacle. SecRespond finds incident-response agents investigating every alert competently and never hypothesizing that something unflagged is on the disk. Shadow evaluations finds instruction drift and poor resource awareness over six days. Three domains, one deficit, and it is the closest relative of RNG-Bench (06-18, non-Markov memory where the answer depends on history the current observation does not show): RNG-Bench tests memory of invisible history, HumanCLAW tests memory of invisible self.
Security evaluation splits into pre- and post-compromise. SecRespond (2607.26791) is the first post-compromise benchmark, 10 cyber ranges from real compromised cloud hosts across 21 ATT&CK techniques and 5 operating systems, 23 frontier models on the OpenCode harness, and no model completes detection and remediation on any single range. StealthBench (2607.26314) adds a compound metric requiring an offensive agent to both succeed and stay hidden, and no model exceeds 54%. Both inherit the single-harness caveat GTA-2 (04-20) established, where harness design mattered more than model capability.
One honest-reporting example worth naming. Grading the Narrators (2607.24117) evaluates a claim-provenance framework on 20,000 physics-textbook claims and reports that two components validated, the grade-recovery loop partially failed by missing the highest-fault narrator, and two analyses were inconclusive. On a day when the top-rated papers in the field are about what evaluations cannot prove, this is what the requested epistemic hygiene looks like in practice.
Current State (as of 2026-07-22)
Agent failure attribution closes the loop from locate to repair. AgentDebugX (2607.18754) organizes debugging as Detect → Attribute → Recover → Rerun, with a DeepDebug root-cause module built on the premise the failure-attribution thread keeps naming: the step where an error surfaces is usually not the step that caused it. It posts the best strict (exact agent-and-step) attribution on the "Who and When" benchmark (28.8% on qwen3.5-9b vs 21.7% single-pass) and, crucially, converts diagnosis to repair, fixing 13/73 failed GAIA tasks in one rerun (vs 4-6 for self-correction baselines), lifting 55.8% → 63.6%. This is the recovery half of the OAT / Failure-as-a-Process attribution thread (2026-07-20 digest): OAT located where a trajectory deviated from a success flow; AgentDebugX adds attribution → repair → a shareable Error Hub (scrubbed diagnosis-repair bundles as debugging memory). Absolute attribution is still low (28.8% exact), which is the honest ceiling. Its same-day companion DataFlow-Harness (2607.16617) extends the harness-as-artifact theme (07-16 Harness Handbook) to data engineering: constrain the agent to typed, live-grounded DAG mutations instead of free-form scripts and get 72.5% lower cost than Vanilla Claude Code at comparable reliability.
Current State (as of 2026-06-18)
Two same-day benchmarks confirm frontier models fail at sustained and stateful tasks, and an active-perception agent shows the escape route. The page's running thread, that short-horizon accuracy does not predict long-horizon robustness, gets its strongest evidence yet on the same day from two opposite axes.
Sustained, long-horizon work. CEO-Bench (arxiv 2606.18543) puts an agent in charge of a fictional startup for 500 simulated days, managing pricing, marketing, and budgeting through a Python interface over noisy interconnected business databases. It bundles four skills short-horizon evals never test together: long horizons under uncertainty, information acquisition in noise, adaptation to a changing world, and orchestration toward one goal. Most state-of-the-art models go bankrupt. Only Claude Opus 4.8 and GPT-5.5 finish above the $1M starting balance, and neither consistently profits. The differentiator is coding ability: the best agents write code that simulates customer cohorts to forecast cash and mines negotiation history for hidden preferences. This pushes the horizon far past PhysicianBench's ~27 tool calls (2026-05-05) and GTA-2's open-ended workflows (2026-04-20), and echoes AcademiClaw's compute-quality decoupling (2026-05-05): raw model size is not the lever, simulator-building skill is.
Stateful, non-Markov memory. RNG-Bench (also 2026-06-18) is a non-Markov memory benchmark where the answer depends on history not visible in the current observation. Frontier multimodal LLMs stay far from saturated even at 128K context. Paired with CEO-Bench it draws a clean two-axis map of what short-horizon evals miss: sustained progress over a long horizon (CEO-Bench) and stateful use of invisible history (RNG-Bench). Frontier models are weak on both.
The escape route: active perception with positive test-time scaling. OmniAgent (arxiv 2606.19341) reframes long-video understanding as a POMDP observe-think-act loop that distills cues on demand into persistent textual memory, decoupling compute from input length. Its load-bearing benchmark result is positive test-time scaling: performance keeps improving as reasoning turns increase, because each turn gathers new information rather than re-refining the same view. This is the direct contrast to LoopCoder-v2 (2026-06-17, where latent-loop internal refinement saturates), and the small-model-beats-large pattern holds: the 7B agent beats Qwen2.5-VL-72B on LVBench (50.5% vs 47.3%). The lesson for long-horizon benchmarks above: test-time compute pays off only when each step adds information, exactly the property CEO-Bench's 500-day loop demands.
Current State (as of 2026-06-06)
The benchmark surface keeps migrating from single answers to environment-grounded trajectories, and safety follows it. Two same-day papers move the evaluation unit from "the response" to "the final state / the adaptive process."
Operational safety, not prompt refusal. SABER (arxiv 2606.01317) is the safety counterpart to the trajectory-quality thread the page has tracked since AgentLens (05-14, 10.7% of passing SWE-bench runs are Lucky Passes). Existing safety evals check whether a model refuses an unsafe prompt; SABER places coding agents in realistic stateful project workspaces and judges safety from the final environment state after a sequence of actions, categorizing violations by cause. Even the best model has a >54% harmful safety-violation rate, and models have distinct safety profiles, so a single aggregate score hides the real risk surface (the same over-aggregation lesson as Soohak/AgentLens/AssetOps). SABER is the controlled-benchmark floor under today's Anthropic RSI / NSA-Mythos story: the autonomous coding agents being handed production write-access and offensive-cyber tooling fail operational safety more than half the time.
Adaptive planning under hidden constraints. AdaPlanBench (arxiv 2606.05622) targets a gap the static planning benchmarks leave open: world and user constraints that are revealed only when a proposed plan violates them, forcing multi-turn re-planning under accumulating feedback. Built on 307 household tasks with a scalable dual-constraint pipeline; the best of ten leading LLMs reaches only 67.75%, performance degrades as constraints accumulate, and user constraints are harder than world constraints (failures trace to weak physical grounding). Together with SABER, the message is consistent: agents look much weaker once the benchmark grades the full interaction and its environmental consequences rather than a single turn.
Current State (as of 2026-06-07)
Research judgment is a distinct, weak layer above research execution (ForeSci). ForeSci (arxiv 2606.00644) is a temporally controlled benchmark for forward-looking AI research judgment: 500 tasks across four AI domains and four decision families, each paired with a cutoff-aligned offline knowledge base, post-cutoff papers hidden during generation and used only for validation (so it avoids rewarding lucky future guesses). The signature finding is an evidence-decision decoupling: agents cite relevant evidence yet forecast the wrong research object — retrieval quality does not become decision quality. This is the research-judgment analogue of the page's recurring "surface signal misleads about real capability" thread (AgentLens "lucky pass" 05-14; the verifier-gaming line): a process metric (good citations) looks healthy while the outcome (correct call) is wrong. It also frames the autonomous-researcher stack as two layers — judgment (ForeSci, choosing what to work on) sitting above execution (the MLEvolve/EvoDS self-evolving-research cluster) — and shows the judgment layer is the weaker one. → summary
Current State (as of 2026-05-14)
Latest additions (2026-05-14): Two papers extend the benchmark-skepticism thread that has been building since Soohak (05-12). AgentLens (summary) shows that 10.7% of passing SWE-bench Verified trajectories are Lucky Passes — regression cycles, blind retries, missing verification, or temporally disordered work. The framework merges per-task passing trajectories into a Prefix Tree Acceptor reference space and uses a context-sensitive intent-stage labeler (Exploration / Implementation / Verification / Orchestration). Some model backends drop 5 ranking positions when scored by quality instead of pass rate, which means pass-rate alone is misleading for between-model comparison. AgentLens-Bench: 1,815 trajectories from 47 tasks across 8 model backends. AssetOpsBench retrospective (raw) reports public-to-hidden score correlation of −0.13 on 234 submissions to the CODS 2025 challenge — public standing does not predict hidden robustness. Three papers in three days (Soohak, AgentLens, AssetOps) all say the same thing from different angles: aggregate leaderboard metrics over-aggregate. MAP (Map-then-Act) (summary) is the architectural complement to AgentLens: frontier models surpass near-zero ARC-AGI-3 baselines in 22 of 25 game environments when they build the environment prior before acting. Training on map-then-act trajectories beats training on expert execution traces, which reframes what good demonstration data looks like for long-horizon agents.
Prior State (as of 2026-05-07)
Standard LLM benchmarks underserve agents. The field has been building agent-specific eval frameworks across several dimensions: decision-making quality, professional domain coverage, multimodal grounding, and robustness under fault injection. Nine benchmarks (OccuBench, GTA-2, DR3-Eval, PRL-Bench, Claw-Eval-Live, InteractWeb-Bench, AcademiClaw, PhysicianBench, ProgramBench) now report frontier-agent failure rates of 0–55% on realistic multi-step tasks. ProgramBench (05-06) at 0% on every model is the floor; PhysicianBench (46%) and AcademiClaw (55%) are the realistic ceiling. 2026-05-07 adds a tenth dimension: BRIGHT-Pro, the first benchmark for evidence-portfolio retrieval rather than top-1 relevance, evaluating retrievers under both static and agentic protocols. MedSkillAudit (also 05-07) shifts the evaluation surface from agent capability to agent skill release readiness: 75 medical research skills audited, 57.3% below Limited Release threshold, system-expert agreement (ICC=0.449) exceeded the human inter-rater baseline (0.300). The Marcus production-agent security paper (05-06) frames the limit case: 91% of 847 deployed agents are vulnerable to tool-chaining attacks, 89.4% drift after 30 turns. Capability ceiling, evaluation methodology, skill audit, deployment security: the agent benchmarks cluster is now four-dimensional.
Key Benchmarks
CEO-Bench (2026-06-18) — long-horizon business-operation benchmark. An agent runs a fictional startup for 500 simulated days, managing pricing, marketing, and budgeting through a Python interface over noisy interconnected business databases. Tests four skills together: long horizons under uncertainty, information acquisition in noise, adaptation to a changing world, orchestration toward one goal. Only Claude Opus 4.8 and GPT-5.5 finish above the $1M starting balance; neither consistently profits. Best agents write cohort-simulation code to forecast cash and mine negotiation history for hidden preferences. Highest horizon (500 days) in the cluster; pairs with same-day RNG-Bench (stateful non-Markov memory, frontier MLLMs far from saturated at 128K). → summary
OccuBench (2026-04-16) — 100 tasks across 65 professional domains using Language World Models (LWMs) to simulate environments. Key finding: no single model dominates all industries; implicit faults are hardest. → summary
Exploration/Exploitation Measurement (2026-04-16) — Policy-agnostic metric for explore/exploit errors in LM agents on 2D grid environments. Reasoning models perform best; harness engineering meaningfully improves both dimensions. → summary
GameWorld (2026-04-16) — 34 browser games, 170 tasks, state-verifiable outcomes for MLLM game agents. Best models still far below human. → summary
MERRIN (2026-04-16) — Search-augmented agent benchmark with noisy multimodal web evidence. Average accuracy 22.3%; agents over-rely on text modalities. → summary
InfiniteScienceGym (2026-04-16) — Procedurally generated scientific analysis benchmark. No model exceeds 45%; abstention on unanswerable questions is a key weakness. → summary
DR3-Eval (2026-04-18) — Deep Research Agent benchmark. Static per-task corpus sandboxes with evidential sources, confounding documents, and noise. Reverse-constructed questions (derived from verified evidential docs) ensure every task is answerable. Multi-dimensional scoring: recall, factual accuracy, citation coverage, instruction following, depth. State-of-the-art models still struggle. → summary
GTA-2 (2026-04-20) — Two-tier benchmark: GTA-Atomic (single-step tool precision) and GTA-Workflow (long-horizon, open-ended multi-tool coordination). Key results: frontier models below 50% on atomic tasks; top models at 14.39% on workflows. Critical finding: execution harness design (Manus, OpenClaw) matters more than underlying model capability. Uses real user queries and deployed tools — not synthetic evals. Recursive checkpoint-based evaluation for open-ended tasks. → summary
PRL-Bench (2026-04-20) — Physics Research by LLMs benchmark. 100 tasks from Physical Review Letters papers (Aug 2025+, post-training cutoff for most models). Covers 5 subfields: astrophysics, condensed matter, high-energy, quantum information, statistical physics. Tasks replicate authentic research: exploration-oriented formulation, long-horizon workflows, verifiable outcomes. All frontier models score below 50%. Expert-validated. → summary
Claw-Eval-Live (2026-05-01) — First live workflow-agent benchmark. Refreshable signal layer (ClawHub Top-500 skills, updated each release) + reproducible release snapshot (frozen fixtures, services, graders). 105 tasks, 13 frontier models, deterministic + structured-LLM grading on execution traces, audit logs, service state, post-run artifacts. Best model: 66.7%; no model reaches 70%. HR / management / multi-system business workflows persistently fail. → summary
InteractWeb-Bench (2026-05-01) — First benchmark to grade clarifying behavior explicitly. Four user-agent personas + persona-driven instruction perturbations from RE defect taxonomies. Unified agent action space: Clarify / Implement / Verify / Submit. Frontier MLLM agents remain trapped in blind execution — generating code that satisfies their misreading of the instruction without ever asking. → summary
AcademiClaw (2026-05-05) — Bilingual academic-level benchmark, 80 multi-step tasks curated from 230 real student submissions across 25+ professional domains (olympiad math, linguistics, GPU-intensive RL, full-stack debugging). Docker sandbox per task; six-technique multi-dimensional rubric scoring + five-category safety audit. Best of six advanced models: 55%. Capability varies sharply across domains; compute does not predict output quality — argues against current "more thinking tokens equal better results" defaults. → summary
PhysicianBench (2026-05-05) — 100 long-horizon physician tasks from real consultation cases inside an EHR environment with vendor APIs. 21 specialties; ~27 tool calls per task. Best closed-source model: 46% pass@1. Best open-source: 19%. Highest tool-call horizon in any of the eight benchmarks; the gap between knowledge tests (where LLMs match physicians) and EHR-mediated workflows (where they do not) is the load-bearing finding. → summary
BRIGHT-Pro and RTriever-4B (2026-05-07) — first benchmark for evidence-portfolio retrieval rather than top-1 relevance. Each query is expanded with multi-aspect gold evidence; retrievers are graded under both static and agentic protocols. RTriever-Synth, an aspect-decomposed synthetic corpus, generates complementary positives and positive-conditioned hard negatives. RTriever-4B (LoRA on Qwen3-Embedding-4B) substantially improves over its base. Aspect-aware and agentic evaluation expose behaviors hidden by standard top-k metrics. → summary
MedSkillAudit (2026-05-07) — first skill-release-readiness audit framework. 75 medical research skills, two human experts, ordinal release disposition (Production / Limited / Beta / Reject). System-expert ICC = 0.449 vs human inter-rater 0.300. 57.3% of skills below Limited Release threshold. Negative ICC on Academic Writing (-0.567) reveals structural rubric-expert mismatch on open-ended generative tasks. Pre-deployment audit complement to the Marcus post-deployment security study. → summary
OpenSearch-VL (2026-05-07) — open recipe for frontier multimodal search agents. Wikipedia path sampling with fuzzy entity rewriting, source-anchor visual grounding, unified text+image+OCR+image-manipulation tool environment, and multi-turn fatal-aware GRPO (mask post-failure tokens, preserve pre-failure reasoning via one-sided advantage clamping). 10-point average gain across seven benchmarks; matches proprietary commercial models on several. The training-time intervention at the same multi-turn surface where the Marcus security paper measures failures. → summary
Patterns Across Benchmarks
Reasoning models consistently outperform base models on agentic tasks
Over-exploration is a common failure mode in strong models
Professional/domain-specific tasks expose different weaknesses than general benchmarks
Deterministic environment generation (OccuBench, InfiniteScienceGym) removes publication bias
Execution harness dominates model capability (GTA-2): the scaffold around the model determines workflow completion more than model capability itself. Confirmed empirically by the Ridge Security pentester benchmark (2026-05-04) at constant model: belief state, evidence-as-invariant, and trust propagation account for >5x finding gaps between architectures using the same Gemini 3 Flash backbone.
Eight benchmarks now converge on the same finding: frontier models fail realistic multi-step tasks reliably — this is a consistent, cross-domain measurement
Middle-band discrimination (Claw-Eval-Live, 05-01): models with similar pass rates diverge in overall completion, suggesting per-task-family routing could outperform any single model
Blind execution (InteractWeb-Bench, 05-01): a distinct, named failure mode where agents guess rather than clarify under ambiguous instructions — the first benchmark to grade this dimension explicitly
Compute-quality decoupling (AcademiClaw, 2026-05-05): computational resource consumption does not predict output quality across 80 academic-level tasks. The compute-as-proxy default is empirically broken
Long-horizon tool-call gap (PhysicianBench, 2026-05-05): 27-call average is the highest horizon in the cluster; the open-source vs closed-source gap (19% vs 46%) is largest at this horizon, suggesting tool-use trace data, not raw capability, is the bottleneck
Static benchmarks leak via memorization (EvoBrowseComp, 2026-06-14): static search benchmarks like BrowseComp let a model score by recalling a fact instead of browsing for it, so the score is a memorization test in disguise and rots once the questions enter a training set. EvoBrowseComp synthesizes 400 EN + 400 ZH contamination-free questions from live-web traversal via a three-agent pipeline, using popularity as the contamination proxy (a fact popular enough to be memorized cannot test browsing), and can be regenerated to stay fresh. See EvoBrowseComp.
Benchmarks should model environment change, not just static tasks (EvoArena, 2026-06-14): the first agent benchmark to make progressive environment updates the unit of evaluation across terminal, software, and social domains. Current agents average 39.6%; chain-level success (consecutive evolving subtasks) is the hardest band. See EvoArena.
Sustained vs stateful are the two axes short-horizon evals miss (CEO-Bench + RNG-Bench, both 2026-06-18): two same-day benchmarks find frontier models fail along orthogonal axes. CEO-Bench grades sustained progress over a 500-day horizon (only Opus 4.8 and GPT-5.5 stay solvent, neither profits); RNG-Bench grades stateful use of non-Markov history (frontier MLLMs far from saturated at 128K context). Together they extend the long-horizon failure thread past PhysicianBench (~27 tool calls) to the 500-step regime. See CEO-Bench.
Positive test-time scaling beats latent-loop saturation (OmniAgent, 2026-06-18): the active-perception counterpoint to the long-horizon failures above. OmniAgent's observe-think-act loop over video shows performance keeps improving as reasoning turns increase, because each turn gathers new information rather than re-refining the same representation. This contrasts directly with latent-loop compute saturation (LoopCoder-v2, 2026-06-17) and holds the small-model-beats-large pattern (7B beats Qwen2.5-VL-72B on LVBench). Implication for benchmark design: test-time compute only pays when each step adds information. See OmniAgent.
Fault-tolerance scales slower than task skill (ToolMaze, 2026-06-08): the first benchmark to break tools on purpose, with a 2x2 perturbation taxonomy (explicit/implicit × transient/permanent) over DAG-structured tool-call tasks. Implicit semantic failures crater the Perturbation Recovery Rate by ~37% because agents over-trust corrupted output, and agentic fault-tolerance improves with model scale 3.66x slower than basic task execution. Dynamic replanning is a distinct bottleneck that scaling and prompting do not fix. See ToolMaze.