AI research wiki
agentic-systems · 2026-11-01 · Tier 2

Benchmarks as difficulty measures, synthetic work environments, and a Platonic theorem (2026-11-01)

Benchmarks as difficulty measures, synthetic work environments, and a Platonic theorem (2026-11-01)

Sources: HF Daily Papers 2026-10-30 · 2610.29927 · 2610.37161 · 2610.37786 · 2610.38174 · 2610.39302 · psychiatric-intake RL via @rohanpaul_ai · raw: raw/huggingface/2026-10-30-*.md

Shorts

  • Your Benchmark Is an Empirical Measure Over Difficulty (2610.29927, 140 upvotes). A benchmark induces a distribution over item difficulty. Two failure modes: measure mismatch (its difficulty mix differs from the one you care about) and coverage gaps (empty regions of the difficulty axis, so local progress is invisible). Two diagnostic metrics with guarantees; 25 benchmarks across 7 domains analysed; many overweight some difficulty regions. BenchFill rewrites items to fill gaps. Ties to the 10-30 "Sharpening Tax" and "what outcome RL buys" findings: if RL mostly improves finishing on mid-difficulty items, a benchmark heavy in that band overstates progress.
  • WorkFactory (2610.37161). Builds executable professional tasks from intents via an information-support graph, then probe-executes to find and repair missing information. 112,686 tasks across 612 occupations; fine-tuned WorkPilot models at 4B, 27B and 35B-A3B; WorkPilot-35B-A3B goes 879 to 1565 Elo on GDPval-AA v2.1. Diminishing returns are model-dependent.
  • WebAnything (2610.37786). Reconstructs resettable website environments from real URLs with an Explorer-Builder and an independent verifier. 244 sites, 12,984 accepted tasks; beats human-assisted WebHarbor on fidelity (12 wins, 2 ties, 0 losses). Fine-tuning Qwen3.5-9B on 5,856 trajectories: +9.9 REAL success, +13.33 WebArena-Verified. Extends agent-training-environments (Trajectory2Gym, 10-28).
  • EasyProver / Aegis (2610.38174). Recursive Lean agents with isolated contexts and parallel tactic probing, an off-the-shelf frontier model, no training: 669/672 PutnamBench, all six IMO 2026 formalizations, 597/600 NTP4VC. Aegis turns competitive-programming Rust into Lean verification tasks; EasyProver proves 52.7% of 2,000 within 30 minutes. Harness, not training, again carries the result.
  • Platonic Representation Theorem (2610.39302, 60 upvotes). Proves JEPAs (joint-embedding predictive architectures) trained on separate modalities learn the same representation up to a rotation when positive pairs are two views of one scene. Next-token prediction is a JEPA with "bad views" (parts of one document), which explains why language models align with vision only at scale. Good views need 42x fewer samples to reach the same alignment; a 109M text JEPA aligns with DINOv2 and CLIP better than every evaluated LLM. Sample-efficiency angle: objective choice, not scale, buys alignment.
  • Psychiatric intake RL (Google DeepMind, 2610.40476). Multi-turn RL against simulated patients who reveal facts only after rapport. Gemini 3.1 Flash-Lite trained this way beats its base in 76.7% of 60 matched pairs; missed acute-risk escalations fall 30.5% to 12.9% on 256 adversarial cases. About 1 in 8 crises still go unescalated.

Relation to prior wiki pages

  • Benchmark-as-measure gives a formal frame for the measurement worries in agent-benchmarks.
  • WorkFactory and WebAnything are two more points in the "synthesize the environment, then distil" pattern in agent-training-environments.