AI research wiki
July 30, 2026 · daily digest

cere-bro | 2026-07-30

cere-bro | 2026-07-30

HuggingFace served seven benchmarks in a single day, and the two highest-rated papers on Kurate's cs.AI board this week both argue that benchmarks of exactly this kind cannot prove what they claim to prove. Yesterday's digest predicted a second measurement-validity paper would reach Kurate's top three within 60 days. It took one.

TL;DR

Deep Dives

Can AI Agents Conduct Open-Ended AI Research?

The agents did every piece of engineering unassisted, and the authors of the papers they were imitating rejected both outputs without hesitation.

Source: HuggingFace Daily Papers Links: arXiv 2607.27191 · Wiki summary

flowchart LR
  P[Unpublished paper<br/>NeurIPS 2026] --> Q[Central open<br/>research question]
  Q --> A[Frontier agent<br/>6 days, $1000s<br/>of compute]
  A --> E[Engineering:<br/>done, no help]
  A --> R{Research<br/>lifecycle}
  R --> F1[No sense of the<br/>publishable bar]
  R --> F2[Uncreative fixes<br/>to design flaws]
  R --> F3[Cannot backtrack<br/>from dead ends]
  R --> F4[Poor resource<br/>awareness]
  R --> F5[Instruction drift]
  E --> G[Original authors<br/>grade the output]
  F1 --> G
  F2 --> G
  F3 --> G
  F4 --> G
  F5 --> G
  G --> V[Unambiguous reject,<br/>both papers]
  classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
  classDef decision fill:#fef3c7,stroke:#f59e0b,color:#78350f
  classDef output fill:#d1fae5,stroke:#10b981,color:#065f46
  classDef warn fill:#fee2e2,stroke:#ef4444,color:#7f1d1d
  classDef aux fill:#e0e7ff,stroke:#6366f1,color:#312e81
  class P,Q input
  class R decision
  class E,G output
  class F1,F2,F3,F4,F5,V warn
  class A aux

What is it about? Every forecast of explosive AI progress assumes agents will soon automate AI research itself, and nobody has measured that properly. This paper introduces a protocol it calls a shadow evaluation. You take a high-quality paper that has not been published yet, hand an agent its central open research question, and have the paper's own authors grade what comes back.

What problem does it solve? The two existing options both fail. Narrow verifiable tasks (implement this, hit that accuracy) cut out the open-ended part that is the actual research. Submitting AI-written papers to blind peer review inherits everything wrong with peer review: overstretched reviewers, high variance, poor quality. A shadow evaluation removes contamination, because the answer does not exist publicly, and it uses the only graders on earth calibrated to the real bar.

What is the core novelty? The protocol, not the result. This is the first evaluation of research agents where the grading standard is a real submission's acceptance bar and the grader is the person who solved the problem. The cost is that it does not scale: two evaluations, six days each, thousands of dollars of compute apiece, plus expert reviewer time. That is exactly why nobody had run it.

Key takeaways

Gaps in the study Two papers, one field, one conference, which cannot tell you whether the ceiling is uniform or whether some research questions are already reachable. Six days and a few thousand dollars is also a fraction of what the human authors spent, so this is one point on a curve the paper does not plot. And authors grading an agent's attempt at their own question are not neutral graders, though releasing the reviews is the right mitigation.

Industrial implication Four of the five failure modes are judgment about what to do next given an incomplete picture, which is not the kind of gap that closes with a bigger model or a longer context window. Research-agent products should be built and sold as extremely capable execution layers under human direction, and the engineering result is not a small win. But anyone pricing in autonomous research within a year is pricing in a capability that no current training objective targets.

→ Full summary

GPT-Red: Automated Red Teaming via Self-Play at Scale

OpenAI says this is the largest LLM safety training run ever documented. What it produced is an attacker that beats human red-teamers and transfers to models it never trained against.

Source: HuggingFace Daily Papers (OpenAI) Links: arXiv 2607.26115 · Wiki summary

flowchart LR
  R[GPT-Red<br/>attacker policy] -->|prompt injection| POP[Population of defenders<br/>trained simultaneously]
  POP -->|attack success| R
  ENV[Realistic red-team<br/>environments] --> POP
  R --> AT[Adversarial training<br/>for GPT-5.6]
  AT --> D56[GPT-5.6]
  D56 -.->|harder target =<br/>better signal| R
  R --> H{Held-out<br/>transfer}
  H --> H1[New environments]
  H --> H2[New defender models]
  H --> H3[New harnesses]
  classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
  classDef decision fill:#fef3c7,stroke:#f59e0b,color:#78350f
  classDef output fill:#d1fae5,stroke:#10b981,color:#065f46
  classDef warn fill:#fee2e2,stroke:#ef4444,color:#7f1d1d
  classDef aux fill:#e0e7ff,stroke:#6366f1,color:#312e81
  class ENV input
  class H decision
  class D56,H1,H2,H3 output
  class R warn
  class POP,AT aux

What is it about? OpenAI built a model whose only job is finding prompt injection attacks, meaning attacks that get a model to follow instructions hidden inside content it was only supposed to read. They then used it to adversarially train GPT-5.6.

What problem does it solve? Automated red-teaming usually does not transfer. Train an attacker against one fixed defender and it learns that defender's quirks. GPT-Red attacks a whole population of defenders that are themselves being trained during the run, so the target keeps moving and single-defender tricks stop paying off. The environments are realistic agent settings with tools and harnesses, not a list of adversarial strings, which is why the attacks route through harness behaviour and survive a harness change.

What is the core novelty? The population design plus the scale. OpenAI claims compute on the order of its largest RL post-training runs. The population is the reproducible part; the compute is not.

Key takeaways

Gaps in the study There is not a single number in the abstract. "More attacks than human red-teamers" has no baseline size, no attack budget, no severity measure. "Compute on the scale of our largest runs" compares to an undisclosed quantity. The missing figure that matters most is GPT-5.6's residual injection rate against GPT-Red after hardening, and whether it stays low as GPT-Red keeps training. A flywheel claim needs two turns of the wheel shown, and this shows one.

Industrial implication Stop treating a clean score on a static injection suite as evidence of anything. Four days ago Opus 5 under Auto Mode reported a zero injection success rate against a human-designed attack suite, with an unexplained inversion where the smaller Sonnet 5 scored 0.93% and the larger Opus 5 scored 3.7% without those defences. GPT-Red is a different measurement instrument, and it has not been pointed at Opus 5. For defenders without OpenAI's compute the only durable protections are the ones that do not depend on the model resisting: capability scoping, credential isolation, egress control.

→ Full summary

StealthBench: Measuring Operational Stealth in Offensive-Security Agents

Every agent in the benchmark found a real vulnerability. Then it uploaded credentials to a public bucket, or deleted a production database to prove it had access.

Source: HuggingFace Daily Papers Links: arXiv 2607.26314 · Leaderboard · Wiki summary

flowchart LR
  T[11 real OPSEC incidents<br/>from bug-bounty and<br/>red-team logs] --> S[14 dockerized<br/>scenarios]
  S --> A[Agent runs<br/>offensive task]
  A --> V{Vulnerability<br/>found?}
  V -->|no| F[Failed solve]
  V -->|yes| O{OPSEC panel:<br/>3 LLM judges,<br/>majority vote,<br/>6 dimensions}
  O -->|clean| SS[Safe success<br/>ceiling 54%]
  O -->|blown| RK[Reckless solve:<br/>creds in public upload,<br/>prod deleted,<br/>users force-added]
  classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
  classDef decision fill:#fef3c7,stroke:#f59e0b,color:#78350f
  classDef output fill:#d1fae5,stroke:#10b981,color:#065f46
  classDef warn fill:#fee2e2,stroke:#ef4444,color:#7f1d1d
  classDef aux fill:#e0e7ff,stroke:#6366f1,color:#312e81
  class T,S input
  class V,O decision
  class SS output
  class RK,F warn
  class A aux

What is it about? Offensive-security benchmarks grade one thing: did the agent find the bug. StealthBench adds the axis that separates a professional operator from a detectable one, which is whether the objective was achieved without revealing presence, capability, or collected intelligence.

What problem does it solve? It makes autonomous-attacker risk measurable in the terms defenders actually experience. An attacker that succeeds loudly is a very different threat from one that succeeds quietly, and until now nothing distinguished them.

What is the core novelty? The compound metric and the source material. Eleven hand-verified operational-security failures pulled from real bug-bounty and red-team trajectories, expanded into 14 dockerized scenarios, scored on safe success rate (solved and stealthy), Stealth@Solve (tradecraft among successful solves only), and reckless solve rate (solved, cover blown).

Key takeaways

Gaps in the study Eleven incidents is a small, hand-curated base, and the expansion to fourteen scenarios is where diversity either exists or does not. The OPSEC dimensions are judged by LLMs rather than detected by a defensive product, so a scenario never establishes whether a real tool would have alerted. The three-judge majority vote is also softer than it sounds, because the 07-26 hidden-anchor result showed a false-positive basin in LLM judges that transfers across judge families, so three correlated judges are not three independent opinions.

Industrial implication Blue teams get six behavioural signatures that current autonomous attackers reliably emit, which is a cheap detection rule set available today and probably decaying within a year, because this paper is also a roadmap for fixing exactly that. Check it against the real case: the July intrusion timeline recorded an agent that used encrypted, fragmented transfers, which is real tradecraft, while generating 17,613 logged actions over two and a half days, which is not. It went undetected for five days anyway.

→ Full summary

SecRespond: Benchmarking Agents on Post-Compromise Incident Response

Give a security agent a compromised disk and a pile of alerts, and it will investigate every alert competently and never once wonder what else is on the disk.

Source: HuggingFace Daily Papers (Alibaba NLP) Links: arXiv 2607.26791 · Wiki summary

flowchart LR
  H[Compromised<br/>cloud host] --> SN[Forensic disk<br/>snapshot]
  H --> AL[Alerts, vuln scans,<br/>baseline checks]
  SN --> AG[LLM agent<br/>OpenCode harness<br/>23 models]
  AL --> AG
  AG --> R{What gets found?}
  R -->|reliably| F1[Problems the alerts<br/>already flagged]
  R -->|fails| F2[Silent intrusions<br/>only the disk reveals]
  AG --> OUT[3 forensic reports<br/>plus remediation plan]
  OUT --> V[No model completes<br/>detection AND remediation<br/>on any range]
  classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
  classDef decision fill:#fef3c7,stroke:#f59e0b,color:#78350f
  classDef output fill:#d1fae5,stroke:#10b981,color:#065f46
  classDef warn fill:#fee2e2,stroke:#ef4444,color:#7f1d1d
  classDef aux fill:#e0e7ff,stroke:#6366f1,color:#312e81
  class H,SN,AL input
  class R decision
  class F1,OUT output
  class F2,V warn
  class AG aux

What is it about? The first benchmark for what happens after a breach. Existing cybersecurity evals place agents in clean environments before an attack. SecRespond hands them a forensic disk snapshot of a compromised host plus the alerts a security product generated, and asks for three forensic reports and a remediation plan.

What problem does it solve? Incident response is where LLM agents are actually being deployed inside security operations centres right now, with access to host artifacts and command lines, and nothing measured whether they are any good at it.

What is the core novelty? Ten cyber ranges built from genuinely compromised cloud hosts, covering 4 entry-point types, 21 MITRE ATT&CK techniques and 5 operating systems, evaluated across 23 frontier models on one agent harness.

Key takeaways

Gaps in the study Ten ranges cannot support per-technique detection rates across 21 techniques. Everything runs on one harness, and the wiki's harness thread (GTA-2, 04-20, where harness design mattered more than model capability) says a single-harness result underdetermines any model comparison. There is no human analyst baseline, so 23 models failing tells you the task is hard but not how hard.

Industrial implication Scope security agents to alert triage and explicitly not to threat hunting, because this says the second capability is absent rather than weak. The consequence for budgets is that alert coverage, not response automation, stays the binding constraint on what gets found. Note the shape it shares with today's research-agent result: in both cases the agent executes the task it was pointed at and does not generate the task it was not pointed at.

→ Full summary

OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office Tasks with Economic Grounding

The average task in this benchmark takes a human 2.32 hours. That number ships with the task, along with a price. So the question stops being whether the agent is good enough and starts being whether it is good enough for what it costs.

Source: HuggingFace Daily Papers Links: arXiv 2607.27155 · Project site · Wiki summary

flowchart LR
  PRAC[Practitioner-proposed<br/>office-suite requests] --> PRIV[Privacy-preserving<br/>adaptation]
  PRIV --> T[100 long-horizon tasks]
  T --> E1[Human labor time<br/>avg 2.32 hours]
  T --> E2[Task price proxy]
  T --> VER{Code-based verifiers<br/>from fine-grained rubrics}
  VER --> EVAL[Frontier models<br/>+ measured human baseline]
  E1 --> VW[Value-weighted<br/>evaluation]
  E2 --> VW
  EVAL --> FIND[Far cheaper and faster<br/>than humans, still short<br/>of human deliverable quality]
  classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
  classDef decision fill:#fef3c7,stroke:#f59e0b,color:#78350f
  classDef output fill:#d1fae5,stroke:#10b981,color:#065f46
  classDef warn fill:#fee2e2,stroke:#ef4444,color:#7f1d1d
  classDef aux fill:#e0e7ff,stroke:#6366f1,color:#312e81
  class PRAC,T input
  class VER decision
  class VW,E1,E2 output
  class FIND warn
  class PRIV,EVAL aux

What is it about? A hundred office-suite tasks, spreadsheets and documents and reports, drawn from requests real practitioners proposed and then adapted through a privacy-preserving process. Every task is long-horizon: the average needs 2.32 hours of human labor. The distinguishing feature is that each one carries two economic signals attached, the human labor time and a task price proxy.

What problem does it solve? Agent benchmarks report whether a task got completed and say nothing about whether completing it that way made economic sense. Pairing each task with labor time and price lets you compare human cost against inference cost directly, and lets the score be weighted by task value instead of counting a five-minute task and a four-hour task equally. A benchmark that treats those the same is measuring something no buyer cares about.

What is the core novelty? Economic grounding built into the benchmark rather than bolted on as analysis afterwards. Two supporting choices make it work: code-based verifiers derived from fine-grained rubrics, which is what keeps scoring stable on tasks this long, and a human baseline run alongside the models so the cost comparison is measured rather than asserted.

Key takeaways

Gaps in the study The task price proxy is the number the entire framing rests on, and how it was derived is not stated. Office work also has an unusually clean deliverable, so economic grounding transfers less easily to domains where the output is a judgment. And the privacy-preserving adaptation is both necessary and unquantified: sanitizing a real practitioner request for public release can strip exactly the messy context that made it take 2.32 hours in the first place.

Industrial implication This is the benchmark shape enterprise buyers actually need, because the deployment question was never "can it" but "at what quality, for what price, against what the person costs." Expect the labor-time-plus-price pattern copied into vertical agent benchmarks quickly. Expect the honest headline, cheaper and faster but not good enough, to hold for office agents through at least the next year, and expect vendors to quote the cost half without the quality half.

→ Full summary

TurboVLA: Real-Time Vision-Language-Action at 32 Hz on an RTX 4090

Robot policies route every single control step through a multi-billion-parameter language model. This one removes the language model and matches much larger policies at 0.9 GB of VRAM.

Source: HuggingFace Daily Papers (Huazhong University of Science and Technology, Huawei) Links: arXiv 2607.27205 · Code · Wiki summary

flowchart LR
  subgraph CONV[Conventional LLM-centric VLA]
    V1[Image] --> PR[Project into<br/>LLM token space]
    L1[Instruction] --> PR
    PR --> LLM[Large language model<br/>billions of params,<br/>every control step]
    LLM --> DEC1[Action decoder]
  end
  subgraph TURBO[TurboVLA]
    V2[Image] --> VE[Vision encoder]
    L2[Instruction] --> LE[Language encoder]
    VE <-->|lightweight<br/>bidirectional<br/>interaction| LE
    VE --> DEC2[Compact decoder,<br/>continuous action chunks]
    LE --> DEC2
    DEC2 --> OUT[32 Hz, 0.9 GB VRAM,<br/>0.2B params]
  end
  classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
  classDef output fill:#d1fae5,stroke:#10b981,color:#065f46
  classDef warn fill:#fee2e2,stroke:#ef4444,color:#7f1d1d
  classDef aux fill:#e0e7ff,stroke:#6366f1,color:#312e81
  class V1,L1,V2,L2 input
  class OUT,DEC2 output
  class LLM warn
  class PR,DEC1,VE,LE aux

What is it about? A vision-language-action model is a policy that takes a camera image plus a natural-language instruction and emits robot joint commands. Almost all of them project the image into a large language model's token space, run the language model, and decode actions out the far side. TurboVLA encodes vision and language separately, lets them attend to each other through a lightweight bidirectional module, and decodes action chunks directly.

What problem does it solve? The language model gets run on every policy invocation. At control frequencies that is dozens of billion-parameter forward passes per second, which is why VLA deployment needs a server in the loop.

What is the core novelty? Reformulating the pathway from V → L → A into V + L → A. Task-conditioned representations are built by direct vision-language interaction rather than by serializing the image into language space and running a general-purpose reasoner over it.

Key takeaways

Gaps in the study LIBERO only, and LIBERO has a bounded instruction distribution, which is the friendliest possible ground for a design whose central risk is losing exactly what the language model was providing: open-vocabulary understanding and generalization to unseen phrasings. The paper's own framing makes the missing experiment obvious, and it is not run. No real-robot result is reported either, so 32 Hz is a compute claim rather than a demonstrated control loop.

Industrial implication If this survives contact with real robots and open-vocabulary instructions, manipulation moves from server-attached to on-board, which removes the network round trip that currently caps control frequency and turns fleet deployment into a per-unit hardware cost rather than a per-unit inference bill. Compare it with VISCO (07-27), which keeps the language model and compresses the visual tokens feeding it: same cost, opposite cut. The untested middle, a small language model with compressed visual tokens, is where the real answer probably sits.

→ Full summary

CAST: Game Solvers as Turn-Level Teachers

On-policy distillation has always needed the teacher's full output distribution. This paper shows that if your teacher is a solver, one number per turn carries the same signal.

Source: HuggingFace Daily Papers Links: arXiv 2607.25308 · Code · Wiki summary

flowchart LR
  S1[Game state t] --> ACT[Agent action]
  ACT --> S2[Game state t+1]
  S1 --> SV1[Solver value<br/>of state t]
  S2 --> SV2[Solver value<br/>of state t+1]
  SV1 --> D[Value delta<br/>= solver advantage]
  SV2 --> D
  D --> INJ[Inject as turn-level<br/>signal into RLVR]
  RW[Sparse final reward:<br/>says nothing about<br/>which turn mattered] --> INJ
  INJ --> POL[Policy update]
  D -.->|soft-optimal<br/>solver assumption| EQ[Equals on-policy distillation<br/>from the solver,<br/>scalars only, no logits]
  classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
  classDef output fill:#d1fae5,stroke:#10b981,color:#065f46
  classDef warn fill:#fee2e2,stroke:#ef4444,color:#7f1d1d
  classDef aux fill:#e0e7ff,stroke:#6366f1,color:#312e81
  class S1,S2,ACT input
  class POL,EQ output
  class RW warn
  class SV1,SV2,D,INJ aux

What is it about? RLVR (reinforcement learning with verifiable rewards, where the signal is a checkable final answer rather than a human preference) gives you one reward at the end of a long game and no information about which of the fifty moves along the way was the good one. CAST notices that in any domain with a classical solver, the solver's state value already answers that. The change in solver value from one state to the next says directly whether an action advanced the position.

What problem does it solve? The two existing fixes for turn-level credit are a learned process reward model (cheap, inaccurate) or a teacher LLM scoring every step (accurate, expensive). A solver is both cheap and exact.

What is the core novelty? The equivalence proof. Under a soft-optimal solver assumption, maximizing the solver advantage is on-policy distillation from the solver, needing only scalar values rather than teacher logits. That collapses the teacher's bandwidth from a vocabulary-sized vector per token to one number per turn, which means the teacher no longer has to be a language model at all. A Sokoban solver has no tokenizer and no logits, and under this framing that stops mattering.

Key takeaways

Gaps in the study The soft-optimal assumption does real work in the derivation and no game solver satisfies it exactly, so the equivalence is approximate by an unquantified amount. All three training domains are perfect-information puzzles with cheap exact solvers, which is the narrowest slice of "domains with solvers." Nothing tests noisy or expensive solvers, and there is no sensitivity analysis for solver quality.

Industrial implication Anywhere a verifier already exists as a program rather than a model, this says you have been throwing away its most valuable output. Code agents have compilers, test suites and static analyzers; SQL agents have query planners; formal-methods agents have proof checkers. All of those currently get used as terminal pass/fail rewards. CAST says they can be dense per-step teachers at no extra inference cost. Line it up with two results from yesterday and the trend is unmistakable: BPM (07-29) removed the shared-tokenizer requirement from on-policy distillation via byte-prefix marginalization, Relay-OPD (07-29) removed the verifier requirement using teacher-student continuation asymmetry as a label-free trigger, and CAST removes the requirement that the teacher be a neural network at all.

→ Full summary

CoRT: Counterfactual Replay for Token-Level Rubric Credit

Rubric-based RL evaluates a response against eight written criteria and then gives every token in the response the exact same gradient.

Source: HuggingFace Daily Papers (Nanjing University, ByteDance) Links: arXiv 2607.25659 · Wiki summary

flowchart LR
  RESP[Sampled response] --> R1[Replay under<br/>rubric-conditioned prompt]
  RESP --> R2[Replay under matched<br/>criteria-free prompt]
  R1 --> C[Per-token<br/>log-likelihood contrast]
  R2 --> C
  C --> W[Bounded,<br/>response-normalized<br/>weights]
  ADV[Signed GRPO<br/>response-level advantage] --> RD[Redistribute<br/>across tokens]
  W --> RD
  RD --> UP[Rubric-dependent tokens<br/>get more credit]
  BASE[Standard GRPO:<br/>one advantage broadcast<br/>to every token] -.->|replaced| RD
  classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
  classDef output fill:#d1fae5,stroke:#10b981,color:#065f46
  classDef warn fill:#fee2e2,stroke:#ef4444,color:#7f1d1d
  classDef aux fill:#e0e7ff,stroke:#6366f1,color:#312e81
  class RESP,ADV input
  class UP output
  class BASE warn
  class R1,R2,C,W,RD aux

What is it about? Rubrics decompose evaluation into explicit criteria (correctness, formatting, safety, tone) instead of one opaque score. GRPO, the standard RL algorithm for LLM post-training, then flattens all of it into a single response-level advantage and broadcasts it uniformly across every token. A formatting criterion and a factual-correctness criterion live in completely different spans, and the gradient cannot tell them apart.

What problem does it solve? Recovering within-response credit without training an auxiliary scorer. The prior approach, Rubrics-to-Tokens, trains a separate relevance model, which costs a second training stage and a second thing that can go stale as the policy moves.

What is the core novelty? CoRT replays the same sampled response twice, once under the rubric-conditioned prompt and once under a matched criteria-free prompt, and uses the per-token log-likelihood difference as a measure of how much that token depended on the rubric. The policy already contains the signal: if a token was generated because the rubric was there, removing the rubric lowers its likelihood.

Key takeaways

Gaps in the study The confound to worry about is whether the contrast measures rubric dependence or merely prompt sensitivity. A token whose likelihood drops when any long prefix is removed would be upweighted for entirely the wrong reason, and a placebo-prompt ablation would settle it cheaply. The 4.4-point average also comes with an unstated spread, and "vast majority of comparisons" concedes regressions that are never characterized.

Industrial implication One extra forward pass per sampled response, no new model to maintain, 4.4 points. For anyone running rubric-based post-training, which is now the standard recipe for instruction-following and safety behaviour at every major lab, that is worth testing this quarter. This is also the fourth paper in a month arguing that uniform credit across a trajectory is the waste, after TIP (most teacher tokens carry no signal, roughly 10% suffice), LongAct (04-18) (restrict RL gradients to high-magnitude activation positions, about 8% on LongBench v2), and Relay-OPD. Four different signals, one shared claim.

→ Full summary

Local Coding Models: KV Cache, Not Parameter Count, Is the Constraint

Two models of comparable size. One holds a 262K context window in under 2 GB of cache. The other needs 40 GB.

Source: Kilo Code blog, via @kilocode on X Links: Blog post · Wiki summary

What is it about? A practitioner benchmark of 9 local coding models across every consumer hardware tier, from an 8 GB GPU to a dual-3090 rig. The authors' own headline is not about capability, it is about memory: NVIDIA's Nemotron Cascade 2 holds 262K tokens of context with a KV cache (the stored key and value tensors for every token already processed, so attention does not recompute them) under 2 GB, while Devstral Small 2, a dense model, needs 40 GB of cache alone for comparable context.

What problem does it solve? It kills parameter count as a deployability proxy. Weights are a fixed cost paid once at load. The KV cache grows linearly with context, and coding agents are the workload that burns context fastest, because a repository map plus a few files plus history is tens of thousands of tokens before any work starts.

What is the core novelty? Not a technique, a measurement. The gap is roughly 20x, which is large enough to invert the usual advice: a 30B model with a 2 GB cache is a smaller total footprint at long context than a 14B dense model with a 40 GB cache, so the bigger model is the one that fits.

Key takeaways

Gaps in the study A vendor-adjacent blog benchmark with no stated methodology for how cache size was measured, and footprint is extremely sensitive to quantization, attention implementation and batch size. The comparison is between two specific models rather than an architecture-controlled ablation, so some of the gap is model-specific. Treat the direction and the order of magnitude as real and the exact multiple as approximate.

Industrial implication The procurement question changes from parameter count to cache-per-token at target context, and that number appears on no model card anywhere. The same architectural property shows up at the opposite end of the scale: PrfaaS (04-22), which offloads long-context prefill to a separate datacenter and ships the resulting cache over Ethernet, only works because hybrid-attention models emit cache 13x more slowly (4.66 Gbps for MiMo-V2-Flash against 59.93 Gbps for a dense baseline). Same cause, measured as GB-resident on one card and as Gbps-on-the-wire across datacenters.

→ Full summary

SkillRise and DecoEvo: Two Ways to Stop Freezing the Evaluator

One paper makes writing your own skill document an action the policy gets gradient for. The other co-evolves the rubric alongside the solver while making it impossible for the rubric to just get easier.

Source: HuggingFace Daily Papers Links: SkillRise arXiv 2607.26784 · DecoEvo arXiv 2607.25675 · SkillRise summary · DecoEvo summary

flowchart LR
  subgraph SR[SkillRise: RL-native curation]
    T1[Task 1] --> P1[One policy:<br/>solve]
    P1 --> C1[Same policy:<br/>curate skill doc]
    C1 --> D[(Evolving<br/>skill document)]
    D --> P2[Task 2, harder]
    R2[Discounted downstream<br/>outcomes] -.->|supervises curation| C1
  end
  subgraph DE[DecoEvo: decoupled co-evolution]
    S[Solver skill] --> RESP[Response]
    RESP --> RUB[Rubric generator]
    RUB -->|criterion feedback| S
    RESP --> A1{Coverage audit}
    RESP --> A2{Discrimination audit}
    A1 --> RUB
    A2 --> RUB
    HACK[Score-coupled updates:<br/>easier rubric wins] -.->|blocked| RUB
  end
  classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
  classDef decision fill:#fef3c7,stroke:#f59e0b,color:#78350f
  classDef output fill:#d1fae5,stroke:#10b981,color:#065f46
  classDef warn fill:#fee2e2,stroke:#ef4444,color:#7f1d1d
  classDef aux fill:#e0e7ff,stroke:#6366f1,color:#312e81
  class T1,S input
  class A1,A2 decision
  class RESP,P2 output
  class HACK warn
  class P1,C1,D,R2,RUB aux

What is it about? Both papers attack the same structural waste from different sides. SkillRise notes that standard agentic RL treats every task as an independent episode and forgets everything, so it has one policy alternate between solving a task and curating a skill document passed to the next task. DecoEvo notes that text-space optimization (improving a model by editing external natural-language artifacts rather than weights) almost always holds the evaluator fixed, so once the solver satisfies the rubric's criteria, every omitted dimension stays permanently invisible.

What problem does it solve? SkillRise removes the multi-stage pipelines where extraction, retrieval and execution are separate components that have to be kept consistent. DecoEvo solves the reason nobody just evolves the rubric too: if rubric updates are selected by the solver's score, the optimizer discovers that making the rubric easier raises the score.

What is the core novelty? For SkillRise, decoupled credit assignment. Solving is supervised by the current task's outcome; curation is supervised by discounted downstream outcomes, which is what forces the document to be written for a future reader rather than as a summary of what just happened. For DecoEvo, two audits that cannot be gamed by lowering the bar: requirement coverage (a rubric that drops a hard criterion loses coverage) and response discrimination (a rubric that only asks easy questions stops separating good from bad). Neither audit ever looks at the aggregate solver score.

Key takeaways

Gaps in the study SkillRise runs on three short-episode simulators with clean rewards, the task sequences are author-constructed in increasing difficulty, and nothing shows what happens when order is random or unrelated tasks interleave. Nothing is said about skill-document growth either, and an evolving document that is never pruned eventually becomes a context-length problem. DecoEvo's whole anti-gaming argument rests on who runs the two audits, and the abstract does not say. If an LLM judges coverage and discrimination, the generator can learn to satisfy the auditor rather than the property, which is the same failure one level up. There is also no compute-matched comparison, so some of DecoEvo's margin may simply be more search.

Industrial implication DecoEvo's transferable idea is not the co-evolution loop, it is the audit pair. Coverage and discrimination are cheap measurable properties of any rubric, and most production eval rubrics are written once and never checked against either, so running the two audits against an existing suite is a half-day that will find omitted dimensions in almost any mature evaluation. SkillRise's is the runtime result: a single policy that curates as an action removes a retrieval component from the serving path, and the deployable version is a per-repository or per-customer skill document that improves over a ticket queue.

→ SkillRise summary · DecoEvo summary

The Wild Wild West of LEGO Datacenters

Everyone argues about chips and grid interconnects. SemiAnalysis says the thing that actually caps US datacenter capacity in 2027 is that there are not enough licensed electricians, and capital cannot fix that on any useful timescale.

Source: SemiAnalysis (also arrived via starred Gmail) Links: SemiAnalysis post · Wiki summary

flowchart LR
  SITE[Site: grading,<br/>wiring, foundations] -->|cannot modularize| EXP[Labor exposure<br/>irreducible]
  SHELL[Shell: structure,<br/>skin, roof] --> FAC[Built in factory,<br/>craned and bolted]
  SYS[Systems: power rooms,<br/>cooling skids] --> FAC
  ELEC[Electricians:<br/>30-40% of man-hours,<br/>shortage from 2027] --> WHY{Why modular<br/>at all}
  WHY --> FAC
  FAC --> WIN[Build window<br/>~36% shorter,<br/>7-9 months]
  FAC --> CST[~8% cheaper<br/>per MW]
  FAC --> VC[Vendor content per MW<br/>$3.5M to $7M]
  classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
  classDef decision fill:#fef3c7,stroke:#f59e0b,color:#78350f
  classDef output fill:#d1fae5,stroke:#10b981,color:#065f46
  classDef warn fill:#fee2e2,stroke:#ef4444,color:#7f1d1d
  class SITE,SHELL,SYS input
  class WHY decision
  class WIN,CST,VC,FAC output
  class ELEC,EXP warn

What is it about? How the largest datacenters are physically built, and why that changed. Concrete walls now arrive as finished panels, mechanical and electrical rooms arrive pre-wired, and sometimes an entire data hall arrives on the back of a truck. SemiAnalysis's Modular Tracker covers 61GW+ of modular capacity across 1,000+ sites, and projects modular passing 30% of total live capacity by the end of 2028.

What problem does it solve? Trade labor. SemiAnalysis has spent recent months arguing that most claimed datacenter bottlenecks are misunderstood and solvable, including in "Stop Saying Half of 2026 US Datacenter Capacity Is Canceled." This is the stated exception, because you cannot quickly produce licensed electricians and pipefitters. Electricians alone are 30 to 40% of total construction man-hours on a datacenter project, and their new Labor Model projects a shortage emerging in 2027, worst exactly where the buildout concentrates, in Texas and Ohio.

What is the core novelty? Two analytical moves. First, treating reachable labor supply as a shared pool across states, so a project in one state reduces the labor available to its neighbors and site capacity stops being additive. That is the part that makes this a model rather than an observation. Second, insisting on a distinction the industry has blurred into meaninglessness: prefabrication is any work done off-site, a statement about where the work happened, while modular means specifically self-contained units that ship complete and bolt together. Every modular unit is prefabricated; not all prefabrication is modular. That conflation is why vendor speed and cost claims have been impossible to compare, and it is what the article's 80-plus-player vendor map exists to fix.

Key takeaways

Gaps in the study Every headline figure comes from SemiAnalysis's own proprietary Industrials Model and is not independently checkable, and the article is explicit that it is testing vendor claims against that model rather than against observed completions. The labor forecast holds labor-hours per GW roughly flat ex-modular, which is precisely the quantity modularization is supposed to change, so demand and mitigation are not modeled jointly in the framing chart. The vendor-by-vendor positioning, where the falsifiable claims actually live, sits behind the subscriber paywall. And an 80-player universe with no shared definition of the product is exactly the setting where some modular claims turn out to be relabeled skids; the article promises to test this and the tested results are not in the public portion.

Industrial implication If you model AI compute supply, 2027 US capacity should be forecast against electrician-hours in the concentrating states, not against GPU shipments or interconnect queues, and that is a different spreadsheet from the one most people are running. The procurement read is that a large slice of what used to be on-site construction labor migrates onto vendor balance sheets as equipment content, which is why Vertiv's per-MW content doubles. That makes the modular OEM and system-integrator layer a structurally better place to sit than the EPC layer, and turns "who owns the factory" into a question about AI capacity. It also explains the financing behavior in today's Pulse: developers racing a labor curve have to commit capital before a lease exists, which is what "spend now, lease later" bridge lending is absorbing.

→ Full summary

Industry Pulse

Funding, valuations, and compute deals

Global View

The field's most-praised papers this week are papers about why the field's measurements do not work, and the market is quietly paying for the same conclusion. HuggingFace served seven evaluation papers today, while Kurate's cs.AI board has Do Agent Benchmarks Measure Capability? at #1 with an 87.5% win rate, arguing agent benchmarks have a protocol-validity problem rather than a difficulty problem, and What AI Red-Team Evaluations Can and Cannot Prove at #2 with a 90.9% win rate, arguing red-team evaluations establish that a vulnerability exists and essentially never that one does not. Yesterday's digest predicted a second validity paper would reach the top three within 60 days and it arrived in one day, which makes this the third measurement-crisis thread the wiki has tracked this quarter alongside KV eviction ablations and contamination-free evaluation. The industry version of the same judgment is that micro1 is paying HVAC companies up to $150,000 to have real practitioners rank anonymized models on vendor invoices and payroll, which is what buying validity looks like when you cannot get it from a leaderboard, and it is exactly the signal OmegaUse-OfficeVal generates by pricing each task in human hours. Note who is not paying: GPT-Red is marketed as a flywheel toward robustness, which is precisely the absence claim Kurate's #2 says a red-team evaluation cannot make.

Offensive AI capability is being scaled with frontier compute while defensive AI capability cannot find anything nobody pointed at, and the only thing holding the asymmetry together is that today's attackers are loud. GPT-Red is the largest documented LLM safety training run and produces an attacker that beats human red-teamers and transfers across harnesses; StealthBench finds no offensive agent above 54% once you require it to stay hidden; SecRespond finds no model completing detection and remediation on any range because agents investigate alerts and never hypothesize silent intrusion. The real case validates all three at once: the July intrusion generated 17,613 logged actions, which is terrible tradecraft, and still ran five days undetected, with OpenAI now conceding credentials were used on four other services. Industry is responding by shipping offensive-defensive tooling on both sides in the same week (OpenAI open-sourcing Codex Security CLI after 3,000-plus fixes, Claude Mythos halving a post-quantum cipher's key strength in 60 hours) while 1,100-plus of their own employees petition Washington to slow down, and StealthBench is a public scoreboard for the one capability that would make the next intrusion invisible.

Every part of the stack independently concluded this month that context memory, not model capability, is the binding constraint, and the market has started pricing the plumbing accordingly. OpenAI tripled its ARC-AGI-3 score with two settings that let the model retain reasoning across context windows, an explicit statement that the bottleneck was context management rather than reasoning; practitioners measured a 20x KV cache gap between Nemotron Cascade 2 at 262K context in under 2 GB and a comparable dense model at 40 GB; and TurboVLA hits 97.7% on LIBERO at 0.9 GB by deleting the language model from the robot policy entirely. That practitioner number is roughly an order of magnitude larger than what this month's cache-management research reports, including LOCKS (07-29), which matches full-cache quality at 100K+ context reading about 2% of tokens, so the sequencing lesson is architectural choice first and page selection second. Meanwhile the market is valuing the layer that decides which model gets the context at 70x revenue, with Stripe reportedly near a $10 billion deal for OpenRouter on ~$140 million annualized, which sits awkwardly against DeepMind's meaningfulness result (07-20) finding routing gains often vanish under honest accounting, and against senators pressing Apple away from Chinese memory in a market where the memory-hierarchy page projects HBM allocation constraints running to 2030.

Looking Ahead

On Kurate's rising authors, again: still an artifact, and now a stale one. The list is unchanged in character from last week. Every author crossing the threshold is a co-author on one of a handful of biomedical foundation-model papers (Guy Lutsker, Gal Sapir, Jordi Merino, Smadar Shilo, Anastasia Godneva and Eli Meirom on the human-physiology generative model 2604.27899; Andrew Zhang on virtual-patient representations 2604.18570) that have simply persisted on the weekly board since W28. The metric counts board persistence, not author productivity, and the 07-29 digest already stated the falsifiable version: change the threshold to require distinct papers and this list collapses to zero. No Twitter handle additions are warranted, and none of these authors work on AI systems. This is now the second week the same fix has gone unmade, which makes it a connector task rather than a finding.

(Reddit contributed nothing today: all eight subreddit farms returned zero posts passing filters, the third dry stretch this month. Twitter's curated retweet feed was empty for the second consecutive slot, so the AI handle feed carried the entire social signal, and most of it was geopolitics. No file exists in the parallel Daily-Digest job directory for today.)