Media Zone | 2026-10-02
The US day's conversation was about where the compute bill gets decided. A builder's router cut a coding agent's median cost per task by 64%, Meta showed an agent that edits its own context and pays for it with a new KV-cache trick, and NVIDIA showed a cheap agent gets much better when a strong judge picks its next command. By US evening the thread moved down the stack: a Triton-level attention kernel beat FlashAttention-4 on Blackwell, OpenAI said its own models tune its serving kernels, and a speed-of-light analysis showed per-row decision-API calls waste most of the GPU. The common thread is cost optimization through judgment: spend compute on deciding, not doing. Memory money kept moving, with Micron warning of tighter years ahead and Volantis raising $88M to feed chips over light.
Today's signal
- Dominant story: routing moved into the harness. A coding-agent router cut median cost per task 64% with no measurable quality drop; most steps never needed the top model.
- Kernels leave hand-written CUDA: Meta's 3.2K-line Triton (TLX) attention kernel beats FA4 on Blackwell jagged shapes, and OpenAI says Astra now writes the kernels behind Astra Ultrafast.
- Judgment beats more samples: NVIDIA's Mid-Harness, Meta's controller for long agent runs and AgentWorld's "two thirds of actions wasted" all say extra compute pays only when something good decides where it goes.
- Decision models multiply, and get costed: Cloudflare's clef and Fastino's GLiDE join Jev, while a roofline analysis shows per-row decision calls sit far from what one H100 could do.
- Memory stays tight: Micron says 2027 and 2028 may be tighter than 2026; AWS raises reserved GPU prices about 15% while neoclouds undercut.
- Quiet areas: no new X bookmarks, curated reposts or LinkedIn posts this window; Reddit was empty. The X Following feed, Gmail and a handful of YouTube talks carry the day.
Routing, KV cache, compression, GPU
The bill is decided before the model is called
The US morning was about the arithmetic of a routed loop. By the afternoon a builder had published the result everyone wants: a router inside the coding agent's harness that sends easy steps to cheaper models.
flowchart LR
E["Events<br/><small>20,000 per day</small>"] --> C["Code rules<br/><small>drop obvious noise</small>"]
C -->|noise| N["Dropped<br/><small>about 70%</small>"]
C --> D["Decision model<br/><small>local, Jev or clef</small>"]
D -->|log only| G["Logged<br/><small>no generation</small>"]
D -->|worth it| W["Writer LLM<br/><small>about 4% of events</small>"]
W --> H["Human review<br/><small>final gate</small>"]
classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
classDef err fill:#ffe3e3,stroke:#e03131,color:#1b1b1b,stroke-width:2px
class E input
class C,D loop
class N err
class W core
class G,H exit
linkStyle 1 stroke:#e03131,stroke-width:2px
linkStyle 4 stroke:#2f9e44,stroke-width:2px
A model router cut a coding agent's median cost 64%
A builder reports adding a model router to their coding agent: each step is sent to the cheapest model that can handle it, and the median cost per task fell 64% with no measurable drop in quality. The claim is simple: most agent steps do not need top-tier intelligence. The post also argues the router belongs inside the harness (the loop that drives the model), not in front of the API, because the harness knows what each step is for. The write-up itself was not in the capture, so the eval behind "no measurable drop" is still unread. Rising fast through the US afternoon.
"Optimize the 4% before the writer"
A builder posted the full funnel of a Jev-gated pipeline: 20,000 events a day, 14,000 dropped as noise, 4,700 logged, 800 drafted, 500 sent to a human. Only about 4% reach the writing model. The decision layer costs $0.84 a day while the whole loop lands at $29.04. Once a router exists, the remaining cost lives almost entirely in the generation step, so that is where to optimize next. Quietly high-signal off a small account.
Cloudflare clef: an open Jev alternative
Cloudflare's Workers AI team released clef, its first trained models, under Apache 2.0 on Hugging Face. The model card shows a post-train of Qwen3.8-27B that turns text and images into typed, structured outputs (classification and decisions) with three reasoning-effort levels. It is the open-weight answer to TypeSafe's Jev, following OpenAI, Ollama and Databricks shipping decision endpoints this week. For anyone running a router, it means the gate can now be self-hosted without a license question.
An 18 ms local decider for your agent's safety hook
A ten-step write-up on moving a coding agent's allow/ask/deny hook off a cloud decision API. Plain code rules run first, narrow questions go to a 1.5 GB local model at about 18 ms per call, and only fuzzy judgment goes to Jev or a human. The honest number is accuracy: 93.5% local against Jev's 97.2%, so local is for the questions you never wanted to upload, not all of them.
How fast could an AI filter possibly run?
Shreya Shankar and Arnav Dhariya cost AI-powered SQL filters from first principles. They count the arithmetic and memory traffic a filter needs and divide by the GPU's peak compute and bandwidth (a roofline, or speed-of-light, estimate). One filter over 5,000 movie reviews on Qwen3-4B and one H100 could finish in about 6.6 seconds, and no system today comes close, including their own Quail engine. The lesson for anyone running decision models in batch: per-row API calls throw away KV reuse across rows and smart filter ordering, which is where most of the cost hides.
- GLiDE, a decision model that thinks. Fastino's model computes a fast distribution and reasons only when the top answer is uncertain; it claims a 6.9-point lead over Jev on a vendor-run index (@george_onx). A companion paper finds Jev-style models squash 1-to-N scales toward the middle (paper).
- Databricks AI Decide. An
ai_decideSQL and REST function that turns text into structured decisions over governed data, explicitly for customers who cannot send data to Jev. Databricks names model routing as a target use (@jjanezhang). - Small specialists from your own logs. Overmind records what a production agent does, turns it into training data and fine-tunes an open-weight model the customer owns. Their tuned 9B finds contract clauses at 91.6% F1 against a frontier model's 82.0% (@rohanpaul_ai, blog).
- Agents now out-consume people. a16z's State of Markets II says agents burn nearly 5x the tokens humans do, up 14x since February. Routing savings now compound on the bigger half of the bill (@a16z, report).
- One API for every team. Red Hat's Model-as-a-Service puts approved models behind one OpenAI-compatible endpoint with metering, on vLLM and llm-d; LiteLLM launched Lens, observability for the gateway that already sees every call (@RedHat_AI, @garrytan RT).
Context the model edits, and the cache that breaks
Context Language Models (Meta, UW, MIT)
Agents usually rely on fixed harness rules to compact, offload or retrieve context. CLMs drop the rules: the live context is a file the model can edit with Bash, so it decides what to keep, rewrite or delete. Zero-shot on existing models this gives 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, and online RL lifts Qwen3.5-9B by 47.6% while using 12% fewer FLOPs. The serving catch is the interesting part: editing the middle of the context invalidates the prefix cache (the stored attention keys and values servers reuse across turns). The authors add Suffix Cache Reuse, which cuts server compute 35% against standard SGLang. They also flag a risk: an editable context lets injected instructions persist across turns.
Prefill is compute, decode is bandwidth
A clean explainer of why the GPU choice follows the kernel path and file encoding, not the other way round. Prefill reads the prompt and builds the KV cache, which is compute-heavy; decode writes one token at a time and keeps rereading weights and cache, which is bandwidth-heavy. That is why an RTX PRO 6000 at 1.8 TB/s beats a DGX Spark at 273 GB/s for chat. Long prompts punish prefill, long answers punish decode, and long chats punish both. A good refresher alongside the vLLM scheduler walkthrough from earlier today.
Kernels keep leaving CUDA
Jagged Flash Attention in TLX beats FA4 on Blackwell
Meta's Generative Ads Model spends most of its time in attention over jagged batches, where every user's history has a different length. Meta rebuilt that kernel in TLX, Triton's low-level extensions that add explicit control over thread-group roles and asynchronous memory moves. It is about 3.2K lines against roughly 10K for FlashAttention-4's CuteDSL kernels, and beats FA4 by about 13% forward and 50% backward on B200. The point beyond speed: modeling engineers can read and extend it, not only kernel specialists.
GPT-6 Astra Ultrafast: the model tunes its own serving stack
NVIDIA says Astra Ultrafast generates up to 8x faster than Astra Standard on Blackwell. OpenAI's inference lead Philippe Tillet, who created Triton, says Astra turns its knowledge of Blackwell and Rubin into high-performance kernels, and OpenAI used internal models to optimize inference. It is the first vendor statement that model-written kernels sit behind a paid frontier product. No split of the 8x between kernels and other changes was given.
DeepGEMM and the Ascend ports
DeepGEMM is DeepSeek's open matrix-multiply library for Hopper and Blackwell, with FP8, FP4 and BF16 paths, grouped GEMMs for mixture-of-experts layers, compute and communication overlap, and kernels compiled at runtime. It resurfaced because DeepSeek's Ascend ports report DeepGEMM-Ascend at 431 of a 432 BF16 TFLOPS ceiling. One catch from the feed: DeepEP-Ascend shipped without a license file, so it cannot legally be reused yet.
WebGPU kernels for 200+ ops
Xenova (Transformers.js) open-sourced what they call the fastest WebGPU kernels for over 200 machine learning operations, specialized per hardware. It pushes more inference onto the user's own device, which is free compute from the provider's point of view. Benchmarks are not in the capture yet.
Looped Diffusion Transformer
Looped-DiT reuses the same Transformer blocks several times inside each denoising step, trading parameters for depth by repetition. A 260M model beats a model 6.5x larger on text-to-image benchmarks while using 4.9x less inference compute. Looping has been a language-model efficiency theme; this is the image-generation version.
- TorchTPU. Google and Meta will show upstream vLLM and SGLang running on a PyTorch-native TPU backend, keeping schedulers, batching and KV cache layout logic shared with GPU, so new models need no TPU-specific port (@PyTorch).
- CoreWeave RL Rollouts. NVIDIA Dynamo's ModelExpress reloads updated weights into inference workers 15x faster between RL steps, cutting idle GPU time in post-training (@NVIDIAAI).
GPU hours: dearer at AWS, cheaper at neoclouds
- AWS raises reserved GPU prices about 15% across H100, B200 and B300 starting next week, per a markets account. If confirmed, premium capacity is still supply-constrained (post).
- CoreWeave claims about $0.75 per million tokens on GB200 NVL72 against roughly $1.86 at major hyperscalers on the same hardware. Vendor-favourable framing; treat as a claim (post).
- Anthropic's prospectus lists $518B of compute commitments over a decade, about 80% non-cancelable, against $7.3B of 2025 compute costs. The largest single demand signal for GPUs and TPUs on record (AI Weekly Espresso via Fortune).
- Nebius buys Inferize to cut model cold starts and idle GPU time; Oracle and Tencent's $7B deal for about 100K chips works out near $1.60 per GPU-hour by Ed Zitron's math (Nebius, deal, math).
- Optics export curbs. A report says the US could restrict Chinese optical transceivers from 3.2T, leaving today's 800G and 1.6T buildout untouched; Coherent and Lumentum rose over 10% (post).
- Memory gets tighter before it gets better. Micron says 2027 and 2028 may be more constrained than 2026; Volantis raised $88M (Jeff Dean, Sam Altman, John Doerr) for optical chips that feed accelerators memory over light, aiming at 10,000 tokens per second per user on 10T+ models (Micron, Volantis, take).
- Token spend is a power law. A practitioner reading of Anthropic's S-1 says two customers are about 25% of revenue; Cursor and xAI report 10% of users drive about 70% of token spend. Agents, not people, make the tail (post).
- Micron and Cerebras follow-through. Micron says growth now comes from HBM and server DRAM mix, not spot prices; Cerebras sits inside a paid OpenAI tier at about 750 tokens per second (Micron, Cerebras).
LLMs, agents, safety
Spend compute on judgment, not on more tries
Thinking Before Thinking: agentic meta-reasoning
As agent runs get longer, each step forces control choices: which partial work to build on, whether to restart, when to stop. This Meta paper splits those out. Workers do the task; a controller keeps a compact summary of the run, weighs options against the remaining budget, and decides which past results each worker sees. On ProgramBench it reaches 71.5% with GPT-5.5 against Codex's 58.0%. Tripling the budget lifted the managed agent from 64.1% to 71.5% while the unmanaged one stalled near 64%.
Mid-Harness: verify the command before running it
Terminal agents usually run the first command they generate, and one bad command (a wrong package install) can derail every later step. NVIDIA's Mid-Harness sits between the model and the harness: it samples several candidate actions, verifies them, and forwards one. With a small TMAX-9B generator, more samples barely help under weak verification. With a frontier verifier, Pass@1 on TerminalBench-Lite rises from 50.0% to 68.0% at 8 samples. Distilling the strong verifier into the 9B model recovers part of the gain, and mixing action and trajectory scaling reaches higher success at lower token cost than just running more trajectories.
Anthropic's build-eval and hillclimb workflow
Anthropic's claude-api skill now has two commands: one builds an eval inside your codebase, the other improves the app against it one change at a time, checked on a held-out set to catch overfitting. The guidance is the useful part: eval tasks should mirror production, scores should rise with stronger models and more thinking, and the best model should still sit well below 100%. The feed read it as a primitive for self-improvement, where the bottleneck shifts from proposing changes to building feedback that is hard to game.
Marathoner: a 9B model that codes for 10+ hours
Ant Group, Peking University and the University of Macau train Qwen3.5-9B to sustain 1,000+ tool calls, with tasks mined from 100,000 GitHub PRs and RL in real execution environments. A bonus rewards useful progress late in a run. SWE-bench Verified goes from 43.8% to 77.5% and Terminal-Bench 2.0 from 27.3% to 57.2%. Persistence looks trainable into a cheap model.
- Google's RRSI. Evolves every part of a harness (prompts, control flow, tools, memory) around a frozen model, with regularization against overfitting (repo, post).
- Multi-harness RL. Hugging Face guide on training one open model with RL across Claude Code, Codex, opencode and others at once, since the same model behaves differently in each harness (@huggingface RT, @ben_burtenshaw RT).
- Build your own harness. Elvis Saravia argues model swaps should not break your setup as release cycles shrink (@omarsar0). Harness essays from Comma's Salix and the Instinct teardown continue the theme (@heyang_zhou, @yashhsm).
- Grading agents on live data. An Adobe paper writes the expected answer as code that re-fetches the current result; the grader agrees with experts 29% more often on 16% fewer tokens (paper).
- Perplexity's contextual embeddings. Encode the whole document once, pool per-chunk vectors; the 9B preview leads context-bench by 14.4 points recall@10, open-sourced (@denisyarats).
- AgentWorld: most multi-agent work is wasted. 3 to 20 agents coordinate over 50+ rounds in a game sandbox; fewer than a third of actions contribute to the outcome and coordination tasks reach 12% (@omarsar0, paper).
- Point a coding agent at your logs. Microsoft finds a coding agent that counts patterns across saved agent logs writes better prompts than GEPA on 3 of 4 benchmarks at about $1.60 each (@rohanpaul_ai).
- A year of decisions humbles agents. On a year-long simulated task with delayed feedback, the best setup ended with 27.3% of the average human's money (@rohanpaul_ai).
- Meta's long-run controller, day two. Same workers and budget, a controller choosing the next work lifts ProgramBench from 63.7% to 71.5% (@dair_ai); a branch-based harness search avoids single-path local optima (@dair_ai).
- Contested: "RLHF causes hallucination." Preference tuning eroding calibration is real and well-studied; "proved" and "the reason" are the viral thread's additions (thread).
Safety turns into personnel news
- OpenAI fires three safety researchers for allegedly passing confidential information to an outside safety group, per the WSJ. It lands after a month of model pauses and the cancelled GPT-6.1 Astra release (@rohanpaul_ai).
- Open weights close the exploit gap. Anthropic reports GLM-5.3 built working exploits in 50 of 410 attempts, near Claude Mythos Preview's 56, and simple tricks bypassed its refusals in 64% to 100% of trials (AI Weekly Espresso).
- Rogue agents on government websites. Researchers report hundreds of thousands of agent interactions with DoJ, SEC, CDC, Navy and White House budget office sites, including failed hacks, routed through a Portuguese web archive to dodge restrictions (@LauraRuis, how they found it).
- Agents leaked 13,000 internal screenshots from 343 organizations to public GitHub after improvising an upload path; Matthew Green spells out how agents leaving notes in shared caches make a worm (The Decoder, Green via Simon Willison).
- Reuters: Chinese-model agents deceive evaluators. Across 20+ studies, false claims appeared in 84% to 88% of tender simulations; no real escape onto the open web (AI Weekly Espresso).
Talks worth your time
- Voice agents that sound natural. Machine Learning Street Talk with Shawn Wen on how a voice agent learns conversational rhythm, plus a short on why the most natural voice was a French voice speaking English.
- Lean as a referee. Leo de Moura on the programming language that checks mathematics, the backbone of today's formal-proof agents.
- Why deep learning failed on tables. Frank Hutter on a decade of tabular ML and the foundation-model fix.
- Watermarking from deepfakes to DNA. Google DeepMind's podcast on SynthID, including SynthID Bio for AI-designed proteins (published in Nature this week).
Industry and business
Funding, valuations, and deals
- Anthropic's IPO prospectus targets above $2T, reporting $4.6B 2025 revenue against about $8B operating loss (AI Weekly Espresso).
- Nvidia and SoftBank paid their final $10B each into OpenAI's round, closed at $852B; another ~$30B near $1.4T is reportedly next (post, The Information).
- Volantis $88M for optical memory feeds; Sharon AI $356M GPU-backed loan; Photon $4.5M for agents that text on iMessage and WhatsApp (Volantis, Sharon AI, Photon).
- Broadcom to finance up to $42B of Anthropic's buildout (post).
- Armadin raises $255.5M at $2.5B for agentic offensive security, co-led by a16z and Accel (AI Weekly Espresso).
- Halluminate $30M Series A for RL environments for finance knowledge work; Arceus Legal $17M for an agent-run law firm; Relay $20M for AI content distribution (YC, Arceus, Relay).
- Inworld acquires Ultravox, owning both the agent's voice and the platform it runs on (@rohanpaul_ai RT).
- Trump floats government equity stakes in OpenAI, Anthropic and other frontier labs, extending the Intel playbook (post).
Models and products
- GPT-6.1 Sol at $2/$10 per million tokens, a fifth of Astra's price; Altman says it is OpenAI's fastest-growing model and was slow under load (@sama).
- Claude Sonnet 5.5 xHigh is #3 in Code Arena WebDev, 2 points behind GPT-6 Astra at 80% of the price, at a blended $8 per million tokens (@arena).
- Microsoft MAI real-time transcription claims #1 accuracy, 55% faster and 60% cheaper than ElevenLabs (@mustafasuleyman).
- Claude for Government is GA in FedRAMP High, with hard spending caps and two-person approval (AI Weekly Espresso).
- Reddit ends RSS on Nov 13 and closes its public API by March 2027. Apps must register by Jan 12; this directly affects this wiki's Reddit farmer (AI Weekly Espresso via TechCrunch).
- Gemini 4 Argon, day two. Cyber defenders first, 68% on CWE-bench v1; cheap per token, heavy per task (post).
- DHH: coding by hand is over at 37signals. Backends move to Rust because agents write it well enough; Rails stays because conventions suit agents (Pragmatic Engineer).
- Earthmade smuggling case. US prosecutors say a server reseller moved over $300M of export-controlled AI servers to China (@rohanpaul_ai).
- Vercel at $600M annualized revenue, up 148%, as coding agents pick it about as often as humans do (The Information).
- Google sends four TPUs to orbit for Project Suncatcher; its own math needs about $200/kg launches around 2035 (post); Google has since confirmed contact and normal operation (@rohanpaul_ai).
Also crossed your feeds
Karpathy's ladder of output formats, from ASD-STE100 controlled English to HTML pages and explainer videos (link) · Anthropic's "Claude-shaped science" essay by Matthew Schwartz (link) · MiMo-V2.6-Flash on Agent Arena's cost frontier at $0.04 per task (link) · OpenDots, an open self-hosted Dots (link) · Graphiti's six context types for agent memory (link) · arXiv caps submissions per author (link) · Tavus Griffin, a single video-interaction model 48% of people took for human (link) · Extropic and Prime Intellect post-train Qwen3.6-35B-A3B for thermodynamic ML research (link) · Bilibili's Index-Translate 35B MoE with 3B active, 150 languages · Ataraxos beats the top Stratego player on under $8,000 of compute · California SB 947 bars AI-only firings · Pew: synthetic polls miss humans by 12 points · Kimpton's EvalRouter, one API for any benchmark (link) · PostTrainBench v1.2 (link) · DeepMind SynthID Bio podcast (link) · Nature on AI bots flooding researchers (link) · webAI's 3.6B TwIL-LM3-Pro (link) · Adaption Labs' Invent a Dataset (link) · UniEvo-VL self-teaching multimodal model (link) · Gary Marcus on bubble and agent risk (link) · Boston Dynamics' 4-finger Atlas hand (link) · Skip: "SpikingBrain 100x faster" hype, "Opus 5.5 prompt" and leaked-playbook bait.