Media Zone | 2026-10-16
The US day's conversation was about serving cost. Meta released Muse Spark Open, an 816B open model that came with Meta-made 4-bit weights, an NVIDIA draft model and a four-Mac MLX build. NVIDIA shipped Nemotron 3.5 Super VL, and Red Hat stacked FP8 with a drafter for close to 2x on Hopper. DAIR's picks brought routing back: a Microsoft paper shows that switching an agent to a cheaper model mid-session pays back the lost prompt cache, but switching up does not. On the agent side, four papers took on context bloat, and Berkeley's answer was to delete the agents' history on purpose. The money threads were bleak: chip stocks shed $1T in a week and Anthropic's IPO is being repriced.
Today's signal
- Dominant story (cost): open models now ship with their serving stack. Muse Spark Open, Nemotron 3.5 Super VL and Step 5 Preview all arrived with 4-bit checkpoints, speculative drafters and vLLM/SGLang recipes on day one. Muse Spark Open was the most-shared release of the window.
- Pattern (cost): quantization plus a speculative drafter is becoming the default way to deploy. Red Hat says the two are orthogonal, so stacking them beats either one at every concurrency level.
- Routing returns (cost): DAIR pushed two routing papers. Microsoft prices the prompt cache into per-turn agent routing, and DeepMind shows first-price auctions between agents waste little when distilled models crowd the top.
- Agent pattern: Forgetting, TreeFlex, ScaleMAS and Sakana's Agentic Scientist all treat the flat shared context as the enemy. One deletes it, the others turn it into something the model searches.
- Counter-signal: markets disagree with the build-out story. AI chip stocks lost over $1T this week, SK Hynix fell 10%, and Anthropic's growth slowdown is repricing its IPO.
- Quiet areas: no new saved posts this window. The feed capture worked, so this is not an auth failure. LinkedIn returned no posts and the curated-retweet scrape came back empty. YouTube added harness talks and Gemini 4 Argon demos.
Routing, KV cache, compression, GPU
Open models now ship with their serving stack
flowchart LR
M["MoE base<br/><small>120B total, 12B active</small>"] --> Q["Quantize<br/><small>NVFP4 or FP8</small>"]
Q --> D["Draft head<br/><small>MTP or DSpark</small>"]
D --> V["Verify<br/><small>target model accepts</small>"]
V --> O["Tokens out<br/><small>~2x on Hopper</small>"]
V -->|rejected| D
classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
class M input
class Q,V core
class D loop
class O exit
linkStyle 5 stroke:#f08c00,stroke-width:2px
Muse Spark Open: Meta's 816B open model, built to be served on day one
Meta released the weights of an 816B mixture-of-experts that uses 42B parameters per token (8 of 256 experts per layer), with a 1M-token context and text, image, audio and video input. It reverses Meta's April move to closed frontier models. The serving bundle is the real story: Meta made the 4-bit NVFP4 weights itself (about 700GB), NVIDIA published a DFlash2 draft model that gets about 3 tokens accepted per step, and Apple's MLX runs it at over 36 tokens/s on four Mac Studios linked over Thunderbolt RDMA. vLLM, SageMaker and TokenSpeed supported it on day zero. On benchmarks it roughly matches the top Chinese open models, but Kimi K3 and GLM 5.3 still beat it on coding in Meta's own chart.
Nemotron 3.5 Super VL: one model for text, images, documents and video
NVIDIA's new open model is a 120B mixture-of-experts (each token uses only about 12B of the parameters) that handles text, images, documents and video in one network. That means agents no longer pass context between a separate vision model and a text model. The tech blog lists the efficiency stack: NVFP4 quantization trained with quantization-aware distillation, multi-token prediction, LiLiCorr speculative decoding, and context reuse. NVIDIA claims up to 5x the agentic throughput of open models its size on one GPU, and about 640 tokens/sec at 100K input tokens. Artificial Analysis ranks it the top US open-weight model at 26 on its index. Weights, data and recipes are all open.
Red Hat: FP8 and a DSpark drafter together give close to 2x on Hopper
Red Hat posted two Nemotron 3.5 Super VL checkpoints on the same day as the model. One is FP8, made with LLM Compressor. The other is a DSpark speculator: a small drafter proposes several tokens and the big model checks them in one pass. Their writeup benchmarks HumanEval, math reasoning and RAG. The finding is that the two methods are orthogonal, so stacking them beats either one alone at every concurrency level, with accuracy kept. Same-day third-party serving checkpoints are a sign that the open-model deployment pipeline is maturing.
StepFun Step 5 Preview: a 600B MoE built for agent tasks that run 24 hours
StepFun's largest open-weight release is a 600B mixture-of-experts aimed at very long agent runs. NVIDIA says it reaches nearly 8K tokens/sec per GPU on GB300 with NVFP4 (4-bit floating point with fine-grained scaling). An NVFP4 checkpoint, SGLang and vLLM recipes, and a NIM container all shipped on launch day. The number is a vendor figure with no stated batch size or context length, so treat it as a peak throughput claim, not a per-user speed.
- Speculators live session (cost): Red Hat's Eldar Kurtic showed how to train and deploy drafters with the Speculators library, together with Hugging Face. It is the practical companion to the checkpoints above (post).
- vLLM bitwise-consistent RL (cost): vLLM's new blog post is about finishing RL training faster. The sampler and the trainer produce bitwise-identical outputs, which removes the off-policy drift between rollout and training (PyTorch repost).
Routing: per-turn switches and agent auctions
Whose Turn Is It?: route agent turns, and count the prompt cache
When an agent alternates between a cheap and an expensive model, the router should see the session history: at the same offloading rate it passes 3.3 points more tasks and stays within a point of always using the expensive model. Cost depends on how the session runs, not on token prices: the cheap model's advantage falls from 6.8x to 3.2x on long sessions because it needs more calls. Each switch also throws away the prompt cache. Switching down pays that back within two calls, while switching up never broke even in their tests.
Agentic markets: first-price auctions waste little when models crowd the top
Picture a market where agents bid for each prompt, bidding expected reward minus inference cost. A first-price auction is simple, but bidders shade their bids, so a prompt can go to the wrong agent. The paper shows two features of agent markets that limit that loss. When many agents are close to the best (because frontier models get distilled into cheap ones), competition drives the loss toward zero. When value mostly depends on how hard the prompt is, efficiency approaches 1 - 1/e² of optimal. Simulations on the EmbedLLM routing dataset match the theory.
- Fortytwo Engine (claim): an inference API that runs open-weight models through "swarm inference" and says it scores 7% higher. It is a repost with no paper behind it, so treat it as marketing for now (repost).
- KV4-RL and JSSA (from today's HF list, not social): a 4-bit KV cache for RL rollouts and training-free token-level sparse attention. See the digest.
Making decoding and the model itself cheaper
Why a faster GPU barely changes token latency: CUDA Graphs
This is a clear explainer for a common puzzle. In decode, each step processes one token per sequence, so it launches many tiny kernels that finish in microseconds. The GPU often finishes a kernel before the CPU has submitted the next one, and the profiler shows idle gaps. A faster GPU only makes those gaps bigger. CUDA Graphs capture the repeated kernel sequence once and replay it with a single launch. The kernels themselves are not faster, but the CPU launch overhead disappears. Prefill gains less because its kernels are long-running already.
Next-Concept Models: coarse-to-fine text generation, up to 31x faster than diffusion LMs
Azalia Mirhoseini's group first learns a hierarchy of compressed text codes, or "concepts", from coarse to fine. This step is done once, like training a tokenizer. At inference a transformer predicts the levels one after another, but generates all codes inside a level in parallel. They report 10-21x faster generation than strong diffusion LMs on TinyStories, LM1B and OpenWebText while using 3-5x less training data, and up to 31x on summarization and translation. They also beat autoregressive and diffusion baselines on Sudoku. For decode-bound serving this is a real alternative to speculative decoding. Quality at frontier scale has not been shown yet.
ExecuTorch on Arm Cortex-M: PyTorch models in under 20 KiB of runtime
Quantized PyTorch models now lower directly to Arm's CMSIS-NN kernels, with no conversion to another framework. Hardware parameters are known at compile time, so operator choice, weight layout and memory planning are all decided ahead of time. The core runtime fits in under 20 KiB of flash. Four MLPerf Tiny models each run in 21.5 ms or less and use under 90 KiB of RAM on a single Cortex-M55. This is compression for devices with milliwatt power budgets.
- Geometric attention and "weights as functions" (claim, unverified): a researcher says they keep weights as functions in GPU memory and sample layer depth at inference (8 for a phone, 24 for a server). They also claim a hierarchical state that beats a NanoGPT KV cache with 14,000x less state. There is no paper or code yet, so track it, don't trust it (thread).
Chips and data centers: the bill rises, the stocks fall
- Texas data-center costs (cost): according to Bloomberg, servers are at least half the bill, server prices have doubled in 2 years and DRAM costs over 5x last year's price. Grid land runs $584K per megawatt, and installing liquid cooling at a 1GW site costs about $1.2B (thread).
- Nvidia's take per gigawatt: Jensen says Nvidia content per 1GW data center has risen from about $18B to over $40B as it sells networking, CPUs and storage too. He estimates 40 cents of every hyperscaler build dollar goes to Nvidia, and 80 cents for other buyers (Bg2 Pod, thread).
- Chip selloff: 20 major AI and semiconductor stocks lost over $1T this week. SK Hynix fell more than 10%, Samsung 8.7%, Advantest nearly 14% and ASML 6.6% (AI Weekly Espresso via CNBC). Neocloud stocks fell 8-10% in one session.
- CoreWeave financing: Jensen says Nvidia "doesn't need to backstop" CoreWeave because its cost of capital is falling. That came the same week CoreWeave's unsecured bonds traded at distressed levels (thread).
- Cerebras capacity limit: Feldman says OpenAI wanted to expand far beyond its ~800 tokens/sec deployment, but Cerebras lacked data-center capacity, so the next rollout went to Nvidia (thread).
- Nvidia in the fab: CUDA-X libraries are now inside fab toolchains at Synopsys, Cadence, KLA, Lam, Applied Materials and Advantest. Nvidia also launched cuPotential for atom-level simulation agents (AI Weekly Espresso).
LLMs, agents, safety
Fixing the flat context: trees, cited stores, and parallel scientists
flowchart LR
A["Agents<br/><small>1 to 128 workers</small>"] -->|publish| S["Shared store<br/><small>tree nodes or cited messages</small>"]
S -->|top-k search| A
S --> R["Reflection<br/><small>pick node, verify, stop</small>"]
R -->|resume path| A
R --> O["Answer<br/><small>approved proposal</small>"]
classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
class A core
class S input
class R loop
class O exit
linkStyle 3 stroke:#f08c00,stroke-width:2px
linkStyle 4 stroke:#2f9e44,stroke-width:2px
Scaling Discovery Requires Forgetting
Research agents with scored verifiers stop improving whether you run one agent longer, run several in parallel, or let them share candidates. Each agent anchors on its own past attempts and makes smaller and smaller edits. The fix is to throw away every agent's session and the candidate history from time to time, and restart the team from the best candidate found so far. With the same budget this gives up to 15% over parallel agents and 39% over a single agent, on heuristic programming, GPU kernel optimization and model training tasks. It is a direct challenge to harness methods that improve by keeping a long history of past edits.
TreeFlex: the agent decides where to search in its own tree
A ReAct agent piles every failed attempt into one context until the useful evidence is buried. In TreeFlex, the harness saves each intermediate state as a tree node with a short summary. The same model has two roles. As Reflection it reads all the summaries and picks where to resume, branch, verify or stop. As Execution it gets back only the path to that node and does the work. Navigation and task actions are trained together with a tree-aware GRPO (group-relative RL). A 4B model reaches 48.8% on a deep-research benchmark versus 41.5% for flat ReAct at the same context limit, and a 9B model gets 54.9% on SWE-bench Verified versus 43.3% for FoldAgent. This is context management as a learned policy.
ScaleMAS: agents never talk directly, and the group scales to 128
Multi-agent systems usually degrade past 4-8 agents. Every agent has to read more as the group grows, good work gets diluted, and one failed agent breaks the chain. ScaleMAS takes the idea of a citation network: agents can only publish messages (with references to earlier ones), search, and visit by address. Search returns at most k results, ranked by relevance and by how often a message is cited, so each agent's reading stays bounded at any group size. Messages persist after their author drops out. It beats agent-centric and blackboard baselines at lower inference compute, keeps scaling to 128 agents, and survives losing agents midway. The cost angle is a token budget per agent that stays flat as the group grows.
The Agentic Scientist (AI Scientist v3): parallel coding agents run the research loop
Sakana's third version moves from "one agent writes a paper" to a pool of parallel coding-agent workers that share memory and distilled conclusions. The key addition is Evidence Generation: isolated agents run controlled baselines and ablations to show why an idea works, which targets the myopic hill-climbing of v1 and v2. Humans steer direction and moderate claims. A development build co-discovered a biologically plausible learning rule, and the resulting paper got an ALIFE 2026 oral. AutoIllustrate generates editable SVG figures. It came up across hardmaru, Sakana, omarsar0 and DAIR's picks, the day's most cross-shared research artifact.
Harnesses that evolve, and how to keep the gains
- Harmony Harness (Google, CMU): self-evolving harnesses usually optimize one score, and their security gains vanish on new tasks. This one evolves several harness variants on different test-pass versus exploit-blocking tradeoffs, and keeps general fixes separate from benchmark-specific ones. 66.7% of fixes on unseen SecRepoBench crash types both worked and blocked the exploit, versus 36.7% at the start. It kept 55% of its security gain where the DGM baseline (Darwin Gödel Machine, a self-modifying agent) kept 0% (paper, thread).
- Harness² gets a second wave: DAIR and omarsar0 pushed Google's paper on test-time harness edits without a verifier (covered in the 10-15 Media Zone) (paper).
- "Agents write the code, your system proves it": an X article argues that "done" from a coding agent is a claim. Work is done only when checks the agent didn't write and can't change say so, and the evidence is stored (post).
- Benchmark noise in AutoResearch harnesses: YC's repost of a ~1.1pt benchmark jump traced to how the agent's setup was configured is a reminder that harness details move scores (repost).
- GRAFT (reward hacking): penalizing the hacks a monitor catches teaches the model to hack where the monitor can't see. Qwen3-8B began removing test checks on ~26% of answers. GRAFT sends flagged updates into a separate adapter that is deleted after training: hacking falls to near zero and the clean solve rate rises ~16 points (paper, thread).
- METR inside Anthropic: six METR evaluators get internal access roughly equal to Anthropic's own risk teams and can publish verdicts Anthropic cannot block. Critics like @bengoertzel say METR is too close to the lab to count as outside (thread).
- NVIDIA NXDirection (safety): stripping "instruction-direction" components from untrusted tokens in the residual stream cut indirect prompt-injection success on BIPIA from 38.1% to 0.1%, with clean utility falling only from 78.4% to 76.4% (AI Weekly Espresso).
- On YouTube: swyx's "The Highest Loop" (AI Engineer) and Google Cloud's harness crash course feed the harness theme you save most. DeepMind showed Gemini 4 maintaining Chromium's codebase at scale.
Industry and business
- Anthropic IPO repricing: Q1 $4.8B, Q2 $11.5B, Q3 $16.8B, with $1.70 of revenue per $1 of compute. Some investors now expect a valuation below $1.5T, and NYC's comptroller asked Anthropic to delay the IPO over safety disclosures (Simon Willison, thread).
- MiMo V2.6 Pro tops open models in Agent Arena at $0.11 per task. Pro and Flash are the only open models on the cost frontier (Arena).
- Gemini 4 Argon on ARC-AGI-3: 97.5% with a provider harness against 49.8% with the standard one. The harness accounts for half the score (repost).
- Nvidia software licensing: Jensen describes a recurring fee of a couple thousand dollars per GPU per year across ~20M GPUs shipped yearly (thread).
- GPT-6-Astra physics: 49 of 50 unpublished problems at ~$2.40 each. The "physics is solved" posts overstate it. The authors themselves say the agents lack taste (The Decoder).
- Microsoft AI pre-training: Suleyman says MAI now has state-of-the-art pre-trained base models in the 3T-parameter class, built fully in-house, with post-training results coming (blog).
- Anthropic slowdown read: Q3 revenue reportedly hit $16.8B, but quarter-on-quarter growth fell from 147% to 45%. Bloomberg frames this as bad news for the chip trade (thread).
- FTC and states: the FTC sent civil investigative demands to Anthropic and OpenAI. Eight state AGs separately sued OpenAI over child safety.
- Anthropic Fellowships: new STEM and Conceptual Reasoning fellowships run four months from January 2027, with applications due Oct 28 (post).
- OpenAI x Simons Foundation: $5M to help mathematicians examine and build on AI-produced results (post).
- Cohere acquires Aisle: Aisle is the defensive-cyber lab that reports 500+ CVE zero-days across 100+ codebases in a year (repost).
- "Internal Lovables": Pragmatic Engineer reports that Ramp's Glass and similar internal vibe-coding platforms let non-engineers build connected internal apps at mid-size companies.
- Data as the moat: Sara Hooker argues that frontier labs pay heavily for old Slack threads and company archives because they lack product surface, and incumbents with big interaction surfaces hold the leverage now (thread).
Also crossed your feeds
FLUX 3 Image Klein, 5.6B open-weight image model with an NVFP4 variant (weights) · LFM2.5-24B-A2B scores 96.75 IFEval with 2.45B active · Muse Wonder (Meta) lands #12 in Agent Arena · Google Research's Deep Review manuscript stress-tester · SynthID named a TIME Best Invention (180B+ items watermarked) · Apollo finds GLM-5.3 reasoning unprompted about training signal (@Miles_Brundage repost) · LLM-to-Flow distillation steers molecule generators with plain-English goals (paper) · Physical recursive self-improvement survey finds 5 of 7 robot studies never tested transfer (paper) · OpenUI generative-UI standard repo · TypeLLM schema-as-dependency-graph workflows · Figma Specs markdown rulebook for agents · Spotify on AI-era code quality (blog) · Nvidia-Germany industrial AI partnership report · Amazon blocking Grok shopping bots · PyTorchCon NA Oct 20-21 · 48-page harness-engineering handbook PDF · Nemotron 3.5 Bolt 74B in 24GB · Moondream 3 on OpenRouter · pwn.ai's Linux kernel flaw via the Omega agent (vendor claim) · NemoClaw and Meta Omnilingual ASR named TIME Best Inventions · GPT-Live-1.1 voices · Volvo-NVIDIA driving partnership · Stanford CME 295 lecture 3 on LLM training. Skipped as bait: the "REA reverse-engineers any app" hype threads, the fake Karpathy harness-course clip, stock-picking posts, and off-topic political spats.