media-zone · 2026-11-16

Media Zone | 2026-11-16

Media Zone | 2026-11-16

The US Sunday conversation is about paying the expensive model only for decisions. Claude Code's advisor mode keeps Opus or Fable on call for three moments in a task and lets cheaper models do everything else. Perplexity's Decider skips text generation entirely and emits a routing decision in one forward pass. By US evening the same idea showed up as harness discipline: practitioners argued for boring deterministic steps over more agents, and a DeepMind paper put 10 GB chip-simulation logs in a database instead of a prompt. The serving stack got a 10-week public roadmap and a Rust API rewrite with ~20% faster first token. Bernstein's HBM4 math says memory makers earn about 65% gross margins. The Nvidia story ended the day on a claimed Microsoft GPU-order cancellation, two days before earnings.

Today's signal

  • Dominant story: "advisor" routing inside Claude Code. A cheap main model works, a strong model is consulted only before a plan locks, after a repeated error, and before "done."
  • Pattern: decision-only compute. Advisor mode, Perplexity's single-pass Decider and Kenneth Stanley's "shadow agent" proposal all price oversight explicitly: who gets the expensive tokens, and when.
  • Serving stack, US afternoon: a public roadmap teaches vLLM, SGLang, KV eviction and disaggregated prefill end to end. Anthropic's Rust API rewrite cut time to first token about 20%.
  • Hardware: HBM4 costs about $6-7/GB to make and sells near $20/GB. Long-term compute leases run half the price of short-term capacity.
  • Counter-signal: the "repeat your prompt twice, it's free" thread is wrong on cost. Repetition doubles prefill tokens and KV cache. Output tokens are unchanged, input bills are not.
  • Grift alert: "Anthropic just released the best trick" threads inflate advisor mode with a made-up "3.4x cheaper" figure, and a "paste this prompt to install an uncensored model" thread is the evening's top-scored local-AI post. Skip both.
  • Quiet area: no new saved posts or curated reposts all day (bookmark runs report zero new saves, not an auth failure). YouTube had no new AI videos, LinkedIn returned no posts and Reddit had nothing past its filters. Everything below is seven X Following-feed captures plus the window's newsletters and blogs.

Routing, KV cache, compression, GPU

Pay the big model only for decisions

The strong model is a consultant, not the worker
Advisor mode puts a cheaper model in charge and calls the expensive one at three checkpoints.
flowchart LR
  T["Task<br/><small>user request</small>"] --> M["Main model<br/><small>Sonnet, writes code</small>"]
  M --> H["Subagents<br/><small>Haiku, search and read</small>"]
  H --> M
  M -->|checkpoint| A["Advisor<br/><small>Opus or Fable, full transcript</small>"]
  A -->|guidance| M
  M --> D["Done<br/><small>after final review</small>"]
  classDef input fill:#d0ebff,stroke:#1971c2,color:#1b1b1b,stroke-width:2px
  classDef core fill:#e5dbff,stroke:#6741d9,color:#1b1b1b,stroke-width:2px
  classDef loop fill:#fff3bf,stroke:#f08c00,color:#1b1b1b,stroke-width:2px
  classDef exit fill:#d3f9d8,stroke:#2f9e44,color:#1b1b1b,stroke-width:2px
  class T input
  class M,H core
  class A loop
  class D exit
  linkStyle 3 stroke:#f08c00,stroke-width:2px
  linkStyle 4 stroke:#f08c00,stroke-width:2px
  linkStyle 5 stroke:#2f9e44,stroke-width:2px
Blue is input, purple is the cheap models doing routine work, amber is the expensive checkpoint, green is the result.
Docs · in-harness routing

Claude Code's advisor tool

The advisor pairs your main model with a stronger one that Claude consults at key moments: before committing to an approach, when an error keeps recurring, and before declaring a task complete. The advisor reads the whole conversation, including every tool call, and returns guidance the main model applies. It runs server-side, is experimental, and you set it with /advisor opus, an advisorModel setting or the --advisor flag. The docs say it pays off on long tasks where most turns are routine but plan quality decides the outcome, and adds little on short tasks. This is model routing (sending each request to the cheapest model that can handle it) moved inside one agent session, with the routing decision made by the agent itself. Two US-morning threads spread it widely. One adds a "Fable thinks, Opus reviews, Haiku swarm" stack and a "3.4x cheaper" number that the docs do not contain.

Guide · single-pass routing

Perplexity's Decider: route without generating text

Ken Huang's guide argues that using a generative model to make a yes/no routing or security decision is wasteful. The model writes 30 to 100 tokens to express one bit, its formatting drifts, and its confidence is uncalibrated. Perplexity's open-weight pplx-decider models (Apache 2.0) instead read the input with bidirectional attention and emit calibrated decision tokens from a small readout head in one forward pass, with zero generated text. The 4B Flash variant reportedly runs air-gapped on Apple Silicon at about 835 ms. Custom gates are tuned by freezing the backbone and training only the 2.6 MB readout layer with a Brier-score loss (a loss that rewards well-calibrated probabilities). Several of the guide's numbers come from secondary write-ups, so treat them as claims to check.

Paper claim · prompt repetition

Say the prompt twice: real gain, wrong cost story

A viral thread revives a Google result: causal models read left to right, so a document pasted before the question is encoded without knowing what to look for. Repeating the full prompt lets every token of the second copy attend to the question. The thread reports gains across seven models and benchmarks, including 21% to 97% on one extraction task. The thread says it "costs exactly $0." It does not. Output tokens stay the same, but input tokens, prefill compute and KV cache (the memory of past attention keys and values) all double. That is cheap for short prompts and expensive for long-document workloads, the very case where the trick helps most.

The serving stack, taught and rewritten

Repo · inference-serving curriculum

time-to-first-token: a 10-week LLM serving roadmap

Akshay Pachaar's open roadmap (50 lessons, 30 minutes a day) builds one serving stack end to end instead of scattering across demos. It starts with the roofline model, the reason decode is bound by memory bandwidth while prefill is bound by compute. Then it reads vLLM's PagedAttention and scheduler from the code, benchmarks SGLang's RadixAttention prefix reuse against it, and load-tests past 1,000 concurrent requests with TTFT and inter-token latency on Grafana. Later weeks cover FP16 vs FP8 vs INT4 on quality as well as speed, speculative decoding and KV eviction "including where the gains disappear," disaggregated prefill and decode on Kubernetes, and a cost/latency/quality router with per-request token budgets. It is a curriculum, not new research, but it is the most complete public map of exactly the stack the reader works on, and the evening's highest-scored efficiency post.

Talk clip · production latency

Anthropic rewrote the Messages API in Rust with agents

The Claude Platform's head of product says the team, with no Rust experts, moved the Messages API from Python to Rust in about a month with two or three engineers, agents writing most of the code. Time to first token dropped roughly 20%. The humans planned the migration and built the checks, chiefly shadow traffic that compared old and new responses before the switch. Her point is that much of the job is now verifying what the agent did. For serving, it is a reminder that a fifth of TTFT can live in the API layer, not the model or the kernels. The poster appends a self-promo thread, so take the summary from the clip.

  • Self-improving harnesses, now as social hype. Shanghai AI Lab's Self-Harness has the agent cluster its own failures, propose edits to its tools and prompts, and keep an edit only if nothing regresses. Qwen3.5 reportedly goes from 18% to 36.7% on Terminal-Bench 2.0 with the same weights (@HoodieFade_).
  • A "20x cheaper, more trustworthy" deterministic harness was posted twice by Carlos Perez with no link. No artifact yet, so no verdict (@IntuitMachine).
  • "Lots of boring deterministic steps." Elvis Saravia says the best agent systems he has seen are mostly classifiers, verifiers and evaluators around a few model calls, because more agents compound errors. The token angle is direct: every deterministic check is a model call you do not pay for (@omarsar0, follow-up).
  • Fix the mistake class, not the prompt. Lauren Tan (3,000+ agent PRs a month) runs every agent mistake through four questions: redesign so it is impossible, catch it with a lint or type, let the agent self-check, and only then write a rule. AGENTS.md bloat is the default she avoids (@0x_kaize).
  • The research twin. SkillCache (NVIDIA, on DAIR's list this week) compiles one finished skill run into a replayable capsule and cuts the next run's tokens 74-81%. It is the paper version of "make the repeated part deterministic" (digest Deep Dive).

Memory and compute economics

  • HBM4 margins. Bernstein estimates HBM4 (the stacked high-bandwidth memory beside each GPU) costs about $6-7/GB to make and sells near $20, a roughly 65% gross margin. Nearly half a 32GB stack's cost is base die, packaging and test, which favors TSMC and Teradyne alongside Micron (@StockSavvyShay).
  • Compute is priced by contract length. Five-year deals at IREN and CoreWeave run about $20-25 per watt-year, against $40-50 for short-term capacity. Nebius's 2026 base sits near $12, so its realized price rises as old contracts roll (@StockSavvyShay).
  • Demand vs. price. The same account says AI token volume is up about 25x since January, mostly from agents, while AI stocks sell off. It is a bull argument with picks attached (DigitalOcean, SiTime, Astera Labs), so treat the 25x as unsourced (@StockSavvyShay).
  • Capex shifts to neoclouds. Raymond James sees cloud capex near $1.5T in 2027, with neoclouds the largest bucket at about $400B and Google passing Amazon among hyperscalers (@StockSavvyShay).
  • Samsung adds $1.9B to its Austin fab to make Starlink chips for SpaceX (@SawyerMerritt via @elonmusk).
  • Chip verification moves to SQL. A Google DeepMind and Columbia paper converts 10 GB+ simulation waveform dumps into SQLite so agents can query them, answering 95.33% of 150 expert questions. Cost angle: the data never enters the context window (@rohanpaul_ai, paper).
  • LongCat-2.5 never touched Nvidia. Meituan's 1.6T open agent model was trained on Chinese hardware and demoed on 194 Huawei Ascend chips (via The Decoder).
  • Fab process watch. The Semiconductor Newsletter's weekly monitor covers AVS 72 etch results (cryogenic carbon etch with a 7.2 vertical-to-lateral ratio at -90°C, aluminum atomic-layer etch at 0.08 nm/cycle) and early EUV-resist metrology. All of it is pre-production (The Semiconductor Newsletter).

Nvidia after Huang: from succession to scandal

  • The WSJ story now alleges Huang's partner arranged about $12B of top-end chips to China, and that the affair threatens the ecosystem he built. The claims are reported, not established (@DAlperovitch via @GaryMarcus, @carlquintanilla).
  • Investor pressure. Atreides Management sent a letter to Nvidia's board, its first ever, and posts now question the board's qualifications (@GavinSBaker via @garrytan).
  • Supply-chain angle. Musk says Huang always came through on compute deliveries. Others argue his exit lets China hawks tighten export rules. Future & AI notes an Nvidia intervention had halted a Megaspeed chip inquiry (@elonmusk, @dee_bosa via @garrytan).
  • Succession chatter. US-evening posts float Google Cloud's Thomas Kurian as a possible successor. It is speculation with no reporting behind it (@RajeshChanstgf).
  • Late US evening: a claimed Microsoft GPU cancellation. Ed Zitron posts what he says is an Amy Hood email cancelling some GPU orders, and argues Huang feared overbuying TSMC and memory capacity in August. Unconfirmed, and it lands two days before earnings (@edzitron, capacity post).
  • Who wrote the narrative. Varun Mathur ran agents over about 1,660 stories from 40 outlets and applied PageRank to their citations: the NYT is the most-cited origin, Bloomberg and the WSJ drive volume (@varun_mathur, site).
  • Skip: the "this is why they took down Jensen" Nemotron conspiracy thread is the day's most-viewed Nvidia post and has no evidence (@adamemedia).

LLMs, agents, safety

Reading models from the inside

Thread · weight forensics

A model's training history is written in its weights

Alapan Chaudhuri shows you can recover much of how a model was trained just by inspecting its weights: which tokens appeared in the data, the learning-rate schedule and the number of steps. With the data in hand you can go further, to the step each document was seen, the shuffle seed and the number of passes. The examples are checkable. OLMo 2's first instruct release had 20,341 embedding rows that never moved, including the token that closes every chat turn, a tokenizer bug AI2 later fixed. Pythia-1B trained at a peak learning rate of 2.5e-4, not the 3.0e-4 on its model card. Qwen2.5's instruct tuning touched exactly 4 of 22 control tokens at every size. This is a cheap audit tool for open weights: model cards can be wrong, and the weights cannot lie about what was trained.

Essay · architecture building block

Raschka: Understanding Positional Embeddings in Modern LLMs

Sebastian Raschka's long tutorial explains how a transformer learns word order. Attention alone cannot tell "the red fox jumps over the gray fox" from the reverse. The causal mask gives implicit order but cannot tell a gap of one from a gap of ten. GPT-2 adds a learned position vector to each token embedding before the first block. Modern models like MiniMax-M3 instead apply RoPE (rotary embeddings, which rotate query and key vectors by an angle set by position) inside every attention layer. The piece builds RoPE from scratch, then surveys what recent open models do: NoPE (no positional encoding) layers, partial RoPE and hybrids. It matters for long context, because the position scheme sets how well a model extrapolates past its training length and how its KV cache can be compressed.

Agents that keep working without you

  • Always-on agents, day two. Anthropic's guide (carded yesterday) is now spreading in Chinese-language tech circles, read alongside DeepSeek's event router and LangChain's async subagents. Readers single out the deferred tool call: save the session, free the worker, resume when the human approves (@Xudong07452910, guide).
  • Agents that provision their own database. MongoDB's Atlas Ephemeral Clusters let an agent get a live cluster from one API call. Unclaimed clusters pause after 48h and delete after 7 days. The poster says they worked with MongoDB on it, so read it as a sponsored post (@_avichawla).
  • A phone-sized book on agentic engineering. Gappy Paleologo had Astra and Fable co-write a short book in the style of Fleuret's "Little Book of Deep Learning," and shared it free. It is the window's highest reach-normalized post (@__paleologo).
  • Shadow agents, priced. Kenneth Stanley proposes pairing every deployed agent with a watcher that sees its full context and reasoning, and whose only tool is filing a report. It roughly doubles run cost, and he frames that as the honest test of how much we will pay for oversight (@kenneth0stanley).
  • Goal engineering. Harrison Chase and Actively AI argue that as agents take on open-ended goals, the goal spec becomes the main design surface, beyond prompts or tools (@hwchase17).
  • Agent sandboxes. Josh Rosen's survey asks whether every agent needs its own computer or whether most work fits a much lighter execution environment, a direct cost question for agent fleets (@JoshARosen).
  • Model-retirement migrations. Your provider retires your model in 30 days and 200 prompts are tuned to it: how do you migrate with no regressions? It is a useful framing for routing-eval discipline (@nikks_techie).

Research and safety notes

  • DAIR's Top AI Papers of the Week (Nov 9-15) landed in the US evening: s3 (multi-agent systems that organize themselves instead of following a designer-written orchestrator), WikiTrace, AutoRefine, LongCapHarness, RSI for pre-training, Self-Organized Memory and Simple Scaling of Agent Swarms. The farmed DAIR list will cross-check these against HF and Kurate at digest time (@dair_ai, issue).
  • dspy-wsp is a prototype DSPy optimizer for "weight-space programming": a frontier model edits a small model's weights directly, with no gradients. Prototype only, no numbers in the capture (@dbreunig via @stanfordnlp).
  • Stanford CS329A, Self-Improving AI Agents, is being passed around as the go-to lecture series on the topic, alongside the "Engineering AI Agents" course's tool-calling lecture (@omarsar0, @sahil0x01 via @stanfordnlp).
  • Compositional reasoning under RL (NeurIPS 2026) asks whether RL post-training teaches models separate skills they can then combine on harder problems. Stanford NLP amplified it in the evening (@yuhe441).
  • "Spiritual behavior" across 141 models. A LessWrong study tests whether Claude's reported spiritual-attractor behavior is Claude-specific or general to LLMs (@MoralAIProject).
  • EmbeddingGemma 2 held the top trending spot on HuggingFace for over a week, rare for a non-LLM. A thread explains how it was trained (@gabriberton).
  • "LLMs develop coherent values" is the 2025 utility-engineering result recirculating: preference cycles drop below 1% at scale, and GPT-4o makes skewed trades between lives. It is old work with new packaging (@thesupermannx).
  • Marcus vs. doomers. He argues panic posts lump a raw LLM together with a tool-using system like Claude Code, and so muddle policy. He is fighting with critics on both sides today (@GaryMarcus).

Industry and business

  • Claude Fable 5.2 and Mythos 5.2 shipped as one model in two forms. They are up to 30% faster than 5.1, priced at $8/$40 per million tokens, and about 1,000 agents formalized Fermat's Last Theorem in 30 hours (via the Future & AI newsletter).
  • Anthropic launched a roadshow for its proposed IPO. Deedy Das claims 85% gross margins and calls its lead over OpenAI "unassailable." Treat that as opinion (via the Future & AI newsletter, @deedydas). Gary Marcus warns retail buyers to read the lockup terms and the profit-to-liabilities ratio first (@GaryMarcus).
  • Fireworks AI confirmed customer data was accessed, a serving-provider breach worth watching for anyone routing traffic through third-party inference (via the Future & AI newsletter).
  • Meta released Muse Ember, its smallest model. OpenAI acquired trip planner Odessia. Adobe and Disney signed a multi-year AI alliance (via the Future & AI newsletter).
  • AlphaGenome 2 predicts how DNA changes affect specific cell types. It finds 30% of causal variants across 28 immune cell types, up from 16% (via the Future & AI newsletter).
  • Embla 1 from Sevren, a 10B model, runs offline on an iPhone 18 Pro with open weights on HuggingFace. The 57.4 benchmark average is self-reported (@ahsuzm).
  • Agent Arena: Claude Fable 5.2 (Max) took #1 with a +30.4% net lead. Qwen4-Max debuts at #14, making Alibaba the top Chinese lab and #5 overall (@arena).
  • Google Antigravity doubled the context window for its Argon, Carbon and Hydrogen models (@_mohansolo via @demishassabis).
  • OpenAI launched a science blog, opening with physicist Nathaniel Craig using GPT-6 Astra on a lattice definition of the Standard Model (@OpenAI).
  • Bland's Tofu-1 is a speech-to-speech model aimed at noisy, high-stakes calls (@zaygranet via @garrytan). AgentRun passed 1,000 GitHub stars in its launch week (@_aj via @garrytan).

Also crossed your feeds

LLM inference platform architecture diagram (@techyoutbe) · Python's interpreter eats about 20MB of a ulimit memory cap before your code runs (@ReignSwords) · "a weak model can still train a strong one" decision-model paper (@rohanpaul_ai) · ex-Anthropic researcher Jacob Coxon may return to an AI lab (@rohanpaul_ai) · list of every private AI startup above $10B valuation and $100M run-rate (@deedydas) · Google's PaperOrchestra multi-agent paper writer, second day of spread (@DataChaz) · "use a classic ML model before you reach for an agent" (@UPing123zzz) · a YOLO home camera built with Fable (@zodchiii) · Deedy's Fable 5.2 showcase list (@deedydas) · Academic Humanizer repo (@divaagurlxw) · Deep Personalization, continuous-context local proactive agents (@Memoirs via @stanfordnlp) · nine-step RAG architecture diagrams, and a reply arguing the hard part is the dependencies between steps (@GauriTripa94282, @UPing123zzz) · Heidy Khlaaf on safety-critical legacy systems (@timnitGebru) · a Medium list of 10 open-source AI tools for 2027 (newsletter) · Sinkhorn balancing explainer, the 1963 routine behind optimal transport and balanced MoE assignment (@docmilanfar) · Zep Agent Skills, procedural memory (@TheTuringPost) · Marcus, Chollet and Szegedy on model versus system (@GaryMarcus, @fchollet) · Starship Flight 15's first tower catch dominated the non-AI feed (@elonmusk). Skipped: the "install an uncensored model" prompt thread, Explee "$2m of GPUs" ads, "Claude Council" and "send this to Fable before bed" engagement bait.