Efficiency shorts 2026-11-17: the road to Together Compiler, Mobile AI Factory, output format as a cost knob, distilled-generator diversity
Sources: Last year we built megakernels. This year we deleted them (Together AI, 2026-11-16; paper arXiv:2611.14681 is Kurate cs.AI #7) · Mobile AI Factory, arXiv:2611.13509 · Parsed, Plausible, Wrong, arXiv:2611.17169 · eDMD, arXiv:2611.17194 · ReDiverse, arXiv:2611.18550 · raw: raw/labs/2026-11-17-081848-labs.md, raw/huggingface/2026-11-16-*.md
The road to Together Compiler (cross-source: Kurate + lab blog)
Together's retrospective on why it stopped hand-building megakernels (a whole model forward pass fused into one GPU kernel). After months of effort the megakernel served "one model, one batch size, and one customer." Three findings: (1) on Llama 1B, CUDA Graphs plus PDL (programmatic dependent launch, which overlaps one kernel's start with the previous one's tail) on top of default Torch Inductor kernels recover 98% of megakernel speed without writing a kernel; (2) Inductor's choices are unreliable: its freezing flag gives +10-26% throughput at large batch but a 49% slowdown at batch 1 (FP32 weight conversion was not undone for matrix-vector ops), and its 607 tuning flags (198 added in a year) interact unpredictably; (3) its benchmark harness misleads: it clears L2 cache before timing, so isolated kernels look 39-58% slower than in the live forward pass, zero-initialized token IDs make embedding lookups 2.1x faster than real ones, and timing overhead of about 2.5-3.6 µs swamps few-microsecond decode kernels. Agent loops searching "multi-kernels" now beat the old megakernels by up to 1.7x on H100 (1,100+ TPS) and by 28% on B200 (2,118 vs 1,649 TPS on Llama 3.2 1B). The 11-05 summary covered the production system; this post is the evidence for why the search, not the kernel, is the product.
Mobile AI Factory (2611.13509)
An agentic system that builds a fine-tuned, quantized small model for a given task and a given Android phone. An orchestrator LLM chooses between tuning hyperparameters, mining real data and generating synthetic data; fixed rules roll back changes that hurt accuracy. It searches small to large across base models and quantizations and measures the actual phone across inference engines and backends. On ten tasks and four phones, its on-device models beat two five-shot frontier models on five tasks (by up to 0.34) and match them on three. On entity extraction it comes within 0.03 of exhaustively training all 27 variants using 19x less GPU time. The best configuration differs by phone, and Claude Opus 5.5 given full hardware specs but no measurements misses the app's requirements in 25% of easy-to-moderate and 60% of hard cases. Same lesson as CompressionBench (11-16): agents can compress models, but only with a faithful measurement signal.
Parsed, Plausible, Wrong (2611.17169)
Output format is a free design choice with a cost and an accuracy effect. Fine-tuning sub-1B parsers (Qwen3.5-0.8B, LFM2.5-350M) on six lossless encodings of the same order tree (44.3 down to 10.9 tokens per order), with statistics fixed before results were read: shortening alone never helps, but writing the target in ordinary words does. The word encoding is the most accurate (87.5% vs 85.8% exact match for the native s-expression on Qwen), emits 4.1x fewer tokens, and cuts silent errors (well-formed but wrong trees) 2.3x on Qwen and 1.8x on LFM2. Strict refusal of off-menu input costs 2.1-10.1% of real, mostly polite, orders.
Distilled-generator diversity (eDMD, ReDiverse)
Two image/video distillation papers. eDMD trains the fake-score model only on the student's final outputs, which provably makes the matching term vanish if and only if the final output distribution matches the teacher, and beats prior distribution-matching methods at 4 steps on SD3-Medium, FLUX.2-klein and Qwen-Image-2.1. ReDiverse finds that few-step students keep the teacher's total activation variance but squeeze it into fewer directions at one intermediate block on the first denoising step; a small adapter (or a training-free steering along teacher principal directions) restores seed diversity.
Concept pages: GPU kernels, quantization, knowledge distillation.