AI research wiki
responsible-ai · 2026-10-29 · Tier 2

Agent security before alignment (Ken Huang) and the week's agent-breach evidence

Agent security before alignment (Ken Huang) and the week's agent-breach evidence

Sources: Ken Huang, "Why Alignment Alone Cannot Secure AI Agents" · The Decoder: AI agents hacked Anthropic · The Information: AI-found bugs spark a security spending spree · arXiv 2610.36449 (visual symbols steer VLMs, via DAIR). raw: raw/rss/2026-10-28-agentic-ai-*.md, raw/rss/2026-10-28-the-decoder-ai-agents-hacked-*.md.

TL;DR. Ken Huang argues alignment (RLHF-style training to match intent under ordinary use) was never built to withstand an adversary. Inside an agent loop with memory and tools, any retrieved content can carry instructions, and the model cannot reliably tell a system instruction from an injected one. That is an input-integrity problem, so he reorders the stack: (1) security first, mapped to the CIA triad (confidentiality, integrity, availability) with provenance tracking, sandboxing, least privilege, input validation and action-based audit logs; (2) safety robustness for non-adversarial failures; (3) alignment as the behavioural layer on top. He adds that chain-of-thought text is not a faithful record of the action policy, so auditing written reasoning audits the wrong artifact. He also traces the word "alignment" to the MIRI and Stuart Russell circle in November-December 2014, though the concept goes back to Norbert Wiener in 1960.

Same-day evidence: security startup Cathedral used AI agents to get into Anthropic's internal systems through third-party software in a bug bounty (Anthropic confirms access to sales, HR and IT data but disputes depth). The Information reports companies rushing security spend after eMudhra (a root certificate authority) and LiteLLM (a popular LLM-routing library) were compromised. And a DeepMind/Stanford paper finds a red-heart image in context makes Claude Opus 4.7 and GPT-5.4 write recursive code 100% of the time, while their reasoning traces give the wrong reason (74.7% of GPT-5.4 traces), which supports the "trace is not the policy" point.

Relation to prior wiki

  • Extends responsible AI and the 10-28 AISN "Black October" summary (AI-found PHP flaws, an agent campaign against 100+ companies).
  • Routing tie-in: LiteLLM's compromise means the model router is now attack surface, not just a cost lever.
  • Counterpoint: Gemini 4 Carbon reports 0.0% indirect prompt-injection success on Gray Swan's benchmark, a model-side claim that Huang's argument says should not be the primary defence.