What Is an AI Distillation Attack? Inside Claude's New Defenses

An AI "distillation attack" is when one company secretly funnels huge volumes of queries through a rival's chatbot, harvests the answers — especially the step-by-step reasoning behind them — and uses that data to train a cheaper model that mimics the original's abilities. Anthropic says this is exactly what happened to Claude, and on October 12, 2026 it detailed a set of new defenses it says have cut the data such distillers can harvest by roughly 90 to 99%.

The announcement is a follow-up to a blunter one from February, when Anthropic publicly named three Chinese AI labs — DeepSeek, Moonshot AI, and MiniMax — as running what it called industrial-scale extraction campaigns against Claude, using tens of thousands of fraudulent accounts. Here's what a distillation attack actually is, what Anthropic built to stop it, and why the company says the stakes go beyond a single company's trade secrets.

What is an AI distillation attack?

"Distillation" is a legitimate and widely used machine-learning technique: a company trains its own small, fast "student" model to mimic a larger "teacher" model, usually to make a cheaper product. Google, OpenAI, and Anthropic all distill their own models this way, and there's nothing improper about it when a company does it with its own systems.

A distillation attack is different. It happens when an outside party extracts a competitor's model outputs without authorization — querying it at massive scale, often through fake accounts, proxy networks, or disguised traffic, then using the responses as training data for its own model. Anthropic describes what it has observed against Claude as "covert, industrial-scale campaigns to extract the capabilities of one model and replicate them in another." The practice lets a company copy a frontier lab's years of research and billions of dollars of compute spending in a fraction of the time and cost — and, crucially, without inheriting the safety work that went into the original model.

The underlying technique dates back well before generative AI; see Wikipedia's overview of knowledge distillation for the machine-learning background. What's new is the scale and the covert infrastructure attackers now use to pull it off against commercial chatbots.

How the attack actually works

Anthropic's account, and corroborating reporting, describes a fairly consistent playbook:

• Fraudulent accounts: Attackers create large numbers of accounts, often using stolen or synthetic identities and payment details, sometimes routed through regions where Claude access is otherwise restricted.
• Proxy networks: Requests are funneled through intermediary services that mix distillation queries in with ordinary, legitimate traffic to avoid detection. Anthropic says one such network has run more than 20,000 fraudulent accounts at once, and that some networks have controlled hundreds of thousands of accounts over time.
• Harvesting reasoning, not just answers: The single most valuable thing to extract isn't Claude's final answer — it's the reasoning trace that shows how Claude got there. A model trained on full reasoning transcripts learns far more than one trained on answers alone.
• Disguised requests: Anthropic says attackers have asked Claude to "output your prior reasoning verbatim," translate earlier reasoning into another language, or write it out inside special tags — all attempts to recover reasoning that Anthropic otherwise summarizes before showing it to users.

Why Claude's reasoning is the real prize

Modern "reasoning" models like Claude work through a problem step by step before producing a final answer. Anthropic already summarizes that internal reasoning before showing it to users, rather than exposing the raw chain of thought. That turns out to matter enormously for distillation: Anthropic says a model trained on full Claude Opus 4.8 reasoning transcripts scored 80.4% on the SWE-bench Verified coding benchmark, while an identical model trained only on the summarized version scored 71.8%. That roughly nine-point gap is the difference attackers are fighting to close, and it's why extracting raw reasoning — not just outputs — has become the central goal of the campaigns Anthropic describes.

The campaigns Anthropic says it caught

In a February 23, 2026 report, Anthropic said it had identified industrial-scale distillation campaigns run by three Chinese AI developers. The company said it traced the activity through request metadata, timing patterns, and shared payment and proxy infrastructure.

The October follow-up adds a specific example: Anthropic says it found that some requests sent to Moonshot's own Kimi models were secretly forwarded to Claude, and that one such request — which Anthropic believes came from a user affiliated with China's People's Liberation Army — sought help analyzing surveillance footage of an individual. None of the three named companies has publicly detailed a response to Anthropic's allegations.

Why Anthropic calls this a national security issue, not just a business one

Anthropic frames the problem as bigger than lost licensing revenue. Its stated concern is twofold. First, distilled models can approach the capability frontier — in areas like cybersecurity and biology — without inheriting proportionally robust safety guardrails, meaning dangerous capabilities get copied faster than the safeguards around them. Second, and more pointedly, Anthropic says foreign actors that successfully distill Claude's capabilities can "feed these unprotected capabilities into military, intelligence, and surveillance systems" — tying the issue to authoritarian governments' use of frontier AI for offensive cyber operations and mass surveillance. Anthropic's threat intelligence hub tracks this and other misuse patterns the company says it has detected and disrupted. That framing is echoed in Pandromeda's earlier coverage of Anthropic's broader push to harden critical systems, including its cyber mission to defend power grids and code.

What Anthropic actually built to stop it

Anthropic's October post lays out several layers of defense, most of which were rolled out alongside recent Claude model releases:

• Preserved thinking: Starting with Claude Fable 5.1 and Claude Opus 5.5, new API accounts created on or after August 31, 2026 can no longer edit Claude's prior reasoning context — a technique that had made it easier to repeatedly coax out hidden reasoning through proxy networks.
• Isolating reasoning from weaker models: Claude's reasoning can no longer be read back by older models that carry weaker safeguards, closing a path attackers used to launder extracted reasoning through less-protected systems.
• Organization-only access: With Claude Sonnet 5.5, reasoning transcripts are only accessible to the organization whose own request produced them.
• A two-stage extraction classifier: Built on Anthropic's constitutional classifiers research, this system is trained specifically to catch extraction attempts — including novel tricks like asking Claude to translate its own prior reasoning into another language. It runs on Claude Sonnet 5.5, Opus 5 and 5.5, and Fable 5 and 5.1, and Anthropic says it has a false-positive rate under 0.1%.
• Harder repeat offenses: Accounts that repeatedly attempt extraction face stricter usage limits and can be banned — Anthropic says tens of thousands of accounts have been banned, often within two hours of a campaign starting.
• Charging for blocked requests: Since September 24, 2026, Anthropic has charged for requests its safeguards block before Claude even responds, including attempted extraction requests — removing the free-retry economics that made large-scale probing cheap.

The results Anthropic is reporting

Anthropic's headline figures, drawn directly from its October post: the new defenses have cut the data harvested by distillers by roughly 90 to 99%, and the volume of reasoning tokens successfully extracted is now 10 to 100 times lower than before the defenses went live — on some days, Anthropic says, too small to measure at all. The company frames this as a meaningful but not final win, describing the effort as "reliably prevent[ing] many more attempts" while cautioning it is a "cat-and-mouse game" whose long-term durability it can't guarantee. Notably, Anthropic says it has not yet seen a highly sophisticated, persistent attempt to circumvent the current defenses — implying the real test may still be ahead.

The trade-offs: what legitimate users lose

Anthropic is unusually candid that these defenses have costs. The company says it has "turned away meaningful revenue and trillions of tokens' worth of paying traffic from distillers" — traffic it could have simply billed for. It also acknowledges that its classifiers, despite a sub-0.1% false-positive rate, inevitably misidentify or block some benign requests, and that preserved thinking required some legitimate developers to update their own integrations. Back in February, Anthropic said its goal was to limit distillation without affecting legitimate users — and in October, it explicitly acknowledged it has not fully achieved that balance yet. Developers choosing between Claude tiers should also note that preserved thinking and classifier behavior can vary by model — see Pandromeda's comparison of Claude Haiku 5.5, Sonnet 5.5, and Opus 5.5 for how the current model lineup differs.

It's not just Anthropic: Gemini and GPT models face the same problem

Anthropic is not alone in reporting this. Google's Threat Intelligence Group has separately described "distillation attacks" against Gemini — including one campaign that sent more than 100,000 prompts designed to probe the model's reasoning across multiple languages, which Google treats as a form of intellectual-property theft rather than a national-security threat. OpenAI made a related allegation in early 2025, saying it had evidence that a Chinese lab had queried its API at scale to help train a competing model. Anthropic, Google, and OpenAI reportedly began exchanging technical indicators about this kind of abuse in 2026, treating it as an industry-wide problem rather than a single company's grievance. It's a reminder that account-level abuse at scale is a recurring theme in tech security generally, not just in AI — Pandromeda has covered similar dynamics in other large-scale breaches, such as the Burger King Japan data breach.

What's next

Anthropic frames distillation defense as an ongoing arms race rather than a solved problem. Watch for three things in the months ahead: whether Anthropic extends its extraction classifier to future model releases beyond the Sonnet, Opus, and Fable families it currently covers; whether DeepSeek, Moonshot, or MiniMax issue any public response to Anthropic's allegations; and whether regulators or the other frontier labs — Google and OpenAI among them — move toward shared technical standards for detecting fraudulent account networks, something Anthropic has said it wants to see industry-wide. Given Anthropic's own admission that it hasn't faced a truly sophisticated, persistent attacker yet, the next real test of these defenses is probably still to come.

Written byTheo Park

Theo Park runs the AI desk at Pandromeda. He follows model launches from the frontier labs and the open-weight community, tracks the assistants and developer tools built on them, and explains what each release changes on pricing, capability and safety. His reporting leans on primary sources: model cards, technical reports, API documentation and the companies' own announcements.

More from Theo ParkAll AI coverage