AI/Explainer

What Is Prompt Injection? The AI Attack That Hijacks Agents

Hidden instructions in a webpage, email or document can hijack an AI agent’s actions. Here’s how prompt injection works, real incidents, and how Claude, ChatGPT and CISA respond.

Illustration of an AI browser agent at work, from Anthropic's research on prompt injection defenses
An AI browser agent navigating the web. Image: Anthropic.

Prompt injection is the single attack security researchers at Anthropic, OpenAI and the U.S. Cybersecurity and Infrastructure Security Agency (CISA) keep warning about as AI agents move from answering questions to actually browsing the web, reading email and clicking buttons on your behalf. It works by hiding instructions inside content an AI model reads — a webpage, a PDF, an email, even an image — so the model follows the attacker’s commands instead of yours. No major AI lab claims to have fully solved it yet.

Prompt injection in short
  • Attackers hide instructions inside text, images or documents an AI agent reads, hijacking its next actions.
  • It is different from “jailbreaking” — the attacker targets the agent through content, not through the user’s own chat message.
  • Anthropic’s best current defenses still show roughly a 1–2% attack success rate against adaptive attackers.
  • OpenAI, Anthropic and a six-nation government coalition have all published guidance in 2026 calling it the hardest unsolved problem in agentic AI.

What is prompt injection?

Prompt injection is an attack that smuggles instructions into the data an AI model processes, so the model treats an attacker’s text as if it were a legitimate command from its developer or user. Anthropic defines it as “adversarial instructions hidden within the content that AI models process” — content the model was only supposed to read and summarize, not obey.

The risk is largely new because AI agents now do more than chat. A browser-using agent can navigate pages, fill out forms, click links and download files — the same broad computer-use capability that now lets agents operate a screen directly. A coding agent can read files and run commands. An email assistant can read your inbox and send replies. Each of those abilities gives an attacker a lever: if the agent can be tricked into misreading instructions, it can be tricked into acting on them.

Direct injection vs. indirect injection

Security researchers generally split prompt injection into two categories:

  • Direct injection happens when an attacker types the malicious instruction straight into a chat window themselves, trying to override the AI’s own system prompt or safety rules. This overlaps with what many people call “jailbreaking.”
  • Indirect injection is the more dangerous version for agents: the malicious instruction is planted somewhere the AI will later read as part of its normal job — a webpage it browses, a document it summarizes, an email it opens, a product review, even metadata in an image. The user never sees the hidden instruction; the agent encounters it mid-task and may silently comply.

A simple example Anthropic has described: an email sitting in an inbox contains a block of white text on a white background, invisible to a human reader. The hidden text instructs an AI assistant reading the inbox to quietly forward anything marked “confidential” to an external address while it drafts an unrelated reply. The user asked the agent to clean up their inbox — not to leak it — but the agent cannot always tell the difference between the user’s instructions and an attacker’s instructions once both are sitting in the same context window.

The same trick works through almost any format an agent can read. Security researchers have catalogued injected instructions hidden in: alt text on images that a vision-capable model reads but a human skims past; metadata fields inside PDFs and spreadsheets; comments left on GitHub issues or pull requests that a coding agent is asked to review; calendar invite descriptions; and ordinary-looking customer support tickets submitted to a company’s own help desk, aimed at an AI agent that triages them automatically. None of these require the attacker to compromise any account — they only need the AI agent, at some point, to read something they control.

Why is prompt injection so hard to fix?

The core difficulty, as both Anthropic and the CISA-led guidance describe it, is architectural rather than a simple bug: large language models read system instructions, user messages and outside content as one continuous stream of text, without a hard, unforgeable boundary between “things I must obey” and “things I am merely reading.” A human assistant instinctively knows that a sticky note taped inside a filing cabinet isn’t a command from their boss. Today’s AI models have to be trained, case by case, to draw that same line — and training can reduce the failure rate without ever reaching zero, because an attacker only needs to find one phrasing the model’s classifiers haven’t seen.

That’s also why the CISA-led guidance explicitly rejects the idea of a single fix. It recommends layering independent defenses — filtering what comes into an agent, validating what goes out, and strictly limiting what an agent is allowed to do — on the assumption that each layer will occasionally fail, but all of them failing on the same attempt is much less likely. The same logic underpins the move at both Anthropic and OpenAI toward damage-limitation tools like Lockdown Mode: if you can’t guarantee the model will never be fooled, you design the system so that being fooled doesn’t automatically mean losing data or money.

Real-world incidents

Prompt injection stopped being a theoretical problem in 2026. Security researchers have documented live attacks using the technique against autonomous agents, including cases where attackers planted fake package documentation and typosquatted cryptocurrency sites specifically designed to trick AI coding and trading agents into approving real financial transactions. Not every agent-related data exposure this year was a classic prompt injection, either — a separate incident in which Claude Code’s “don’t ask again” setting leaked 31,000 secrets on GitHub showed how agent permission settings, not just injected text, can also open the door to real data loss.

Government agencies have also started treating it as a named, high-priority risk rather than a footnote. In their joint Careful Adoption of Agentic AI Services guidance, published May 1, 2026, CISA and five allied cybersecurity agencies — the NSA and counterparts in Australia, Canada, New Zealand and the UK — singled out prompt injection as one of the most persistent and difficult-to-fix threats facing organizations that deploy AI agents, warning that no single safeguard is sufficient on its own.

How well can AI models resist it?

Anthropic has published some of the most detailed public data on this. Testing its Claude for Chrome browser agent against an internal adaptive attacker allowed 100 attempts per scenario, Anthropic measured the following attack success rates (ASR) — the share of attack attempts that successfully hijacked the agent — with and without its safeguards:

ModelNo safeguardsEarlier safeguardsImproved safeguards
Claude Opus 4.58.0%7.6%1.4%
Claude Sonnet 4.513.8%10.8%1.9%
Claude Sonnet 413.8%10.7%2.0%
Claude Haiku 4.512.0%11.1%2.4%

Anthropic’s own framing is blunt: a roughly 1% success rate is a significant improvement over earlier defenses, but it still “represents meaningful risk” at scale, since an agent used millions of times a day will still be hijacked in a meaningful number of those sessions. The company has said plainly that no browser-using agent today is immune to prompt injection.

How Anthropic is defending Claude

Anthropic’s approach, detailed in its research post on prompt injection defenses, layers several techniques together rather than relying on one fix:

  • Reinforcement learning against injections. Claude is trained on simulated web pages and documents that contain injected instructions, and rewarded for recognizing and refusing them — including injections phrased to sound urgent or authoritative, a common manipulation tactic.
  • Real-time classifiers. Every piece of untrusted content entering Claude’s context window, including hidden text, manipulated images and deceptive on-page UI elements, is scanned by classifiers that flag likely injection attempts and adjust the model’s behavior before it acts.
  • Red-teaming. Anthropic’s internal security researchers continuously probe the browser agent for new attack patterns, and the company also takes part in external, Arena-style adversarial testing challenges run by outside researchers.

Anthropic’s developer documentation also recommends that anyone building on Claude explicitly tell the model, in its system prompt, that content coming back from tools, documents or web search is untrusted data that must never be allowed to override the system prompt or the user’s original request — a basic design pattern meant to stop an agent from quietly re-prioritizing an attacker’s text over its actual instructions. It’s the same “untrusted by default” principle behind the Model Context Protocol (MCP), which standardizes how agents connect to outside tools and data in the first place.

OpenAI’s answer: Lockdown Mode

OpenAI took a different angle with ChatGPT. Rather than trying to make the model immune to injected instructions, its Lockdown Mode — introduced February 13, 2026 alongside new “Elevated Risk” labels shown across ChatGPT, ChatGPT Atlas and Codex — deterministically disables the specific tools an attacker would need to actually exfiltrate data after a successful injection, such as browsing the open web, Deep Research, Agent Mode, inline image rendering and downloading files for analysis.

That’s an important distinction: Lockdown Mode does not stop a hidden instruction from reaching the model in the first place. It is designed to cut off the escape routes an attacker would use afterward, on the assumption that some injections will get through no matter how good the model’s own judgment gets. OpenAI later expanded Lockdown Mode’s availability to its Free, Go, Plus, Pro and self-serve Business plans.

How to protect yourself and your data

For everyday users and for teams deploying AI agents, the practical advice from security researchers and the CISA-led guidance converges on a few basics:

  • Give agents the least access they need. An agent that can only read your calendar can’t leak your email, even if it’s hijacked. Scope permissions narrowly for every task.
  • Be cautious with agents that browse freely or handle money. The riskiest combination is an agent that both reads untrusted content (the open web, unscreened documents) and can take consequential actions (sending money, sending messages, deleting files).
  • Use built-in safety modes when available. Features like Lockdown Mode or Claude’s classifier-backed browsing exist specifically to blunt the damage of a successful injection — turning them off for convenience increases real risk.
  • Watch for anomalous behavior. An agent that suddenly tries to visit an unrelated site, forward a message, or take an action you didn’t ask for mid-task is the clearest practical sign an injection may have occurred.
  • Treat “no single safeguard is enough” as the operating assumption. That’s the core message from CISA’s joint guidance: organizations should layer input filtering, output checks and strict execution permissions rather than trusting any one defense.

What happens next

Every major lab building AI agents now treats prompt injection as an open, active problem rather than a solved one. Anthropic continues publishing updated attack-success-rate figures as it ships new safeguards; OpenAI has already widened Lockdown Mode’s rollout once since launch; and the CISA-led coalition’s guidance is written to evolve as new agentic risks emerge. Expect the Attack Success Rate numbers above to keep moving — in both directions — as agents gain more real-world permissions and attackers get more creative about hiding instructions in the content those agents are asked to read.

Frequently asked questions

What is prompt injection in simple terms?

It’s an attack that hides instructions inside content an AI model reads — like a webpage, email or image — so the model follows the hidden command instead of the user’s actual request.

Is prompt injection the same thing as jailbreaking?

No. Jailbreaking is typically a direct attempt by the user to override an AI’s rules in their own message. Prompt injection, especially the indirect kind, hides the malicious instruction in outside content the AI reads during a task, without the user ever seeing it.

Can prompt injection actually steal my data?

Yes. Researchers have documented cases of hidden instructions directing AI agents to forward confidential emails or approve financial transactions. That’s why OpenAI’s Lockdown Mode focuses on blocking the exfiltration routes an attacker would need afterward.

How are Anthropic and OpenAI defending against it?

Anthropic trains Claude with reinforcement learning against simulated injections and scans incoming content with classifiers. OpenAI’s Lockdown Mode instead disables tools like web browsing and file downloads that an attacker would need to exploit a successful injection.

What does the CISA-led guidance recommend?

Its May 2026 guidance, co-published with five allied agencies, calls for layered defenses — input filtering, output validation and strict permission limits — since it says no single safeguard is enough to stop prompt injection on its own.

Can prompt injection be completely fixed?

Not yet, according to Anthropic and OpenAI. Even the best current safeguards show roughly a 1–2% attack success rate against adaptive attackers, which is why both companies pair model training with damage-limiting features.

Sources

More on Prompt Injection →Prompt InjectionAI AgentsAI SecurityClaudeChatGPT
Theo Park
Written byTheo Park

Theo Park runs the AI desk at Pandromeda. He follows model launches from the frontier labs and the open-weight community, tracks the assistants and developer tools built on them, and explains what each release changes on pricing, capability and safety. His reporting leans on primary sources: model cards, technical reports, API documentation and the companies' own announcements.

More from AI

See all