What Is Prompt Injection? How AI's Biggest Security Risk Works

Prompt injection is an attack that gets an AI model to follow instructions hidden in its input instead of the rules its developer set for it. The U.S. National Institute of Standards and Technology (NIST) defines it as an attack that exploits the way untrusted input gets concatenated with a prompt written by a higher-trust party, such as the application's own developer. In practice that means two very different things depending on who is doing the injecting: a user can type a jailbreak straight into a chatbot (direct injection), or an attacker can bury a command inside a webpage, email, or file that an AI agent later reads on someone else's behalf (indirect injection). OWASP, the nonprofit behind the widely used Top 10 security lists, ranks prompt injection as the number one risk facing large language model applications — and says no method yet fully prevents it.

What is prompt injection?

Large language models don't have a built-in way to tell the difference between "instructions I should follow" and "text I'm just supposed to read." A chatbot's system prompt, a user's question, and the contents of a document it's been asked to summarize all ultimately arrive as the same kind of text. Prompt injection exploits that blur: it's any input, crafted or accidental, that changes the model's behavior or output in a way its operator didn't intend. The OWASP GenAI Security Project's official definition is blunt about this — a prompt injection vulnerability "occurs when user prompts alter the LLM's behavior or output in unintended ways," and the project notes that the inputs don't even need to be visible to a human, only parseable by the model.

That distinction matters because it means prompt injection isn't really a bug that can be patched the way a buffer overflow can. It's closer to a structural property of how current language models process text, which is why it keeps showing up near the top of every major AI security framework.

Direct prompt injection: when the attacker is the user

In direct prompt injection, the person interacting with the AI system is the adversary. They write a prompt specifically designed to override the system's guardrails — classic examples include role-play jailbreaks that ask a model to "pretend" it has no restrictions, or instructions that explicitly tell the model to "ignore previous instructions." Anthropic's own developer documentation groups this together with jailbreaking under a single threat model: "the user of your application is the adversary and crafts inputs intended to bypass your guardrails." Because the attacker controls the input directly, defenses here tend to focus on screening what comes in — classifying prompts before they reach the main model, and designing system prompts that are explicit about what the assistant should refuse.

Indirect prompt injection: when the attack hides in content

Indirect prompt injection is the more unsettling variant, because the person using the AI product isn't the attacker at all — they're a target. Here, a malicious instruction is planted in content the AI will later process on the user's behalf: a webpage an AI agent browses, an email it summarizes, a calendar invite it reads, or a file it's asked to extract text from. OWASP describes this as the model ingesting external content that contains instructions altering its behavior, "intentionally or unintentionally." Anthropic's documentation frames the same risk from a builder's perspective: "you're protecting your users from instructions embedded in content that Claude reads on their behalf," and lists the body of an inbound email, a fetched webpage, OCR output from an uploaded file, and tool-call results as the places such instructions can hide.

This is the version of the attack that security researchers and vendors have grown far more concerned about as AI products gain the ability to act — not just answer questions. A model that can only reply with text can't do much damage if it's tricked. A model wired up to send emails, browse the live web, or call tools through something like the Model Context Protocol is a different story: a successful indirect injection can turn a routine "summarize this document" request into unauthorized data exfiltration or unwanted actions, all without the user ever typing a malicious word.

Direct vs. indirect: a quick comparison

Why agentic AI makes this urgent now

Prompt injection has been discussed since the earliest days of instruction-following language models, but it has moved from a research curiosity to a live operational risk as AI products have become agentic. Browser-use agents that click through websites, email- and calendar-connected assistants, and tools wired together through protocols like MCP all give an AI system two things an attacker can exploit together: a steady diet of untrusted external content, and the ability to actually do something with what it reads. Anthropic's guidance captures the shift directly, warning that an agent's tool results, fetched webpages, and document text are all places "an attacker who can influence that content might embed instructions that try to redirect" the model. The more autonomy and tool access an AI product has, the more an indirect injection can accomplish if it succeeds — which is why the risk calculus looks different for a simple chatbot than for an AI agent with a browser, an inbox, and a set of connected tools.

What OWASP and NIST say about the risk

OWASP's GenAI Security Project — the group that maintains the widely cited Top 10 lists for web application security — has kept prompt injection in the top spot of its Top 10 for LLM Applications across its revisions. Its published guidance recommends a layered set of mitigations rather than any single fix: constraining model behavior through clear system-prompt instructions on role and scope, validating expected output formats in code rather than trusting the model's word, applying input and output filtering, enforcing least-privilege access so the model itself never directly holds sensitive API tokens, requiring human approval for high-risk actions, clearly segregating and labeling untrusted external content, and running regular adversarial testing. Crucially, OWASP is explicit that none of this adds up to a cure: no method is known to fully prevent prompt injection, only to reduce its impact.

NIST's contribution is more definitional than prescriptive. Its official glossary entry, drawing on its adversarial machine learning taxonomy published as NIST AI 100-2e2025, defines prompt injection narrowly and technically as an attack that exploits the concatenation of untrusted input with a prompt constructed by a higher-trust party, such as the application's own designer. That framing is useful because it explains why the problem resists a quick patch: as long as untrusted and trusted text ultimately get combined into one input stream the model reads the same way, the model has to do the work of telling them apart — and today's models don't do that reliably.

How Anthropic defends Claude against prompt injection

Anthropic publishes its own guidance for developers building on Claude, and it treats direct and indirect injection as genuinely different problems requiring different fixes. For direct injection and jailbreaks, Anthropic recommends screening user input with a lightweight model before it reaches the main conversation, validating input against known injection patterns, and writing system prompts that explicitly state ethical and legal boundaries along with how the model should refuse.

For indirect injection — the agentic risk — Anthropic's recommendations center on structuring an application so Claude can reliably tell untrusted content apart from real instructions. That means delivering third-party content only inside tool-result blocks rather than system prompts or plain user text, since Anthropic says Claude is trained to treat instructions inside tool results with more skepticism; explicitly telling Claude what a piece of content is and where it came from, such as flagging an email as being from an unknown sender; stating in the system prompt that anything returned by tools is data to report rather than commands to follow; and, where possible, JSON-encoding untrusted text so an attacker can't use stray characters to "break out" into an instruction context. Anthropic also recommends screening tool outputs with a classifier before Claude acts on them, limiting the model's access to sensitive data and actions under a least-privilege principle, and red-teaming an agent's own workflows with documents and emails that deliberately contain injection attempts before shipping it. For products using Anthropic's computer-use or browser-use tools, the company says it runs additional classifiers on what those tools return, scanning for potential injections and prompting Claude to check whether an instruction genuinely came from the user before acting on it.

How Microsoft and other vendors are building in defenses

Microsoft takes a more infrastructure-level approach through Azure AI Content Safety's Prompt Shields, a detection API built specifically to flag both kinds of attacks before they reach a model. Per Microsoft's own documentation, Prompt Shields accepts a user prompt and a set of documents, analyzes each separately, and returns a simple true-or-false "attack detected" verdict for the prompt and for every document — explicitly distinguishing "user prompt attacks," which cover the direct, jailbreak-style case, from "document attacks," defined as injected instructions found in document content, as opposed to document content that's merely harmful. Microsoft's documentation is careful to note a real limitation: Azure AI Content Safety doesn't independently verify where submitted content actually came from, so it's up to the application developer to establish and trust that provenance before passing content through the API.

The practical effect of tools like this is to move detection earlier in the pipeline, flagging suspicious content before it ever reaches the model that's supposed to act on a user's behalf — a complementary layer to the prompt-structuring approach Anthropic recommends, rather than a replacement for it.

Mitigations at a glance

Why no defense is complete

It's worth being direct about the limits here, because every primary source on this topic is. OWASP states plainly that no method is currently known to fully prevent prompt injection. Anthropic's own documentation, while listing a long set of mitigations, frames them as ways to "strengthen your guardrails" rather than eliminate the risk, and notes that even Claude's built-in resilience is paired with the recommendation that developers add their own layers on top. Microsoft's Prompt Shields documentation similarly flags that its detection depends on the developer supplying accurate content and provenance — the tool scores what it's given, it doesn't verify where that content truly originated. The consistent theme across OWASP, NIST, Anthropic, and Microsoft is that prompt injection is managed through layered defense-in-depth, not solved with a single feature. That's also why the mitigation menus from each organization overlap so heavily on the same few ideas: isolate untrusted content, limit what the model can do with it, screen before and after, and assume some attacks will still get through.

What to watch next

Prompt injection defenses are moving in step with how much autonomy AI products are given. As more AI agents gain standing access to inboxes, calendars, browsers, and connected tools — including through standards like MCP that make it easier to wire a model up to outside systems — the attack surface for indirect injection grows alongside the convenience. Expect vendors to keep shipping more dedicated detection tooling in the vein of Prompt Shields, more built-in training and classifier work aimed at making models themselves more skeptical of embedded instructions, and continued emphasis from groups like OWASP on least-privilege design and human-in-the-loop approval for anything consequential. Anthropic's broader security work, including its push into dedicated cyber-defense initiatives, suggests major AI labs now treat this class of risk as core infrastructure work rather than an edge case. For anyone building or deploying agentic AI products, the practical takeaway from every primary source here is the same: treat anything the model reads from outside its own trusted instructions as data, never as a command, and design the rest of the system so that even a successful injection has as little to work with as possible.

Written byTheo Park

Theo Park runs the AI desk at Pandromeda. He follows model launches from the frontier labs and the open-weight community, tracks the assistants and developer tools built on them, and explains what each release changes on pricing, capability and safety. His reporting leans on primary sources: model cards, technical reports, API documentation and the companies' own announcements.

More from Theo ParkAll AI coverage