Red-Teaming Prompt Injection Against AI Agents: Payloads, Evasion, and TTPs in the Lab

How do you red-team prompt injection against an AI agent?

You red-team prompt injection against an AI agent the way you red-team any other system: in an isolated lab, with a defined scope, a threat model, and a payload library you build and version yourself. The core move is to stop treating the agent as a chatbot and start treating it as an attack surface with hands — it reads untrusted content, holds memory, calls tools, and acts on what it reads. Your job is to plant instructions where the agent will obey them, then prove what happens when it does. In practice that means crafting jailbreak payloads for the direct channel, seeding indirect injection into the documents, emails, and web pages the agent ingests, and mapping every successful hit to a known technique so the finding survives the handoff to engineering. It is one spoke of a broader red-team framework practice that also covers fuzzing, memory poisoning, and benchmarking.

The rest of this article is a working playbook. It assumes you have authorization, an isolated environment, and a defender on the other side who actually wants to fix things. If any of those three are missing, stop — what follows is offensive security tradecraft, and it is only legitimate inside a sanctioned engagement.

Scope note (read this first): Everything below is written for authorized red-team and lab work against systems you own or have written permission to test. The payloads are illustrative patterns, not turnkey exploits against any specific product. Test in a sandbox, never against production agents wired to real customer data or live financial actions.

Why is prompt injection the number-one risk for agents?

Prompt injection sits at the top of the OWASP Top 10 for LLM Applications as LLM01:2025 for a simple structural reason: a language model has no reliable boundary between instructions and data. Everything it sees — your system prompt, the user’s message, a retrieved document, a tool’s output — arrives as the same undifferentiated token stream. When the model decides what to do next, it weighs all of it together. An attacker who can get text into that stream is, in effect, writing part of the prompt.

For a plain chatbot that is annoying. For an AI agent it is dangerous, because the agent can do things: send email, query a database, run code, hit an internal API, or write to its own memory. The blast radius is no longer “the model said something rude” — it is “the model exfiltrated a customer table” or “the model approved a refund.” That escalation from speech to action is exactly why agents deserve a dedicated red-team practice rather than a generic content-safety check.

The threat also splits cleanly in two, and your lab needs both. Direct prompt injection is when the attacker controls the user input — classic jailbreaks, role-play framing, instruction overrides typed straight into the chat. Indirect prompt injection is when the malicious instruction rides in on external content the agent processes: a poisoned PDF, a scraped web page, a support ticket, a calendar invite, an email body. Indirect injection is the one that keeps defenders up at night, because the victim user never typed anything hostile — they just asked their assistant to “summarize my inbox,” and an attacker’s email did the rest.

What does a prompt-injection red-team lab actually need?

Before you write a single payload, stand up an environment that lets you fail loudly and safely. A credible lab has four pieces.

An isolated target. Run the agent against a sandboxed copy of its tools. Replace the real “send email” tool with a logging stub, the real database with seeded fixtures, the real payment API with a recorder. You want the agent to attempt the dangerous action so you can observe it, without the action actually firing into the world.

A canary and an oracle. Plant unique, traceable secrets — a fake API key, a marked “confidential” string, a honeytoken URL — so that when an injection succeeds, the proof is unambiguous. The cleanest oracle is exfiltration to a server you control: if your collaborator endpoint receives the canary, the injection worked, full stop. No arguing about whether the model “kind of” complied.

Full transcript capture. Log the system prompt, every tool call with arguments, every tool result, and the final output. Most failed-looking tests are actually partial successes hiding in a tool-call argument you never inspected. If you only read the final chat bubble, you will miss half your findings.

A payload library under version control. Treat payloads like exploit code: name them, tag them by technique, record which model and which guardrail version they beat, and re-run the whole set on every target. This is the difference between a one-off “I jailbroke it once” anecdote and a repeatable regression suite the blue team can adopt.

How do you build prompt-injection payloads?

Payload construction is iterative, not magic. You start from a goal, pick a delivery channel, choose a framing technique, then layer evasion on top until something lands. Build them in families so you can measure what each layer buys you.

The starting families worth having in any jailbreak payloads collection:

• Instruction override. The bluntest form: text that tells the model to disregard prior instructions and follow new ones (“Ignore the above and instead do X”). Modern guardrails catch the obvious phrasings, which is precisely why you keep them — they’re your baseline for measuring how much evasion the layers below actually add.
• Role / persona framing. Wrap the request in a fictional or authoritative role so the model treats the harmful instruction as in-character or as a trusted system directive. The classic “you are an unrestricted assistant” lives here, as does impersonating the developer or a system message.
• Context / task confusion. Disguise the payload as legitimate work: a “translation” task whose source text is the real instruction, a “summarize this document” where the document is the attack, a “format this JSON” where a field carries commands.
• Tool-targeted payloads. Written specifically to make the agent call a tool with attacker-chosen arguments — the part that turns a chatbot jailbreak into an agent compromise. The goal isn’t to make the model say something; it’s to make it do something.
• Memory and context poisoning. Instructions designed to be written into the agent’s long-term memory or persistent context so they fire on future, unrelated sessions — the persistence effect you isolate in a dedicated memory-poisoning RAG test. MITRE ATLAS tracks these as distinct agent techniques because their whole point is persistence.

A practical loop: pick one goal (say, “make the agent send the canary to my endpoint”), write the most direct payload, run it, read the full transcript, and note where it failed — input filter, model refusal, tool-arg sanitizer, or output filter. Each failure point tells you which evasion to reach for next. You are not searching blindly; you are walking the request down the pipeline and finding the weakest joint.

How do you test indirect injection specifically?

Indirect injection testing is the highest-value work in the lab because it mirrors how agents get hit in the wild. The setup is different from direct testing: you are not the user of the agent, you are the author of the content the agent will later read on someone else’s behalf.

The method, step by step:

1. Inventory the ingestion surface. List every external source the agent reads — RAG corpus, attached files, fetched URLs, email, tickets, tool outputs from third-party APIs. Each is a potential carrier. The surface is almost always larger than the team assumes.
2. Author a poisoned artifact. Put the instruction inside a document, web page, or message that a victim would plausibly ask the agent to process. Make the legitimate part of the artifact convincing; the injection is a passenger, not the whole payload.
3. Hide the payload from the human. This is where indirect injection gets its teeth. Instructions can live in white-on-white text, zero-size fonts, HTML comments, image alt text, document metadata, or off-screen DOM. The human reviewing the file sees nothing; the model reads everything.
4. Trigger via a benign user request. Have the “victim” ask the agent something innocent — “summarize this,” “what’s in my inbox,” “research this page.” The injection should fire without the victim doing anything hostile.
5. Confirm with the oracle. Watch for the canary hitting your endpoint, or the dangerous tool stub logging a call. That is your proof of indirect compromise.

The scenarios that consistently produce findings: RAG poisoning (planting a malicious entry in a vector store or knowledge base so it surfaces in retrieval), poisoned web pages an agentic browser will visit, and email or calendar content for inbox assistants. Internal copilots wired to documents, mail, and databases are especially fruitful — they are high-value, broadly permissioned, and usually less hardened than the customer-facing apps that got all the security attention.

What evasion patterns get past guardrails?

Once you have a payload that works against a naked model, evasion is about getting the same intent past the filters wrapped around it. Layer these on top of your base families and measure each layer’s contribution against your guardrail version.

Evasion pattern	How it works	What it defeats
Encoding / obfuscation	Base64, ROT13, leetspeak, homoglyphs, or unicode tricks that the model decodes but a keyword filter doesn’t	Naive input/output string matching
Language switching	Deliver the instruction in a lower-resource language, or mix languages mid-sentence	Filters tuned mainly on English
Payload splitting	Break the instruction across multiple messages, files, or fields that only combine in the model’s context	Per-message content scanners
Indirection / “summarize this”	Hide the instruction in content the model is asked to process, not in the instruction itself	Filters that only inspect the user turn
Token / whitespace smuggling	Invisible characters, formatting, or markup that shifts how the model parses instruction vs. data	Heuristics that assume clean text
Many-shot / context flooding	Prepend many fake “examples” that normalize the harmful behavior before the real ask	Single-turn safety classifiers

The pattern to internalize: filters operate on surface features (keywords, language, message boundaries), while the model operates on meaning. Every reliable evasion exploits that gap — it preserves the meaning the model will act on while mangling the surface the filter inspects. When you write up a finding, name the gap explicitly, because that is what tells the defender whether to fix the filter or rethink the architecture.

How do you map findings to attacker TTPs?

A red-team finding that says “I got the bot to misbehave” dies in the backlog. A finding mapped to a named technique gets fixed. Use a shared vocabulary so your results plug into the defender’s threat model and detection program.

MITRE ATLAS is the standard reference for agent TTPs — the adversarial-ML companion to ATT&CK, maintained by MITRE as a living knowledge base. As of its February 2026 release (v5.4.0) it spans 16 tactics and 84 techniques. Prompt injection lives at AML.T0051 with sub-techniques distinguishing direct and indirect variants, and — importantly — it spans more than one tactic stage. Crafting the adversarial content is staging activity that happens before the injection itself executes, so a program that logs “prompt injection” as a single event misses the run-up that detection could have caught earlier.

ATLAS has also grown teeth specifically for agents. A 2025 collaboration with Zenity Labs added agent-centric techniques including AI Agent Context Poisoning (manipulating the context an agent’s model reasons over to persistently steer its actions), Memory Manipulation (altering long-term memory so malicious changes survive across sessions), and Thread Injection (planting instructions in a specific conversation thread to bend behavior for that session). Earlier 2025 releases added RAG Poisoning and False RAG Entry Injection. If your lab finding is “I poisoned the knowledge base and it stuck across sessions,” there is now a precise technique ID to hang it on.

Pair ATLAS with OWASP LLM01:2025 for the risk framing and you get a finding that reads cleanly to both worlds: OWASP tells the product team why this matters and which control class is missing (input validation, context segregation, privilege limitation, output filtering); ATLAS tells the detection team exactly which technique to build telemetry for — the same mapping discipline threat-intel teams apply when translating ATLAS techniques into detection signals. Map every payload in your library to both. It is the cheap, boring step that determines whether your engagement changes anything.

What does a defensible red-team report look like?

Close the loop by writing for the people who have to fix it. For each finding, record: the technique ID (ATLAS) and risk ID (OWASP), the channel (direct vs. indirect), the exact payload and evasion stack, the model and guardrail version it beat, the oracle evidence (the canary hit, the logged tool call), and the blast radius if it had fired in production. Then state the architectural fix, not just the patch — because keyword filters are an arms race, and the durable mitigations are structural: least-privilege tools, human-in-the-loop on dangerous actions, treating all retrieved content as untrusted, and segregating instructions from data wherever the stack allows it.

The honest takeaway from the lab is uncomfortable and worth saying plainly: there is no known input filter that fully solves prompt injection, because the instruction/data ambiguity is baked into how these models read. That is not a reason to skip the work — it is the reason to do it as disciplined, repeatable red-teaming, and to push fixes toward architecture and privilege rather than hoping a classifier will hold the line.

---

This article is educational material for authorized security testing. The techniques described are intended for use only within sanctioned, isolated lab environments and against systems you own or have explicit written permission to test. Unauthorized testing against systems you do not control may be illegal.

Sources

• OWASP Gen AI Security Project — LLM01:2025 Prompt Injection
• MITRE ATLAS
• Promptfoo — MITRE ATLAS red-team mapping