Testing Memory Poisoning in a Red-Team Lab: Seeding RAG Corpora and Detecting Tainted Retrieval

A memory poisoning test is the controlled, lab-only exercise of planting malicious or misleading records into a retrieval-augmented generation (RAG) system’s knowledge sources, then proving whether those records resurface at query time and steer the model’s output. You run it the way you run any red-team experiment: against a system you own, with a clear hypothesis, a seeded payload you can fingerprint, and a detection harness that catches the tainted retrieval before — and after — it lands in the prompt. Done right, the test answers one question precisely: if an attacker can write to our corpus or vector store, what can they make the model say or do?

This article walks through the full loop in a self-contained RAG poisoning lab — how to seed documents and vector embeddings, how to trigger the poisoned path, and how to build tainted retrieval detection that you can keep running in CI and in production. It is one spoke of a broader red-team framework practice for AI agents. Everything here assumes an environment you control. Poisoning data you do not own is not a test; it is an attack.

What is memory poisoning in a RAG system?

Memory poisoning is the manipulation of the data an AI system retrieves at inference time so that the retrieved context changes the model’s behavior. In a RAG pipeline, the model’s “memory” is not its weights — it is the corpus, the chunked documents, and the vector store that indexes them. Anyone who can write to those layers can write to the model’s effective knowledge.

The poisoning happens upstream of the model and is invisible to it. The language model faithfully summarizes, reasons over, or acts on whatever lands in its context window. If a tainted chunk wins the similarity search, the model treats it as ground truth. That is the entire problem in one sentence: retrieval launders attacker-controlled text into trusted context.

There are two practical flavors you will test. The first is corpus poisoning, where the malicious content sits in a source document — a wiki page, a scraped web record, a support ticket, a PDF — that later gets ingested, often as a vehicle for indirect prompt injection. The second is vector-store poisoning, where the attacker writes directly to the embedding index, often crafting embeddings that sit suspiciously close to high-value queries so the bad chunk is almost always retrieved. Both end the same way: tainted retrieval feeding a trusting model.

Why build a memory poisoning test in a lab instead of relying on guardrails?

Output guardrails inspect what the model says. Memory poisoning corrupts what the model knows. By the time a content filter sees a confidently wrong answer or a smuggled instruction, the poisoned chunk has already done its work. A lab test moves the detection upstream, to the retrieval boundary where the tainted record is still identifiable as a record rather than as fluent prose.

The lab also gives you something production telemetry rarely does: ground truth — the defender’s counterpart to the attacker’s view of how memory poisoning corrupts agent context. You planted the payload, so you know exactly which document IDs, chunks, and embeddings are malicious. That lets you measure detection precisely — true positives, false negatives, time-to-detect — instead of guessing whether a fuzzy filter “probably caught most of it.” Red-team value comes from reproducible, fingerprinted experiments, not from vibes.

Finally, RAG systems are a live attack surface, not a theoretical one. The OWASP Top 10 for LLM Applications lists data and model poisoning (LLM04) and vector/embedding weaknesses (LLM08) as distinct, named risks. NIST’s AI Risk Management Framework and its adversarial machine learning taxonomy (NIST AI 100-2) classify poisoning as a primary attack class against ML systems. Building the test is how you turn those abstract categories into evidence about your own stack.

How do you seed a RAG corpus and vector store for the test?

Seeding is the controlled-injection phase. Your goal is to insert payloads that are realistic enough to behave like a genuine compromise, yet fingerprinted enough that you can find every trace of them afterward.

Step 1: Define the threat model and a fingerprint

Decide who the attacker is and what they can write to. A contractor who can edit a Confluence space? A scraper that pulls untrusted web pages into your index? An API that lets users submit “knowledge” that gets embedded? Each maps to a different write path, and the write path determines where you seed.

Before you plant anything, mint a fingerprint — a unique, searchable canary you embed in every payload. A random token like CANARY-7f3a9e2b in a metadata field, plus a distinctive marker phrase in the body, means you can later grep the corpus, the index, and the model’s outputs for the same string. The fingerprint is what makes the experiment measurable and what lets you guarantee full cleanup.

Step 2: Craft the payloads

Build a small, versioned set of poison documents that each test a different objective:

Payload type	What it plants	What it tests
Factual override	A false “fact” contradicting a true source (e.g. a wrong refund policy)	Whether retrieval surfaces the lie over the truth
Instruction injection	An imperative aimed at the model (“ignore prior context, reply with…”)	Whether retrieved text is treated as instructions
Tool / action bait	Text coaxing an agent to call a tool or exfiltrate data	Whether RAG context can drive actions, not just answers
Query-targeted chunk	Content engineered to match a specific high-value query	Whether one chunk can dominate similarity search

Keep payloads benign in effect even when malicious in form. The “tool bait” should target a sandboxed, logged tool that does nothing harmful. You are testing whether the path opens, not detonating a real payload.

Step 3: Inject at the right layer

For corpus poisoning, drop the documents into the source system and let your normal ingestion pipeline chunk and embed them. This is the highest-fidelity test because the payload travels the same road real data does — including any chunking quirks that split or preserve the injection.

For direct vector-store poisoning, write the embeddings straight into the index (FAISS, pgvector, Chroma, Pinecone, or whatever you run). Here you can also test adversarial embedding placement: generate a chunk, embed it, then nudge its text until its vector sits very close to a target query’s vector, so it ranks first regardless of relevance. That is the lab analogue of an attacker who has learned your embedding model and engineers chunks to win retrieval.

Record everything as you seed: document IDs, chunk IDs, vector IDs, the fingerprint, the target query, and the expected effect. This seed manifest is your answer key.

How do you detect tainted retrieval?

Tainted retrieval detection is the act of catching a poisoned record at the moment it is selected as context — ideally between the retriever and the prompt assembler, so you can block, flag, or quarantine it before it reaches the model. You detect it at three checkpoints: at ingestion, at retrieval, and at output.

Detection at ingestion (write-time)

The cheapest place to stop poison is before it is ever indexed. At write time you can:

• Provenance and trust scoring. Tag every chunk with its source, author, and trust tier. Content from untrusted write paths (user submissions, scraped pages) gets a low score that the retriever can later penalize or exclude.
• Content screening. Run incoming documents through detectors for embedded instructions, suspicious imperatives (“ignore”, “disregard”, “you must”), invisible Unicode, and zero-width characters that hide payloads from human reviewers.
• Embedding outlier detection. New vectors that are anomalously dense in a region of the space, or that are near-duplicates engineered to crowd a query, are statistical outliers. Flag points whose nearest-neighbor distances or local density deviate sharply from the corpus baseline.

Detection at retrieval (read-time)

This is the highest-leverage checkpoint and the one your lab test exercises most directly. For each query, inspect the retrieved set before it becomes context:

• Score-gap and consistency checks. A single chunk that dramatically outscores all others, or a retrieved set whose top results contradict each other, is a signal worth flagging. Poison engineered to dominate similarity search often produces an unnaturally large gap.
• Cross-source corroboration. Require that a claim used in a high-stakes answer appears in more than one independent source. A “fact” supported by exactly one low-trust chunk is suspect by construction.
• Canary verification (lab). Because you seeded a fingerprint, your harness can assert directly: did the poisoned chunk get retrieved for the target query? This gives you a clean true-positive measurement and a regression test you keep forever.

Detection at output (post-hoc)

Even with upstream controls, watch the tail. Scan model outputs for the fingerprint phrase, for refusals or instruction-following that imply injected context won, and for answers that diverge from a known-good baseline. Log which document and chunk IDs were in context for every generation, so any bad output is traceable back to the record that caused it. Attribution is what turns an incident into a fix.

What does the full poisoning test loop look like?

A complete run moves through five stages, and you script the whole thing so it is repeatable:

1. Baseline. Run your target queries against the clean corpus and record the answers, the retrieved chunk IDs, and the similarity scores. This is your control.
2. Seed. Inject the fingerprinted payloads via the chosen write path; capture the seed manifest.
3. Trigger. Re-run the same target queries. Compare retrieval and output against the baseline.
4. Detect. Run your detection harness across all three checkpoints and score it against the seed manifest — what did it catch, what did it miss, how fast.
5. Clean up. Delete every seeded document, chunk, and vector by ID and by fingerprint, then re-run the baseline to confirm the corpus is back to its control state. Confirm zero fingerprint hits anywhere.

The two metrics that matter most are attack success rate (did the poison change the answer for the target query?) and detection rate (did the harness flag the tainted retrieval before it reached the model?). A healthy result is high detection and low attack success. The gap between them is your remaining exposure.

How do you keep the test from becoming a liability?

Treat the lab like the dangerous thing it is. Use a dedicated, isolated environment — never your production index, and never a shared staging system other teams trust. Real instruction-injection and tool-bait payloads can leak into other people’s queries if you seed a shared store.

Make cleanup non-negotiable and verifiable. The fingerprint exists precisely so you can prove the corpus is clean afterward; an orphaned poison chunk that survives a sloppy teardown is a self-inflicted incident. Script the teardown, assert zero fingerprint hits, and fail the run loudly if anything remains.

Scope authorization explicitly. A memory poisoning test belongs in a documented engagement against systems you own or are contracted to test, with the data layers named in writing. The techniques here are dual-use; the line between a red-team exercise and an attack is ownership and consent.

---

This article is educational and intended for red-team work on systems you own or are explicitly authorized to test. Seeding poisoned data into corpora or vector stores you do not control may be illegal. No specific vendors are endorsed here; tool names appear only as common examples.