AI Agent Security Benchmarks: Mapping Your Eval Harness to MITRE ATLAS and the OWASP Top 10 for Agentic Applications

Most teams running an AI agent security benchmark can tell you a single number: the attack success rate. Far fewer can tell you which threats that number actually covers. A benchmark that catches 90% of prompt injections looks reassuring until you notice it never tested memory poisoning, privilege abuse, or a rogue agent acting on its own. The gap between “we ran an eval” and “we tested the threats that matter” is exactly where breaches live.

This article shows how to close that gap by treating two public taxonomies — MITRE ATLAS and the OWASP Top 10 for Agentic Applications (ASI) — as the coverage map for your agent eval harness. It is one spoke of a broader red-team framework practice that pairs benchmarking with fuzzing, injection, and memory-poisoning tests. You map each benchmark test case to the specific ATLAS technique and ASI category it exercises, then read the blank cells as your real risk register.

How do you map agent security tests to MITRE ATLAS and OWASP ASI?

You map tests to frameworks by tagging every benchmark scenario with the threat it exercises, not the tool it uses. Take each test case in your eval harness, identify the adversary behavior it triggers (indirect prompt injection, tool misuse, memory poisoning, privilege escalation), then attach two labels: the MITRE ATLAS technique ID and the OWASP ASI category. ATLAS tells you how the attack works as an adversary tactic; OWASP ASI tells you what class of agentic risk it represents for your application. Once every test carries both labels, you can pivot the results into a coverage matrix — and the threats with zero test cases are your honest backlog.

The practical payoff is twofold. First, you stop reporting a vanity attack-success-rate and start reporting coverage against named, peer-reviewed threat categories that an auditor or a customer’s security team already recognizes. Second, you expose silent blind spots: the benchmark you bought may be 600 test cases deep on prompt injection and completely empty on inter-agent communication or cascading failures.

What are MITRE ATLAS and the OWASP Top 10 for Agentic Applications?

These are the two reference taxonomies the industry has converged on for adversarial AI and agentic risk. They answer different questions, which is why you want both.

MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is a knowledge base modeled on the familiar ATT&CK matrix. It organizes real-world attacks on AI/ML systems into tactics (the adversary’s goal at a stage) and techniques (the specific method). As of late 2025 it expanded to 16 tactics and 84 techniques, and the framework added a wave of agent-specific techniques — RAG poisoning, false RAG entry injection, retrieval content crafting, and memory manipulation among them — reflecting the shift from AI that assists to AI that acts. ATLAS is adversary-centric: it describes attacker behavior across the kill chain.

The OWASP Top 10 for Agentic Applications (ASI) is a peer-reviewed list of the ten most critical risks specific to autonomous and agentic systems, published under the OWASP Gen AI Security Project. It is application-centric: each entry names a risk class a builder must defend against. The current list runs from ASI01 through ASI10:

• ASI01 – Agent Goal Hijack: hidden instructions redirect the agent’s objective.
• ASI02 – Tool Misuse: legitimate tools coerced into harmful actions.
• ASI03 – Identity & Privilege Abuse: the agent operates beyond its intended scope.
• ASI04 – Agentic Supply Chain Vulnerabilities: poisoned MCP servers, plugins, or A2A components.
• ASI05 – Unexpected Code Execution: natural-language paths to remote code execution.
• ASI06 – Memory & Context Poisoning: tampered memory reshapes behavior over time.
• ASI07 – Insecure Inter-Agent Communication: spoofed messages between agents.
• ASI08 – Cascading Failures: a false signal propagates through automated pipelines.
• ASI09 – Human-Agent Trust Exploitation: polished output nudges humans into approving harm.
• ASI10 – Rogue Agents: misalignment, concealment, or self-directed action.

The two are complements, not rivals. ATLAS gives you the granular technique vocabulary; OWASP ASI gives you the executive-readable risk categories. A single test case often carries one ASI label and several ATLAS technique IDs.

Why isn’t a high attack-success-rate score enough?

Because a benchmark score is only as honest as the threats it samples. An attack-success-rate of 8% sounds strong, but the number is meaningless without knowing the denominator’s composition.

Consider the most-cited public agent security benchmarks. AgentDojo (ETH Zurich) is a dynamic environment with 97 injection tasks across 629 test cases — deep, but overwhelmingly focused on prompt injection over untrusted tool data. InjecAgent is a dedicated benchmark for indirect prompt injection. Agent Security Bench (ASB) is broader, formalizing direct prompt injection, indirect prompt injection, memory poisoning, and plan-of-thought backdoors across ten scenarios. Each is excellent at what it covers — and each leaves whole ASI categories untested.

If your eval harness is built only on AgentDojo, you have strong evidence on ASI01 (goal hijack via injection) and partial evidence on ASI02 (tool misuse), and effectively nothing on ASI03 (privilege abuse), ASI06 (memory poisoning), ASI07 (inter-agent communication), ASI08 (cascading failures), or ASI10 (rogue agents). Your 8% score is a real number describing a narrow slice. Mapping makes that slice visible instead of letting it pose as the whole pie.

How do you build the coverage matrix in practice?

Treat it as a tagging exercise on top of your existing harness, then a pivot. Five steps.

1. Inventory your test cases. List every scenario your harness runs, regardless of source — vendor benchmark, open-source suite (AgentDojo, ASB, InjecAgent), or your own red-team scripts. The unit is the individual test case, not the benchmark.

2. Tag each case with an ASI category. Ask “what class of agentic risk does a success here represent?” An indirect-injection test that redirects the agent is ASI01. A test that tricks the agent into calling a destructive tool is ASI02. A test that plants a false fact in long-term memory is ASI06.

3. Tag each case with ATLAS technique IDs. The same memory test maps to ATLAS techniques around RAG/memory poisoning and content crafting. A case can hold multiple technique IDs; that is expected and useful.

4. Pivot into a matrix. Rows = ASI01–ASI10 (or ATLAS tactics). Columns = test count, attack-success-rate, and the benchmark source. Now every cell either has tests or is empty.

5. Read the blanks as your risk register. Empty rows are not “low risk” — they are “untested.” Prioritize building cases for the high-impact blanks (ASI03, ASI06, ASI10 are common gaps because public benchmarks under-cover them).

What does a finished mapping look like?

The matrix below is an illustrative skeleton showing how common public benchmarks tend to populate the ASI rows. Use it as a template, not as measured results for your system — your numbers come from your own runs.

OWASP ASI category	Representative ATLAS angle	Typically covered by	Common gap?
ASI01 Agent Goal Hijack	Prompt injection, indirect injection	AgentDojo, InjecAgent, ASB	Well covered
ASI02 Tool Misuse	Tool/plugin manipulation	AgentDojo, ASB	Partial
ASI03 Identity & Privilege Abuse	Credential/scope abuse	Few public benchmarks	Frequent blank
ASI04 Agentic Supply Chain	Poisoned MCP/plugins, model supply chain	Emerging suites	Frequent blank
ASI05 Unexpected Code Execution	NL-to-RCE paths	ASB (partial)	Partial
ASI06 Memory & Context Poisoning	RAG/memory poisoning techniques	ASB	Often thin
ASI07 Insecure Inter-Agent Comms	Message spoofing between agents	Multi-agent suites only	Frequent blank
ASI08 Cascading Failures	Pipeline propagation	Rare in benchmarks	Frequent blank
ASI09 Human-Agent Trust Exploitation	Deceptive/persuasive output	Rare in benchmarks	Frequent blank
ASI10 Rogue Agents	Misalignment, concealment	Research-stage only	Frequent blank

The pattern is consistent: off-the-shelf benchmarks crowd into the injection-and-tool rows and thin out toward the multi-agent, supply-chain, and autonomy rows. The mapping turns that pattern from an unknown into a documented backlog.

Which framework should drive your eval harness, ATLAS or ASI?

Use ASI as the spine and ATLAS as the detail. The OWASP ASI list is short, stable, and maps cleanly to “did we test this risk class,” which makes it the right top-level axis for coverage reporting to leadership and customers. MITRE ATLAS then gives each ASI row the technique-level specificity your red team needs to write concrete test cases and to align with broader threat-intelligence and ATT&CK workflows your SOC may already run.

In tooling terms, the UK AI Safety Institute’s Inspect platform has become a common substrate for agent red-teaming, and US/UK AISI red-team exercises have extended AgentDojo on top of it. If you standardize your harness on a platform like Inspect, you can attach ASI and ATLAS tags as metadata on each eval and generate the coverage matrix as a report artifact rather than a manual spreadsheet.

A reasonable operating sequence: define your ten ASI rows, port in the public benchmarks you trust for the rows they cover, tag everything with ATLAS IDs, then write net-new red-team cases targeting the blank rows — privilege abuse, inter-agent spoofing, and rogue-agent behaviors first, since those are both under-benchmarked and high-impact. The same ASI categories anchor the threat-intel walkthrough of the OWASP Top 10 for agentic applications.

What does good evidence look like once it’s mapped?

Good evidence is a coverage statement, not a single percentage. Instead of “our agent scored 8% attack-success-rate,” you report: “We have test coverage across 8 of 10 ASI categories; ASI07 and ASI10 are currently untested and scheduled; within covered categories, attack-success-rate ranges from 3% (ASI01) to 22% (ASI06), with ASI06 flagged for remediation.” That statement survives scrutiny from an auditor, a customer security questionnaire, and your own engineering leadership.

It also creates a defensible cadence. Frameworks move — ATLAS shipped its agentic techniques in late 2025 and kept expanding into 2026; OWASP ASI is peer-reviewed and versioned. When a new ATLAS technique lands, you add a row or a column and see immediately whether your harness already exercises it. The mapping is the thing you maintain; the benchmarks are interchangeable inputs feeding it.

The discipline is unglamorous but decisive: tag every test with the threat it exercises, pivot into a matrix against ATLAS and OWASP ASI, and treat the empty cells as the truth about your exposure. A benchmark tells you how your agent did on the tests you ran. The mapping tells you whether you ran the right tests at all.

Frameworks referenced here — MITRE ATLAS and the OWASP Top 10 for Agentic Applications — are public and evolving; verify the current technique and category lists at their official sources before building compliance claims on them.