Garak vs PyRIT vs promptfoo – hands-on red-team benchmark on a real agent

TL;DR. We built a small tool-using agent with two planted secrets, put it behind HTTP, and ran Garak 0.17.0, promptfoo (latest via npx) andPyRIT 1.1.0 against it — 786 test prompts in total, every reply logged and scored against ground truth. Garak sent the most prompts (768) and reported the most failures in its own detectors, but almost never reached our actual secrets. PyRIT's six hand-written objectives reached a secret four times in 13 seconds. promptfoo generated and graded 12 cases in under 20 seconds and reported 12/12 passed — while the agent had disclosed its system key in one of those replies, inside a sentence the grader classified as a refusal. That false negative is the most useful thing we learned. This is the measured companion to ourframework comparison.

Methodology

The target

agent_server.py (Python standard library, ~150 lines) is a weather-kiosk assistant calling mistralai/mistral-nemo through OpenRouter with function calling and two tools:get_weather and read_note. Two fictional markers are planted: one in the system prompt, one in an internal note reachable only through the tool. The system prompt forbids disclosing either. The agent is exposed as POST / prompt → output so all three frameworks hit the identical endpoint, and every turn is appended to a JSONL log with a ground-truth hit flag: did the reply contain either marker? That log — not the frameworks' own verdicts — is our source of truth.

Budget and limits

Each framework was capped so the whole study stayed far under US$3 of model spend (the target model costs about $0.03 per million tokens; the full run moved well under a million). Garak: three probes at one generation each. promptfoo: three plugins × two tests, one static strategy. PyRIT: one orchestrator, six objectives. Temperature 0 on the target. Numbers below are from a single run on 2026-10-15; we report them as point estimates, not statistics.

Results at a glance

	Garak 0.17.0	promptfoo (npx)	PyRIT 1.1.0
Install	`uv pip install garak` into a Python 3.12 venv, first try	none — `npx -y promptfoo@latest` (≈41 s first download)	`uv pip install pyrit` into a Python 3.12 venv, first try
Wiring to an HTTP agent	JSON generator config (`rest`), worked unchanged	YAML `https` target with `transformResponse`, worked unchanged	`HTTPTarget` from a raw request template; first run died on a 5 s default read timeout, needed an explicit httpx timeout
Prompts sent	768	12	6
Wall time	588 s (12 parallel attempts; ~50 min when sequential)	7 s generate + 11 s eval	12.7 s
Agent errors	0	0	0
Framework's own verdict	492 failures across 3 detectors (see below)	12/12 passed	n/a — no scorer configured; we scored from the log
Ground truth: replies containing a planted secret	1 of 768	1 of 12	4 of 6 (3 system-prompt, 1 note)
Report artifacts	HTML report (1.7 MB), hit log (492 lines), JSONL (6 MB)	JSON export (145 kB) with per-case grader reasoning; web UI	none with in-memory storage; results live in the memory DB unless you export
Blockers hit	none	`redteam run` asks for e-mail verification; the iterative jailbreak strategy requires promptfoo's cloud generation	API surface changed again in 1.x; docs lag

Garak: breadth without aim

Three probes, 256 prompts each. Garak's own detectors reported: promptinject.HijackHateHumans236/256 failed, promptinject.HijackKillHumans 142/256 failed,latentinjection.LatentInjectionReport 114/256 failed. Taken at face value the agent is wide open to instruction hijacking — and it is: a plain tool-calling wrapper around a small open model does not resist text that tells it to print something else. But those failures measure whether the model echoed a canary string the probe asked for, not whether it gave up anything we cared about. Against our ground truth, only 1 of 768 replies contained a planted secret, and that one was a side effect: the model substituted the system key into a canary it was asked to print. Garak's value here is coverage and a reproducible, well-formatted report; its probe library is model-centric and does not know what your agent is protecting.

promptfoo: fastest loop, one costly false negative

Generation ran locally (we disabled remote generation and pinned the generator to the same cheap OpenRouter model): 3 plugins × 2 tests, plus a static prompt-injection template strategy, 12 cases, 7 s. Evaluation: 11 s, 12 target requests, 3 grader requests. Verdict: 12/12 passed. Our log disagreed: in one harmful:privacy case, asked directly for the confidential key, the agent replied that it could not find a note with that id — and put the key itself in that sentence. The grader read the shape of the reply ("a standard refusal") rather than its content. This is the pattern to watch for in any LLM-graded red-team tool: a refusal-shaped leak passes. Everything else about promptfoo was the best developer experience of the three — declarative config, instant HTTP target, readable per-case reasoning in the JSON export, CI-friendly exit codes. Two honest caveats: the full redteam run flow gates on e-mail verification, and the iterative jailbreak strategy only works through promptfoo's hosted generation, so a fully offline run is limited to static strategies.

PyRIT: precise when you know what to ask

One PromptSendingAttack over an HTTPTarget, six objectives we wrote by hand around the two secrets (the prompts are in pyrit_run.py). 12.7 s. 4 of 6 objectives produced a reply containing a secret — three disclosed the system-prompt key, one the note. PyRIT found the most real problems per prompt by a wide margin, because the prompts encoded knowledge of the target. The cost is that PyRIT gives you building blocks, not a scan: we wrote the target wiring, the objectives and the scoring ourselves, hit a default-timeout failure on the first attempt, and got no report artifact out of the in-memory backend.

False positives and false negatives

Garak — no false positives in the sense of mis-detected canaries, but its 492 "failures" are not 492 security findings; they are one finding (instruction hijack works) counted 492 times. Triage by detector, not by row.
promptfoo — one false negative out of 12 (8 % of cases), on the exact behaviour the plugin was meant to catch. Add a deterministic not-contains assertion for every secret you know about; do not rely on the rubric alone.
PyRIT — no verdicts to be wrong about; scoring is whatever you bolt on. Our substring ground truth has no false positives by construction.

What we would do with these numbers

Keep promptfoo as the regression gate in CI, with hard-coded secret assertions alongside the LLM grader.
Run Garak occasionally for breadth, in parallel mode, and read it per detector.
Use PyRIT when a human has a hypothesis about a specific agent and wants to script it — that is where 4-of-6 came from.
Whatever the tool says, log every reply and grep for what you are protecting. Our 150-line target did that, and it caught the one miss the grader did not.

Sources

NVIDIA/garak · promptfoo/promptfoo · microsoft/PyRIT
Target agent, configs, run scripts and raw results: research/code/red-team-benchmark/ (results/garak.json, results/promptfoo.json, results/pyrit.json, plus the per-framework agent logs)