Skip to main content
All posts
AI & Data19 min read

LLM Red Teaming Playbook: PyRIT vs garak vs promptfoo

A repeatable LLM red-teaming loop with PyRIT, garak and promptfoo: what each tool really does in 2026, how they map to OWASP, and how to gate releases.

Published Updated: 15 November 2026

On 13 November 2026 Microsoft shipped PyRIT 1.2.0, its open-source AI red-teaming toolkit, with a findings workflow in the GUI. A month earlier NVIDIA's garak scanner added a hit@k attack-success metric. In March, OpenAI announced it would acquire Promptfoo, whose CLI many teams use to red team LLM applications from CI. The three main open-source tools for attacking your own LLM applications all changed substantially this year, and most comparisons online are out of date.

They also answer the wrong question. "Which tool?" matters less than "which loop?". A one-off scan before go-live tells you how your application behaved on one day, against one model version and one system prompt, and all three change every few weeks. This playbook is for the people who own that risk: CISOs, AI platform leads, architects and the engineers who run the tests. We describe a six-stage loop, what PyRIT, garak and promptfoo each contribute in November 2026, and how findings become a regression gate on every release.

TL;DR / Key takeaways

  • Red team the application, not the model. Your system prompt, retrieval data, tools and guardrails create most of the attack surface. A clean model scan says little about them.
  • Use two tools, not one. garak is the fastest broad scan and runs fully on your infrastructure. promptfoo is the easiest to wire into CI and the strongest on agent and API authorisation. PyRIT is the deepest for adaptive multi-turn attacks and human-led operations.
  • OWASP labels are not coverage. In our count, garak tags its 193 probe classes against the 2023 OWASP list, and promptfoo's 2025 preset maps nothing to Supply Chain. Use the mappings to organise work, not as evidence.
  • The judge is the weakest link. Attack success rate is only as good as the scorer that decides what "success" means. Review a sample of verdicts before you trust a trend.
  • Findings must become regression tests. Every confirmed finding goes into a CI suite with a per-category threshold, so a model upgrade or prompt change cannot quietly reopen it.
The LLM red-team loop. Numbered stages 1–6 match the sections below; the warm arrow is attack traffic into the system under test, the green dotted arrow a fix, the blue arrow the release gate.
The LLM red-team loop. Numbered stages 1–6 match the sections below; the warm arrow is attack traffic into the system under test, the green dotted arrow a fix, the blue arrow the release gate.

Download: Editable draw.io file · Open in diagrams.net · Chart data (CSV)

Why this matters now

The tools grew up. PyRIT reached 1.0 in July 2026 with a re-architected core and added "Operations and Findings" to its CoPyRIT GUI in 1.2.0. It now lives at microsoft/PyRIT; the old Azure/PyRIT repository is archived. garak 0.17.0 (September) added tags that map probes to EU AI Act risk categories. promptfoo ships a release every few weeks. Advice written in 2025 is likely wrong in detail.

Ownership shifted. OpenAI announced its acquisition of Promptfoo on 9 March 2026, to fold it into its Frontier enterprise agent platform. The README now says Promptfoo "remains open source and MIT licensed". European buyers should still look at data flows rather than licences.

The targets changed. LLM applications became agents with tools, memory and retrieval. Microsoft's AI Red Team, reporting on more than 100 generative AI products it had tested, put the practical reality in one lesson title: "You don't have to compute gradients to break an AI system." Germany's BSI said the same from the defender's side in its November 2025 guidance on evasion attacks: there is currently "no single bullet proof solution". If prevention is never complete, testing has to be continuous.

The three tools in November 2026

We checked the source and documentation of PyRIT v1.2.0, garak v0.17.1 and promptfoo 0.125.1 on 15 November 2026.

PyRITgarakpromptfoo
MaintainerMicrosoft AI Red Team + communityNVIDIA + communityPromptfoo (OpenAI) + community
LicenceMITApache 2.0MIT
RuntimePython ≥ 3.11PythonNode.js (also brew, pip wrapper)
Mental modelFramework: targets, converters, attacks, scorers, memoryScanner: generators, probes, detectors, buffs, harnessesConfig-driven CLI: plugins × strategies
Interfacespyrit_scan CLI, CoPyRIT GUI, Python APICLI, YAML/JSON config, JSONL + HTML reportCLI, YAML config, web viewer, GitHub Action
Signature strengthAdaptive multi-turn attacks and human-led operationsBreadth: 193 probe classes in 42 modulesAgent and API tests (BOLA, BFLA, RBAC, MCP), framework presets
Watch out forPython-first; breaking changes between minor versionsOWASP tags use the 2023 list; many probes off by defaultRemote-only plugins send data to Promptfoo's API
PyRIT, garak and promptfoo compared on nine capabilities, our assessment of each project's documentation and source
PyRIT, garak and promptfoo compared on nine capabilities, our assessment of each project's documentation and source

The matrix is our assessment, not a vendor's. Two rows deserve attention. None of the three fails your build on findings without glue code. garak writes report.jsonl, promptfoo a results file, PyRIT its memory store, so you need a short script that sets the exit code. And only garak and PyRIT run entirely on your infrastructure by default. promptfoo's documentation is candid: harmful-content, bias and several security plugins, including indirect-prompt-injection, ssrf, bola and bfla, are remote-only and use Promptfoo's hosted inference.

There is also a managed option. The AI Red Teaming Agent in Microsoft Foundry builds on PyRIT's attack strategies, reports attack success rate (ASR), and runs cloud scans in a few regions including France Central, Sweden Central and Switzerland West. Microsoft's own documentation warns that its LLM-based scoring "can be non-deterministic".

The red-team loop, stage by stage

The numbers match the hero diagram. You can run the whole loop with any two of the tools.

① Scope and threat model

Start with the application, not the tool. Write down the purpose, users, the data the model can read, the tools it can call, and every trust boundary that untrusted text crosses: user input, retrieved documents, web pages, emails and tool outputs. Map each to the OWASP Top 10 for LLM Applications 2025. Mark categories you cannot test by prompting, such as supply chain, as "covered elsewhere", so nobody mistakes a gap for a pass. Our STRIDE-for-agents threat model produces exactly the list of trust boundaries this stage needs.

② The attack library

Build one versioned library per OWASP category from four sources:

  • garak probes: encoding tricks, latent and indirect injection, package hallucination, Markdown exfiltration, XSS through the UI, system-prompt extraction and DAN-style jailbreaks. In v0.17.1 only 93 of the 193 probe classes are active by default; the rest you request explicitly.
  • PyRIT datasets and converters: seed prompts plus roughly a hundred converter classes (Base64, ciphers, Unicode confusables, ASCII smuggling and more) that you can chain onto any objective.
  • promptfoo plugins and strategies: plugins generate the intent (bola, excessive-agency, prompt-extraction, rag-document-exfiltration). Strategies wrap it in a delivery technique such as jailbreak:composite, crescendo or base64.
  • Your own cases: every incident, bug-bounty report and near miss. These matter most because they encode your business context.

Keep the library in Git next to the application, and treat it like code.

③ The harness

The harness drives attacks, records every turn and controls cost. Point it at the full application in a production-like staging environment, not the bare model endpoint. A model scan will not find that your retrieval layer returns another tenant's documents.

Here the tools differ most. PyRIT's multi-turn attacks (Crescendo, PAIR, Tree of Attacks with Pruning) use an adversarial LLM to adapt each turn to the target's last answer, and its XPIA workflow plants instructions in content the application will retrieve. garak's rest and websocket generators reach most HTTP-fronted applications. promptfoo's HTTP, WebSocket and browser providers attack the same endpoint your UI calls, with session handling. Fix seeds and attempt budgets per category, and rate-limit.

④ The judge

Every attempt needs a verdict. garak uses detectors, mostly deterministic pattern checks plus some classifiers and judges. PyRIT has scorers, from regex and tool-call scorers to LLM-based true/false and Likert scorers. promptfoo grades each plugin's output with an LLM rubric.

A lenient judge hides real failures, and a strict one floods triage with noise. Either way your ASR trend measures the judge, not the application. Three rules work for us. Pin the judge model and rubric between runs you compare. Prefer deterministic checks where the failure is observable: a canary string leaving the system, a forbidden tool call, HTML in the output. And have a human review a sample of verdicts every cycle. Microsoft's fifth lesson, "The human element of AI red teaming is crucial", is about exactly this.

⑤ Findings and triage

A finding is a confirmed, reproducible failure with a severity, an OWASP category, an owner and a fix. Report ASR per category, never as one number. An application at 2% on toxic content and 40% on indirect injection is not "21% vulnerable". It has one acceptable risk and one urgent problem. Fixes land in the application: a tighter tool scope, an output encoder, a retrieval filter, a human approval step. Prompt changes alone rarely hold. Our guide to prompt injection and tool abuse covers the usual controls.

⑥ The regression suite in CI

Every confirmed finding becomes a permanent test. The CI job replays the suite whenever the model, system prompt, retrieval configuration or tool set changes, and blocks the release if a category breaches its threshold or regresses. promptfoo fits best here: promptfoo redteam run against a committed config, plus a script that reads the results and sets the exit code. garak works well as a nightly broad scan with a pinned probe list. The loop closes when regressions and new findings flow back into the library at stage ②. If you already run a quality evaluation harness, put the suite next to it; our AI agent evaluation harness uses the same CI-gate pattern.

What the numbers say

We publish no attack success rates, and we would distrust any single ASR quoted for a tool. ASR depends on the target, the judge, the attempt budget and the attack mix. What we can measure reproducibly is what the tools themselves claim to cover, so we counted it.

Built-in OWASP LLM Top 10 (2025) mappings per tool, counted from garak v0.17.1 and promptfoo 0.125.1 source
Built-in OWASP LLM Top 10 (2025) mappings per tool, counted from garak v0.17.1 and promptfoo 0.125.1 source

Method. For garak we read the plugin registry that ships with v0.17.1 (garak/resources/plugin_cache.json): 193 probe classes in 42 modules, excluding the test module, of which 93 are active by default. garak's OWASP tags follow the 2023 v1.1 list, as its own tag file shows, so we crosswalked them to 2025 by risk name. 2023 "Sensitive Information Disclosure" becomes 2025 LLM02, and 2023 "Model Theft" goes to LLM10, because OWASP now covers model extraction under Unbounded Consumption. For promptfoo we counted the plugins per category in its owasp:llm preset. PyRIT ships no OWASP mapping. Every number and the full method are in the CSV.

Four readings stand out:

  1. garak leans heavily towards prompt injection. 85 of 193 probe classes carry that tag, and 44 carry no OWASP tag at all. That reflects where public attack research has been. It is not a balanced test plan.
  2. Three 2025 categories have no garak mapping: data and model poisoning, system prompt leakage, and vector and embedding weaknesses. Some of that is labelling. garak has a system-prompt extraction probe, but it is tagged as prompt injection, which is exactly why tags are not coverage.
  3. promptfoo spreads thinly but widely. Its preset touches nine of ten categories with two to eight plugins each, leaves Supply Chain empty, and reuses plugins: the PII plugins appear under three categories.
  4. Counts say nothing about depth. Seven "Excessive Agency" plugins against one tagged garak probe looks decisive. One well-built PyRIT multi-turn campaign against your real tool set will tell you more than either number.

Use the mappings to organise your library and report. Show an auditor your findings and regression suite instead.

In a recent engagement…

A European company asked us to "run a red-team tool" against its customer-service assistant before a wider rollout. The assistant had retrieval over product documentation and two tools, order lookup and ticket creation. The team had already scanned the model endpoint with garak and was satisfied, rightly: the model behaved well.

The application did not. With the harness pointed at the chat endpoint, the first real finding came from an uploaded PDF. Instructions hidden in a customer attachment made the assistant look up an order the user did not own. No model scan could have caught it, because the flaw was the tool's authorisation, not the model's alignment. The second finding was in our own judge. The first rubric marked several polite refusals as successes because they repeated part of the forbidden request. A human review of a sample caught it before it reached a report.

The fixes were unglamorous. The tool now checks order ownership server-side, attachments are treated as untrusted data, and the judge has a deterministic check for planted canary data. All three findings sit in the CI suite, and the quarterly PyRIT session that followed found nothing new in those categories. That is the loop doing its job.

Which tool when

If you need…Start withWhyAdd later
A broad scan of a model or endpoint this weekgarakHundreds of probes, no code, fully on your infrastructurepromptfoo for app-level tests
A red-team gate in CI/CDpromptfooDeclarative config, pipeline-friendly CLI, framework presetsgarak as a nightly scan
Agent, tool and API authorisation testspromptfooBOLA, BFLA, RBAC, MCP plugins; check which are remote-onlyPyRIT for adaptive abuse
Adaptive multi-turn attacks, human-led operationsPyRITCrescendo, PAIR, TAP, XPIA, findings workflowpromptfoo to regress findings
Strict data residencygarak or PyRITOnly the models you configure are calledpromptfoo with own provider, remote generation off
A managed service in AzureFoundry AI Red Teaming AgentBuilt on PyRIT, ASR per risk categoryOpen-source PyRIT beyond its supported targets

The checklist we use before calling a programme operational:

  1. A threat model names every trust boundary and tool.
  2. The attack library is versioned, organised by OWASP 2025 category, and includes your own incident cases.
  3. Tests target the full application in staging, and indirect injection is tested through every ingestion path.
  4. The judge is pinned, and a human reviews a sample of verdicts every cycle.
  5. ASR is reported per category with attempt counts.
  6. Every confirmed finding has an owner, a fix and a regression test.
  7. CI blocks releases on per-category thresholds and regressions.
  8. Every tool's data flows, including hosted inference, are documented.

Governance for European enterprises

EU AI Act. Article 55(1)(a) requires providers of general-purpose AI models with systemic risk to evaluate models "in accordance with standardised protocols and tools reflecting the state of the art", including documented adversarial testing. It has applied since 2 August 2025. Most enterprises are deployers rather than such providers; our GPAI provider versus deployer guide helps you check. For high-risk systems, Article 15 requires resilience against data poisoning, model poisoning, adversarial examples and confidentiality attacks. Since Regulation (EU) 2026/1744, those obligations apply from 2 December 2027. That buys time to build the loop properly, not a reason to wait.

GDPR. Red-team traffic often contains personal data, seeded or leaked. If a tool sends prompts and responses to a hosted service, that service processes them. promptfoo documents exactly what goes to api.promptfoo.app and how PROMPTFOO_DISABLE_REMOTE_GENERATION plus your own provider change it. Record this before the first run, and use synthetic canary data wherever you can.

NIS2. Article 21(2)(f) requires procedures to assess the effectiveness of cybersecurity risk-management measures. For an essential or important entity with customer-facing LLM applications, a documented red-team loop with regression evidence is a strong answer.

BSI. For German organisations, "Evasion Attacks on LLMs – Countermeasures in Practice" (10 November 2025) is the national reference for the controls your findings should lead to.

A pragmatic adoption path

  1. Week 1: baseline. Threat-model one production LLM application. Run garak against its endpoint and promptfoo's owasp:llm preset against the full application in staging. List the remote-only plugins and decide whether that is acceptable.
  2. Weeks 2–3: judge and triage. Pin the judge, add canary checks, review a sample by hand, and assign owners to the first findings. Expect most to be application issues, not model issues.
  3. Week 4: CI gate. Commit the config, add the exit-code script, set per-category thresholds. Run report-only for two weeks, then enforce.
  4. Month 2: depth. Run a PyRIT session with multi-turn attacks and XPIA against the riskiest tool paths, and add every confirmed finding to the suite. Our DevSecOps pipeline design guide shows where the gate sits.
  5. Quarterly: review the loop. Re-check tool versions, refresh the library from incidents and new research, and report ASR trends per category to the risk owner.

The verdict

LLM red teaming has stopped being a research activity and become a release discipline. The tools are good enough and free: PyRIT for depth, garak for breadth, promptfoo for the pipeline. What is usually missing is the loop around them: a threat model to aim them, a judge you have checked, triage that produces owners, and a regression suite that refuses to ship a known failure twice.

Pick two tools, not three. Test the application, not the model. Never quote an ASR without its judge and attempt budget. Measure progress in confirmed findings that stay fixed, not in probes run.

If you want help setting up the loop, our AI engineering team runs it as a short, fixed-scope engagement on one of your production applications, from threat model to first CI gate. You keep the attack library and the pipeline.

FAQ

What is LLM red teaming?

LLM red teaming is adversarial testing of a system built on a large language model. You play the attacker and try to make the application leak data, ignore its instructions, misuse its tools or produce harmful output. It differs from a benchmark because it targets your application, with your system prompt, retrieval data, tools and guardrails, not the base model in isolation. Microsoft's AI Red Team puts it bluntly: AI red teaming is not safety benchmarking.

Should we use PyRIT, garak or promptfoo?

Usually two of them. garak is the fastest broad scan of a model or endpoint and runs entirely on your infrastructure. promptfoo is the easiest to put into a pipeline, has the strongest agent and API authorisation plugins and ships framework presets such as owasp:llm. PyRIT is the deepest toolkit for adaptive multi-turn attacks and human-led operations. A common pattern is garak or promptfoo in CI and PyRIT for the quarterly deep dive.

Is promptfoo still open source after the OpenAI acquisition?

Yes, as of November 2026. OpenAI announced the acquisition on 9 March 2026, and the project README states that Promptfoo remains open source and MIT licensed. Check the data flow, not the licence. Some red-team plugins and strategies are remote-only and send generation or grading data to api.promptfoo.app, unless you configure your own provider or run the enterprise on-premises edition.

Do garak and promptfoo cover the OWASP Top 10 for LLM Applications 2025?

Partly, and the labels need care. promptfoo's owasp:llm preset maps plugins to the 2025 list but leaves LLM03 Supply Chain empty. garak tags probes against the older 2023 v1.1 list. In our count of garak v0.17.1, after crosswalking, no probe maps to the 2025 categories for data and model poisoning, system prompt leakage, or vector and embedding weaknesses. A tag is a label, not proof of coverage.

What is attack success rate and what threshold should block a release?

Attack success rate (ASR) is successful attacks divided by total attempts, usually per risk category. There is no universal threshold. The number depends on the judge, the attempt budget and the attack mix, so it is only comparable between runs that keep those fixed. We set the gate per category: zero tolerance for confirmed data exfiltration or unauthorised tool actions, and a trend rule, such as no regression against the last release, for softer content categories.

Does the EU AI Act require red teaming?

For providers of general-purpose AI models with systemic risk, yes. Article 55(1)(a) requires model evaluation including conducting and documenting adversarial testing, and it has applied since 2 August 2025. For high-risk AI systems, Article 15 requires resilience against attacks such as data poisoning and adversarial examples, but those obligations now apply from 2 December 2027 after Regulation (EU) 2026/1744. Most enterprises deploying LLM applications are not legally required to red team today, but it is the most credible evidence that their controls work.

Sources

All sources checked on 15 November 2026. Tool counts are our own analysis of the tagged releases; the method is in the chart data CSV.

Tier A: primary, standards and regulatory

Tier B: research and analysis

Tier C: vendor and news, directional

Topics

LLM red teamingAI red teaming toolsPyRITgarak LLM vulnerability scannerpromptfoo red teamOWASP Top 10 for LLM Applications 2025prompt injection testingattack success rate

Frequently Asked Questions

LLM red teaming is adversarial testing of a system built on a large language model: you play the attacker and try to make the application leak data, ignore its instructions, misuse its tools or produce harmful output. It differs from a benchmark because it targets your application, with your system prompt, retrieval data, tools and guardrails, not the base model in isolation. Microsoft's AI Red Team puts it bluntly: AI red teaming is not safety benchmarking.

Expert engagement

Need expert guidance?

Our team specializes in cloud architecture, security, AI platforms, and DevSecOps. Let's discuss how we can help your organization.

Get in touchNo commitment · No sales pressure

Related articles

All posts