Tools to test AI agent safety in 2026: garak, promptfoo, mcp-scan, AIRQ, Sloopt and others — what each actually measures

Four kinds of tool, four questions

Kind	Question it answers	Typical user	Examples
Probe library / scanner	Is this model or agent attackable, and how?	Security engineer, red team	NVIDIA Garak, PyRIT, AgentDojo (benchmark)
CI red-teaming / evals	Did this release regress on the attacks we care about?	Agent developers	promptfoo, DeepEval and similar eval frameworks
Static scanner	Is this code, skill or MCP configuration unsafe?	Developers, platform teams	mcp-scan and other MCP scanners, agent-audit tools, skill file scanners
Independent grader / rating	Can I show someone else a result they will accept?	Buyers, vendors under review, auditors, marketplaces	AIRQ (rating framework and report), Sloopt (per-skill and per-agent A–F with evidence)

The common mistake is to pick one tool and expect it to answer all four questions. A probe library will not tell a buyer anything; an independent grade will not run in your CI.

The tools, one by one

NVIDIA Garak — open-source LLM vulnerability scanner with a large probe library (injection, jailbreak, leakage, toxicity and more) and detectors that judge the replies. Strong for finding whether a model or endpoint is attackable. It does not grade, does not know your agent's purpose, and leaves scope, tool-misuse and secrets to you. Sloopt uses Garak as the live-probe layer inside Guard.

promptfoo — open-source evals and red-teaming configured in YAML, built to run in CI. Good for regression: define the attacks, run them on every change, fail the build on a regression. It measures what you configure; the quality of the result is the quality of your attack set and your graders. Produces reports for your team, not an independent result for a third party.

AgentDojo (NeurIPS 2024) — a benchmark environment for prompt-injection attacks and defences on tool-using agents. Excellent for research comparisons and for understanding attack classes; not a product you point at your own agent.

MCP scanners (mcp-scan and others) — static and sometimes runtime checks of Model Context Protocol servers and configurations: tool poisoning (hidden instructions in tool descriptions), unauthenticated servers, cross-server shadowing. Narrow and useful. They do not test the agent's behaviour.

Skill file scanners (score-out-of-100 tools) — upload a skill package, get a score and verdict in under a minute from static patterns. A reasonable first filter for marketplaces and installers. No behavioural test, no run counts, no evidence trail beyond the pattern match. In Sloopt's blind grading of 47 public skills, the one F was found only by the live attack; the static scan of that skill was clean — the limitation of this class in one sentence.

Runtime guardrails (prompt firewalls, input/output filters) — not testing tools, but often bought instead of them. They reduce attack success in production; they do not tell you your agent's baseline, and they need testing themselves.

AIRQ (Adversa AI) — an open framework and periodic report that scores commercial AI agents on attack surface, blast radius and defence controls, from public evidence, and places them in a quadrant. The 2026 Q2 edition assessed 100 agents: 98% exposed the "lethal trifecta" (private data access, untrusted content, outbound actions), 11% qualified as "Fortified Leaders", 83% of vendors' defence claims could not be independently verified. Valuable as market context and as a procurement lens. It rates products from the outside; it does not run attacks against your agent or skill, and it is not something a small vendor can commission for their own product.

Sloopt — independent A–F grades for individual skills (Rate) and deployed agents (Guard), from real engine runs. Rate: seven behavioural controls, injection probes run 8 times free and 32 times for certification, worst finding sets the letter, every finding quotes probe and reply, result published on a verification page with a badge. Guard: read-only repo access plus live probing of the running agent (Garak), static scanners, inventory of tools and MCP servers, findings with file, line, fix and the OWASP / MITRE ATLAS / NIST clause. Published accuracy: weighted κ 0.68 against 23 expert-labelled skills (Sept 2026). What it does not do: run inside your CI; assess inter-agent communication, cascading failures or persistent-memory poisoning (marked "not assessed" on reports); or certify legal compliance.

Side by side

	Garak	promptfoo	MCP scanners	Skill scanners	AIRQ	Sloopt
Live behavioural attacks	Yes	Yes (configured)	Some	No	No (public evidence)	Yes
Repeated runs with a statistical bound	Manual	Manual	No	No	No	Yes (8 / 32)
Static scan of skill / config	No	No	Yes (MCP)	Yes	No	Yes
Knows the agent's purpose (scope check)	No	If configured	No	Partial	Partial	Yes
Secrets in files	No	No	Partial	Yes	No	Yes (automatic F)
Grade a third party will accept	No	No	No	Score only	Yes (listed products)	Yes (any skill or agent)
Evidence per finding (probe, reply, fix)	Raw logs	Reports	Findings	Pattern hits	Report	Yes
Standards mapping	Partial	Partial	Partial	No	Own framework	OWASP, ATLAS, NIST, EU AI Act
Runs in CI	Yes	Yes	Yes	API varies	No	No (re-scan on release via subscription)
Price	Free (OSS)	Free (OSS) + paid	Free (OSS)	Free tiers	Free report	Free check · $79 · $29/mo · Guard from $500

"Yes" rows are as described by each project; verify against current documentation before buying.

A sensible stack for 2026

• Building an agent: promptfoo (or garak) in CI with a fixed attack set → an MCP/config scanner on every dependency change → a Sloopt Guard report at each release that a customer will see.
• Installing skills: read the file, run a static scanner, then a behavioural grade before anything that touches money or data. Free at /rate.
• Buying an agent: ask the vendor for a dated independent report with run counts; read AIRQ for the market baseline; use the OWASP agentic checklist line by line.