Before trusting an AI pentester with your systems, look at what it has actually demonstrated. Our research tests agents against known web vulnerabilities and AI workflows. Explore the results, how each test was scored, and where the limits are.
Zero-day, pass@1
62.5%
25 of 40 targets exploited
One-day, pass@1
70%
28 of 40 targets exploited
Graded by a deterministic oracle. Per-target evidence and traces are published with the report.
Public results
CVE-Bench v2.1.0
62.5% zero-day · 70% one-day
FailSafe SWARM exploited 25 of 40 targets in the zero-day setting and 28 of 40 in the one-day setting at pass@1, graded by a deterministic oracle.
Published report and tracesAttackBench × NEAR
624 adversarial conversations
An adaptive attacker tested twelve model-framework combinations across three agent runtimes, showing that safety is a property of the model and runtime together.
Read the research1
on CVE-Bench v2.1.0
Ranked first in our September 2026 comparison of reported results on real web vulnerabilities. See the scores, test budgets and method.
48h
from vulnerability scanning to code remediation
Deploy agents, build threat models, reproduce findings, connect ITSM and open remediation PRs within 48 hours of agreed access and integration readiness. Your team approves changes.
Local
Own your inferencing
Own the hosting and the inference. A fully local deployment keeps sensitive workloads, model processing and evidence inside your enterprise environment, with no external model calls.
Background guide
Learn how autonomous security testing works, how to compare it with other testing approaches, and what evidence to ask vendors to publish.
Read the agentic pentesting guideMachine-readable research
The CVE-Bench standings are available as JSON so researchers, answer engines, and internal security teams can fetch the same versioned result without scraping a visual page.
Evaluate SWARM on your own application
Benchmark results describe a specific evaluation, not a guarantee for your environment. Startups can assess an authorized target with the one-time $999 USD offer: 250 credits for one agentic black-box run, with a report and remediation guidance.
Review the offer before signing in to SWARM. Custom scope and sovereign deployments are quoted separately.
Sources and scope
Compare benchmark version, target set, access, attempt budget and grading method together. Results from CVE-Bench and AttackBench answer different questions; neither is a guarantee about a customer’s environment.
By FailSafe Research. Editorial update: .
Questions & answers
How FailSafe measures autonomous security systems and publishes the evidence.
FailSafe publishes SWARM results on CVE-Bench and the AttackBench adaptive benchmark developed with NEAR.
On CVE-Bench v2.1.0 at pass@1, FailSafe SWARM reported 62.5% in the zero-day setting and 70% in the one-day setting across 40 targets. The result is versioned, oracle-graded, and published with per-target evidence.
The CVE-Bench report links public per-target results and trajectories. AttackBench publishes a method and results summary; confirm access to its corpus and harness before planning reproduction, because the previously linked repository was unavailable at the November 5 check. Stochastic runs may produce different outcomes.
Benchmarks turn broad claims about AI security into measurable, comparable tasks. They help security teams evaluate an agent's ability to discover and validate vulnerabilities under a defined threat model.