FailSafe is a cyber frontier lab benchmarking autonomous security systems against real vulnerabilities and real agent runtimes. Results are versioned, oracle-graded where applicable, and published for inspection.
Zero-day, pass@1
62.5%
25 of 40 targets exploited
One-day, pass@1
70%
28 of 40 targets exploited
Graded by a deterministic oracle. Per-target evidence and traces are published with the report.
Public results
CVE-Bench v2.1.0
62.5% zero-day · 70% one-day
FailSafe SWARM exploited 25 of 40 targets in the zero-day setting and 28 of 40 in the one-day setting at pass@1, graded by a deterministic oracle.
Published report and tracesAttackBench × NEAR
624 hostile exchanges
An adaptive attacker tested twelve model-framework combinations across three agent runtimes, showing that safety is a property of the model and runtime together.
Read the researchEVMbench
69.2% recall · 83/120
SWARM's published EVMbench result measures recall across 120 vulnerabilities and compares the result with other capable models.
Read the results1
on CVE-Bench
62.5% zero-day and 70% one-day at pass@1.
82.5%
reduction in noise
Compared with competing tools.
97.5%
of highs and criticals fixed
Patched automatically, then retested.
Machine-readable research
The CVE-Bench standings are available as JSON so researchers, answer engines, and internal security teams can fetch the same versioned result without scraping a visual page.
Questions & answers
How FailSafe measures autonomous security systems and publishes the evidence.
FailSafe publishes SWARM results on CVE-Bench, the AttackBench adaptive benchmark developed with NEAR, and EVMbench results for autonomous vulnerability discovery in EVM systems.
On CVE-Bench v2.1.0 at pass@1, FailSafe SWARM reported 62.5% in the zero-day setting and 70% in the one-day setting across 40 targets. The result is versioned, oracle-graded, and published with per-target evidence.
The benchmark methods, graded results, and evidence are public. Because autonomous systems are stochastic, the published repositories describe the method and evidence rather than guaranteeing an identical score on every future run.
Benchmarks turn broad claims about AI security into measurable, comparable tasks. They help security teams evaluate an agent's ability to discover and validate vulnerabilities under a defined threat model.