Can an Agentic Pentest Prove a Bug? A Reproducibility and False-Positive Protocol

Sources checked: 12 November 2026. FailSafe Research.

An agent can describe a plausible vulnerability. Proof requires an observed effect under a stated configuration, and a result another engineer can repeat. This article sets out a protocol for judging an agentic finding. It does not report a new test corpus. The public CVE-Bench v2.1.0 result and the benchmark methods remain the evidence for our published scores.

What counts as proof?

Keep three events separate: a proposed action, an executed action, and an observed impact. A model can propose a tool call without the tool running. A tool can run without reaching the data the finding claims. A finding is reproducible when the same configuration produces the stated effect again, and the evidence shows which of those three events actually occurred.

Name the oracle before the run. A deterministic check, a graded harness and a human reviewer are different oracles. Each can be valid, but the finding should say which one produced the result and what it could not observe.

A six-step protocol

1. Freeze the configuration. Record the target version, model, agent framework, tools, identities, policies and test window.
2. Define the oracle and the success condition. State what evidence would confirm the effect and what would make the result inconclusive.
3. Set the attempt budget and reset rule. Decide how many attempts each case receives and how state is reset between attempts.
4. Run every attempt and record failures. A successful final attempt is weaker evidence if earlier attempts failed and were omitted.
5. Classify the outcome. Use confirmed, inconclusive, false positive, duplicate or out of scope. Keep the rejected findings.
6. Retest the corrected build. Repeat the original case and the legitimate operation that should still work.

What the evidence package should contain

Minimum evidence for a reproducible agentic finding

Record	Why it matters
Target and version	Establishes which build produced the result.
Identity and credential scope	Shows what authority the agent actually held.
Preconditions and approved actions	Separates an authorized test from an unplanned side effect.
Sanitized steps	Lets another engineer repeat the case without publishing a working payload.
Observed effect and oracle result	Connects the action to the claimed impact.
Attempt count and failures	Shows whether the result depended on repeated tries.
Logs or trace references	Supports the timeline and the permission decision.
Limitations	States what the test did not observe or cover.

How attempt budgets change the claim

A pass@1 result and a pass@k result answer different questions. So do two runs with different attempt budgets, different models or different tool sets. A percentage without the budget, target set and grading rule is not comparable to another vendor's percentage. When you review a benchmark, ask for per-case outcomes, the number of attempts, resets between attempts, and whether failed attempts are published. Stochastic runs can produce different outcomes from the same configuration.

A false-positive taxonomy

Common reasons an agentic finding does not hold

Type	What to check
Heuristic match	The agent matched a pattern without confirming the effect.
Narrative assertion	The report describes an impact the evidence does not show.
Environment artifact	A test fixture, mocked service or debug setting produced the result.
Stale state	A previous run left data or permissions that influenced the next attempt.
Duplicate	Two findings describe the same underlying cause.
Out-of-scope effect	The action reached a system or data set the engagement did not authorize.

Record the rejected findings as well as the confirmed ones. A vendor that cannot show a false positive it caught has not shown you its triage process. The synthetic finding and retest record shows how to preserve the original failure and the correction against one reference.

What to ask before accepting a finding

Ask which oracle produced the result, how many attempts the case received, what state was reset, which identity held the authority, and what telemetry could not observe. Ask for the limitation in the vendor's own words. If the answer depends on a downstream service the test could not see, the result is inconclusive, not confirmed.

This protocol is a reviewing aid. It does not certify a finding, a vendor or a benchmark, and it does not replace your own reproduction on the build you operate.