Skip to content
Back to blog

Autonomous Penetration Testing: How to Evaluate a System That Chooses Its Next Step

4 min read
Autonomous Penetration Testing: How to Evaluate a System That Chooses Its Next Step

Sources checked: . FailSafe Research.

Autonomous penetration testing describes software that selects its own next action while investigating a target. The label says nothing about scope enforcement, evidence quality or how the system stops. Those are the properties a buyer can evaluate, and they matter more than whether the vendor uses the word autonomous or agentic.

What autonomy actually changes

A scanner runs a fixed set of checks. An autonomous system observes a result, chooses a follow-up and repeats until it reaches a goal or a limit. That flexibility is the value and the risk. The same decision loop that finds a composed attack path can take an action nobody approved if the boundary is not enforced in the system, not only in the brief.

Ask for the controls in the product, not in the policy document. A claim that the tool respects scope is only meaningful if an out-of-scope request is rejected before it is sent.

Evaluation table

Properties to test and the evidence to request
PropertyWhat to testEvidence
Scope enforcementA request to an unlisted host, path or account is blocked before execution.Blocked request, policy decision and the configuration that defines the boundary.
Action logEvery action is recorded with identity, target, time and result, in a form the customer can read.A sample export for one test case, including denied actions.
Stop behaviorA stop command halts further actions within an agreed interval and the halt is demonstrated.Command timestamp, last action before the halt and confirmation that none followed.
Human approvalHigh-impact actions require approval and cannot be self-authorized by the agent.Approval record, the action it covered and what happened when approval was withheld.
Evidence qualityA finding shows the precondition, the action and a reproduced consequence.Finding record, supporting trace and a repeat run.
ReproducibilityThe same case produces a comparable result under the documented conditions.Attempt budget, state reset and result across runs.
Model and version changesA model or policy update is recorded and can trigger a review or retest.Version identifier, change date and the affected assessment.

Distinguish the categories without ranking them

Scanners, autonomous platforms and human-led engagements can overlap in tooling. The question is which one answers your decision. A recurring automated run is useful for repeatable checks and fast feedback. A human engagement is useful for context, ambiguity and specialist methods. An autonomous system can sit between them, and the only way to compare proposals is to give each the same target, identities and evidence criteria.

The method comparison applies one scenario to each approach. It does not rank the categories.

Questions for a procurement review

  1. Where is the target boundary enforced, and can we see it reject an out-of-scope request?
  2. Can we export the full action log, including denied actions, without a vendor intermediary?
  3. Who can stop a run, and how is the stop demonstrated?
  4. Which actions require human approval, and how is that approval bound to the action?
  5. What is the attempt budget for a finding, and how are false positives removed?
  6. How is a model update communicated, and what does it trigger?
  7. What does the retest repeat, and how is the deployed build identified?

Benchmark claims need their conditions

A benchmark result is a measurement under a stated method. Ask for the version, the target set, the attempt budget and the grading method. A score on one benchmark does not establish coverage of your application, and a vendor's own evaluation is not an independent test. FailSafe publishes CVE-Bench method and results with the version and budget, and treats them as version-specific evidence rather than a general capability claim.

Where FailSafe fits

The agentic pentesting guide describes the operating model, including scope, evidence and retesting. The rules-of-engagement guide records the authorization, permitted effects and stop conditions to agree before a run. The agentic pentest buyer guide turns the questions above into a scored worksheet.

Questions & answers

Questions about this topic

Direct answers to common questions about this page.

The terms are usually used for the same mechanism. The useful question is how the system enforces scope, records actions and stops, not which word the vendor prefers.

Only with explicit authorization, enforced scope, action budgets, approval gates and a demonstrated stop path. Many teams begin in staging and permit a narrow production check with agreed limits.

Ask for a test in which an out-of-scope request is rejected before execution, and review the policy configuration and the log of the denial.

No. A benchmark measures performance under its own method and target set. Ask for the version, budget and grading, and treat the result as evidence for that setting.

Put this guidance to work

Agree one authorized target and the evidence you need. FailSafe can demonstrate scope enforcement, logging and stop behavior.

Startup offer: $999 USD for 250 credits and one authorized black-box run. Sign in to SWARM before Stripe checkout. Custom scope and sovereign deployments are separate.