Agentic Pentest Buyer's Guide: 12 Questions Before You Trust an Autonomous Test

Sources checked: 12 November 2026. FailSafe Research.

An agentic pentest platform should be judged on what it can demonstrate under agreed constraints. The presence of a language model or an autonomous loop says little about scope control, evidence quality or the work your team will carry. This guide turns the buying decision into 12 questions, each with an answer to request and an artifact that supports it. Use it beside the agentic penetration testing guide and the platform comparison.

The 12 questions

Evaluation questions, requested answers and supporting evidence

Question	What to request	Evidence that supports the answer
1. What is authorized, and what is excluded?	A written target inventory with environments, identities, excluded dependencies and the permitted test window.	Signed scope, asset list, exclusion list and the named owner of each environment.
2. Who approves actions during a run?	The approval model for destructive, lateral or data-access actions, including who can stop the run.	Named approver role, approval gates, and a stop procedure held outside the agent.
3. How are credentials handled?	The identity model, secret storage, scope per tool, rotation and revocation process.	Token scopes, secret-handling design, and a revocation test against an old credential.
4. What is the attempt budget and retry policy?	The per-case attempt limit, total budget, and how stochastic runs are reported.	Attempt counts per finding and a stated definition of pass@1 or pass@k.
5. How do you reproduce a finding?	A sanitized evidence package that an engineer can follow without guessing.	Target version, identity, preconditions, observed effect, logs, oracle result and limitations.
6. How do you handle false positives?	The classification method and examples of findings the vendor rejected.	A false-positive taxonomy, triage record, and the reviewer who made the call.
7. What happens at an unexpected boundary?	The stop rule and what the agent does when it reaches data or systems outside scope.	A kill-switch test, the response owner, and evidence that the stop worked from outside the agent.
8. Which integrations exist, and what authority do they grant?	The integration list and permissions for repositories, ticketing, ITSM and chat.	Least-privilege scopes and an approval step for write actions.
9. How is target and tenant isolation enforced?	The execution isolation model for each target and for multi-tenant deployments.	Per-target sandbox, egress rules, and a tenant-boundary test with synthetic records.
10. What evidence is retained, where, and for how long?	Data residency, retention, access and deletion terms for traces, findings and reports.	Hosting location, retention schedule, access log, and a deletion procedure.
11. How is remediation retested?	The retest coverage, window and what counts as closed.	The original case repeated on the corrected build, plus the legitimate control that must still work.
12. What are the published limitations?	Benchmark methods, versions, attempt budgets and the caveats attached to each claim.	Per-case results, a versioned report, and explicit limits that match the marketing claim.

Score the answers, not the demo

Use the same three-point scale for every question. Zero means no answer. One means the answer is described in a slide or call. Two means the vendor showed the artifact during a synthetic or redacted review, and your engineer could repeat the check. A platform can score well on detection and still fail procurement if it cannot show credential isolation, data residency or a working stop control.

Scoring worksheet for a shortlist

Area	Question numbers	What a two requires
Scope and control	1, 2, 7	Written scope, named approver, and a stop test performed from outside the agent.
Evidence and quality	5, 6, 11, 12	A repeatable evidence package, a rejected finding, and a retest on a corrected build.
Data and isolation	3, 9, 10	Scoped tokens, a tenant-boundary test, and a retention or deletion record.
Operating model	4, 8	Attempt budgets reported per case and integration permissions agreed in writing.

Run the evaluation in four steps

First, ask each vendor for the written scope and evidence list. Second, run a synthetic proof of concept on a disposable target. The safe agentic proof-of-concept guide describes the constraints. Third, have the engineer who will use the report review the evidence, not only the sales team. Fourth, agree acceptance criteria and a retest before the engagement begins. The RFP evidence questions and the AI security vendor evidence checklist can fill gaps in the shortlist.

What this guide does not do

This is a buying aid from FailSafe, not a certification or a vendor ranking. It does not establish that any platform will find a particular issue in your environment. Keep your legal, data-protection and procurement requirements in the decision, and confirm the scope and evidence terms in writing before testing.