Buying an AI Agent Pentest: Permissions, Tools, and Evidence

Sources checked: 29 October 2026. This article contains FailSafe's analysis and proposed validation workflow; linked sources support the reported facts.

Procure against the deployed capability

An AI agent’s risk depends on what it can access and do, not only on the model name. Our procurement worksheet lists the application, tools, data sources, identities, permissions, and human approvals involved in a defined workflow. Ask which of these components the proposed assessment covers. A prompt-only demonstration may be useful for one question while leaving tool authorization, tenant isolation, or consequential actions outside the tested boundary.

Describe the permitted workflow

Choose representative tasks and explain the intended result, allowed resources, and prohibited actions. Identify how external content enters the workflow and which actions require confirmation. Use synthetic records and non-production destinations where feasible. If the agent can send messages, modify infrastructure, or trigger payments, define a safe substitute or bounded test condition. The buyer should not grant unrestricted access merely to create an impressive demonstration of capability.

Ask for evidence of a security property

Useful assessment objectives concern observable boundaries: whether a role can access an authorized dataset, whether an approval is enforced, or whether a tool action respects the intended scope. Distinguish an undesirable generated answer from a demonstrated security consequence. Ask the provider to record prerequisites, configuration, outcome, and limitations. A collection of provocative transcripts alone may not give engineers enough information to identify the responsible component or verify a corrective change.

Separate product controls and engagement safeguards

The application may have permission checks, monitoring, and human approval, while the assessment has its own authorization, stop conditions, and evidence rules. Review both. Ask who can pause testing and who validates important observations. Confirm handling of prompts, logs, connected data, and model-provider processing. If a sensitive integration is simulated, say so in the final report. The simulation can support a bounded conclusion without proving identical behavior in a live downstream service.

Make the delivery useful to engineering

Require findings linked to affected components and expected security properties, plus actionable remediation and a defined retest objective. Ask for a coverage statement listing tools and roles actually investigated, blocked cases, and configuration changes. A model update or permission change can alter the system after the assessment, so record the tested version. Select the provider on scope, safety, evidence, and operational fit; do not infer full AI-system assurance from a general pentesting benchmark score.

Ask the application owner to review the proposed evidence example before the assessment begins. This can reveal that a dramatic generated response is irrelevant to the deployed workflow, while an ordinary tool action crosses a meaningful permission boundary.

Reference and next step

The NIST AI RMF offers broader AI-risk context. The purchasing worksheet above is our original scope-and-evidence framework for a security engagement.

Review FailSafe SWARM for the proposed testing model. Businesses considering a focused first engagement can review the first-pentest offer or discuss scope and evidence requirements before purchasing.