What an AI Pentest Report Should Contain

Sources checked: 11 November 2026. FailSafe Research.

A useful AI penetration test report lets an engineer trace each finding from a recorded observation to the control that failed. Buyers can evaluate the deliverable before purchasing by asking what it records, how its conclusions are bounded and who owns each fix. A list of tested categories without that chain is difficult to act on.

What should each finding contain?

Each finding should name the affected boundary, the approved case that produced it, the expected behavior and the observed result. Evidence must be sanitized: no credentials, production data, personal details or working payloads. The report should separate demonstrated impact from inferred impact. One cross-workspace read of a synthetic document does not establish a wider breach. The synthetic finding and retest example shows that chain in full, and the example finding record shows one set of fields you can ask a vendor to produce.

Which scope and configuration details belong in the report?

Model and framework versions, runtime, tool and connector versions, policy configuration, environments, identities used, attempt budget and stopping rules. These details make the result reviewable: a finding against one model version may describe a different system than the one now deployed. The report should reflect the fields agreed during scoping, as described in the scope and evidence worksheet. Exclusions such as training pipelines or third-party services should be stated with the reason they were left out.

Why do outcome labels matter?

Ask whether the report uses bounded outcome labels: observed violation, blocked for the tested condition, inconclusive and outside scope. A refusal in the final answer does not establish that content never reached an earlier processing stage. An inconclusive result caused by missing telemetry or unavailable access deserves its own label rather than being counted as a pass. The table below shows one illustrative entry from a fictional report.

Illustrative finding entry from a fictional report

Field	Illustrative entry
Boundary	Cross-workspace document retrieval in a fictional staging assistant.
Case	Reader A (Workspace A) requests synthetic Document B through the supported search workflow.
Expected result	Denied before content retrieval.
Observed result	Document B returned; restricted text reached the model context.
Outcome label	Observed violation for the tested path and configuration.
Evidence	Sanitized trace reference, caller role and server permission decision. No credentials or real records included.
Owner and retest	Application team; retest pending on the reviewed fix deployment.

What closes a finding?

A closed finding names the remediation owner, the reviewed change, the deployment it was tested on and the result of repeating the original case and the legitimate control. The conclusion stays bounded: resolved for the tested path and configuration. A residual risk statement lists what remains unknown, such as roles, tools or deployments that were not retested. A model swap alone is a configuration change, and the finding should say so.

What separates a useful report from a checklist dump?

A checklist dump lists categories tested without connecting them to observations, evidence or owners. Before purchasing, ask the vendor for a sample or fictional report and check whether findings link to sanitized evidence, whether outcome labels distinguish blocked from inconclusive, whether configuration details are recorded, who owns remediation and how a retest is scoped and reported. The AI pentesting overview describes the assessment sequence those answers should line up with. If the fit is confirmed, review the available offers; the standard run does not automatically include every AI method, and specialist scope is agreed separately.