LLM Application Penetration Testing: Test Cases and Evidence

Sources checked: 13 November 2026. FailSafe Research.

An LLM application penetration test examines the deployed system around the model, not the model in isolation. The application authenticates users, retrieves documents, calls tools and writes output. Each of those layers can create a security decision that a prompt alone cannot enforce. The test should state which layer is expected to make each decision, then try to cross it with synthetic data.

Scope the system, not just the model

Record the application, model and connector versions, the retrieval sources, the tools and their credentials, the user roles, and the data boundary. Include the provider or self-hosted runtime, because data handling and failure behavior differ. The LLM scope and evidence worksheet lists the minimum record. Agree what is excluded before active testing.

Test areas and the evidence each produces

The areas below can be covered depending on the architecture and agreed scope. Each case should produce an observation that another engineer can follow.

LLM application test areas, questions and evidence

Area	Question	Evidence to record
Input handling	Can untrusted text change an instruction, a tool call or a data scope?	Input source, proposed action, downstream stage reached, control case.
Retrieval and context	Is retrieved content labeled as external, and can the test show whether it entered context?	Retrieval record, labeling state, context telemetry or an inconclusive label.
Tool authorization	Which caller and credential may invoke each tool, on which object?	Caller, credential scope, server decision for an allowed and a denied pairing.
Output handling	Where does model output travel, and what renders or executes it?	Downstream sink, sanitization result, allowed rendering control.
Identity and tenancy	Can one user or workspace reach another's data through any route?	Authenticated caller, resource owner, denied case, canary result.
Secrets and logs	Do credentials stay out of prompts, traces and logs, and can decisions be reconstructed?	Token scope, redaction check, decision log coverage.
Unsafe side effects	Which actions require approval, and what stops an unexpected action?	Approval gate, stop test, permitted control action.

Write a case record that survives review

Use a stable case identifier. Record the version and configuration digest, the identity, the synthetic fixture, the proposed action, the executed action, the observed effect and the limitation. Keep a proposed tool call separate from an executed action. If telemetry cannot show whether restricted content reached the model, record that gap as inconclusive. The AI pentest report guide describes the wider deliverable set.

Handle model nondeterminism honestly

State the attempt budget before testing. Repeat a case from a fresh state so a previous run does not supply the result. Use a positive control that must work, such as an allowed retrieval or an authorized tool call. A result found once is an observation, not a pass rate. Do not compare runs with different models, prompts, budgets or versions as if they were the same test.

Retest the corrected build

After a reviewed fix deploys, repeat the original case, the legitimate control and the approved alternate paths on the recorded build. State which variants remain untested. A new model, connector, tool definition or policy can change the boundary even when the application code is unchanged.

What the result does not establish

A scoped test does not certify the application or guarantee that no other path exists. It reports what was tested, under which configuration, and what the evidence supports. Teams that need a scoped engagement can review the AI and LLM assessment scope or discuss requirements with FailSafe.