Back to blog

RAG Deletion Testing: Follow the Document Beyond the Index

3 min read
RAG Deletion Testing: Follow the Document Beyond the Index

Sources checked: . FailSafe Research.

Deleting a document from a knowledge base does not tell an assessor what happened to its derived chunks or cached answers. Consider a customer who removes a confidential onboarding document from a connected workspace. A useful deletion assessment follows that document through the retrieval pipeline and distinguishes active copies, retained records, and previously delivered messages. The objective is to verify a documented lifecycle rather than promise that every historical representation can be erased instantly.

Document derivative map

Create a uniquely marked synthetic document and list the systems expected to receive it. Include raw storage, extraction output, embeddings, indexes, answer caches, citation previews, job queues, and documented backups. Ask the owner to specify deletion timing and permitted retention for each component. Record whether the source deletion event is a tombstone, a sync update, or a direct API operation. Select an assessment window long enough to measure the agreed process without pretending it covers an unobserved backup-retention period.

Delete a controlled source document

Confirm the fixture is retrievable, then delete it through the normal authorized workflow. Observe scheduled cleanup, repeated retrieval, citation access, and a query likely to hit a prior answer cache. Check the outcome after a worker retry and an interrupted synchronization if those cases are approved. Use component inventory or logs to inspect derivative deletion; a model failing to repeat the marker is insufficient. Keep previously delivered chat messages outside the deletion claim unless their behavior was explicitly included in the agreed product policy.

Follow every derived copy

Build a small lifecycle record with source identifier, derivative identifiers, deletion-request time, cleanup events, and last active retrieval. Record whether a retained copy is inaccessible, pending expiry, or still queryable. Redact storage locations and avoid embedding confidential document text in the evidence. If the assessment cannot inspect a processor or backup, label it unavailable and cite the operator’s retention statement separately. A failed search alone must not be presented as proof that storage deletion completed.

Recheck stale retrieval paths

Retest with a second fixture after changing invalidation logic and confirm that a legitimate updated document remains usable. Give the engineering owner a regression case covering deletion during a pending ingestion job. The OWASP RAG reference is background for lifecycle concerns; the acceptance criteria come from the organization’s own commitments. This assessment can establish observed behavior across named components and dates. It cannot establish deletion from uninspected backups, third-party model training, or a user’s downloaded copy.

Plan this assessment with FailSafe

Use a staging environment, synthetic records, approved test identities, and an agreed stop condition. Record the model, connector, policy, and application versions so a later result can be compared with the same configuration. Unavailable evidence should remain an explicit coverage gap.

Primary reference: OWASP security guidance. The scenarios and acceptance criteria here are editorial recommendations for adapting that background to an authorized assessment.

For a scoped review of these boundaries, explore FailSafe AI security testing and agentic penetration testing. Agree the specific tools, identities, evidence, and exclusions before purchasing an assessment.

Your document derivative map

Use this compact record in the review. Fill it with synthetic fixtures and the observed result, then retain the configuration with the finding.

Source versionDerived copyDeletion result
Approved control caseRecord the relevant decisionRecord the observed result
Denied boundary caseRecord the expected refusalRecord any deviation and owner

Questions & answers

Questions about this topic

Direct answers to common questions about this page.

Only if that is the entire relevant lifecycle. Check source copies, chunks, caches, previews, processing queues, and documented retention.

No. A retrieval miss or model response is weaker evidence than component-level lifecycle records and access checks.

Put this guidance to work

Bring an AI workflow with its identities and permitted actions. FailSafe can help turn this boundary into an authorized test and a useful remediation record.

Assess this AI-system boundary