Sources checked: . FailSafe Research.
Model extraction and training-data exposure are different losses. Extraction is about reproducing a model's behavior or function. Exposure is about recovering information from its training data. A test plan that treats them as one item produces a vague result, and a contract that promises to detect either without constraints is overstating what an assessment can show.
Two questions, two methods
| Concern | Question | Evidence a test can produce |
|---|---|---|
| Model extraction | Can an attacker reproduce enough of the model's behavior to substitute for it or avoid paying for it? | Query budget, agreement between the copied behavior and the reference on a defined task set, and the access required. |
| Training-data exposure | Can a user recover memorized content, personal data or secrets that the model should not reproduce? | Canary insertion results on a controlled model, or plausibility probes with a documented false-positive method when no training access exists. |
| Model disclosure | Can an attacker learn architecture, configuration or system instructions that increase other attacks? | Observable responses, error handling and the information they reveal, checked against a baseline. |
These assessments differ in feasibility. Extraction can be measured against a task set and a query budget. Exposure is harder: without knowledge of the training data, a plausible-looking output is not proof of memorization. Say which condition applies to the model you are testing.
What to fix before testing
- Authorization. Confirm that the model owner permits probing at the agreed query volume. Provider terms may restrict extraction or automated querying.
- Query budget. Record the maximum requests, concurrency and time window. Extraction results depend heavily on this number.
- Task set. Choose the tasks used to compare copied behavior with the reference, and keep them fixed across attempts.
- Canaries. If you control training or fine-tuning, insert known synthetic strings and test whether they can be recovered. This gives a clean positive or negative.
- Baseline and false positives. For a model you did not train, decide how you will separate memorization from plausible generation. Without that method, label the result inconclusive.
- Data handling. Do not place real personal data or secrets into prompts to test whether the model repeats them.
What the result should say
A useful extraction result states the query budget, the reference model version, the task set, the observed agreement and the cost of reaching it. It should also say what the attacker would still need: a serving stack, latency, tool access, safety behavior and whatever else the product provides beyond raw model output.
A useful exposure result is more modest. With canaries, it can say whether a known inserted string was recovered under defined conditions. Without canaries, it can describe behavior that suggests memorization, but it should not claim that a specific record was in the training set. Reviewers and regulators may not accept an inference as proof.
Controls that reduce the risk
- Rate limits and per-account query budgets that make large-scale copying costly.
- Monitoring for automated query patterns, with a response plan that does not disrupt legitimate heavy users.
- Output filtering for secrets and personal data, tested with synthetic values.
- Training-data review and deduplication, including removal of credentials and personal data before fine-tuning.
- Clear terms and technical controls on model access, especially for high-value models.
- Versioned model deployment, so a finding can be tied to the build that produced it.
No control proves that a model cannot be copied or that it memorized nothing. The aim is to raise cost, reduce the value of what is recovered and detect the attempt.
Where FailSafe fits
FailSafe assesses AI systems within an authorized scope, including model access controls, output handling and data boundaries. The model supply chain review questions cover checkpoints, adapters and fine-tuning data. The LLM application testing methodology records how test areas and limits are documented, and the AI pentesting scope describes the assessment sequence. FailSafe does not test a third-party model provider's infrastructure without that provider's authorization.
Questions & answers
Questions about this topic
Direct answers to common questions about this page.
No. A test can show whether known canaries are recoverable under defined conditions. It cannot prove absence of memorization across the full training set.
Extraction describes copying behavior through queries. Theft can also involve taking weights, code or configuration. The assessment should state which one it covers.
It depends on the model, task and access. Record the budget used, because a result without the query budget is difficult to compare or reproduce.
Only within that provider's terms and with authorization for the assessment. Provider infrastructure is not in scope for a customer's test.
Related Articles
Fine-Tuning vs RAG for Cybersecurity Models
Separate behavior training from retrieval context when evaluating a cyber model and its assessment workflow....
SBI data exposure investigation: facts and security lessons
What SBI's October 21, 2026 filing says about suspected customer-data exposure, what remains unknown, and lessons for API and access-control testing....
Model Supply Chain Backdoors: Test Checkpoints, Adapters and Fine-Tuning Data
Microsoft's November studies show a model can pass normal evaluations and still carry a trigger. Review the artifact chain before trusting agent behavior....
Put this guidance to work
Describe the model, the access you control and the data you can use for testing. FailSafe can define a bounded assessment and its limits.
Startup offer: $999 USD for 250 credits and one authorized black-box run. Sign in to SWARM before Stripe checkout. Custom scope and sovereign deployments are separate.