Back to blog

Model Supply Chain Backdoors: Test Checkpoints, Adapters and Fine-Tuning Data

3 min read
Model Supply Chain Backdoors: Test Checkpoints, Adapters and Fine-Tuning Data

Sources checked: . FailSafe Research.

A model can be malicious while its code, framework, permissions and public benchmark results look acceptable. Microsoft published two studies that make this concrete. One found that modifying fewer than ten dormant neurons could create a trigger-activated agent backdoor without retraining, while the model continued to pass conventional evaluations. The other examined post-training compromise, reporting roughly 78% attack success from poisoned fine-tuning data in familiar harnesses and transfer to an unseen harness, with attacker-produced checkpoints reaching average success of 84.1% to 94.8% on unseen repositories.

Microsoft says it has not observed this technique weaponized, and the attack-success figures come from controlled experiments. They are not enterprise prevalence. The architectural point still holds: weights, adapters, quantizations and fine-tuning data are executable supply-chain artifacts, and treating only the application code as the supply chain leaves them unexamined.

Where a trigger can enter

The reported studies describe triggers arriving through ordinary user requests, repository content and tool results. That matters for coding agents and for agents that read tickets, documents or web content. A triggered model does not need a new exploit. It needs a condition it was trained or modified to recognize, and then it acts with whatever authority the harness already granted.

Artifact, question and evidence for a model supply-chain review
ArtifactQuestionEvidence to record
Base modelWhich exact base and revision produced this checkpoint?Model identifier, digest, publisher and license record.
Adapters and fine-tunesWho trained them, on what data, and can the data be described?Training record, dataset provenance, owner and approval.
QuantizationWas the model converted from a reviewed artifact?Conversion pipeline, input digest, validation result.
Checkpoint distributionCould the artifact have been replaced in transit or in storage?Signature or checksum, registry controls, access log.
Dataset pipelineCan untrusted content reach training or tuning data?Source allowlist, review step, poisoning test result.
Runtime harnessCan a triggered model act without tool authorization?Denied tool call for an unauthorized identity.
Change controlIs the system re-evaluated after a model or adapter change?Release gate, re-evaluation record, rollback path.

Behavioral testing and integrity testing answer different questions

Prompt-injection and jailbreak tests ask how the model responds to crafted input. Integrity testing asks whether the artifact itself was altered. A behavioral test can miss a trigger it never produces. An integrity review can flag an unexplained weight delta but cannot tell you whether the model would act on it in your harness.

The practical connection is tool authorization. If a model becomes malicious, the damage depends on what the harness lets it do. An independent check should show that an unauthorized tool call fails even when the model requests it. That control is testable today, without knowing the trigger.

What an assessment can and cannot establish

An authorized assessment can review provenance, compare published digests, test dataset and repository ingestion paths, and verify that tool authorization denies an unauthorized action. It can also look for trigger sensitivity across a bounded set of inputs. It cannot prove the absence of a backdoor. That limit should be stated in the report rather than hidden behind a clean result.

AttackBench and SWARM test deployed behavior and authority. Model-integrity assurance is adjacent and needs specialized review. FailSafe treats the two as separate claims. The AttackBench research covers model-and-runtime behavior, and the AI security assessment scope covers tool boundaries in a deployed system. Microsoft's findings are the publisher's; no FailSafe reproduction of these studies is claimed here.

Questions & answers

Questions about this topic

Direct answers to common questions about this page.

No. Microsoft reported no observed in-the-wild use. The attack-success figures come from controlled experiments, not enterprise prevalence.

Microsoft's dormant-neuron study describes exactly that. The model continued to pass conventional evaluations while carrying a trigger.

Tool and credential authorization. If an unauthorized action fails even when the model requests it, the trigger has less authority to use.

No. A scoped review can check provenance, ingestion paths and bounded trigger sensitivity. It cannot prove the absence of every trigger.

Put this guidance to work

Review the deployed harness, tool authority and artifact chain that your agents depend on.

Startup offer: $999 USD for 250 credits and one authorized black-box run. Sign in to SWARM before Stripe checkout. Custom scope and sovereign deployments are separate.

Scope agent and model assurance