You Now Have a Model Supply Chain
By Maplecode
Software supply chain security has matured into ordinary practice: pinned versions, lockfiles, signature verification, a bill of materials, automated alerts on known vulnerabilities.
AI components arrive through a different door. Model weights pulled from a public hub, a dataset downloaded from a link in a paper, an embedding model chosen in a notebook and never revisited. Same production system, considerably less scrutiny.
Weights are executable in practice
A model file is data, but the loading code around it frequently is not. Serialisation formats that reconstruct arbitrary objects — the Python pickle family in particular — execute code on load. A crafted checkpoint from an untrusted source is a remote code execution vector with a friendly filename.
The mitigation is unglamorous. Prefer formats designed not to execute on load. Verify checksums against a source you trust rather than the one that served the file. Pin to a specific revision rather than a moving tag, because a hub reference without a revision is the moral equivalent of latest.
Load untrusted weights in an isolated environment first, with no credentials and no network path to anything that matters.
Behaviour changes without a version bump
The dependency risk unique to this domain is silent behavioural drift.
A hosted model behind a stable endpoint name can change underneath you. Your code is identical, your prompts are identical, and the outputs shift — sometimes better, sometimes not, occasionally in a way that breaks a downstream parser or a classification threshold you tuned.
Nothing in your dependency tooling notices, because nothing in your dependency tree changed. The only defence is an evaluation set you run on a schedule rather than only on deploy, with alerting on the metrics that matter to you. Teams without this discover drift through a customer complaint.
Where the provider offers pinned model versions, use them, and treat upgrading as a change requiring evaluation — because that is what it is.
Prompts and templates are dependencies too
Prompt templates, few-shot examples, evaluation sets and retrieval configurations all determine system behaviour as directly as code, and they are frequently managed worse: edited in a console, stored in a spreadsheet, or hardcoded and copied between services.
Put them in version control, review changes, and tie each production deployment to a specific revision. When output quality regresses, the first question is what changed, and "someone edited the prompt in the UI on Tuesday" is not an answer you can act on.
Extend the bill of materials
If you produce an SBOM, it probably does not list your models. It should: model identity and revision, the provider, the licence, where inference runs, and what data is sent there.
The licence line matters more than teams expect. Open-weight models carry a wide range of terms — some restrict commercial use, some restrict use by organisations above a size threshold, some impose conditions on derivative models. "It was on a public hub" is not a licence grant, and this is exactly the class of assumption that produces an expensive discovery later.
The failure mode: the notebook that shipped
The characteristic incident is not an attack. It is a prototype promoted to production with its dependencies intact: an unpinned model reference, a dataset from a personal drive, a prompt in a string literal, credentials in an environment variable, no evaluation.
It works, so it stays. Six months later nobody can reproduce how it was built, which model version it was tuned against, or whether the licence permits what the company is now doing with it.
The cheap fix is a promotion checklist that runs before anything AI-shaped reaches production. The expensive version is an archaeology project.
Where we would push back
We would not accept an unpinned model reference in a production path, in the same way we would not accept an unpinned package. And we would not deploy a system whose behaviour cannot be evaluated on demand, because without that you cannot tell drift from a bug.
We would also resist self-hosting a model purely to feel safer. Self-hosting moves the risk rather than removing it — you now own patching, isolation and the loading path — and the decision should follow from data residency or cost, not from a general sense that local is safer.
Your retrieval corpus is an untrusted input
One more surface, and the newest: if a system retrieves documents and feeds them to a model, whoever can write into that corpus can influence its behaviour.
In practice the corpus is often a wiki, a ticket system or a shared drive where write access is broad because it always was. That access was granted on the assumption humans would read the content and apply judgement. Once a model reads it and acts on it, the same access becomes a path to steering an automated process.
Treat the corpus as part of the trust boundary. Know who can write to it, keep an audit trail of what changed, and prefer sources with review before publication for anything the system will act on rather than merely quote.
A minimum standard
Pinned revisions for every model. Checksums verified. Non-executing weight formats where available. Prompts and eval sets in version control, tied to deployments. Scheduled evaluation with alerting. Licences recorded alongside versions. And a promotion checklist that a prototype must pass before it carries production traffic.
Worth adding one review gate: whenever someone proposes a new model, dataset or hub dependency, the same questions a package would face. Who publishes it, under what licence, pinned to what revision, verified how, and what happens to our evaluation numbers if it changes. Four questions, asked once, at the point the dependency enters the codebase rather than at the point it breaks something.
Related: security engineering and zero trust.
