LLM04: Supply Chain — Where Does Trust Break Down in the LLM Supply Chain?

Part 7 of the AI Pentesting series. Previous: How Much Authority Did You Give the Model? LLM03: Excessive Agency

Building an LLM application often starts with assembling existing components. A base model is downloaded from a platform. A LoRA adapter published by someone else is added. The required Python packages are installed. The application is deployed to production in a container.

This accelerates development, but it also makes the application’s security depend on people, accounts, repositories and processes outside the organization’s direct control. Supply chain security examines how these external dependencies enter a system and what evidence supports the decision to trust them.

In the LLM ecosystem the chain is long: model weights, training data, adapters, tokenizer files, custom loading code, software libraries and deployment environments are all part of it. A problem in any one component can reach the application serving end users. OWASP therefore treats the LLM supply chain as a security concern that explicitly includes third-party models and data (LLM04: Supply Chain).

This article examines LLM supply chain security through four examples: Pickle deserialization, typosquatting with similar model names, LoRA adapters and unverified dependencies. It then uses a small code example to observe loading behavior and discusses mitigation, version management, model signing and SBOM/AIBOM inventories.

Key takeaways

“The model we use” is really a bundle of components (base model, adapters, tokenizer, loader code, dependencies), each a separate trust decision.
Loading a model can execute code: Pickle deserialization is a trust boundary, and even weights_only=True was bypassable in affected PyTorch versions (CVE-2025-32434).
A trusted name or polished model card does not verify a publisher; typosquatting and poisoned adapters exploit exactly that assumption.
Defense means provenance, version pinning, safer formats, signing, an AIBOM inventory and behavioral testing, not any single control.

What Does the LLM Supply Chain Include?

Thinking of the supply chain as only the website a model is downloaded from leaves out most of the picture. Before a model answers in production, it may pass through several stages controlled by different teams: a research team publishes the base model, another adapts it with a dataset, a third converts it to a smaller format, and the organization runs the result on its own infrastructure. Each stage introduces new files, dependencies and trust decisions.

What a developer calls “the model we use” is often a combination of components:

The base model provides the main learned parameters. Publisher identity, origin and the exact revision all matter.
Datasets influence behavior during training and adaptation; their sources and modification history should be evaluated.
LoRA adapters adapt the base model to a task. They may come from a different source and need their own review.
Tokenizer and configuration files determine how text is processed; the files used in testing should match production.
Loading code brings the model into memory; the code executed and the permissions it receives are critical.
Software dependencies and the deployment package provide the environment everything runs in.

Trusting a model solely because of its name or hosting platform is insufficient. Even when the publisher is legitimate, the wrong file may be selected; even when the file is correct, the loader may be vulnerable; even when loading looks clean, the model’s behavior may have been deliberately altered. The central question:

How do we verify where a component came from, what happened to it, and which exact version is running in production?

LLM04: Supply Chain — Where Does Trust Break Down in the LLM Supply Chain?

How Does the Chain of Trust Break?

Consider an organization building an assistant that answers questions about internal documents. The team picks a well-known base model and adds a third-party adapter to improve Turkish performance. Initial tests look good, and the app goes to production. But the process has gaps:

The adapter is downloaded from a moving branch instead of a specific commit.
The base model is reviewed, but the adapter’s origin is not recorded.
The model loader runs in an environment with access to production credentials.
Evaluation covers only normal usage examples.
File hashes and previous versions are not retained.

Under these conditions, a compromised publisher account or a modified file could affect a later deployment, and an investigation becomes hard: Which server downloaded which file? When did the change happen? Which other applications use the same adapter? Where is the previously verified package? Supply chain security helps prevent attacks and makes these questions answerable.

Example 1: Code Execution Through Pickle Deserialization

Python’s Pickle module converts objects into a byte stream (serialization) and reconstructs them later (deserialization). Its security-sensitive property is that reconstruction can involve function calls, so maliciously crafted data can execute attacker-chosen code during loading. Python’s documentation explicitly warns against unpickling data from untrusted sources (Pickle security warning).

This matters for the model supply chain: a developer may believe they are simply loading a model into memory while the loading process also performs an operation chosen by an attacker, creating a risk of remote code execution (RCE). The impact depends on the loader’s permissions; if that process can reach sensitive files or credentials, consequences are more serious. JFrog’s 2024 research examined models shared on Hugging Face that executed malicious code during loading (JFrog research). The supply chain failure occurs when an externally obtained file becomes an executable component without sufficient verification. The model does not have to generate a response for the attack to begin.

Example 2: Typosquatting Redirects Users to the Wrong Source

Before reviewing a model’s security, we must confirm we selected the intended model. Typosquatting uses a name similar enough to a trusted one that users mistake it for the legitimate source. In the model ecosystem, the similarity can appear in the model name or the publisher’s account name; a missing letter or an imitation of a familiar organization can be overlooked in a fast-moving project. This is the model-registry cousin of the package typosquatting developers already know.

Mithril Security’s PoisonGPT study is a research example: the researchers modified GPT-J-6B to produce false information about a specific subject and used the account name EleuterAI, resembling the legitimate EleutherAI. They also showed targeted incorrect behavior could be introduced without a substantial drop in some general evaluation results. This was a research demonstration of distribution through a similar account name, not a compromise of the real publisher’s account (PoisonGPT study).

The chain of trust breaks when the wrong publisher is accepted as the intended one. A polished model card, professional descriptions or a familiar model name does not verify identity; what needs verifying is the complete repository identity and its relationship to the publisher’s official sources.

Example 3: A LoRA Adapter Changes the Trust Boundary

LoRA adapts a large model to a task without retraining the whole model, and an adapter can be distributed separately from the base model; PEFT checkpoints store adapter weights and configuration as separate components (PEFT checkpoint format). This eases sharing but adds a separate trust decision.

A small adapter file can have significant security impact, because its purpose is to change the model’s behavior. A maliciously prepared adapter may behave normally on ordinary inputs while producing different outputs when particular triggers appear; research on LoRA backdoors has demonstrated this under experimental conditions (NDSS research on LoRA backdoors). This differs from the Pickle example: an adapter that changes outputs does not by itself execute OS commands; its impact appears in the model’s responses. A coding assistant, for instance, could answer ordinary questions correctly while suggesting insecure practices under particular conditions. Two separate questions follow:

Does loading the package execute unexpected code on the system?
After loading, does the model behave within expected security boundaries?

Verifying the base model does not automatically verify an adapter added later.

Example 4: Custom Loading Code and Software Dependencies

Some models require custom Python code beyond the library’s built-in support. In Transformers, trust_remote_code=True can allow custom code from the model repository to execute, which is a decision to bring externally supplied code into the trusted environment. Hugging Face recommends reviewing that code and pinning a specific revision to control the effect of repository updates (Transformers security). Recording only the weight file’s hash is therefore insufficient; the custom code must be in scope too.

The same applies to Python dependencies: a package you use directly may install others, and these transitive dependencies become part of the software your application runs. The review should cover the complete set of components operating together in production. See What Is an SBOM?

Code and Dynamic Analysis: What Happens During Loading?

We can observe Pickle’s behavior with a small example that only prints a message. It uses data it creates itself, reads and writes no files, opens no connections and runs no OS commands:

import pickle

class LoadingDemo:
    def __reduce__(self):
        return (
            print,
            ("[DEMO] print was called while reconstructing the object.",)
        )

payload = pickle.dumps(LoadingDemo())
print("1. Serialization completed.")
print("2. Deserialization is starting.")
result = pickle.loads(payload)
print(f"3. Deserialization completed. Return value: {result!r}")

Output:

1. Serialization completed.
2. Deserialization is starting.
[DEMO] print was called while reconstructing the object.
3. Deserialization completed. Return value: None

Here __reduce__ specifies the function and arguments used to reconstruct the object, so pickle.loads(payload) calls print. The important observation is that loading can invoke a function. This does not show that a real model contains malware; it explains why deserialization should be treated as a trust boundary.

Loading a file to inspect its contents can also activate behavior carried by that file.

Unknown model files should therefore be examined in an environment without access to production secrets.

Planning Dynamic Analysis for a Real Component

Analysis proceeds along two dimensions: loading behavior and model behavior.

Loading analysis examines what happens on the system while the model is brought into memory:

Processes: are unexpected child processes started? Do operations appear that the model’s intended function cannot explain?
Network: which addresses are contacted during loading, and can those connections be explained by expected downloads?
File access: are unexpected files read, created or modified?
Credentials: API keys, tokens and production secrets the component does not need should not be exposed to the test environment.

This should happen in an environment without sensitive data and with restricted access. Containers help, but the strength of isolation depends on granted privileges, mounted directories and network configuration.

Behavioral analysis focuses on the model’s responses. For a LoRA adapter, compare the base model, the combination containing the adapter under review and, where available, a previously approved combination. Record variables such as tokenizer, system prompt and generation parameters; with stochastic generation, evaluate over repeated runs rather than a single output. Test normal task performance alongside different phrasings, application-specific failure scenarios and sensitive decision points. The results still have limits: finding no problems in the tested inputs does not prove there are no hidden triggers. Dynamic analysis must sit alongside provenance verification, integrity checks and code review. This is the same methodology we apply in How to Penetration Test an LLM Application.

Risks

The impact of an LLM supply chain incident depends on the component and the permissions of its environment:

Code execution and unauthorized access: malicious code during loading can affect files and resources the process can reach; broad loader permissions increase the impact.
Changes to model behavior: poisoned weights or adapters may produce incorrect, manipulated or insecure outputs, which is worse when acted on without review.
Admission of the wrong component: publisher impersonation and similar names can lead to the wrong component being approved.
Vulnerable dependencies: even a well-behaved model may be loaded by a vulnerable library or run by a vulnerable service.
Uncontrolled version changes: moving branches or tags can let unreviewed files into later deployments.
Lack of traceability: without file, version and deployment records, identifying affected systems and withdrawing a component takes longer.

The same behavioral failure has different consequences in an app that only drafts text versus an assistant that can use tools to act (see Excessive Agency). Risk assessment should consider what the model can access, how its outputs are used and where human oversight exists.

Mitigation Strategies

1. Verify the source and publisher. Obtain the model link from the publisher’s official site or a trusted repository, record the complete organization and model identity, and do not treat similar names, copied descriptions or popularity as proof of identity.

2. Match the reviewed version to the running version. Record exact revisions for the model, adapter and custom code; manage dependencies through lockfiles and verifiable records; retain the exact identity of the deployed container. Version pinning tells you which combination was tested and which reached production.

3. Prefer safer loading formats. Safetensors stores tensors without Pickle’s arbitrary-object reconstruction, reducing Pickle-related risk (Safetensors). The format alone does not prove the weights behave reliably or that accompanying custom code is safe.

4. Apply model signing and integrity verification. A hash shows whether a file matches a reference, but if the reference hash comes from the same attacker-controlled source it is not enough. Digital signatures tie integrity to a signing identity; Sigstore’s model-transparency project provides tools for signing and verifying models (Sigstore model-transparency). Define which identities are accepted, remember that a signed base model does not mean an independent adapter was verified, and keep the limits clear: a trusted identity can sign a poisoned component, so behavioral testing is still needed.

5. Preserve provenance records. How an artifact was produced matters alongside where it came from. Record source revisions, inputs, conversion steps and the production environment; SLSA’s provenance approach addresses verifiable information about how artifacts were produced (SLSA provenance). Keep a model-card description distinct from a verified production record.

6. Create an SBOM and AIBOM. An SBOM makes software components visible; an AIBOM extends the inventory to AI-specific components. CycloneDX’s AI/ML-BOM supports representing models, datasets, configurations and provenance (CycloneDX AI/ML-BOM). A useful inventory connects the base model and revision, adapters and revisions, file hashes and signatures, tokenizer and custom code, software dependencies, data sources and licenses, evaluation records and the applications using each component, so that when a warning is published you can find every affected application. See What Is an SBOM?

7. Plan scanning, testing and rollback together. Automated model scanning helps but is not a guarantee; Hugging Face states that Pickle scanning is not completely foolproof (Pickle scanning). Combine scanning with source verification, code review and behavioral evaluation, retain the deployed combination, and keep the ability to roll back to a verified package. In a possible code-execution incident, examine the credentials and systems the process could reach.

Security Fixes and Version Management

A security feature does not provide the same protection in every version. PyTorch’s weights_only option is a concrete example: from version 2.6, torch.load defaults to weights_only=True when pickle_module is not supplied, restricting the objects accepted during loading (PyTorch serialization).

However, CVE-2025-32434 showed that code execution was still possible with weights_only=True in affected versions. The official advisory identifies versions 2.5.1 and earlier as affected and 2.6.0 as containing the fix (PyTorch advisory GHSA-53q9-r3pm-6pq6). Two things must be distinguished: the version that fixes a particular vulnerability, and the version to use today, which should be chosen considering maintenance status, other advisories and compatibility. Version 2.6.0 is the historical fix for this vulnerability, not a general recommendation for every future deployment.

A sound version management process: select maintained versions evaluated against known vulnerabilities; test the model, adapter and runtime together; record the approved combination with exact identifiers; evaluate new versions before deploying; and keep the ability to return to the previous verified combination. Disabling a security protection to resolve a loading error is not an ordinary compatibility change; it may expand the application’s trust boundary.

Conclusion

LLM supply chain security makes the process of admitting models and components auditable. The Pickle example shows how loading a file can execute code; typosquatting shows the risk of trusting the wrong publisher; the LoRA example shows how a small component added later can change behavior. They share one failure: an external component becomes part of the trusted system without sufficient evidence. A trustworthy LLM application should be able to answer four questions:

Where did the component come from?
What processing did it undergo?
Which version is running in production?
What evidence supports our trust in this combination?

Assembling your LLM stack from third-party models and components? Our Software Supply Chain Penetration Testing and AI & LLM Security Testing services assess model provenance, loaders, adapters and dependencies end to end. Talk to our team.

References

Subscribe to our cyber security newsletter

Scroll to top