Cognitive

Part 6 · LLM Application Security

LLM04: Supply Chain

LLM applications inherit risk from every model, dataset, adapter, library, and service they depend on.

3 min readBuilds on AI Threat Modeling

OWASP LLM04. This material is for defense. Only test systems you own or are authorized to test.

One compromised upstream component can reach every application built on it.

What it is

LLM supply chains can be compromised at any stage: training data, pre-trained models, fine-tuning adapters, conversion pipelines, and deployment platforms. Beyond conventional software flaws, model artifacts can be tampered with or replaced, and their behavior is difficult to inspect.

How the attack unfolds

A model artifact is code-adjacent. Unsafe formats can execute instructions the moment they are loaded.

Figure 33.1. A model artifact is code-adjacent. Unsafe formats can execute instructions the moment they are loaded.

Where the risk appears

  • Model files: unsafe formats, substituted weights, and hidden backdoors.
  • Adapters: LoRA and other fine-tuning components from untrusted authors.
  • Packages: vulnerable dependencies and slopsquatting, where attackers register package names that coding assistants tend to invent.
  • Pipelines: compromised conversion, build, and deployment systems.
  • Terms and licensing: unclear data use or model usage rights.

Real-world case

In 2024, JFrog reported about 100 suspicious models on Hugging Face. Some used Python's pickle format to run code when loaded.

Defenses

Verification stops unsafe artifacts before they enter production.
  1. Vet suppliers. Review sources, terms, and security posture.
  2. Keep an inventory. Maintain an AI bill of materials for models, data, and components.
  3. Verify integrity. Check signatures and hashes, and pin versions.
  4. Prefer safe formats. Use formats that store data only, such as safetensors.
  5. Test before promotion. Evaluate third-party models for your own use case.
  6. Patch and monitor. Track vulnerabilities in every layer.

Common misconception

Common belief

"A popular model is a safe model."

In practice

Download counts show popularity. They do not verify integrity.

Key takeaways

  • LLM applications inherit risks from models, data, and dependencies.
  • Provenance, signatures, and safe formats reduce tampering risk.
  • Popularity is not evidence that a component is safe.

Explore next