Cognitive

Part 6 · LLM Application Security

LLM05: Data and Model Poisoning

Corrupting what a system learns or retrieves can plant behavior that stays hidden until a trigger appears.

3 min readBuilds on LLM04: Supply Chain, Large Language Models

OWASP LLM05. This material is for defense. Only test systems you own or are authorized to test.

A small amount of poisoned data can change what a model learns.

What it is

Data and model poisoning occurs when data or model artifacts are manipulated to embed harmful behavior, bias, or exploitable weaknesses. It can happen wherever data is ingested, transformed, retrieved, or reused, including pre-training, fine-tuning, embedding creation, RAG, and model distribution.

Unlike prompt injection, which acts at inference time, poisoning changes what the system has durably learned or stored.

How the attack unfolds

A backdoored model passes normal testing because it misbehaves only when it sees a specific trigger.

Figure 34.1. A backdoored model passes normal testing because it misbehaves only when it sees a specific trigger.

Types of poisoning

StageExample
Pre-trainingPublic web content is seeded with malicious examples
Fine-tuningA contributed dataset contains a backdoor
RAG ingestionA false document enters a knowledge base
Model distributionA poisoned adapter is published for reuse

Research finding

A 2025 study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that around 250 malicious documents could implant a narrow backdoor in models of varying size. The number needed did not grow with the size of the training data in their experiments.

Defenses

Checks across the data life cycle catch poisoned inputs.
  1. Track lineage. Record where each dataset and model came from.
  2. Validate data. Vet sources and check integrity at each stage.
  3. Restrict write access. Control who can change training data and stores.
  4. Version everything. Make it possible to roll back to a known good state.
  5. Red-team for triggers. Test for hidden behaviors as well as average accuracy.
  6. Monitor in production. Watch for drift and unexpected output patterns.

Common misconception

Common belief

"Poisoning needs control of a large share of the data."

In practice

Research shows that some narrow backdoors can be implanted with a small, fixed number of documents.

Key takeaways

  • Poisoning changes what a system learns or retrieves.
  • Backdoors can remain hidden until a trigger appears.
  • Lineage, validation, and rollback are core defenses.

Explore next