Part 6 · LLM Application Security
LLM05: Data and Model Poisoning
Corrupting what a system learns or retrieves can plant behavior that stays hidden until a trigger appears.
What it is
Data and model poisoning occurs when data or model artifacts are manipulated to embed harmful behavior, bias, or exploitable weaknesses. It can happen wherever data is ingested, transformed, retrieved, or reused, including pre-training, fine-tuning, embedding creation, RAG, and model distribution.
Unlike prompt injection, which acts at inference time, poisoning changes what the system has durably learned or stored.
How the attack unfolds
A backdoored model passes normal testing because it misbehaves only when it sees a specific trigger.
Types of poisoning
| Stage | Example |
|---|---|
| Pre-training | Public web content is seeded with malicious examples |
| Fine-tuning | A contributed dataset contains a backdoor |
| RAG ingestion | A false document enters a knowledge base |
| Model distribution | A poisoned adapter is published for reuse |
Research finding
A 2025 study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that around 250 malicious documents could implant a narrow backdoor in models of varying size. The number needed did not grow with the size of the training data in their experiments.
Defenses
- Track lineage. Record where each dataset and model came from.
- Validate data. Vet sources and check integrity at each stage.
- Restrict write access. Control who can change training data and stores.
- Version everything. Make it possible to roll back to a known good state.
- Red-team for triggers. Test for hidden behaviors as well as average accuracy.
- Monitor in production. Watch for drift and unexpected output patterns.
Common misconception
"Poisoning needs control of a large share of the data."
Research shows that some narrow backdoors can be implanted with a small, fixed number of documents.
Key takeaways
- Poisoning changes what a system learns or retrieves.
- Backdoors can remain hidden until a trigger appears.
- Lineage, validation, and rollback are core defenses.