Part 6 · LLM Application Security
AI Threat Modeling
AI reads instructions and information in the same language, so anything it reads might try to steer it.
Key concepts
- Instructions and data share one channel, which is the root of many AI security problems.
- Behavior is probabilistic, so one safe test proves little.
- Every app has five entry points: input, context, foundations, actions, and outputs.
- Blast radius is the worst outcome if the model is fooled.
- The Lethal Trifecta is private data, untrusted content, and an external communication channel in one agent. Remove one leg to break that attack path.
The lethal trifecta. Remove any one element and this data-theft path breaks.
Formal view and evidence
- Established. Threat modeling defines assets, adversaries, capabilities, and trust boundaries. For an attack with per-attempt success rate , the chance of at least one success in independent attempts is .
- Debated. Whether any model-level mechanism can establish a robust instruction-data boundary.
- Open problem. Formal security guarantees for systems whose core component is a probabilistic language model.
The OWASP lists
Top 10 for LLM Applications, 2026
| ID | Risk | Chapter |
|---|---|---|
LLM01 | Prompt Injection | 30 |
LLM02 | Sensitive Information Disclosure | 31 |
LLM03 | Excessive Agency | 32 |
LLM04 | Supply Chain | 33 |
LLM05 | Data and Model Poisoning | 34 |
LLM06 | Unbounded Consumption | 35 |
LLM07 | Misinformation | 36 |
LLM08 | Hidden Context Exposure | 37 |
LLM09 | Vector and Embedding Weaknesses | 38 |
LLM10 | Improper Output Handling | 39 |
Top 10 for Agentic Applications, 2026
| ID | Risk | Chapter |
|---|---|---|
ASI01 | Agent Goal Hijack | 40 |
ASI02 | Tool Misuse and Exploitation | 41 |
ASI03 | Identity and Privilege Abuse | 42 |
ASI04 | Agentic Supply Chain Vulnerabilities | 43 |
ASI05 | Unexpected Code Execution | 44 |
ASI06 | Memory and Context Poisoning | 45 |
ASI07 | Insecure Inter-Agent Communication | 46 |
ASI08 | Cascading Failures | 47 |
ASI09 | Human-Agent Trust Exploitation | 48 |
ASI10 | Rogue Agents | 49 |
Common misconception
Common belief
"AI security just means stopping rude answers."
In practice
The costly failures include leaked data, unauthorized actions, and compromised systems.
Key takeaways
- LLMs process instructions and data in the same channel.
- Blast radius measures how much harm a compromised system can cause.
- Removing one leg of the lethal trifecta breaks that attack path.