Cognitive

Part 6 · LLM Application Security

AI Threat Modeling

AI reads instructions and information in the same language, so anything it reads might try to steer it.

3 min readBuilds on Retrieval-Augmented Generation, AI Agents and Tool Use

Attacks arrive through several entry points, and each one needs controls.

Key concepts

  • Instructions and data share one channel, which is the root of many AI security problems.
  • Behavior is probabilistic, so one safe test proves little.
  • Every app has five entry points: input, context, foundations, actions, and outputs.
  • Blast radius is the worst outcome if the model is fooled.
  • The Lethal Trifecta is private data, untrusted content, and an external communication channel in one agent. Remove one leg to break that attack path.

The lethal trifecta. Remove any one element and this data-theft path breaks.

Figure 29.1. The lethal trifecta. Remove any one element and this data-theft path breaks.
Even a rare attack becomes likely when it can be retried many times.

Formal view and evidence

  • Established. Threat modeling defines assets, adversaries, capabilities, and trust boundaries. For an attack with per-attempt success rate p, the chance of at least one success in n independent attempts is 1−(1−p)n.
  • Debated. Whether any model-level mechanism can establish a robust instruction-data boundary.
  • Open problem. Formal security guarantees for systems whose core component is a probabilistic language model.

The OWASP lists

Top 10 for LLM Applications, 2026

IDRiskChapter
LLM01Prompt Injection30
LLM02Sensitive Information Disclosure31
LLM03Excessive Agency32
LLM04Supply Chain33
LLM05Data and Model Poisoning34
LLM06Unbounded Consumption35
LLM07Misinformation36
LLM08Hidden Context Exposure37
LLM09Vector and Embedding Weaknesses38
LLM10Improper Output Handling39

Top 10 for Agentic Applications, 2026

IDRiskChapter
ASI01Agent Goal Hijack40
ASI02Tool Misuse and Exploitation41
ASI03Identity and Privilege Abuse42
ASI04Agentic Supply Chain Vulnerabilities43
ASI05Unexpected Code Execution44
ASI06Memory and Context Poisoning45
ASI07Insecure Inter-Agent Communication46
ASI08Cascading Failures47
ASI09Human-Agent Trust Exploitation48
ASI10Rogue Agents49

Common misconception

Common belief

"AI security just means stopping rude answers."

In practice

The costly failures include leaked data, unauthorized actions, and compromised systems.

Key takeaways

  • LLMs process instructions and data in the same channel.
  • Blast radius measures how much harm a compromised system can cause.
  • Removing one leg of the lethal trifecta breaks that attack path.

Explore next