Part 6 · LLM Application Security
LLM10: Improper Output Handling
Model output is untrusted input to every system that receives it.
What it is
Improper output handling is insufficient validation, sanitization, and handling of LLM output before it reaches other components. Because prompts can influence output, passing that output downstream can give an attacker indirect access to those systems.
The 2026 edition also covers insecure AI-generated code and unsafe automatic handling of external resources.
How the attack unfolds
Each destination needs its own deterministic safeguard, because the output may be attacker-influenced.
Dangerous destinations
| Destination | Possible result | Safeguard |
|---|---|---|
| Browser | Cross-site scripting | Context-aware encoding |
| Database | SQL injection | Parameterized queries |
| Shell | Command injection | Avoid shells and use typed calls |
| Interpreter | Remote code execution | Sandboxed execution |
| File system | Path traversal | Validate and restrict paths |
| Markdown renderer | Data sent through image links | Block automatic external loading |
Real-world case
Vanna.AI (CVE-2024-5565) generated charting code with an LLM and executed it. JFrog showed that a prompt injection could lead to remote code execution.
Defenses
- Treat the model as an untrusted user. Apply zero-trust validation to its output.
- Encode for the destination. Each sink needs its own handling.
- Use parameterized interfaces. Never concatenate output into queries or commands.
- Validate against a schema. Reject output that does not match.
- Sandbox generated code. Remove network access and secrets by default.
- Review AI-written code. Apply normal scanning and human review.
Common misconception
"The output came from our own model, so it can be trusted."
Attackers can influence that output through the content the model reads.
Key takeaways
- Model output can be influenced by attackers.
- Every destination needs its own validation or encoding.
- Generated code should be reviewed and sandboxed.