Cognitive

Part 6 · LLM Application Security

LLM10: Improper Output Handling

Model output is untrusted input to every system that receives it.

3 min readBuilds on LLM01: Prompt Injection

OWASP LLM10. This material is for defense. Only test systems you own or are authorized to test.

Model output is untrusted input to every system that receives it.

What it is

Improper output handling is insufficient validation, sanitization, and handling of LLM output before it reaches other components. Because prompts can influence output, passing that output downstream can give an attacker indirect access to those systems.

The 2026 edition also covers insecure AI-generated code and unsafe automatic handling of external resources.

How the attack unfolds

Each destination needs its own deterministic safeguard, because the output may be attacker-influenced.

Figure 39.1. Each destination needs its own deterministic safeguard, because the output may be attacker-influenced.

Dangerous destinations

DestinationPossible resultSafeguard
BrowserCross-site scriptingContext-aware encoding
DatabaseSQL injectionParameterized queries
ShellCommand injectionAvoid shells and use typed calls
InterpreterRemote code executionSandboxed execution
File systemPath traversalValidate and restrict paths
Markdown rendererData sent through image linksBlock automatic external loading

Real-world case

Vanna.AI (CVE-2024-5565) generated charting code with an LLM and executed it. JFrog showed that a prompt injection could lead to remote code execution.

Defenses

Each destination needs its own deterministic safeguard.
  1. Treat the model as an untrusted user. Apply zero-trust validation to its output.
  2. Encode for the destination. Each sink needs its own handling.
  3. Use parameterized interfaces. Never concatenate output into queries or commands.
  4. Validate against a schema. Reject output that does not match.
  5. Sandbox generated code. Remove network access and secrets by default.
  6. Review AI-written code. Apply normal scanning and human review.

Common misconception

Common belief

"The output came from our own model, so it can be trusted."

In practice

Attackers can influence that output through the content the model reads.

Key takeaways

  • Model output can be influenced by attackers.
  • Every destination needs its own validation or encoding.
  • Generated code should be reviewed and sandboxed.

Explore next