Cognitive

Part 6 · LLM Application Security

LLM02: Sensitive Information Disclosure

Confidential data can escape through answers, tool calls, logs, embeddings, and other channels around the model.

3 min readBuilds on AI Threat Modeling, Retrieval-Augmented Generation

OWASP LLM02. This material is for defense. Only test systems you own or are authorized to test.

Sensitive data can escape through channels beyond the final answer.

What it is

Sensitive information disclosure occurs when an LLM system exposes confidential, regulated, privileged, or proprietary data through a channel that was not authorized.

The final answer is only one channel. Tool-call arguments, reasoning traces, retrieved context, logs, telemetry, embeddings, and even observable properties such as timing or cache behavior can all reveal information.

How the attack unfolds

An employee asks an ordinary question. The outcome depends on whether permissions are applied before retrieval.

Figure 31.1. An employee asks an ordinary question. The outcome depends on whether permissions are applied before retrieval.

Where data leaks

  • Training and fine-tuning: models can memorize and reproduce rare sequences.
  • Retrieval: search returns documents the current user should not see.
  • Sessions and memory: data crosses between users or tenants.
  • Operations: prompts and outputs are copied into logs, analytics, and vendor systems.
  • Side channels: token counts, confidence values, and cache hits reveal information.

Real-world case

In 2023, Samsung employees pasted internal source code and documents into ChatGPT while seeking help with their work. The company then restricted the use of generative AI tools on its systems.

Defenses

Controls before and after the model reduce disclosure.
  1. Minimize data. Send only the fields a task requires.
  2. Enforce access before retrieval. Use the identity of the actual user, and not a shared account.
  3. Clean data at intake. Classify it, remove personal data, and remove duplicates.
  4. Isolate tenants and sessions. Keep memory and caches separate.
  5. Protect every output channel. Apply redaction to logs, traces, and tool arguments as well as answers.
  6. Review provider terms. Know how long prompts are retained and who can access them.

Common misconception

Common belief

"If the model never shows it in the answer, the data is safe."

In practice

Logs, tool calls, embeddings, and traces can expose the same data.

Key takeaways

  • Disclosure can occur through answers, tools, logs, and embeddings.
  • Apply access control before data reaches the model.
  • Minimize the sensitive data that enters the system at all.