Part 6 · LLM Application Security
LLM02: Sensitive Information Disclosure
Confidential data can escape through answers, tool calls, logs, embeddings, and other channels around the model.
What it is
Sensitive information disclosure occurs when an LLM system exposes confidential, regulated, privileged, or proprietary data through a channel that was not authorized.
The final answer is only one channel. Tool-call arguments, reasoning traces, retrieved context, logs, telemetry, embeddings, and even observable properties such as timing or cache behavior can all reveal information.
How the attack unfolds
An employee asks an ordinary question. The outcome depends on whether permissions are applied before retrieval.
Where data leaks
- Training and fine-tuning: models can memorize and reproduce rare sequences.
- Retrieval: search returns documents the current user should not see.
- Sessions and memory: data crosses between users or tenants.
- Operations: prompts and outputs are copied into logs, analytics, and vendor systems.
- Side channels: token counts, confidence values, and cache hits reveal information.
Real-world case
In 2023, Samsung employees pasted internal source code and documents into ChatGPT while seeking help with their work. The company then restricted the use of generative AI tools on its systems.
Defenses
- Minimize data. Send only the fields a task requires.
- Enforce access before retrieval. Use the identity of the actual user, and not a shared account.
- Clean data at intake. Classify it, remove personal data, and remove duplicates.
- Isolate tenants and sessions. Keep memory and caches separate.
- Protect every output channel. Apply redaction to logs, traces, and tool arguments as well as answers.
- Review provider terms. Know how long prompts are retained and who can access them.
Common misconception
"If the model never shows it in the answer, the data is safe."
Logs, tool calls, embeddings, and traces can expose the same data.
Key takeaways
- Disclosure can occur through answers, tools, logs, and embeddings.
- Apply access control before data reaches the model.
- Minimize the sensitive data that enters the system at all.