Part 6 · LLM Application Security
LLM07: Misinformation
Incorrect output becomes a security problem when people or automated systems trust it and act on it.
What it is
Misinformation occurs when an LLM produces incorrect, incomplete, unsupported, or misleading information that appears credible enough to influence a human decision, an automated workflow, or an agent's action.
The core risk is misplaced trust. In agentic systems, a false belief about system state or evidence can directly trigger a harmful action.
How the failure unfolds
Separating generation from verification keeps a confident error from becoming an action.
Common forms
- Fabricated facts: invented citations, cases, or statistics.
- Unsupported claims: a summary states more than the source says.
- False system state: an agent reports that a task succeeded when it failed.
- Unsafe code: suggested libraries or methods do not exist or are insecure.
- Overreliance: people stop checking because the output sounds authoritative.
Real-world case
In Moffatt v. Air Canada (2024), a British Columbia tribunal held the airline responsible after its website chatbot gave a passenger wrong information about bereavement fares.
Defenses
- Ground claims. Require support from authoritative sources.
- Separate generation from action. Verify a claim before anything acts on it.
- Verify independently. Do not rely on the same model to judge its own output.
- Match checks to impact. Increase review as the stakes rise.
- Communicate uncertainty. Show sources and limitations clearly.
- Evaluate continuously. Test accuracy on realistic cases.
Common misconception
"Misinformation is a quality issue and not a security issue."
When an error drives a payment, a code change, or an access decision, it has direct security consequences.
Key takeaways
- Misinformation becomes a risk when output is trusted without verification.
- Separate generating a claim from acting on it.
- Match the level of verification to the impact of the decision.