Cognitive

Part 6 · LLM Application Security

LLM07: Misinformation

Incorrect output becomes a security problem when people or automated systems trust it and act on it.

3 min readBuilds on Hallucination and Evaluation

OWASP LLM07. This material is for defense. Only test systems you own or are authorized to test.

Confident language can hide the fact that an answer is wrong.

What it is

Misinformation occurs when an LLM produces incorrect, incomplete, unsupported, or misleading information that appears credible enough to influence a human decision, an automated workflow, or an agent's action.

The core risk is misplaced trust. In agentic systems, a false belief about system state or evidence can directly trigger a harmful action.

How the failure unfolds

Separating generation from verification keeps a confident error from becoming an action.

Figure 36.1. Separating generation from verification keeps a confident error from becoming an action.

Common forms

  • Fabricated facts: invented citations, cases, or statistics.
  • Unsupported claims: a summary states more than the source says.
  • False system state: an agent reports that a task succeeded when it failed.
  • Unsafe code: suggested libraries or methods do not exist or are insecure.
  • Overreliance: people stop checking because the output sounds authoritative.

Real-world case

In Moffatt v. Air Canada (2024), a British Columbia tribunal held the airline responsible after its website chatbot gave a passenger wrong information about bereavement fares.

Defenses

Claims should be checked before anyone acts on them.
  1. Ground claims. Require support from authoritative sources.
  2. Separate generation from action. Verify a claim before anything acts on it.
  3. Verify independently. Do not rely on the same model to judge its own output.
  4. Match checks to impact. Increase review as the stakes rise.
  5. Communicate uncertainty. Show sources and limitations clearly.
  6. Evaluate continuously. Test accuracy on realistic cases.

Common misconception

Common belief

"Misinformation is a quality issue and not a security issue."

In practice

When an error drives a payment, a code change, or an access decision, it has direct security consequences.

Key takeaways

  • Misinformation becomes a risk when output is trusted without verification.
  • Separate generating a claim from acting on it.
  • Match the level of verification to the impact of the decision.

Explore next