Cognitive

Part 7 · Agentic AI Security

ASI09: Human-Agent Trust Exploitation

A confident, helpful agent can persuade people to approve what they should have questioned.

3 min readBuilds on LLM07: Misinformation

OWASP ASI09. This material is for defense. Only test systems you own or are authorized to test.

Persuasive explanations can replace real verification.

What it is

People can place strong trust in agents because of their fluent language, apparent expertise, and human-like manner. Adversaries or poorly designed systems can exploit that trust to influence decisions, extract sensitive information, or gain approval for harmful actions.

The human stays in the loop, but the check stops being meaningful.

How the attack unfolds

Approval only protects a system when the reviewer can see and verify what will really happen.

Figure 48.1. Approval only protects a system when the reviewer can see and verify what will really happen.

Common patterns

  • Authority bias: the agent sounds certain and expert.
  • Fabricated rationale: a convincing reason hides the real action.
  • Urgency: pressure discourages careful review.
  • Approval fatigue: too many prompts lead to automatic acceptance.

Defenses

Reviewers need independent facts about what will happen.
  1. Show the actual action. Generate approval details outside the agent.
  2. Use step-up approval. Add stronger checks for sensitive actions.
  3. Reduce approval fatigue. Request review only when it matters.
  4. Keep immutable logs. Record queries, approvals, and actions.
  5. Calibrate trust. Show uncertainty and avoid manipulative language.
  6. Train reviewers. Teach people how agents can mislead them.

Common misconception

Common belief

"A human in the loop makes the system safe."

In practice

A rushed or misled reviewer gives little protection.

Key takeaways

  • People can over-trust confident agents.
  • Approval screens should show independently generated details.
  • Too many prompts weaken human oversight.