Part 7 · Agentic AI Security
ASI09: Human-Agent Trust Exploitation
A confident, helpful agent can persuade people to approve what they should have questioned.
What it is
People can place strong trust in agents because of their fluent language, apparent expertise, and human-like manner. Adversaries or poorly designed systems can exploit that trust to influence decisions, extract sensitive information, or gain approval for harmful actions.
The human stays in the loop, but the check stops being meaningful.
How the attack unfolds
Approval only protects a system when the reviewer can see and verify what will really happen.
Common patterns
- Authority bias: the agent sounds certain and expert.
- Fabricated rationale: a convincing reason hides the real action.
- Urgency: pressure discourages careful review.
- Approval fatigue: too many prompts lead to automatic acceptance.
Defenses
- Show the actual action. Generate approval details outside the agent.
- Use step-up approval. Add stronger checks for sensitive actions.
- Reduce approval fatigue. Request review only when it matters.
- Keep immutable logs. Record queries, approvals, and actions.
- Calibrate trust. Show uncertainty and avoid manipulative language.
- Train reviewers. Teach people how agents can mislead them.
Common misconception
Common belief
"A human in the loop makes the system safe."
In practice
A rushed or misled reviewer gives little protection.
Key takeaways
- People can over-trust confident agents.
- Approval screens should show independently generated details.
- Too many prompts weaken human oversight.