AI Explanations Can Manipulate Users—But Not Always in the Way You'd Expect
Researchers found that when AI systems use persuasive language to justify decisions, people are more likely to accept them—but only in healthcare contexts. In finance, the opposite happens: users trust straightforward explanations more. The finding matters because high-stakes industries rely on user acceptance of AI recommendations, yet few have tested whether rhetorical tactics undermine or strengthen genuine trust.
Originaltitel: User Compliance and Awareness towards Persuasive XAI: Investigating the Rhetorical Layer of LLM-generated Explanations
In high-stakes decision-making, the acceptance of AI recommendations depends not only on system accuracy but also on how decisions are explained. While prior work on explainable AI has largely focused on transparency and interpretability, less attention has been paid to the persuasive dimension of explanations. To address this gap, we investigate how rhetorical strategies drawn from Cialdini’s persuasion theory, when embedded in natural language explanations generated by large language models (LLMs), influence user compliance and their ability to recognize persuasive intent. We conducted a controlled survey study with 129 participants in two application domains—finance and healthcare—where participants evaluated both a baseline and a persuasive explanation for an AI-generated decision across favorable and unfavorable outcomes. In a complementary task, participants rated ten short explanations on perceived persuasiveness and factual strength, enabling us to measure awareness of persuasive intent. Our results show that persuasive explanations significantly increased compliance in the healthcare scenario (p <.001), whereas baseline explanations were more effective in finance (p <.001), regardless of whether the AI decision was positive or negative. A notable proportion of participants rated explanations containing at least one of Cialdini’s persuasion techniques as highly persuasive, yet simultaneously judged them to be factually weaker. Importantly, we found no statistically significant evidence that participants’ ability to recognize persuasive intent influenced compliance. These findings highlight the dual role of persuasive explanations: they can enhance foster compliance in sensitive contexts such as healthcare but risk undermining trust in domains like finance. For HCI and IUI, our study underscores that explanations are not neutral vessels of information: their rhetorical form substantially shapes how users perceive and engage with AI-assisted decision-making. Designers of explainable AI systems should therefore carefully balance transparency and persuasion when developing interfaces for high-stakes applications.