New paper accepted to the 2nd International Conference on Human-AI Interaction and Experience Design. Zekun Wu will present our work at HAXD’26 in June in Valencia, Spain.
How AI Responses Shape User Beliefs: The Effects of Information Detail and Confidence on Belief Strength and Stance
Zekun Wu, Mayank Jobanputra, Vera Demberg, Jessica Hullman, Anna Maria Feit
Abstract:
The growing use of AI-generated responses in everyday tools raises concern about how subtle features such as supporting detail or tone of confidence may shape people’s beliefs. To understand this, we conducted a pre-registered online experiment (N=304) investigating how the detail and confidence of AI-generated responses influence belief change. We introduce an analysis framework with two targeted measures: belief switch and belief shift. These distinguish between users changing their initial stance after AI input and the extent to which they adjust their conviction toward or away from the AI’s stance, thereby quantifying not only categorical changes but also more subtle, continuous adjustments in belief strength that indicate a reinforcement or weakening of existing beliefs. Using this framework, we find that detailed responses with medium confidence are associated with the largest overall belief changes. Highly confident messages tend to belief shifts but induce less stance reversals. Our results also show that task type (fact-checking versus opinion evaluation), prior conviction, and perceived stance agreement further modulate the extent and direction of belief change. These results illustrate how different properties of AI responses interact with user beliefs in often subtle but potentially consequential ways and raise practical as well as ethical considerations for the design of LLM-powered systems.
Link to article:
https://arxiv.org/pdf/2511.09667
Be First to Comment