AI Advice Makes People More Confident But Less Accurate
Researchers from three French and Italian universities have found that access to AI advice dramatically reduces people's willingness to admit they do not know something. The study, published by a team including Valerio Capraro of the University of Milano-Bicocca, Chiara Marcoccia of École Normale Supérieure, and Walter Quattrociocchi of Sapienza University of Rome, shows a stark collapse in judgment suspension.
When participants had access to AI advice, their willingness to say "I don't know" dropped from 44% to just 3%. Accuracy fell from 27% to 9%. Meanwhile, confidence in their answers rose from 30% to 76%. "People became much worse, the accuracy was only one third, but they were twice as confident," Capraro said.
Deliberately Flawed AI Used in Experiment
The researchers designed the study to test whether people were simply delegating to a reliable tool. They used Google's Step 3.5 Flash model, which was deliberately chosen because it typically gives wrong answers on the specific questions asked. The questions involved visual details from films, such as the color of a team's uniform in the movie "Bend It Like Beckham."
Because the AI was usually wrong on these questions, any reduction in judgment could not be explained as sensible delegation to a trustworthy system. Some participants who would have answered correctly on their own asked the AI and ended up giving wrong answers.
Monetary Incentives Helped Only Slightly
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
The study also tested whether financial rewards could counteract the effect. When participants were offered monetary incentives for correct answers, their willingness to admit ignorance rose from 3% to 8% and accuracy from 9% to 16%. But both figures remained far below the no-AI baselines of 44% and 27%.
This finding aligns with earlier research from Wharton, where researchers coined the term "cognitive surrender" to describe people accepting incorrect AI answers 80% of the time while reporting higher confidence than those working without AI. The new study adds a sharper data point: it is not just that people trust wrong AI answers. The mere availability of AI suppresses the cognitive habit of recognizing what you do not know.
Concerns for Children and Education
Capraro expressed particular concern about children growing up with these systems before they have developed critical thinking skills. "For humans, the capacity to say 'I don't know' is very important because it represents the recognition of the limits of our own knowledge," he said.
The study's findings echo broader concerns about AI design. Google's AI search overhaul replaced links with confident AI-generated summaries. Common Sense Media this week called that design an "unacceptable risk" for students. The pattern is consistent: AI products are designed to answer, never to say "I don't know." The humans using them are learning to do the same.