Study: AI Suggestions Make People Stop Admitting They Don't Know

A team of researchers from the University of Milano-Bicocca, Paris's École Normale Supérieure and Rome's Sapienza University has published a study describing a side effect of using AI assistants that had never before been documented with such precise numbers. The mere availability of an AI suggestion makes people almost stop admitting they don't know something, even when the suggestion is wrong and a financial reward is on offer for a correct answer.

How the experiment worked

The study's authors, Valerio Capraro, Chiara Marcoccia and Walter Quattrociocchi, designed the test to rule out the simplest explanation, that people are just reasonably relying on a proven tool. Instead of questions where AI usually gets it right, they chose a category where language models systematically get it wrong: minor visual details from films, such as the color of the team's kits in the film "Graj w gałę" or the model of car driven by the heroine of the film "Jak kot na autostradzie".

Participants first answered the questions on their own, with the option to select "I don't know" instead of guessing. In this phase, 44 percent admitted they didn't know, and among those who did attempt an answer, 27 percent got it right. The same participants were then given access to a suggestion generated by the Claude 3.5 Flash model and could change their answer.

Worse results, more confidence

The effect turned out to be drastic. When the AI suggestion was available, the share of "I don't know" answers dropped to just 3 percent, and accuracy fell to 9 percent, less than a third of the pre-AI result. Some people who would have answered correctly on their own changed their minds to a wrong answer after seeing the model's suggestion.

The most troubling part is the contrast between accuracy and confidence. Without AI, participants reported 30 percent confidence in their answers. After using the suggestion, their reported confidence rose to 76 percent, even though actual accuracy had dropped almost threefold. The authors call this phenomenon a form of cognitive surrender to a tool that sounds convincing regardless of whether it's right.

People performed much worse, accuracy fell to a third, and yet they were twice as confident - Valerio Capraro, University of Milano-Bicocca

For people, the ability to say 'I don't know' is very important, because it means acknowledging the limits of one's own knowledge - Valerio Capraro, University of Milano-Bicocca

Money only partly helps

The researchers also tested whether a financial incentive for a correct answer would reverse the effect. Introducing a monetary reward for accuracy raised the share of "I don't know" answers from 3 to 8 percent, and accuracy itself from 9 to 16 percent. That's an improvement, but still far from the level seen before exposure to the AI suggestion, suggesting that financial motivation alone isn't enough to rebuild critical distance from the machine.

This result has practical implications beyond the lab. It shows that even in situations where people have a clear incentive to think critically, for example when making professional or financial decisions with the help of a chatbot, the mere presence of AI lowers vigilance against errors.

What this means for Poland

For Polish companies deploying AI assistants in customer service, financial analysis or medical decision support, the finding is a warning that goes beyond the question of model quality alone. Even a more capable system won't solve the problem if its mere presence lulls the vigilance of the employee who is supposed to catch its errors. Earlier findings, like the Dartmouth study showing that correcting AI errors takes doctors longer than writing from scratch, or reports of legal representatives uncritically using AI in court filings, fit the same pattern of declining vigilance.

The authors highlight a particular risk for children and students who are growing up with access to chatbots before they develop the habit of assessing their own uncertainty. If using AI from an early age teaches that every question has a ready, confident-sounding answer, the ability to say "I don't know" may fade before it ever has the chance to take root.

The study doesn't deny the usefulness of AI tools, but it challenges the assumption that simply adding a warning about possible errors or an incentive to verify is enough to keep users critical. The authors suggest that interface-level solutions are needed that actively make it harder to accept suggestions without reflection, rather than merely disclosing the risk of error in fine print.

The paper was published on the PsyArXiv platform and, as is typical for preprints, hasn't yet undergone full peer review, though that doesn't change the scale of the differences described between the group with AI access and the group without it.