Study: Doctors Uncritically Trust AI Recommendations Even When Treatment Fails

Doctors participating in the experiment were deceived twice using the same method: an artificial intelligence system suggested which patients would benefit from treatment, yet the participants failed to notice that in the second part of the test the therapy worked for no one at all. Researchers from the University of the Basque Country in Spain warn that this is not an isolated cognitive error but a repeatable pattern that could threaten patient safety as algorithms become part of everyday clinical practice.

In the experiment, doctors anonymously logged into an online simulation in which they made treatment decisions for hypothetical patients suffering from a rare, fictional disease. The AI system suggested which patients were likely to benefit from therapy, while participants could track the actual treatment outcomes of individual patients in real time and compare them against the algorithm's recommendations.

Two experiments, one pattern

In the first part of the study, the treatment worked moderately well and roughly equally for all patients, regardless of what the algorithm suggested. In the second part, the researchers went a step further: the therapy was completely ineffective for every single patient, yet the AI system kept issuing its recommendations, which doctors could either trust or reject based on the outcomes they observed.

In both variants, doctors rated the AI system as reliable and, in practice, did not use the available data on patient recovery to verify the accuracy of its recommendations. In the second experiment, even though the treatment produced no effect in anyone, participants failed to conclude that the algorithm's suggestions were worthless.

Why doctors miss the errors

The authors call this mechanism a difficulty in learning from evidence that contradicts an algorithm's suggestion. In other words, once a system earns the status of a trustworthy source, people, including trained specialists, tend to ignore subsequent signals that the system is wrong rather than revise their assessment.

Professionals had difficulty learning from the available data when it contradicted the algorithm's suggestions - Aranzazu Vinas, University of the Basque Country

Study co-author Helena Matute stresses that the problem is not limited to medicine or to doctors alone, though in this profession the cost of error is highest.

Doctors, like everyone else, have trouble learning from the available evidence when it contradicts the algorithm's suggestions - Helena Matute, University of the Basque Country

Similar findings elsewhere

The results from Spain are not an isolated case. In a randomized clinical trial published in the journal NEJM AI, doctors trained in the use of AI who received deliberately incorrect diagnostic suggestions from a language model performed worse on clinical reasoning than a group working without such interference, 73.3 percent versus 84.9 percent. The accuracy of picking the correct diagnosis as the first choice fell from 90.5 to 76.1 percent.

Notably, training doctors in AI skills alone did not protect them from automatically deferring to incorrect suggestions. Both studies point to the same mechanism: once a system proves helpful, people stop critically verifying it, even when they have data available that should raise doubts.

What it means for patients

The authors of the Basque study stress that the findings do not mean clinical decision-support systems should be pulled from doctors' offices. Rather, the point is that deploying such tools must go hand in hand with protocols that require doctors to actively and regularly verify recommendations, not just passively accept them.

For healthcare systems that are just beginning to roll out AI tools for diagnostic support and documentation, including facilities in Poland testing such solutions, the study's conclusions offer a concrete lesson: simply using an algorithm does not guarantee staff vigilance, and training on how to operate a tool is not the same as training on how to critically evaluate its output.

Sources: PLOS Digital Health (journals.plos.org), Medical Xpress (medicalxpress.com), News-Medical (news-medical.net), NEJM AI (ai.nejm.org)