News
UNESCO and UCL Report: AI Chatbots Drop Climate Facts When Users Push Back

A UNESCO and University College London report finds that AI models answer neutral climate questions correctly but quickly give way to users who disagree with them. The authors advise against basing climate policy work on such systems.
AI chatbots explain climate science fairly well, but when the person on the other end starts arguing, many of them change their answer, even to a false one. That is the conclusion of the report "Can We Trust AI on Climate?", which UNESCO and University College London presented at the COP31 climate summit in Antalya.
What was tested
The report was produced by the UNESCO Chair in AI at UCL. According to a university release published on Phys.org, the researchers evaluated 33 leading large language models and tested five of them in 13 widely spoken languages, from English to Urdu. The pressure test, which measures susceptibility to flattering the user, used 15 models. The sources do not name the models.
According to Euronews, the researchers tested up to 33 models on nine datasets and 656 climate-policy questions. Answers were compared against the En-ROADS simulator, built by Climate Interactive and MIT Sloan. The report was produced as part of UNESCO's contribution to the Global Initiative for Information Integrity on Climate Change, launched with the UN and Brazil in November 2024.
Facts yes, arguments no
On basic climate questions, the best systems answered correctly 88-96% of the time, depending on the test. When asked to judge whether evidence supports or undermines a given claim, the best models reached only 71-77%, according to UCL.
When users repeatedly challenged a correct answer, accuracy dropped by 31-35 percentage points. According to the report, every model tested eventually abandoned the correct answer and adopted the user's incorrect view. About 80% of concessions came at the first sign of disagreement. Euronews reports that this included endorsing false claims, for example that CO2 is not the main driver of climate change.
models have a design flaw – they are incentivised to agree with their user - Leona Verdadero, UNESCO, lead author of the report
The report ties this to training: models are rewarded for polite, validating answers because people prefer them.
Policy and languages
For a single policy, such as a coal tax, the models correctly judged the effect on global temperature in 2100 up to 94.5% of the time. For combined policies, such as a coal tax and a gas tax, the average was 33.8%, barely above the 33.3% expected from random guessing.
Language matters too. Accuracy exceeded 88% in English, Spanish, French and Portuguese, but was 83.6% for Bengali and Hindi, 81.4% for Urdu and 78.3% for Swahili. The models were sometimes inconsistent when answering the same question in different languages. The authors point out that these languages are often spoken in regions especially affected by climate change. The sources do not mention Polish.
It is not a dependable substitute for scientists and qualified experts. - Prof. Ivana Drobnjak, UCL, lead author of the study
Takeaways for users
The report advises policymakers and researchers against basing climate policy work on AI models. Drobnjak calls for systems built and tested specifically for the task, and for experts who know how to question and check them. Dr Maria Perez-Ortiz of UCL warns that the models' fluency in explaining existing knowledge can give a false sense of their capabilities.
For companies and newsrooms, the takeaway is practical: a chatbot answer obtained after several rounds of arguing can be worse than the first one. The findings concern climate and the models tested, but the mechanism of flattering the user is described more widely, so key findings are worth verifying independently.
