Google DeepMind Tests Gemini in Double-Blind Trial to Curb Benchmark Cheating

Google DeepMind has announced the completion of a pilot the company calls the world's first double-blind evaluation of a frontier-class model. Four external organizations took part in the test, with the entire procedure built on cryptographically isolating both sides from each other: evaluators never saw the model's weights, and Google never had access to the test questions.

The problem Google DeepMind is trying to solve has been building for some time. Companies training large language models publish results on popular benchmarks, but suspicions are growing that models have already "seen" the test questions during training, artificially inflating their scores. Research cited by Google DeepMind found this affected roughly half of the 31 models analyzed, while another study published this year showed that data contamination distorts the results of the largest models especially heavily.

How the encrypted evaluation works

The solution relies on Google Cloud's Confidential Computing technology, which uses isolated GPU enclaves. Inside this environment, both the input data and the model weights stay encrypted even during processing itself, and each side receives cryptographic proof that the other side genuinely had no access to their materials.

We can cryptographically verify that both the external evaluation data and the proprietary model remain private to their owners - from the Google DeepMind blog

The mechanism therefore enforces opacity in both directions: the evaluator cannot copy or analyze the tested model's weights, and the model's creator cannot see the content of the questions or memorize them for future versions. Verification of the software environment is handled through remote attestation, which confirms that the code running inside the enclave is exactly what both sides declared it to be.

Who tested it and what was checked

Four independent organizations took part in the pilot. AVERI evaluated Gemini 2.5 Flash Lite using reserved prompts from the AILuminate safety benchmark developed by MLCommons. The test scope covered cyberattack scenarios, chemical and biological threats, hate speech, self-harm, and incitement to violent crime. The Singapore AI Safety Institute tested the model using its own confidential prompts, selected for risks tied to harmful content relevant locally to the region.

The report was authored by William Isaac, Sol Messing, and Kristian Lum of Google DeepMind. In the paper itself, however, the company did not disclose specific numerical results or scores obtained by Gemini 2.5 Flash Lite in these tests - only the methodology and a description of the security architecture were published.

Limits of the method

The technical report accompanying the pilot openly acknowledges gaps in the procedure. External evaluators did not have full visibility into the code running inside the enclave, meaning they had to trust Google's claims to some extent about what was actually happening there. The hardware attestation process itself also rests on trust in the chain of hardware and software suppliers, rather than on full, independently verifiable transparency from the ground up.

These very limitations are why industry commentators are treating the pilot as a promising but preliminary step rather than a finished solution to benchmark contamination. For the method to earn broader trust, the process would need to become independently reproducible and transparent to a larger number of external parties before it could spread as an industry standard.

Why it matters for AI test credibility

For companies and researchers who rely on language model leaderboards to choose tools for their work, benchmark contamination has practical consequences. If a test score does not reflect a model's real ability to solve new, previously unseen tasks, purchasing decisions based on such rankings can lead to disappointment once deployed under real business conditions.

Double-blind evaluation, if refined and adopted more broadly by other AI labs and independent benchmarking institutions, could become a new standard for publishing results. For now, though, it remains a pilot limited to one relatively small model and a handful of partner organizations.