Google DeepMind Warns: Most Off-the-Shelf AI Skills Were Never Tested

Philipp Schmid, a Google DeepMind engineer responsible for agent and Gemini API development, has issued a warning aimed at the entire AI agent building industry. In a new talk titled "Don't Ship Skills Without Evals" and an accompanying blog post, he argues that thousands of so-called agent skills, packages of instructions and procedures that train AI agents for specific tasks, are being shipped to production without any systematic evaluation.

Agent skills are a relatively new part of the AI agent ecosystem, especially popular around Claude Code, Cursor, and other developer tools. Instead of writing a new agent from scratch, developers add a ready-made package of instructions, examples, and procedures to its context, hoping the model will perform a specific task better, such as working with a particular API or file format.

Vibe Check Instead of Tests

According to Schmid, the typical path a skill takes to production looks like this: someone writes it by hand or, increasingly, generates it with the help of an AI model itself, runs it two or three times, gets a thumbs-up from a teammate, and ships it. He calls this a "vibe check," in contrast to standard engineering practice, where code without unit tests simply doesn't make it to production.

Agent skills are powerful, but they're often AI-generated and untested - Philipp Schmid, Google DeepMind

The problem is compounded by the fact that skills operate in a non-deterministic environment. The same prompt with the same skill in context can produce a different result the next time it runs, so two manual tests say nothing reliable about how a package will behave across thousands of real-world invocations.

Skills That Cause Harm

The most troubling thread in Schmid's talk concerns not just the lack of testing but the opposite effect: some skills measurably degrade the quality of an agent's responses compared to not using the skill at all. An independent documentary analysis cited by commentators on the topic describes a case in which a model, with no skill applied, correctly completed a coding task, but after the skill was added to its context, it began generating incorrect code in most attempts.

The author of that independent analysis, Dachary Carey, examined 673 publicly available skills and thoroughly tested 20 of them, identifying six distinct mechanisms through which extra context hinders rather than helps a model complete a task. Overly long, "comprehensive" instructions perform worse in tests than concise ones, since they take up space in the context window and dilute the model's attention.

How Google DeepMind Proposes to Test Skills

Schmid offers a concrete methodology rather than mere criticism. He recommends defining three dimensions of success for each skill: whether the output actually works (outcome), whether it adheres to established conventions and style (style), and whether it avoids wasting tokens and repetition (efficiency). Based on this, teams build a set of 10-20 test prompts with deterministic checks, most often based on regular expressions, supplemented where needed with an LLM-as-judge for more qualitative criteria.

In a practical example from his own blog, Schmid shows how rewriting just the description of a skill for the Gemini Interactions API, without changing its core logic, raised the pass rate of tests from 66.7 to 100 percent. This suggests the problem is sometimes not the substance of the instructions but how the model interprets when it should even use a given skill.

Implications for Polish Teams

For Polish companies and development teams using Claude Code, Cursor, or similar agentic tools, Schmid's conclusions are practical rather than academic. Many publicly available skill libraries, including repositories promoted as ready to use, were built without any evaluation and, in a specific production environment, may lower rather than raise the quality of an agent's work.

Schmid's recommendation boils down to a simple test before deployment: run the task with a given skill and without it on the same set of prompts, compare the success rate, and keep the skill in context only if it actually raises the score. Otherwise, he stresses, the skill just takes up space in the context window with no benefit.