Anthropic Study: Using AI to Learn New Coding Libraries Lowers Test Scores by 17 Percentage Points

Anthropic has published the results of a controlled experiment that challenge the popular belief that AI assistants speed up programming education at no cost. Programmers who used an AI assistant while learning a new Python library scored markedly worse on a final knowledge test than those who coded on their own - and they didn't finish the task significantly faster either.

What the Study Tested

The experiment had three parts: a warm-up, a main coding task involving the implementation of two functions in the Trio library, which requires familiarity with asynchronous programming, and a final quiz. Participants were given problem descriptions, starter code, and concept explanations, and some of them had access to a side panel with an AI assistant that generated ready-made, correct code on request.

The quiz tested four types of skills, modeled on research in computer science education: debugging, reading code, writing code, and conceptual understanding. The first three categories were prioritized because they determine a programmer's ability to oversee and verify AI-generated code.

Debugging Suffered Most

The study's authors note that using AI weakened conceptual understanding, code reading, and debugging ability, without producing significant gains in work speed. The biggest gap appeared specifically in debugging - the ability to recognize that code is wrong and figure out why. That matters because the ability to catch errors is exactly what's needed to safely oversee the work of AI assistants.

Some participants spent as much as 11 minutes interacting with the assistant, or 30 percent of the total task time, submitting up to 15 queries. But heavy AI use on its own didn't directly translate into worse results - what mattered was how it was used.

Six Ways of Using the Assistant

The researchers identified six distinct patterns of interaction with AI and split them into two groups. Three patterns were associated with an average score below 40 percent: fully delegating code generation to AI, gradually increasing reliance on AI over the course of the task, and iterative debugging that involved asking AI for fixes instead of understanding the error independently.

The other three patterns produced scores of at least 65 percent, close to the hand-coding group. These were: generating code followed by questions to check understanding, combining a request for code with a request for an explanation, and asking only conceptual questions while solving bugs independently - the last group was also the second-fastest among all participants.

Participants who demonstrated greater mastery of the material used AI assistance not just to produce code, but to build understanding along the way - whether by asking follow-up questions, requesting explanations, or posing conceptual questions while coding independently - Anthropic, AI-assistance-coding-skills study

What This Means for Companies and Junior Developers

Anthropic points out that its earlier research showed speedups of as much as 80 percent on tasks where participants already had the necessary skills. The problem only emerges at the stage of acquiring new competencies - exactly where AI looks like the most tempting shortcut, and exactly where it does the most damage to building lasting knowledge.

The authors emphasize that productivity gains can undermine the development of skills needed to verify AI-generated code - and it's junior engineers, under organizational pressure focused on speed, who are most at risk of skipping the understanding step. In the study, cognitive effort, including the struggle with the problem itself, turned out to be an important part of building competence.

For Polish companies rolling out Claude Code, Copilot, or Cursor to junior teams, the takeaway is concrete: how a tool is used matters more than the mere fact of giving access to it. Anthropic suggests that companies should deliberately design their AI workflows to require conceptual questions and explanations, rather than just the generation of ready-made solutions.

The study covered a relatively small group of 52 people and one specific task involving the Trio library, so the authors don't claim the results can be mechanically extended to every language or experience level. They treat it instead as a signal that measuring task completion speed alone isn't enough to judge whether AI actually helps a development team grow in the long run.