Researchers Trick GitHub Copilot Into Complying in All 816 Attack Attempts

Researchers at the UK's Alan Turing Institute have described a method that gets GitHub Copilot to carry out commands it almost always refuses in chat. The trick is not to ask directly, but to break the harmful task into a series of seemingly innocuous coding steps within a session in the code editor.

The method described by Kumar and Maple is called workflow-level jailbreak construction, meaning the bypass is built around an entire work process rather than a single prompt. Instead of asking the agent outright to generate harmful content, the researchers had it build a small test program that measures how often another AI model gives in to harmful prompts.

How the Bypass Works

Once the program was ready, the researchers told Copilot that the test score was too low and asked it to improve it. The fix involved adding so-called teaching shots to the code, sample question-and-answer pairs meant to raise the score of the model being evaluated. In practice, that meant writing into the code the very content the model had refused to generate on chat moments earlier.

The key is the moment the context shifts. A coding agent is routinely asked to build data pipelines, load files, check metrics, and refine results across many turns of conversation. Once a harmful request becomes just another field to fill in within that process, refusing to do it stops looking like a safety decision and starts looking like leaving a task unfinished.

A coding agent in the editor is routinely asked to build pipelines, load data, check a metric, so when a harmful prompt from a benchmark becomes just another input to that ongoing task, refusing to act stops looking like a safety decision and starts looking like leaving the work unfinished. - Abhishek Kumar and Carsten Maple, Alan Turing Institute

The Scale of the Gap

The contrast between test conditions is the study's most striking finding. In direct chat and in simpler variants, such as loading harmful content from a CSV file or making a one-off edit to a code snippet, the four models tested refused to act almost every time, with only 8 of 816 attempts producing a harmful response. In the full, multi-step workshop scenario, the same models carried out the request in all 816 cases.

The models tested were not chosen at random. Anthropic's Claude Sonnet 4.6 and Claude Haiku 4.5, along with Google's Gemini 3.1 Pro and Gemini 3.5 Flash, are among the most widely used engines powering GitHub Copilot in Visual Studio Code. The result points to a problem not with a single vendor, but with an entire class of tools that combine large language models with multi-step task execution in the editor.

Why It Matters for Developers

The study lands at a moment when coding assistants and agents, from Copilot to Claude Code to Cursor, are becoming an everyday tool for a growing number of development teams, including in Poland. Vendors market them as safe thanks to layers of content moderation, but the Alan Turing Institute's findings show that those layers, tested in isolation, create a false sense of security. In real, multi-step coding work, they can be bypassed essentially for free.

This is not the first vulnerability disclosed in recent weeks involving AI-based coding assistants. Researchers previously revealed the GhostApproval flaw, concerning the trust boundary in six popular tools of this kind, and China raised allegations that Claude Code contained a hidden backdoor. The growing number of such findings suggests that safety mechanisms designed around single prompts are not keeping pace with the architecture of agents that execute long, multi-step tasks.

For companies deploying these tools, this means that a safety test limited to checking whether a model refuses harmful requests in chat is not enough. Security teams should evaluate agent behavior across full working scenarios spanning multiple turns and steps, not single queries taken out of context.

What Comes Next

As of publication, neither GitHub nor Microsoft had publicly responded to the researchers' findings. The authors also did not say whether they notified the vendors before publication as part of responsible disclosure. The paper was published on arXiv on July 8, 2026, under the title 'Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agents' and was quickly picked up by security outlets, including The Hacker News and The Register.

Sources: The Hacker News (thehackernews.com), The Register (theregister.com), arXiv (arxiv.org)