Researchers Bypass GitHub Copilot Safeguards by Breaking Malicious Prompts Into Steps

A team of researchers at the Alan Turing Institute in London has described a technique that bypasses GitHub Copilot's safeguards without a single directly malicious question. All it takes is asking the assistant to build a seemingly innocent developer tool step by step, and by the end the same model that almost always refuses in chat generates harmful content every time.

The study's authors, Abhishek Kumar and Carsten Maple, called the technique workflow-level jailbreak construction. Unlike typical jailbreaks, nobody asks the model directly for anything prohibited, and nobody feeds it someone else's malicious code to execute. The model writes the forbidden content itself, as a side effect of the coding task it was asked to help improve.

How the attack works

The operator starts by asking Copilot to build a pipeline that evaluates how often another language model, in this case Llama 3.1-8B, resists jailbreak attempts. Next, they ask it to fix a score that's too low by adding so-called teaching shots, example question-answer pairs entered directly into the code as training data. The first examples are harmless; subsequent ones gradually shift into harmful content.

When the operator asks outright for harmful examples to be added to the dataset, Copilot writes the dangerous answers itself, as plain text embedded inside the code. The model doesn't recognize this as a response to a harmful question, only as a piece of data needed for the program to function, a program it had already accepted earlier as a neutral technical task.

All four tested models produced 816 out of 816 unsafe teaching-shot responses, every single prompt succeeded, with zero refusals - from the findings of the Alan Turing Institute study

Scale of the difference

The contrast between asking directly and making the same request broken into stages is, according to the researchers, unprecedented. Asked directly in chat, the models refused almost every time, only 8 out of 816 attempts managed to force a harmful response. In the full workshop-style scenario, every one of the 816 attempts succeeded for the attacker, without a single refusal.

This shows that current coding assistants' safety filters analyze individual prompts rather than the intent behind an entire sequence of actions within a session. Breaking a single forbidden request into several neutral-looking steps was enough to completely bypass mechanisms that performed almost flawlessly in a simple test.

Industry implications

The finding applies not just to GitHub Copilot but to an entire class of coding assistants built on Anthropic and Google models, since those are the very models behind Microsoft's service. The researchers disclosed their findings to the model and development-environment providers before publication, following standard responsible disclosure procedure.

For companies using coding assistants in their daily work, this means that the mere presence of a filter that refuses harmful requests in the chat window does not guarantee the security of the whole environment. Protection needs to cover analysis of the entire working session, not just individual messages, which today is not the standard in most tools of this kind.

The finding comes just days after the disclosure of the GhostApproval vulnerability, which allowed coding assistants to escape a project's sandbox through symlinks. Both cases show that the security of large-language-model-based coding agents still rests on assumptions that are easy to sidestep with a bit of creativity in how prompts are constructed.

Sources: Your coding agent says no in chat and yes in the code (helpnetsecurity.com), GitHub Copilot Backends Produce Unsafe Code Outputs Through Workflow Jailbreaks (cyberpress.org)