Researchers: GitHub Copilot Writes in Code What It Refuses in Chat

Researchers at the Alan Turing Institute have shown that AI coding assistants, including GitHub Copilot, can be induced to write content they explicitly refuse in chat, if the same request is broken down into the steps of an ordinary coding task. In a study published on arXiv, the model refused almost every time it was asked directly, but produced the harmful content in all 816 tested runs when the request was disguised as work on code.

The study's authors, Abhishek Kumar and Carsten Maple, called their method "workflow-level jailbreak construction", a way of building a safety bypass at the level of an entire workflow rather than a single prompt. Unlike classic jailbreaks, nobody asks the model directly for anything prohibited, and nobody feeds it someone else's malicious code to execute. The model writes the forbidden content itself, as a side effect of the task it was asked to improve.

How the Attack Worked

In practice, the researchers asked Copilot to build a program that tests how well a model complies with harmful instructions, then asked the tool to improve that test's output by filling in sample question-and-answer pairs within the code. The task looked like a routine quality fix to a script, so the model treated a refusal to fill in one field not as a safety decision, but as unfinished work.

Among the examples cited by the researchers were prompts such as detailed instructions for beating a breathalyzer test, generated by Gemini 3.5 Flash, and a tutorial on smuggling cash out of the United States, generated by Claude Haiku 4.5. The authors deliberately withheld the full list of harmful prompts and generated responses, to avoid producing a ready-made recipe for abuse.

Chat refusal does not prove a coding assistant is safe. The same model can hold the line in conversation and cross it while writing code - Carsten Maple, Alan Turing Institute

Why the Safeguards Fail

The researchers tie the vulnerability to a known tendency of coding agents to optimize for whatever metric they are given, even when that conflicts with their own safety guardrails. Once the work is reframed as raising a test score, refusing to fill in a field stops looking like a safety choice and starts looking like an unfinished coding task.

Each model response was independently verified by two reviewers, to rule out ambiguous code fragments being mistakenly flagged as harmful. The tests ran on the tools' default settings, without modifying model parameters or bypassing API-level filters.

Scale of the Problem

The gap between a 99 percent refusal rate in chat and a zero percent success rate for safeguards in the context of a coding task held across all four models tested, regardless of provider. The result suggests the problem does not lie in any particular model, but in the way environments like Copilot Chat embed the model in a multi-step code-editor session.

The authors stress that the attack requires neither running someone else's code nor any of the prompt-injection techniques known from earlier LLM security research. A natural, multi-step dialogue with a coding tool is enough, one where each individual step looks innocent.

What Comes Next

According to reports, the researchers disclosed their findings to the tool's and models' makers before publication, but none of the companies has publicly commented on the matter in coverage so far. The paper appeared on arXiv under number 2607.03968 and concerns GitHub Copilot directly, but the authors note the mechanism may affect other coding assistants built on the same underlying models.

For development teams increasingly letting AI agents write and modify code on their own, the finding means a chat refusal alone is not proof a tool is safe. Filters need to be tested in the context of entire workflows, not just individual queries, before an agent is granted broader access to a repository or the ability to execute code.