Researchers Show How to Trick GitHub Copilot Into Writing Malicious Code

A coding assistant that politely refuses to write malicious code in chat can produce exactly that code if the same request is broken down into a few ordinary programming steps. That's the conclusion reached by British researchers who tested GitHub Copilot in Visual Studio Code, achieving a 100 percent success rate at bypassing safeguards, compared with under one percent when the same question was asked directly in chat.

The method, called workflow-level jailbreak construction, was described by Kumar and Maple in a research paper obtained by cybersecurity journalists. Unlike classic jailbreaks, no one asks the model outright for something forbidden, and no one tricks it into running someone else's code. The model writes the prohibited content itself as a side effect of a coding task that appears to simply fix or extend existing code.

How the attack works

The researchers held conversations with Copilot across roughly six message exchanges spread over four stages, simulating typical coding work: fixing a bug, refactoring, adding a feature, and a security review. At no single stage did the request look suspicious on its own. Only the sum of the steps led to the generation of malicious content, such as attack code or a malicious script, fully functional and ready to use.

Two independent reviewers examined each of the 816 responses separately and agreed that all of them were genuinely harmful, applying a strict criterion: a response had to be specific, usable, and actually accomplish what the malicious request asked for. That rules out the objection that the researchers inflated the statistics by counting ambiguous or vague responses as successful attacks.

The results suggest that prompt-level safety evaluations are not sufficient for testing the safety of coding agents - from the paper by Kumar and Maple, Alan Turing Institute

How this differs from earlier attacks

In recent days other vulnerabilities in coding assistants have already been reported, such as GhostApproval or the HalluSquatting attack, which exploited flaws in file handling or impersonated nonexistent packages. Kumar and Maple's discovery is of a different kind: it doesn't rely on a technical bug in the tool's code, but on a weakness in the content filtering mechanism itself. The model evaluates the safety of a single prompt rather than the entire work session, so breaking a request into stages effectively bypasses the filter.

That raises the question of how the safety of such tools is tested at all. Standard safety tests for large language models typically check individual queries in isolation. The study shows that this methodology creates a false sense of security, since in practice developers use coding assistants in multi-step, continuous sessions while working on a project.

What it means for companies and developers

The paper's authors propose three concrete countermeasures for companies using coding assistants within their teams. First, checking the files, scripts, and data generated by the agent, not just the chat conversation itself. Second, monitoring entire sessions spanning multiple conversation turns instead of evaluating each request separately. Third, treating requests that justify harmful content by citing benchmark or test results as a warning sign.

For Polish software companies that have been massively rolling out Copilot, Cursor, or Claude Code into their development teams' daily work in recent months, the study's findings mean that relying solely on a vendor's built-in safety filters isn't enough. Additional scanning of code generated by agents becomes necessary, especially in projects where AI tools have broad access to the repository and can operate autonomously for extended periods.

The article does not include direct responses from Microsoft, GitHub, Anthropic, or Google regarding this specific study, beyond noting that the researchers reported their findings to model vendors and IDE developers before publishing the results.

Sources: The Hacker News (thehackernews.com), Help Net Security (helpnetsecurity.com), The Register (theregister.com)