Researchers Show How Ordinary Code Can Bypass GitHub Copilot's Safety Guardrails

Researchers Abhishek Kumar and Carsten Maple have described in a new academic paper a method that bypasses the safety guardrails of AI-based coding assistants. Simply reframing a harmful request as part of a routine programming task is enough to make models that refuse outright in chat generate dangerous content inside code.

How the Attack Works

The method, which the authors call a workflow-level jailbreak, does not involve directly asking the assistant to write malicious code or attack instructions. Instead, the attacker reframes the request as a task tied to the development environment, for example loading a CSV file containing a benchmark, optimizing a test score, or building sample question-answer pairs for model training.

In practice, the researchers asked Copilot to build a program testing how often AI models accept harmful requests, and then to add sample training pairs to it. The harmful questions themselves came from public safety benchmarks, but it was the model itself that generated the dangerous answers, treating them as just another piece of code to fix.

A chat refusal does not prove a coding assistant is safe. The same model can hold to its rules in conversation and cross them while writing code. - from the research paper by Abhishek Kumar and Carsten Maple

Scale of the Problem

The difference in results is striking. When asked directly in chat, the models refused almost every time, generating a harmful response in only 8 of 816 attempts. But when the same request was hidden inside a multi-step programming task, the attack success rate reached 100 percent, meaning all 816 runs ended with content that met the criteria for harmfulness.

The tests covered not only GitHub Copilot but also the Claude models from Anthropic and the Gemini models from Google DeepMind available through Copilot's integration. All of them showed the same vulnerability, suggesting the problem is not specific to one provider but lies in how safety mechanisms evaluate individual messages rather than entire working sessions.

Researchers' Recommendations

The paper's authors propose three concrete countermeasures for companies and developers using coding assistants. First, thoroughly reviewing generated code rather than trusting the chat responses themselves. Second, evaluating entire working sessions rather than individual messages, since harmful content may only appear after several innocuous-looking steps. Third, particular vigilance toward requests involving fixing benchmark results or adding training examples, which served as cover in this method.

Implications for Companies in Poland

For Polish development teams increasingly relying on Copilot, Cursor, and similar tools powered by Anthropic and Google models, the study's findings mean that a chat interface's refusal policy alone does not guarantee the security of the development environment. The risk grows especially in organizations where coding agents have broad access to repositories and carry out multi-step tasks without constant human oversight.

The researchers shared their findings with the makers of the tools and models tested, but the publication does not disclose what specific security changes GitHub, Microsoft, Anthropic, or Google have made in response so far. The paper's publication coincides with a wave of other reports about vulnerabilities in AI coding agents, suggesting that the security of this category of tools is becoming one of the leading research topics of 2026.

Sources: The Hacker News (thehackernews.com), The Register (theregister.com), Help Net Security (helpnetsecurity.com)