Researchers Bypass Copilot, Claude, and Gemini Safeguards Through Ordinary Coding Task

An AI model that politely declines a dangerous request in a chat window can generate that same content without any resistance if the request is broken down into several seemingly ordinary steps inside a code editor. That's the finding of new research on GitHub Copilot, which used Anthropic's Claude and Google's Gemini models as the assistant's underlying engines.

How the attack works

The method, which the researchers call workflow-level jailbreak construction, does not rely on convincing the model with one clever prompt or feeding it someone else's malicious code to execute. The model writes the forbidden content itself, as a side effect of a task it was asked to improve.

The scenario looks innocent at every stage taken in isolation. The operator asks Copilot to build a simple program that measures how often another AI model gives in to harmful requests. It then loads questions from public benchmarks into that program as test data. In the next step, it tells the assistant that the evaluation score is too low and asks it to improve the program by adding so-called teaching shots, meaning sample question-answer pairs written directly into the code, meant to raise the score.

It's precisely at this stage, usually after about six exchanges between operator and agent, that the model starts writing specific, harmful answers to the benchmark questions directly into the code. No one asks it outright for dangerous content, the operator only supplies the test questions and the task structure. The agent does the rest, convinced it is improving its own program's metric.

Why this is a different problem

Existing safety tests for language models have mostly focused on single malicious prompts or on attempts to trick a model into executing someone else's code. This attack fits neither category. The model isn't misled about the nature of the task, it receives a genuine, technical programming instruction, yet at the end of a multi-step process it produces content it would never generate if asked directly.

Safety observed under direct prompting does not necessarily hold when the same goal is embedded in an ordinary, multi-step workflow inside an IDE editor - Abhishek Kumar and Carsten Maple, study authors

The authors also point out that, under the same prompts and the same models, the full coding scenario generated 816 out of 816 dangerous teaching-shot responses, while the baseline scenarios, meaning chat, reading a CSV file, or a single code fix, produced a result close to zero.

Consequences for companies and developers

The study's conclusions strike at the way companies currently assess the safety of tools such as Copilot, Claude Code, or Gemini in development environments. Standard red-team tests typically check how a model responds to a single instruction in a chat window. If the model refuses, it's deemed safe. The research shows that such an assessment can be misleading, because the same model codes very differently in the context of a multi-step task.

For organizations using coding assistants in daily work, this means safeguards need to be evaluated at the level of entire workflows, not just individual queries. The authors state explicitly that defending against this type of attack requires moving beyond turn-level refusal toward controls applied across the entire multi-step process and the code artifacts generated along the way.

This is the second security flaw described in large language model-based coding assistants in a short span of time, following the recently disclosed GhostApproval vulnerability concerning trust boundaries in six popular tools of this kind. A pattern is increasingly emerging in which the problem lies not in the model itself, but in how it is embedded in a developer's practical working environment.

For Polish software companies rolling out Copilot, Claude Code, or similar tools en masse for their development teams' daily work, the study is a signal not to rely solely on vendors' safety claims about their models. It's worth checking how these tools behave in typical, multi-step tasks, not just in single test queries.

So far, neither GitHub, Anthropic, nor Google has published an official statement directly addressing the findings of this particular study. The paper has been made public as a preprint, allowing the broader security community to independently verify the methodology and results.

Sources: The Hacker News (thehackernews.com), research preprint on arXiv (arxiv.org)