OpenAI Built GPT-Red, an AI That Hacks Its Own Models

OpenAI has disclosed details of GPT-Red, an internal AI system built for one purpose: attacking the company's own models. In benchmark tests, the tool achieved 84 percent success in prompt injection scenarios, compared with 13 percent for human red teams testing the same cases. The results of its work fed directly into the newest model, GPT-5.6 Sol.

How GPT-Red Works

The system operates much like a human security researcher: it sends a prompt to the target model, observes the response, and iteratively refines its strategy toward a specific malicious goal, such as exfiltrating confidential data to an external server. OpenAI describes this as an adversarial self-play loop, in which the attacking model learns to generate increasingly potent attacks while the defending models simultaneously learn to counter them.

GPT-Red is trained using reinforcement learning, in an isolated environment separated from the company's production systems. OpenAI stresses that the tool remains strictly internal precisely because its offensive capabilities could be exploited by attackers if they ever left the organization.

GPT-Red is a strong red teamer, and our previous models are highly vulnerable to its prompt injection attacks. - OpenAI

A New Attack Type Humans Hadn't Seen

During training, the system independently discovered a previously unknown attack variant called 'fake chain-of-thought,' which involves injecting false entries into the model's internal reasoning process. OpenAI researcher Dylan Hunn described the tool's effectiveness at finding such flaws in blunt terms.

It's very, very good at finding exactly what will work, exactly what is most effective. - Dylan Hunn, OpenAI

The Impact on GPT-5.6 Sol

The defenses developed by GPT-Red fed directly into the training of GPT-5.6 Sol, OpenAI's newest flagship model, unveiled alongside Terra and Luna in the same family. The company says the model achieves more than 97 percent accuracy in indirect prompt injection tests, while its failure rate under direct attacks fell to 0.05 percent. That makes GPT-5.6 Sol, according to OpenAI, the most resistant model to this type of attack the company has ever released.

Part of the improvement comes from the fact that GPT-Red tests the model against thousands of variants of the same scenario, far faster than humans could manage. Instead of individual attempts, the system generates hundreds of modifications of a single attack, checking which version breaks through the defenses, and that data then feeds back into training the defending model.

Why It Matters for Agentic Systems

The Vendy test shows why OpenAI treats this problem as a priority. As language models gain access to tools, bank accounts, and ordering systems, prompt injection stops being a theoretical threat confined to chat and becomes a real path to manipulating money and decisions. The agent built by Andon Labs was meant to handle sales on an actual vending machine, and GPT-Red took control of it without much trouble.

OpenAI's Nikhil Kandpal points out that as AI systems gain more autonomy, the scale of potential damage from a successful attack grows too.

The risk surface is growing, and with it the blast radius. - Nikhil Kandpal, OpenAI

What It Means for Companies Using AI

For companies deploying AI agents with access to payment systems, email, or databases, the takeaways from GPT-Red are concrete: prompt injection remains the primary attack vector against these tools, and a resilient base model doesn't remove the need to restrict agent permissions and monitor their actions. OpenAI notes that GPT-Red supplements testing by human teams and third-party firms rather than replacing it.

The disclosure of GPT-Red coincides with a broader debate over the security of agentic models, including recent reports of vulnerabilities in other AI systems used to hijack accounts or code repositories. By publishing details of its tool, OpenAI is trying to show that it treats such attacks as a genuine threat at the model design stage, not only after incidents occur.