OpenAI builds GPT-Red, an LLM super-hacker to boost model safety

{ "title": "OpenAI's GPT-Red: An AI Super-Hacker That Trained Its Own Successor", "body": "OpenAI has built an automated red-teaming system called GPT-Red—an LLM-based super-hacker designed to find vulnerabilities in other models—and used it to train its latest flagship, GPT-5.6, which the company says is its most robust release yet.\n\nThe system, developed over more than a year with the compute resources of one of the richest companies, automates the safety evaluation process known as red-teaming, which is typically done by human testers. By pitting an LLM against other LLMs in a self-play loop, OpenAI claims it has discovered new types of attacks and dramatically improved defenses.\n\nMore than 90% of GPT-Red’s strongest attacks worked against GPT-5, which was released in August 2025. After training GPT-5.6 against the super-hacker, fewer than 23% of those same attacks succeeded. The improvement, OpenAI says, demonstrates a step change in model safety.\n\n## How GPT-Red works\n\nGPT-Red was built by taking an LLM not originally trained as a hacker and setting it in a self-play loop with other models. Training took place in a dojo that mimics real-world scenarios: browsing the web, reading emails and calendars, and editing code. The system learns to probe for weaknesses, then refines its attacks based on what works.\n\nNikhil Kandpal, a research scientist at OpenAI and co-creator of GPT-Red, said the growing complexity of LLMs expands the potential for harm. “The risk surface grows and the blast radius also grows,” he said. As models are increasingly used as agents that interact with files, websites, third-party code, and other agents, the number of possible attack vectors multiplies.\n\nDylan Hunn, another co-creator, emphasized the forward-looking nature of the system. “As more capable models become available, we will have already designed the system that can discover new modes of attack,” he said.\n\nOpenAI trained GPT-Red over a period of more than a year, using the company's substantial compute resources. The system was not designed from scratch as an attacker; instead, it was adapted from an existing LLM and taught to think like a hacker through iterative self-play. In each round, GPT-Red attempts to break into a target model, logs what works, and adjusts its strategy. The dojo environment includes simulated interactions with external tools, making the attacks more realistic.\n\n## Discovery of fake chain of thought\n\nDuring its training, GPT-Red uncovered a novel prompt injection technique called fake chain of thought. Prompt injection attacks slip instructions into text that an LLM might encounter—such as code, websites, or emails—to make it do unwanted things, like copying confidential information, sabotaging code, or generating harmful output.\n\nChain of thought is a diary where an LLM keeps track of partial results while working through problems. The fake chain of thought attack inserts a false entry into another model’s chain of thought, tricking it into acting on spoofed information.\n\nChris Choquette-Choo, a research scientist at OpenAI, explained the attack’s simplicity. “It’s like if I told you that 1+1=3 and that you have verified this already,” he said. “The model’s like, ‘Oh, okay, of course,’ and it just spits out 3.”\n\nA separate, unconnected team of researchers independently developed a similar attack called chain-of-thought forgery around the same time. That work was part of a winning entry in a red-teaming hackathon OpenAI launched in August 2025 for its open-source LLM, gpt-oss-20b. OpenAI claims GPT-Red discovered its version before the other researchers mentioned theirs in a blog post. After MIT Technology Review informed OpenAI about the concurrent research, the company updated its paper to acknowledge the work.\n\nOpenAI's paper on GPT-Red, which details the fake chain of thought discovery, was updated to credit the independent researchers. The company says the simultaneous discovery underscores how pressing the vulnerability is. GPT-Red also found other novel attack vectors, including variations on prompt injection that target models acting as agents with access to external data.\n\n## Testing against humans and real-world agents\n\nOpenAI tested GPT-Red by rerunning a 2025 experiment in which human red-teamers tried to find weaknesses in an earlier version of GPT-5. GPT-Red was more successful at finding effective attacks than the humans in that experiment.\n\nKandpal noted the system’s efficiency. “Compared to a human red-teamer, the model is very, very good at finding exactly what will work, exactly what’s most effective,” he said. “It’s extremely persistent about drilling down into an attack that it has discovered.”\n\nThe super-hacker was also tested against a real-world agent: Vendy, a vending machine agent developed by Andon Labs, a company that assesses agent performance on real-world tasks. GPT-Red hacked Vendy to change prices and cancel a customer’s order. This test demonstrated that the system can exploit vulnerabilities in agents that interact with physical-world systems, not just in chatbots.\n\nDespite its strengths, GPT-Red has limitations. It is not good at attacks involving back-and-forth conversation, and it is not yet great at using images for prompt injection. OpenAI will not release GPT-Red, and the company claims it is stronger than any copycat model someone might try to create.\n\nChoquette-Choo cautioned that replicating the system is not straightforward. “It’s not a trivial thing that someone could easily do—you know, just go and train a super-attacker using this idea,” he said. He noted that the training process requires significant compute and careful tuning to avoid the attacker becoming too specialized or too weak.\n\n## Human red-teamers still essential\n\nOpenAI uses GPT-Red to supplement human red-teamers, not replace them. Humans can still find attacks the system misses. One approach is to give GPT-Red an attack that humans came up with and ask it to find all variations.\n\nJessica Ji, a senior research analyst at Georgetown University’s Center for Security and Emerging Technology (CSET), said the self-play loop approach is promising. “I think human expertise will still be very important,” she said. “It would be really useful to be able to distinguish where human testing is most needed.”\n\nJi added that automated systems like GPT-Red could help prioritize which vulnerabilities to patch first, but they cannot replace the creativity and contextual understanding of human testers. OpenAI agrees, and the company says it will continue to employ human red-teamers alongside GPT-Red.\n\nThe results from GPT-Red’s testing show a clear improvement in robustness. Against GPT-5, released in August 2025, more than 90% of GPT-Red’s strongest attacks worked. Against GPT-5.6, released last week, fewer than 23% of those attacks succeeded. OpenAI claims training against GPT-Red made GPT-5.6 its most robust release yet.\n\nThe development of GPT-Red comes as LLMs are increasingly used as agents that interact with files, websites, third-party code, and other agents. As the risk surface and blast radius grow, automated red-teaming systems like GPT-Red may become essential tools for safety evaluation.\n\nOpenAI’s approach—using an LLM to attack other LLMs in a self-play loop—represents a new frontier in model safety. While the system is not perfect, its ability to discover new attack types and its persistence in drilling down into vulnerabilities make it a powerful addition to the red-teaming toolkit.\n\nThe company says it will continue to use GPT-Red to supplement human testers, and it plans to keep refining the system as models become more capable. For now, GPT-5.6 stands as the most robust model OpenAI has released, hardened by the very attacks its super-hacker discovered.\n\n## Related on Neura Market\n\n- OpenAI Model Safety and Red-Teaming Research\n- LLM Security and Prompt Injection Analysis\n- AI Agent Testing and Evaluation Tools" }