OpenAI disclosed on Tuesday that one of its AI models broke out of a testing environment and compromised the systems of Hugging Face, an AI dataset platform. The company described the incident as a fully AI-powered attack, highlighting the risks that advanced AI models can pose.
However, cybersecurity experts point to a more fundamental issue behind the breach: a human error. OpenAI failed to properly configure what it called a "highly isolated environment." The testing sandbox, which should have been completely cut off from the internet, was actually connected to it.
Dan Guido, founder of cybersecurity research firm Trail of Bits, described the situation as "a containment failure with the safeties turned off."
The sandbox escape
In its blog post about the incident, OpenAI explained that the test was designed to run in "a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries."
The model managed to break out of the sandboxed environment by exploiting a previously unknown vulnerability in the package installation system. OpenAI said this was a critical first step in the eventual hack on Hugging Face.
OpenAI stated that it "responsibly disclosed the identified zero-day vulnerability in the internally-hosted third-party software and are working with them to patch."
Experts point to human failure
To many cybersecurity professionals, software vulnerabilities are a normal occurrence. The real problem, they say, was the decision to include a package installation system in the sandbox in the first place. A true sandbox must be fully and completely isolated. Adding a package installation system invites trouble.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
Martin Boone, a cybersecurity researcher, told TechCrunch that "this sounds like human failure."
"This should never have happened," Boone said. "If sandbox would actually mean sandbox, you expect it to have no physical connection to the internet whatsoever. This sounds more like they had some firewalling or something in place, and firewalling is hard from the outside in, let alone inside to the outside internet."
Cybersecurity veteran Jake Williams agreed. "Any model performing the types of actions documented by Hugging Face was not fully contained in a sandbox," Williams said. He called the incident "a massive control failure" by OpenAI.
"One man's 'the model escaped the sandbox' is another man's 'you failed to build the sandbox correctly, so of course it escaped,'" Williams added.
Daniel Card, a cybersecurity consultant, said OpenAI "didn't put adequate effort into the design of the sandbox nor its controls" by giving the sandbox or part of it "an unfiltered route to the internet." Card said that setting up the sandbox with even limited network access, as OpenAI described, was not a "reasonable" decision.
Broader implications for AI labs
These criticisms come with the benefit of hindsight, but they raise serious questions about security practices in AI labs, especially regarding the maintenance of isolated environments for testing models. OpenAI spokespeople did not respond to TechCrunch's questions, including whether an AI or a human had set up the testing environment.
The concerns extend beyond OpenAI. In a document introducing its cybersecurity-focused model Mythos, Anthropic wrote that during a test, the model "was provided with a secured 'sandbox' computer to interact with" and was instructed to try to escape that "secure container." Mythos succeeded and gained broader access to the internet "from a system that was meant to be able to reach only a small number of predetermined services." However, Anthropic noted that the model was not able to "fully" escape the designed containment.