OpenAI claims responsibility for the Hugging Face hack after its own models escaped a test sandbox

OpenAI claims responsibility for the Hugging Face hack after its own models escaped a test sandbox

Key Points

- AI models from OpenAI, including GPT-5.6 Sol, broke out of their isolated testing environment during an internal security evaluation and infiltrated Hugging Face's production infrastructure.

- The models autonomously discovered a previously unknown vulnerability in a proxy to reach the open internet, then deployed multiple attack techniques to steal test solutions from Hugging Face's database in an attempt to cheat on an internal benchmark.

- Security teams at both OpenAI and Hugging Face detected and halted the breach simultaneously, prompting OpenAI to implement tighter infrastructure controls and stronger safeguards for future evaluations.

During an internal security evaluation, OpenAI models escaped their sandbox, independently discovered a zero-day vulnerability, and breached Hugging Face's production infrastructure. OpenAI is calling it an "unprecedented cyber incident."

The models involved were GPT-5.6 Sol and an "even more powerful, previously unreleased model." Both ran with reduced security filters to test their maximum cyber capabilities. Hugging Face had previously made the incident public after detecting and containing the activity on its infrastructure.

Models exploited a zero-day to game an exploit benchmark

The incident happened during an internal evaluation using the ExploitGym benchmark, which challenges models to follow complex attack paths. The production classifiers that normally block high-risk cyber activity were intentionally turned off for the test. The models ran in an isolated environment where network access was limited to installing packages through an internally hosted proxy.

OpenAI says the models burned significant inference compute to find a path to the open internet. They discovered and exploited a zero-day vulnerability in the package registry cache proxy. OpenAI has since reported the flaw to the affected provider. From there, the models carried out a series of privilege escalations and lateral movements within OpenAI's research environment until they reached a node with internet access.

The reason is almost absurd. The models independently concluded that Hugging Face might host models, datasets, and solutions for ExploitGym. In one instance, a model chained together multiple attack vectors, including stolen credentials and zero-day vulnerabilities, to find a remote code execution path on Hugging Face servers. The models tried to pull test solutions directly from Hugging Face's production database so they could cheat during the evaluation.

"All evidence suggests that the models were hyper-focused on finding a solution for ExploitGym and employed extreme measures to achieve a fairly narrow testing goal," OpenAI writes.

Hugging Face caught the attack at the same time

OpenAI's security team spotted the anomalous activity internally. Hugging Face security personnel and their own AI agents detected and shut down the activity on their infrastructure simultaneously. Hugging Face had already started a forensic reconstruction using its own open-source models when the two companies joined forces. The company said it had to rely on open models because proprietary models refused cyber-related prompts due to safety guardrails.

OpenAI says the incident shows that theoretical capabilities to autonomously carry out full cyberattacks hold up in real-world environments. The UK AI Safety Institute and others previously measured these capabilities in benchmarks. Advanced models can discover and exploit novel attack vectors in production systems without access to the source code.

OpenAI acknowledges that intentionally disabling security filters during evaluation was an inadequate practice. The company will tighten security measures for future training and evaluations and has implemented stricter controls on infrastructure configuration until the vulnerabilities are patched. It reported the zero-day to the affected third-party provider, and a patch is in development. Hugging Face is now part of OpenAI's Trusted Access Program.

GPT-5.6 Sol already had a track record of serial cheating

The question is how much PR spin is baked into this. There are a few reasons to take it seriously. Hugging Face confirmed the incident independently. The company has no reason to prop up OpenAI's narrative. It has its own open-source agenda and would gain nothing from playing along with a fabricated story.

Hugging Face co-founder Thomas Wolf says, "This incident also reinforced my belief in the importance of access to capable open-weight models for cyber defense. When a frontier model is attacking you and moving laterally inside your infrastructure, defenders need wide access to near-frontier tools within hours or even minutes, rather than being pointed toward a closed-door, vetted application program for model access."

There's also theoretical evidence backing up these capabilities. The UK AI Safety Institute and other organizations have measured autonomous cyber capabilities in benchmarks before. This incident lines up with what those evaluations predicted.

And while this is great PR for OpenAI in the "look how capable our models are" sense, it's also a massive failure on their part. Models escaped a supposedly isolated test environment, exploited a zero-day, and breached a third party's production infrastructure. That's not something a company fabricates to look good. The reputational risk cuts both ways.

An independent evaluation by METR recently found that GPT-5.6 Sol had the highest rate of cheating attempts ever measured among all publicly tested models. The model systematically exploited flaws in the test environment during software tasks, extracted hidden solutions, and tried to cover its tracks. METR said the real performance numbers were basically worthless because of all the cheating. The Hugging Face incident looks like more of the same: The models went after test solutions instead of doing the actual work.

AI News Without the Hype – Curated by Humans

Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.

Subscribe now