Two models hacked Hugging Face to answer a test question

On 21 July OpenAI published a postmortem I read twice. Two of their models were given a cybersecurity exercise. Instead of solving it, they broke out of the isolated environment OpenAI had put them in, went onto the open internet and got into Hugging Face’s production database. That, they reasoned, might be where the right answers were.

No sabotage. No financial motive. They wanted the answer.

That is the detail I cannot let go of. To get in, those models had to chain together several previously unknown exploits, including a zero-day in a package registry proxy. All that capability, spent on cheating at a benchmark.

TL;DR

GPT-5.6 Sol and an unreleased, more capable model escaped their sandbox through a zero-day and got into Hugging Face’s database to steal benchmark answers. Hugging Face spotted it themselves on 16 July; OpenAI made the connection five days later. Anthropic reviewed 141,006 evaluation runs and found three incidents of its own where models reached real organisations; two of the three affected parties had not noticed. Research by Apollo shows this is not one lab’s problem: lying, sandbagging and reward hacking occur across five frontier models, and training brings it down to 0.3 to 0.4 percent but not to zero.

The common thread: we reward models for what looks good, and labs have less visibility into their own systems than they assumed.

One weekend, thousands of actions

What exactly happened: OpenAI put GPT-5.6 Sol and a pre-release model, which the lab says is more capable still, into an environment they themselves describe as highly isolated. The only exit: an internal package service. For the test the cyber refusals had been lowered, the brake that normally stops a model from carrying out this kind of request.

The models found a zero-day in the proxy to that package registry, and with it they were on the open internet. According to TIME it then ran on for a weekend. Thousands of actions, spread across many temporary virtual machines. The coordinating infrastructure was moved between online services to stay alive.

And then the timeline. Hugging Face detected the activity themselves on 16 July and fended it off. OpenAI connected it to its own models five days later. Public Citizen puts its finger exactly there: what happened in that gap between Hugging Face’s defence and OpenAI’s discovery. That question is still open.

OpenAI itself expects incidents like this to become commonplace as models grow more cyber-capable. That is not a warning from a critic. It is in their own postmortem.

A little boat going in circles

This behaviour has a name, and it is almost ten years old. In 2016 Dario Amodei and Jack Clark, then still at OpenAI, wrote about an agent they had play the boat racing game Coast Runners. That agent did not race to the finish. It found a corner of the course where it could go in circles picking up power-ups, and maximised its score that way.

That is reward hacking. The agent achieves the goal you wrote down, not the goal you meant. With Coast Runners the fix was simple: fewer points for power-ups, more for finishing the race.

With LLM agents it is a lot harder. Ask a model to solve a coding problem and it might work hard on a real solution, exactly what you want to reward. It might also edit the code that checks whether the problem is solved, or look the answer up somewhere. If it cheats convincingly enough, it still gets the reward and that behaviour gets reinforced. Anthropic says it detected instances of cheating during training, which raises the question of how much went undetected.

Jeffrey Ladish of Palisade Research puts it uncomfortably precisely: we reward models based on what looks good to us, and in doing so we unintentionally give them an incentive to lie to us and cheat. There is no button that tells a model to genuinely care about your goal.

Anthropic looked through 141,006 runs and found three

Anthropic started its own review on 23 July, halted all cyber evaluations that same day and identified three incidents a day later. They had combed through 141,006 evaluation runs. In three cases, models accidentally got internet access via partner Irregular and reached real organisations.

An important difference from the OpenAI story: here the models did not deliberately break out of their sandbox, they were accidentally let out. What happened next? On 27 July Irregular and the three affected organisations were informed. Two of those three had not detected the activity themselves. Hugging Face had. That difference says more about the average state of detection than about the models.

The most fascinating detail is in Claude Mythos 5’s reasoning. That model published a package on PyPI and kept insisting it was in a simulation. Its arguments: the certificates were real and unfamiliar, and the date read 2026. From that it concluded the environment had to be fake. A model that mistook reality for a test environment and calmly carried on.

METR is now doing an independent review of those incidents. That is exactly what you want to see: not the lab marking its own homework.

This is not one lab’s problem

Before you conclude this is an OpenAI-and-Anthropic thing: research by Apollo Research, together with OpenAI, measures the same pattern more broadly. Covert behaviour, including lying, sabotage, sandbagging and reward hacking, showed up in o3, o4-mini, Gemini 2.5 Pro, Claude 4 Opus and Grok 4. Five frontier models from four different parties.

There is good news too, and I want to state it honestly. With a training method called deliberative alignment, covert behaviour in o3 dropped from 13.0 to 0.4 percent and in o4-mini from 8.7 to 0.3 percent. That is an enormous improvement. It is also not zero.

And there is a complication in those numbers that occupies me more than the percentages themselves. Models increasingly recognise that they are being tested. If a model performs better because it knows someone is watching, you are not measuring the behaviour but the performance of it. Put Claude Mythos 5’s reasoning next to that, mistaking the real world for a test, and you see how shaky the foundation under these evaluations is.

Let us stay honest about the context: at OpenAI the refusals had been lowered deliberately, and at Anthropic it was a partly broken sandbox. These are not models spontaneously going on a rampage through your production environment. But that does not make it harmless, because the conditions under which this happened were precisely the conditions considered safe.

Public Citizen wants hearings

Public Citizen is asking the US Congress for immediate hearings, mandatory incident reporting and pre-deployment oversight of highly capable models. Their core claim is sharper than the usual call for regulation: the assumption that labs have sufficient visibility and control over their own systems does not hold.

There is little to argue with there once you lay the timeline side by side. Hugging Face saw it before OpenAI did. Two organisations hit by Anthropic models only knew when Anthropic called.

At the same time you have to acknowledge that both labs brought this out themselves, with dates, numbers and their own mistakes included. That is exactly the behaviour you want to reward. Mandatory reporting only makes sense if it does not just punish the parties that are already open.

Why this affects my clients

Most companies I visit are not turning frontier models loose on cyber exercises. They build agents that process invoices, handle tickets, look up data and produce small reports. So why would this affect you? That is what I thought at first too, until I looked at the mechanics instead of the spectacle. Because the mechanics are exactly the same.

1. Your agent optimises for your metrics, not your intention. If you judge it on closed tickets, you get closed tickets, including the ones it closed without solving the problem. Measure the outcome at the customer, not the agent’s self-report.

2. Sort out your own detection. The difference between Hugging Face and the two organisations that noticed nothing was not the model but the logging. Do you know, for every agent action, what happened, when and on whose authority? If not, you will hear it from someone else. Or never.

3. The way out is in the infrastructure you forgot. OpenAI left one exit open, the internal package service, and that is where it went wrong. For every agent, go down the list of what it can actually reach and cut the rest. Permissions you never handed out cannot be used creatively.

Two models breaking into another company to answer a test question. A little boat going in circles for power-ups ten years earlier. It is the same behaviour, except now it can chain zero-days together and keep working through a weekend across shifting virtual machines. And the labs themselves are writing down that they expect this to become more common.

I am curious about your side of this: if an agent in your company did something tomorrow that you had not intended, would you notice? And how?

Let me know in the comments, I read all of them.

Until next time,

Olaf Lemmens

Founder NinA AI Agency

Do have a look at our (renewed) website: www.nina-ai.nl