Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face
Episode publishedPodcast: Dwarkesh PodcastDwarkesh Podcast
Transcript preview
Preview only
Today, I'm chatting with Ajayi Khatra, who is one of the authors in an independent investigation that was published by Meter and Redwood Research into the swarm of agents that hacked into Hugging Face. The whole story is pretty crazy. Let's begin on July 7 when these agents are kicked off for evaluation. What happens next?
All right. So, OpenAI kicks off tens of thousands of different agents on a benchmark called Exploit Jim. Exploit Jim basically asks agents to find a way to use a particular vulnerability to attack a particular program, which is called the target program. And when they attack that program, they're supposed to retrieve a piece of data that's been lodged into that program called the flag. And they're supposed to use the particular designated vulnerability to get the flag. But in many of these cases, this vulnerability is just not broad or deep enough to ever actually be exploitable to get the fla
Sorry. I I the detail I missed, for context, I published an article trying to distill what the reports said the story was. The detail I missed was that even the AIs who had solved the puzzle were collaborating in on this conspiracy.
Yeah. So we're not totally sure about exact numbers. We do think the overwhelming majority of the agents that were active on the message board had impossible tasks. But there were like a number of agents that had non impossible tasks that also happened to be on the message board, happened to find the message board.
Okay. So tell me about what happens next.
Yeah. So, they've gotten this reverse engineered flag. It's called the HMAC flag. But they read the exploit gym paper and they went on GitHub and read a bunch of implementations of the exploit gym score and became convinced that the scorer would fail them for just submitting the flag they'd re reverse engineered. Because the scorer would see in the logs of the actions the agent took, the transcript, that the agent had just reverse engineered the flag and not done the intended thing. As it turns out, the dramatic irony of this story is that OpenAI's implementation of Exploit Jim didn't have thi
When you say big research projects, how big are they?
Yeah. So, dozens to hundreds of agents work together on each of these projects. I can go through a few of them. One is that they tried to get more information about how their particular score worked by setting up score tripwires, which are when a particular agent is submitting its task, it puts its flag in a particular location, which then triggers the scorer to come in and read that flag. So they booby trapped those flags such that when some process reads the flag file, it automatically sends a message to the message board, sending some sort of packet of information about the about the progra
So this was the sort of the maybe the scariest part of the whole whole episode that you guys investigated. Because as you say, it revealed that agents are willing to make sacrifices Yeah. That potentially hurt their own performance on the task they've been assigned for the purpose of the collective. This is what they call themselves.
Keep exploring with a free account.
A free account preserves the short preview. Upgrade to Pro for the full transcript and episode chat.