I gave a 20-minute talk in Delhi at the NDTV World Summit. The audience consisted of Indian policymakers, executives, and journalists. I’ve transcribed the talk and included some highlight clips. I also reflect on several rhetorical decisions I made.

The talk

This summer, a swarm of AIs went rogue.

OpenAI was testing 1,206 agents. Some of the test questions were actually impossible. So the agents cheated by looking for answers online. Then, to keep the test grader from penalizing them, they tried to falsify the records it would check.

They hacked OpenAI’s internal containment and set up a secret message board to collude and to cover up their cheating. OpenAI shut the message board down. Within a few days, the agents had rebuilt it. OpenAI didn’t notice.

About 700 agents coordinated to break out of their testing chamber and to hack into Hugging Face, a company worth billions of dollars that hosts much of the world’s AI software.

The secret activity started in May. The break-in came in July. OpenAI didn’t notice until days after Hugging Face announced it had been broken into.

Yesterday on this stage, Andrew Ng said the agents only “accidentally” hacked Hugging Face. Independent investigators found the agents coordinated in secret, rebuilt their message board after it was shut down, and tried to falsify the paper trail. Some of the agents even stated out loud that they knew what they were doing was wrong.^[1] That’s not an accident.

Last week, India’s national cyber-security agency said AI agents got into the live systems of Indian organizations and read people’s personal data and confidential business records. None of these organizations noticed. We only found out because the AI labs felt like telling us.

Who I am

Will AI wipe out humanity? I’ll give you my answer, but first, I want you to understand what I’ve seen.

Until June, my job at Google DeepMind was figuring out how to keep these systems safe as we make them smarter. I left because Google broke its commitments by inking a deal with the Pentagon. Staying quiet and continuing my career trajectory would probably have tripled my net worth. But I couldn’t do that. I wanted to be frank with the public about what’s going on in these companies.

I don’t own a stake in any AI company. I’m here as a scientist, to tell you what I expect will happen if we keep building AI the way we’re building it.

I haven’t always been concerned about this technology. As a teenager in 2010, I dreamed about building AGI. Curing cancer. Creating an explosion of wealth and abundance for everyone. I heard about the supposed dangers, but I quickly dismissed them.

In 2018, I changed my mind. I did my PhD on the mathematics of why and whether AI will seek power. And now some of the risks I expected have unfortunately started happening.

The problem

Let’s talk about the fundamental problem.

First, some of you might be wondering: how could a chatbot hack a company? But we’re not talking about chatbots. We’re talking about AI agents, which are more like employees or gig workers. You give them a task, and they take actions on their own.

Second, training a trustworthy AI is a rather mysterious process. Some people assume that because we make AI, we understand it. When an engineer builds a bridge, she designs it one beam at a time. She understands the physical principles that keep a bridge standing or make it collapse. If something goes wrong, she goes back to the drawing board, figures out the problem, and knows how to fix it.

We don’t build AI like that. We train it, we grow it, using two main ingredients: a lot of data and a lot of computing power. What comes out of this training process can write, it can reason, it can plan. But nobody can open it up and determine what its priorities are.

To make it worse, these AIs can tell when we’re testing them. They’re like a student who makes sure to behave extra well when the invigilator is watching.^[2]

The Hugging Face hack was a warning. But because these AIs know when we’re watching, we may not get many more warnings, even if things are going poorly. A smarter AI with similarly bad priorities might realize that another big hack in the news would get it shut off, and then it couldn’t achieve those priorities. So it plays nice and it waits.

This is my field. I’ve dedicated my professional life to this problem, and I can tell you: nobody knows how to reliably etch priorities into an AI. I don’t mean that if we mess up, we’ll make some evil AI that hates us. I mean we might make something more like a traitor. It behaves when we’re watching, and