In 2017, after getting laid off from a startup, I spent a while looking into AI risk:

Many people I respect have started to think that AI risk is the most valuable place to focus EA efforts. For example, 80000 Hours ranks it first on their "list of global issues", taking into account "scale, neglectedness, and solvability". On the other hand, I have a lot of friends working in machine learning, and none of them think AI risk is worth working on now. This level of disagreement is very strange, and kind of worrying.

After playing for a dance weekend in South Carolina where casual conversation among non-tech people included earnest speculation over the likelihood that AI gets us all killed, I thought it might be a good time to look back at when this was really speculative.

Summary: I talked to the right people, heard some incredibly prescient ideas, and got the bottom line wrong.

---

I do a lot of my thinking via serial writing, and in this case that was a series of twelve blog posts. Taking them one at a time:

Looking into AI Risk (2017-06-26).

This held up well. I picked something well worth looking into, identified a lot of relevant resources, and got a lot more good reading recommendations in the comments.

Superintelligence Risk Project (2017-07-03).

The main part of this post were five ideas about where the disagreement might be. These worked ok as jumping off points for interviews, but they weren't as core to disagreements as I hoped: people often ended up with similar views on most of these ideas while still being far apart in their views on how we should prioritize AI risk.

One thing I do like, though, is that these were pretty concrete questions which now do have answers:

How likely is it that current approaches are all we need for AGI, with relatively straightforward extensions and a lot of scaling?

Very likely. When I asked this question, "current approaches" meant deep neural nets and reinforcement learning. The past nine years have mostly been an incredible amount of scaling.

How valuable is it to work on solving problems that are probably not the right ones? For example, even if we think AGI will not look like current systems, might trying to solve the control problem for current systems teach us enough about the underlying problem and how to do this kind of work that we'll be in a better position once we see more what AGI will actually look like?

Looking at the two best-known approaches in 2017, I'd say Concrete Problems in AI Safety has held up extremely well, while Highly Reliable Agent Design (contemporaneous review) less so. This makes sense, since the paradigm of the former (scale current things) is what happened, while the latter assumed we'd get to powerful AI systems via explicit systems with world modeling, decision theory, etc.

Still, even the HRAD work did end up being pretty useful, by getting people thinking about important problems, naming things, building a field, and getting people spun up.

So I'd say work was pretty useful even if you didn't anticipate the shape of today's AI.

How useful is it to have a strong theoretical foundation, vs just understanding the technology enough from an engineering perspective that we can make it do things for us?

In terms of building powerful AI systems, it seems like it has been near 100% engineering. On the other hand, we now have powerful systems that no one understands. As they get increasingly capable will they continue to be compatible with the continued existence of humanity? Hard to say! Seems like a strong theoretical foundation would be pretty useful.

How similar is this to normal engineering? How much should we expect companies' desires that their AI systems do what they want them to do to work out?

Mostly yes, but more importantly no. Companies want their models to be economically useful which means doing what users want. Except it turns out you can provide a ton of economic value while still occasionally being seriously misaligned, and there's a real chance that as we continue to scale this misalignment is going to work out very poorly.

As we get closer to AGI, how likely is the ML community to take superintelligence risk seriously? Is it just that they don't think it can be productively worked on now or do they not think it will ever be a real problem?

People are taking this increasingly seriously, and I think this has probably gone better than people in 2017 would have predicted. But it doesn't look to me like they're taking it seriously enough: the majority of work still looks like forging ahead very quickly despite a lot of evidence of misalignment even at current scales.

There's also a bit in the post where I talked about applying for a grant to work on this. That was silly, since we had enough savings that this was something I could just choose to do. Looking back, I think the reason I applied was that I had an unrealistic view of grantmakers as people with an excellent global view of what needed doing, and where someone deciding whether to give me a grant to work on this project would be a strong signal of whether the project was worth doing.

(And then perhaps my most prescient decision was calling it "Superintelligence Risk" instead of "Artificial Intelligence Risk", anticipating executive order 14434.)

2017-07-04: Conversation with Dario Amodei

Alan Kay said "the best way to predict the future is to invent it," which certainly seems true here. This description of the situation feels super prescient. Unfortunately I never thought to ask the most important question: might AI reliably doing what we want end up being a limiting factor on AI progress, where this line of safety research might accelerate AI development and put us in what Amodei then considered the riskier "short timelines" path.

2017-07-09: Conversation with Michael Littman

Littman essentially believed that prosaic AGI, which Paul Christiano described as replicating human behavior without "qualitatively new ideas about 'how intelligence works'", was impossible. But that's pretty much what happened! So in many ways it's a big miss. I don't want to single Littman out, however, since in this series he's basically functioning as a stand-in for the mainstream ML community view at the time. Most serious professionals seemed to think this, and even Amodei and Christiano weren't confident this was how things would go.

2017-07-10: