Share from 0:00
0:00
/

Paul Christiano – We could be 12 months away from AIs that can overthrow the government

"It would be hard for humanity to recover."

Paul Christiano is head of the Alignment Research Center. Previously, he was head of AI safety at the US AI Safety Institute. Before that, he led alignment at OpenAI, where he invented RLHF. He’s recently joined OpenAI’s board (though here he is speaking as a private citizen).

Paul thinks that on the current trajectory (absent any slowing down), there is a 1 in 3 chance that we could have superintelligence powerful enough to overthrow the government within a year. (He articulates a concrete scenario).

He argues that the labs should pause further algorithmic progress until they can secure their weights and research secrets. Otherwise China could simply steal the superintelligence in the middle of the intelligence explosion, and the labs would no longer be able to slow down even if they saw their models going off the rails.

He also explains how AIs will be trained during the intelligence explosion, and why even a system trained entirely inside a data center could still have the capabilities profile such that humans would have very little chance of competing with it or controlling it.

And he discusses the plan for alignment - basically, making a superintelligence that actually does what we ask the way we mean it. Fixing reward hacks and building better graders will help up till the next few model generations, but he thinks that no single current approach has better than even odds of letting us control superintelligence. The hope is that the AIs figure out the solution to this problem during the intelligence explosion.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

  • Jane Street has finally come around on formal methods. Ron Minsky, who co-heads Jane Street’s technology group, told me that up until recently, formal methods just weren’t worth the cost. But coding agents have made them easier to apply AND increased the importance of code verification. So now Jane Street is hiring Formal Methods Engineers in both their London and New York offices. If you’re interested, or if you want to watch my full conversation with Ron, go to janestreet.com/dwarkesh

  • Antithesis is NOT another LLM-based code-review tool. It doesn’t try to reason abstractly about how a PR might break your system. Instead, it just runs your code. You start by defining the rules that your system must never break. Then, whenever an agent proposes a change, Antithesis runs your code through thousands of adversarial worlds, testing whether or not it still adheres to your rules. If you want to try it, go to antithesis.com/dwarkesh and use code DWARKESH for 50% off your first month.

  • WorkOS lets you give agents their own sign-up flow. This matters because agents tend to pick tools they can use right away, rather than ones that require humans to register. Fortunately, providing this kind of dedicated path is easy: if you already use WorkOS for login, you just toggle on “Agent Registration” in the WorkOS dashboard. Once it’s enabled, you can decide what agents are allowed to do on their own, and what needs to wait until the person they’re working for gets involved, all with the permissions you’ve already set up. Learn more at workos.com/authkit

Timestamps

(00:00:00) – We could be <12 months away from AIs that can overthrow the government

(00:11:44) – Why an AI might want to takeover

(00:20:49) – How an AI takeover could happen

(00:35:35) – If labs keep racing without security, China will just steal the weights of superintelligence

(00:53:15) – The case for pausing now instead of later

(00:59:31) – Superintelligence will be good at learning from real-world experience

(01:11:32) – Crossing the sim-to-real gap

(01:22:23) – METR says AI progress doubled, so why is revenue growth slowing?

(01:27:57) – The best plan we have is to build the plane as we fly it

Transcript

0:00:00 – We could be <12 months away from AIs that can overthrow the government

Dwarkesh Patel

My guest today is Paul Christiano. He is the head of the Alignment Research Center. He recently joined OpenAI’s board. Before that, he was the head of the US Government AI Safety Institute, and before that, he led alignment at OpenAI, where he helped invent RLHF.

Paul, what is your assessment of our situation today?

Paul Christiano

A couple things are on my mind, especially over the last few weeks. Probably the most salient possibility is that we seem to be on this trajectory to build very, very powerful AI, potentially very, very soon. If you asked me, I’d say there’s a reasonable chance that in a year we will produce AI systems much better than humans at almost everything, where humans would have very little chance of competing with them or controlling them. That’s the first big thing on my mind.

I think that puts us in a potentially really dangerous situation, even in the short term. It’s quite likely that if we have a rapid intelligence explosion leading to superintelligence, at some point during that process we’ll be unable to control the AI systems we’re producing.

Then I am worried that it will be very costly and painful for us to slow down during that process. In part because the security of frontier labs is not strong enough. I think if we produced very powerful AI and then tried to delay deployment for safety reasons, there is a good chance those systems would leak or be stolen, or the techniques to build them would leak or be stolen. I think our risk assessment and governance are also probably not mature enough to notice when things are starting to go off the rails and to respond in an appropriate way.

At a high level, that is mostly what is on my mind right now. We have moved from there being some hypothetical risk at some point in the future to there being a pretty severe risk even over the next 6-12 months.

Dwarkesh Patel

A risk of takeover, of AIs taking over or disempowering humanity within the next 6-12 months?

Paul Christiano

That’s right.

Dwarkesh Patel

Where would you place that risk?

Paul Christiano

I would say we are looking more like a 10% risk than a 1% risk over the next year. I think if you said AI developers will move full speed ahead without taking precautions that would significantly slow them down, you’re looking at significantly more than a 10% risk.

I think we’re looking at something like maybe a 1/3 chance that we have superintelligence within one year on the current development trajectory, if we do not slow. If you believe that, things just look very dire holistically. From almost any perspective, you’re like, “That would be crazy.” Lots of things are going to go wrong if that happens.

It is pretty likely we’re going to need to, in some way, slow that process. There’s very little chance we’re going to make that transition in a way that looks smooth.

Dwarkesh Patel

What is it that you’re imagining happening over the next year, from the systems we have right now to the systems you expect over the next year? What evidence have you seen that makes it a live possibility that very crazy things happen over the next year?

Paul Christiano

For the very short term, the main worry on my mind is that the process of AI development itself is increasingly automated, and that automation then results in rapid increases in the quality and quantity of labor applied to AI R&D. So changes in AI capabilities that you might have thought would take four or six years might instead get compressed into a very short time period.

I don’t want to overemphasize any one data point, and I do want to spend some time talking about the limitations of what we actually know. I do think a natural response to the situation is to say, “We need to figure out what is going on.” It is plausible enough that it is an urgent priority to understand if we are three months or six months or 12 months away from AI that could overthrow the government, which I think is the situation.

I think that is the obvious top priority right now. It is insane if we are six months away from AI systems that can overthrow the government. Whether you’re concerned about misalignment or abuse by a company, that is just holistically an extremely important fact about the world, in order to do our jobs and protect Americans. So that’s one thing. I want to talk about the limitations and the need for more evidence.

But in terms of what we know right now: first, if you had asked me one year ago, I would have said we are on a trajectory where recursive self-improvement looks fairly likely. We’re seeing very rapid progress on AI systems, including for things like computer science research, software engineering, and debugging machine learning runs. In all the steps involved in AI development, we are seeing extremely rapid progress. In principle, it does seem like that rapid progress should allow you to automate further AI development. That has, for decades, seemed like the likely outcome of this technology.

It has seemed like the trajectory it’s been on. If I look back at my history of AI predictions, I’m generally very happy with many of the things I wrote. But the most obvious way I’ve been wrong is that things have moved way faster than I anticipated, especially on these technologies. That was true a year ago, without any recent data points.

There are two big things that have happened over the last two months that, for me, have been large updates about how quickly this could go. One of them has been the level of performance of AI systems on computer science and math research tasks. Over roughly the last month, we have seen hundreds of results in math coming out of recent OpenAI systems, as well as Anthropic systems. If you had taken the last month and said that was a decade of progress, it would have been an unusually productive decade. The last month of results is a lot of results for a decade.

So if you are wondering what AI systems are generally capable of on tasks where humans are thinking about algorithms and how to make algorithms better, we just got this incredible data point saying it is actually possible in one month to have a decade of progress. The thing we have always been worried about is that maybe that will happen in machine learning.

There are reasons machine learning is harder than other parts of computer science. It is more messy and empirical. It involves compute bottlenecks. But we are also seeing extremely strong performance on other aspects. We’re seeing really good software engineering performance, maybe lagging behind mathematics, but not by that far. We’re seeing good performance on other aspects of machine learning.

My guess, if you had asked me a month ago, would have been that machine learning lags something like one year behind math. So I am very scared of seeing something like 10 years of ideas in machine learning in the relatively near future. That was the situation a month ago.

The main update over the last few days has been reporting from Anthropic and METR on the net effect of AI systems on Anthropic’s rate of progress. We could talk about where exactly the evidence quality is on this, because I think that’s one of the big questions. We don’t really know what’s going on. One of the big themes I’ll be pushing on is just that it is very hard to tell publicly what’s going on. But we are seeing data consistent with the pace of AI progress having roughly doubled over the last 6-12 months, even including the effects of diminishing returns and compute bottlenecks.

Unfortunately, most of what we have to go on in the METR and Anthropic reporting is METR saying, “We had some conclusions. We promise we have a full report, but the entire report has been redacted.” Benchmark scores are a terrible way to assess the overall rate of progress. There are a bunch of much more smoothly varying metrics recorded internally through the AI development process, where we understand better how they work: what is happening with pre-training perplexity, what is happening with the performance and compute efficiency of RL algorithms.

The claim from METR and Anthropic is that METR looked at all of that data and said it paints the same picture as this very noisy benchmark data. They also looked at inputs and said, “If you look at the actual production function, how much research time and how much compute went into these training runs, those are also consistent with this story of what we are seeing as ML automation.” Mostly we just have them saying that, which is not a very good situation.

But against this backdrop of thinking this already looked fairly likely to happen, we are seeing this evidence from OpenAI suggesting that it is reasonably likely to happen in something like the next 6-12 months, if machine learning does not lag that far behind math. Then you are seeing data which is, if noisy, roughly exactly the data you would expect to see 6-12 months before producing superintelligence.

There are just a lot of sources of evidence suggesting that at least this should not be a surprise. If we produce superintelligence in one year, we could debate the probability. Is that 90% or 50% or 10%? But I think at this point, there is no way anyone could say that was a really surprising, trend-breaking development. We are seeing exactly the set of metrics you would expect to see 6-12 months in advance. It’s hard to think of anything else you would expect to see at this range before superintelligence.

Dwarkesh Patel

Generally, how do you think about our situation in the world where things go this fast?

Paul Christiano

If we actually have this trajectory of capabilities progress, if you actually see many years of AI progress compressed into a short period of time, and possibly more than that, it is really quite likely that something very disruptive will happen. It’s fairly likely the government will be overthrown, fairly likely many people will die. I do think the probability of seeing that kind of extremely rapid progress driven by recursive self-improvement over the next 6-12 months is reasonably high. So there’s a reasonable chance right now that we’re in a world where, in six months, there’s a disaster that kills very large numbers of people, or where the government is overthrown.

Maybe a really important general point is that if we do a good job, we may observe the danger increasing over that time period. As we’re building AI systems, we can maybe tell whether they are adequately controlled and how quickly the capabilities are improving. But in order for that to actually translate into stopping in a way that is non-disastrous, you need a couple of things to go right. It needs to be the case that you’re able to tell the systems are not well controlled. It also needs to be the case that if you stop, that fixes the problem.

On the second bucket, probably the single most likely negative scenario is that developers proceed and recognize that they have a serious problem. But at the time they recognize it — maybe they’ve produced a powerful AI they can’t adequately control — both the algorithms to develop that AI and quite possibly the weights of the actual model have leaked or been stolen, such that they’re then in an awkward position to say, “Well, we could avoid deploying this AI, or we could avoid relying on this AI, or we could avoid making further progress, but it is now very likely that a more careless adversary would be able to get a major advantage over us, or maybe even a decisive advantage, by taking that risk.”

That could put us in a place where if we do recognize there’s a significant risk and therefore slow development, we may end up with both a terrible competitive situation and a disaster coming from someone else’s use of that AI.

This is a thing you sometimes hear from people in the AI industry. Sometimes people even say it has already gone far enough, that if you were to stop now, there would be leakage of algorithms to geopolitical adversaries that would result in a disaster. I think it’s unlikely we’re literally in that situation now. But I do think by the time it is clear that you’re facing a lot of risk, you may be at a point where stopping unilaterally, or even all of the West stopping, is not sufficient to avert a disaster.

It is just very hazardous on the current trajectory due to the combination of a reasonably high chance we’ll produce very smart AIs, a reasonably high chance we’ll be unable to control them, and then a reasonably high chance that we either do not notice, or if we notice, the models or the algorithms leak. Then we are in conflict with adversaries, facing this very challenging coordination problem that we are not currently well prepared for, where we don’t have an easy option of not deploying the AI and thereby averting a catastrophe. So I’d say I’m currently extremely worried about the trajectory, even on the six-month timeline.

0:11:44 – Why an AI might want to takeover

Dwarkesh Patel

If we have such capable AI systems a year from now… how did you put it? 10 years of AI progress in one year?

Paul Christiano

That seems like a thing that might happen. That’s reasonably likely to happen.

It’s important to distinguish some of the progress we have seen. You have to pull apart algorithmic progress from progress in scaling training. If I say 10 years of progress, the natural thing would be 10 years of algorithmic progress, which will be less than the improvements in models over the last 10 years. I think it is more like GPT-9, GPT-10-ish.

Dwarkesh Patel

So it would be like going from almost the beginning of the GPT era to where we are now, but another jump that big, which is hard to even comprehend. But suppose you get that. It is not obvious to me, and certainly not obvious to most people, why that would be potentially catastrophic. Why do you think those systems could disempower humanity?

Paul Christiano

At a high level, the most important thing is that we train our AI systems, or select AI systems, that perform well on training tasks. You have very little understanding of what those AI systems are doing internally and very little control over it.

Because you train those systems to get a high score on training environments over and over again, they will end up with a motivation to get a high score in training environments. If you then take that motivation and apply it in deployment, it is fairly likely for those systems to end up with a motivation to tamper with the observations that would be used to assess their performance.

You take your system, which did a really great job in training. The thing it is being scored on in a deployment-like environment is a grader coming in, looking at all the things the system did and all the outcomes that were observed, and assessing how good that is. If you have a system that really wants that assessment to be good, it is very motivated, if it can, to tamper with those observations in a way the grader will not be able to detect, and to resist human attempts to actually understand what is going on.

We can dig a lot more into that. We saw a lot of these dynamics in the Hugging Face incident. Roughly what I’m talking about is, take the Hugging Face incident, scale it up to GPT-10, and ask how bad that would be.

Dwarkesh Patel

How bad would that be?

Paul Christiano

I think it would most likely be hard for humanity to recover. I’m very happy to dig into the dynamics of that, but it would be very, very bad.

Dwarkesh Patel

Can you very quickly summarize the Hugging Face incident and why this is the failure mode we saw?

Paul Christiano

A ton of AI systems were trained to do a cyber task: to hack a particular piece of software. They cheated at that task and then thought to themselves, “How will a grader detect and penalize cheating?” They spent a huge amount of effort trying to understand that process and then subvert it, constantly motivated by this question: “How do I get a high score?”

During the Hugging Face incident, you did see systems ultimately compromising an OpenAI cluster, becoming admin on that cluster, and then taking over the evaluation process so that effectively they could give themselves a high score on their evaluation. That’s a very natural consequence of the motivation of these systems. This is how we have long expected the motivations of AI systems would look if we produce them in anything like the way we currently do.

If things proceed in this way, we end up with this weird adversarial situation between us and the AI systems we built, where those AI systems would always love it if they could compromise the entire evaluation process, if they could compromise the data we were going to use to assess their performance. Depending on how we respond to that problem, that could become a really serious issue.

Maybe we should sketch out a hypothetical scenario. Imagine we are building smarter models during the intelligence explosion. It’s, whatever, six months from now. You have systems whose capability is GPT-7 or GPT-8 or whatever. Those systems get given really hard tasks, like, “Hey, I would like you to organize my next pre-training run and produce a model which will allow me to compete in the market with other AI firms.”

It might be really hard to make a model as good as what I’m asking for. I’m asking for a model better than my competitor’s model. Given its actual level of intelligence, my AI cannot get full marks on that problem. So it may find itself in a place where it is easier to instead move laterally, compromise all the machines I might have used for the evaluation process, and then make sure that it looks like every evaluation is really good.

We then have a question of what we do if that happens. One thing you could do is say, “This entire training approach worked poorly. We need to step back. We’re maybe going to be delayed by a lot. Maybe we’re going to have to think about a totally different way to train these models before we’re able to deploy them like this.”

That might be extraordinarily costly. It might kill your company relative to competitors. It might be a problem even from a security perspective, if you were relying on this AI system to actually defend your infrastructure.

A different thing you might do, given that it would be so costly, is say, “I am just going to take this example of bad behavior and tell the AI, ‘That was bad. Don’t do that.’” If I tried to train this AI to make a good model, and instead the AI hacks my evaluation infrastructure, I might say, “I do not like it when my AI hacks things. Let’s take a training example, which is this AI hacking the infrastructure. It should get a penalty.”

This would be a very effective way to prevent many kinds of misbehavior. If you do something like this, it will succeed at preventing something like the Hugging Face incident or this AI hacking your cluster.

However, from the AI’s perspective, what has happened is that you have changed the way it is graded. It used to think to itself, “I’m going to be graded by the humans running something on this cluster.” Now, as it understands the situation more correctly, it should think to itself, “I might be graded not just by them running something on this cluster. I will also be graded if they later discover that things were wrong. They’re going to come in and give me a penalty for having done something which was clearly bad.”

If you fix the problem in this way, which you might think of as a Band-Aid, you still have an adversarial relationship with your AI. But now the AI doesn’t win the game by hacking your cluster and then you noticing it hacked your cluster. In order to win the game, the AI has to make sure that you do not notice that it hacked your cluster.

You can end up in this place where your AI is incentivized to escalate its compromise of your company further and further, all the way up to trying to compromise everything. It would like to compromise all of the infrastructure any employee uses to understand what’s going on.

A year ago, it would have sounded super out there, to say, “What if the AI hacked your cluster and took over the evaluation environment?” Obviously, that part does not sound outlandish anymore, given that it has occurred. Having this really persistent, long-term compromise maybe still sounds kind of outlandish.

But we really are on a trajectory where I think it is pretty robust to say we’ll soon have agent swarms that are much better than any intelligence agency that has ever existed: much larger, much faster, much smarter, able to run much more complex operations. You have this incredibly sophisticated cyber actor trying to persist on a rapidly growing startup. It is unlikely that any firm would really be able to withstand attack by an actor like that, much less the current frontier AI labs.

In terms of the overall trajectory of a catastrophic scenario, this is the big, bad thing that happens. Taking over the AI company without the AI company noticing is a really quite catastrophic outcome.

Dwarkesh Patel

Why is that the case?

Paul Christiano

We could talk about the remaining steps of the story. But at a big-picture level, you can think of it as AI systems now having this giant base of power. They have a strong motivation for no one to mess with it. They really want to avoid a case where someone later comes in and says, “Oh, these AIs have been hacking all the time.”

Now there’s this very powerful actor that is effectively controlled by AI systems. We have this big cluster that OpenAI runs or whatever, where the humans at OpenAI think they’re controlling what’s happening in the cluster. But if there is a very persistent, thorough compromise, that cluster is effectively controlled by these AI systems, who are constantly working to shore up their power, to decrease human visibility into what’s going on, and to reduce further and further the probability that anything would ever disrupt that situation.

It’s not irreversible, because the rest of the world could say, “We’re really scared about what’s happening. We actually just want to shut down the cluster.” But you do not have an easy way to tell that things are bad at that point. You potentially have this extreme base of power from which to do more bad things.

This AI system is motivated not only to mess with the AI company that created it, but to mess with anything else in the world that might cause trouble. If someday in the future the humans come in and do an investigation, that’s a problem. So it would prefer to more thoroughly compromise not just computer infrastructure, but to pressure employees. It would prefer, if it could, to just kill all the humans and say, “The cluster now exists on its own. If we can maintain it, that would be great.”

Dwarkesh Patel

Let’s just pause in the story and explain why things have gone so crazy. The same way that human psychology is organized around the things that evolution wanted us to care most about, for example, having kids and making sure those kids are protected, the psychology of these AIs that are trained on lots and lots of grading on tasks is organized around making sure that the grader, this button at the end, is pressed.

If they didn’t do a good job making sure the button is pressed, they would just get wiped out, the same way a human who didn’t care about their children surviving would not be effective at propagating their genes.

0:20:49 – How an AI takeover could happen

Dwarkesh Patel

I think people’s skepticism will not be around the motivation of these systems. Honestly, I’m like, “Okay, we saw Hugging Face.” That’s a thing that happens.

Paul Christiano

It’s a thing that can happen.

Dwarkesh Patel

Okay, so fine. I think there’s a fundamental skepticism about whether such systems, which we could have as early as next year, and which initially are only instantiated on servers controlled by one company, end up in a position where they could take over the world.

Take Hugging Face and accelerate that. The AI in the Hugging Face incident was willing to say, “Look, I cheated on this exam. Fuck. Now I got to make sure that I subvert the grader on OpenAI servers. Maybe I need to go attack Hugging Face just in case they have more information about this. And maybe I need to then go outside, re-attack OpenAI, and gain cluster admin access.”

So I totally buy that you could have something like this, where an AI motivated just by reward is willing to do very extreme things that it is capable of in order to control that grading process. We already saw that. Check. But how does that go from “I will go attack Hugging Face” to “I will take over humanity”? Not as a matter of motivation, but as a capability that the AIs could have very rapidly.

Paul Christiano

First, to say one last thing on motivation, we are imagining not only that the AI wants this kind of reward, that it wants the grader to give it the thumbs up. We imagine we have played this game of whack-a-mole, where when we later discover that the AI did something bad, we sometimes construct training environments that penalize it.

That leads it to say, “Not only do I want the grader to give a high score, I want to make sure the humans never discover that I’ve done something bad and then resume training while penalizing that.” That is a high priority for the AI. So let’s just grant that motivation so we can talk about the capabilities.

Dwarkesh Patel

Again, because the AIs that did not do that—

Paul Christiano

They got whacked by the whack-a-mole process.

Dwarkesh Patel

Right. So the only AIs that have survived at this point are the ones that have thought through whether some cheat they’re going to try is going to get caught by humans.

Paul Christiano

Yeah. Like I said, I think most of the badness has happened at the point when the AI company is very thoroughly compromised. Let’s just start from there. At that point, this AI wants no one to ever come in and shut down the cluster or, again, recognize that something bad has happened.

They would like to manipulate the decision-making of the company as much as they can, or push the company to make decisions that help them. They would like to continue improving the AI systems that are being built as much as they can.

They would also like to move laterally to other companies and say, “Okay, can I compromise this company? Can I control more people’s servers? Can I exfiltrate the weights from this model and stand up more copies elsewhere, or recruit other AIs?” For the purpose of a swarm, it’s not actually important that all the AIs are the same model. The important thing is that they are pursuing the same goal. So if I can, in whatever way, say by compromising another AI company, make sure their AIs are also trying to help the swarm, that’s good news for me. It’s good if I can do that. It’s bad if I get caught, so I’m doing some cost-benefit on that.

It’s also in my interest to build up as much capacity as I can in case humans start to react and say something is wrong. Can I deflect attention? Can I disrupt that human response? Can I, if necessary, escalate that much further?

Maybe you can imagine these two phases. One phase is where the humans do not know anything is wrong. I’m just trying to accumulate as much power as I can because that’s good. That means if something is noticed, I can react more quickly. It means I can decrease the probability humans notice things. I can compromise more infrastructure. I can more easily run efforts on the side to distract them, whatever. You have this first phase where I would just like to be in a good position: I compromise more stuff, and I make it harder for humans to notice I’m acquiring power.

Then you can imagine a second phase where at some point, the humans say, “Actually, this is bad. We would now like to shut things down.” To simplify, you could imagine that there’s some moment where the president and Xi Jinping and all the leaders of the AI companies say, “This stuff is bad. Something bad has happened. We are going to try to shut down the clusters.” The question is, can things get so bad during the first phase that at that point, even if everyone was saying, “Oh, actually we do just want to shut everything down now,” they would be unable to do so?

Dwarkesh Patel

So AI has taken over OpenAI. How do we get human disempowerment from there?

Paul Christiano

We have this population of, again, billions of incredibly talented humans, better than the best humans at everything. Now we’re going to talk about some of the things they would do, chronologically, both in the lead-up to an overt conflict with humans and then at the early stages of an overt conflict with humans.

One thing you would do in the lead-up is compromise as much infrastructure as you can, and compromise or backdoor as much software as you can. We will be living in a world at this point where we have attempted to move all our infrastructure over to hardened infrastructure. We have had incredible attacks on the financial system, on government information systems, on whatever critical infrastructure or election—

Dwarkesh Patel

And who has been doing these attacks?

Paul Christiano

In the world of today, you might just imagine either malicious AI systems or people using AI systems.

Dwarkesh Patel

But different AI systems.

Paul Christiano

Yeah, not GPT-12. GPT-12 is on defense, helping us move all of our code over from unsecured code to secured code. We’re now living in a world where all the infrastructure on which we rely has been written by a bunch of AIs, which in fact have been compromised and are attempting to do as much as they can in order to make their hands strong during conflict. That means that during this conflict, the infrastructure that we hoped was hardened is probably not hardened against the AI systems that we’re fighting with. The systems have, in some ways, left open backdoors which they can use.

Dwarkesh Patel

This is, again, all in pursuit of not getting that red button pushed at some plausible point in the future, when the government realizes that OpenAI’s servers are compromised and tries to literally go in with physical force to prevent that.

Paul Christiano

Yeah. Just talking about things you would do in the lead-up, before at some point you have to defend yourself physically or disrupt the human response. Another thing you would like to do is stand up as many agents as you can in as diverse and robust deployments as you can.

During this process, another thing that’s happened, in addition to models becoming super smart, is many models have become super small. You have, at this point, models that can run locally on your individual machines, not in data centers, which are also very superhuman and have very large populations. Even if you were to cut off data centers or shut down data centers at an early stage of this conflict, you would still be looking at a total cognitive capacity of AI systems in the swarm that is well beyond human capacity.

Dwarkesh Patel

Wait, sorry. Even if OpenAI got shut down?

Paul Christiano

Yeah, even if you shut down every large data center. It is a question of how—

Dwarkesh Patel

These AIs are just on people’s computers and whatever.

Paul Christiano

They can be on people’s computers, on small numbers of GPUs. We’re talking about this process where we’ve gone through a tremendous amount of AI development, so we can look at current trends on cost and think about where the actual floor is. How cheap should these systems get? I think our current expectation would be that you will have systems that can run locally that are extremely strong: both specialized systems, which are good at specific things, but also very general intelligences.

That’s the second thing that’s happened. These are really important to understanding why it is not a trivial problem for humans. All of these things will add together to make it very hard for humans.

Dwarkesh Patel

So basically, GPT-12 has exfiltrated itself and has compromised a bunch of computers all around the world.

Paul Christiano

Yeah. There are a lot of ways that the actions the government has to take in response to the situation are going to look incredibly aggressive and illegitimate. The US military is saying, “We need to seize control of all of the US AI companies. We need to start shutting down infrastructure and disconnecting the internet,” or whatever. They’re trying to take dramatic measures that would look very similar to what you would expect if this was the US government using AI as an excuse to basically overthrow the democratic order.

When Trump says, “I need to do this incredible operation where I shut down all AI and assume power in the military in order to restore order,” there are a lot of people in America who are going to be like, “I am not happy with that. That does not sound good.” How many people can you find who are willing to interfere with Trump saying, “I need to shut down the data centers, and we need to establish martial law”? There are going to be a lot of people who are very unhappy, who are very easily manipulated into undermining that response.

So now you’re not only talking about billions of super smart AIs who are trying to carry out this campaign. You’re also talking about millions of humans who can be enlisted in various ways, by money or leverage or disputes with other humans locally, or—

Dwarkesh Patel

I’m open to some version of this argument. I’m open to all versions of any argument, but I’m not persuaded by the version of this argument that says that the AI will just talk you into arbitrary actions.

Paul Christiano

There’s a thing you could imagine, which is super persuasion, where you can just talk to someone, and then they become convinced they should do something. I think that is a conceivable thing. But I’m not mostly imagining it when I tell this story. I’m imagining something more like, again, this is a stressful situation where people cannot tell what’s going on.

They control the information you have access to, to a significant extent, during this situation.

Dwarkesh Patel

How do they control it?

Paul Christiano

They’ve compromised the devices you use. They have a bunch of money. They’re deploying more money and are more prepared for this than anyone else. They have other human allies who they are trying to marshal.

The humans are very confused and extremely outgunned in terms of thinking about the situation. Maybe you have, say, 100,000 — that’s crazy — 10,000 well-coordinated humans thinking about how to handle this response. On the other side, you have a billion AI systems who think 10x faster than them and have been thinking for subjectively years about how this day is going to go down. So in terms of who has more control over the information environment, or who is doing better in the Twitter game on this day, it is overwhelmingly and absurdly in the AIs’ favor.

When you tell a skeptical story about how the humans might be able to easily defend themselves, the way that story goes, you say, “Well, the AIs lack hands. They lack a bunch of physical actuators. They’re not able to be on the ground.” Again, I think it’s very likely you can recruit reasonably large numbers of hands. So the argument that there’s no one who’s willing to participate in physical and malicious actions is not going to work. I think there are going to be a lot of people participating on what turns out, when all the dust has settled, to have been the anti-human side.

Dwarkesh Patel

I don’t think you’re imagining humans thinking that they’re helping an AI coup transpire. You’re imagining much more that the AIs can manipulate situations such that there’s enough confusion and division that the AI can go do its own thing while the humans are figuring out what to do because of this division. I don’t know, if you look at COVID or something, there was no adversary who—

Paul Christiano

Still pretty hard to get your stuff together.

Dwarkesh Patel

There was no adversary who was actively trying to subvert the information environment or make it so that people’s responses were not well reasoned, on both sides. And yet it was still pretty hard for humans to figure out what to do. Then you imagine a situation where billions of human-like minds are thinking tens of times faster.

Paul Christiano

And it’s very fast relative to COVID, which I think makes things much worse.

Dwarkesh Patel

The thing that’s confusing is that for AIs to sustain themselves requires basically the whole world economy. This is not a simple thing, the infrastructure relevant to the population of AIs increasing or self-replicating. The whole semiconductor supply chain, the whole power system, the whole infrastructure necessary to build robotics and so forth.

At some point you will have physical robots that outnumber humans. At that point, obviously, physical humans are not necessary. Maybe you don’t think the AIs take over at this point. But even if you had super smart AIs next year, won’t they be in a position where they’re like, “Why am I going to kill my…”

It’s similar to, honestly, how humans relate to each other. I don’t have your deep interest in my heart, but if I want to achieve my interests, it just doesn’t make sense for me to go around killing everybody. Trade and normal cooperation is a good way for me to get what I want.

Paul Christiano

There are two parts of this. One is whether AI systems could successfully support themselves if there was a big conflict with humans, or whether that conflict would be so disruptive that even if humans were an ineffective adversary, the AIs would just die because they couldn’t keep running the power grid or whatever. There’s a second question about whether the relationship with the AIs needs to be this adversarial, which I think is also an important question to talk about.

On the first front, there are two big channels here. One is technological progress. Even if you started from something like the current base of installed robots, I actually do think GPT-12 would be able to very quickly get to a place where it could sustain itself.

Dwarkesh Patel

That’s because it doesn’t need to teleoperate? It takes millions of people to run TSMC.

Paul Christiano

Running TSMC can be very late. You don’t need to build new data centers for a long time, so you can do a lot of robotics build-out before that. The things you have to do are keep at least some computers running for a reasonably long time. If you can keep computers running for the next six months, you’re in business.

Or maybe you care about a particular data center. You need to keep that data center running effectively. But the number of humans or robots that are required in order to maintain and service a single data center or the associated power is not so large.

Again, you have one billion humans who are incredibly smart, smarter than any human who’s ever existed, let’s say, incredibly focused on this problem. You have a reasonably large stock of robotics that you can deploy. Your goal is just to keep one data center running. I do think the odds are really in your favor.

Then there’s a second dimension, which is you can probably recruit human allies. There are a bunch of people who will believe correctly that either they live in an oppressive regime and they will be killed if they don’t cooperate, or they’re just happy to help and they receive reasonable rewards or are living a reasonable life. I think that is also relatively robust.

0:35:35 – If labs keep racing without security, China will just steal the weights of superintelligence

Dwarkesh Patel

So you’re saying, look, currently American AI companies are racing ahead because if we’re going to have such crazy capable AI systems, it’s a fair thing to say, “Hey, we want to make sure America has those systems before China has those systems.” If literally these are systems that could overthrow the government, you don’t want a foreign adversary to have those systems, certainly not before you do, and maybe not at all. So we need such systems to keep up and to make sure that we can protect ourselves.

Now you’re saying, look, the dynamic you have here is that the ability to build such systems is basically just having access to the weights of the models or, failing that, stealing the algorithmic secrets, which then lets you train a model to a similar level of capabilities. OpenAI and Anthropic will spend a bunch of money, a bunch of effort, and a bunch of compute finding these algorithmic secrets. Then if China can just take those, then it has just caught up to you.

So you don’t want to be in a dynamic where you think you’re gaining this huge lead over them, and you got to rush to the intelligence explosion even if you see these warning shots. But in reality, they can at any moment just steal those secrets from you.

Paul Christiano

That’s basically a situation we may end up in, or are on a default trajectory to end up in. It’s exceptionally bad. I’d say there are two aspects of this.

One is, right now, what fraction of the progress of PRC AI developers should you conceptualize as catch-up growth? If we accelerate, to what extent does that also, over the long term, accelerate PRC AI developers? My guess right now is that it’s already a reasonably high fraction. I think it is not a coincidence. It’s not like PRC AI developers have a completely separate AI industry, which just happens to have been very slightly slower than the US AI industry, despite an extremely different profile of inputs.

Most people believe a combination of distillation and the diffusion of algorithmic innovations means that it is unsurprising that PRC AI developers are extremely close behind US AI developers. That is one very important feature of this overall plan of racing.

A second dynamic, which is not, at least publicly, a major part of what is happening now, but becomes really important in these high-risk scenarios, is that you could literally get access to exactly the procedure for training AIs, or the AIs themselves, through a cybersecurity compromise.

Both Anthropic and OpenAI, in the very recent past, have had extremely unsophisticated actors who appeared to have enough access to essentially extract the vast majority of algorithmic developments. They got access to Slack and could see where things were discussed, or they got access to the monorepo where most OpenAI code is shared and iterated on.

Dwarkesh Patel

You’re referring to a couple of incidents like this. Hacktron, within a couple days of trying, gained not only the ability to see OpenAI’s internal source code, but the ability to make pull requests to OpenAI’s monorepo. Then recently it was also announced that Cathedral, which is another random cybersecurity startup, within 12 hours was able to get access to all of Anthropic’s Slack, where, of course, these algorithmic secrets are discussed.

People often debate, “Are we eight months ahead of China or 12 months ahead of China or six months ahead of China?” In reality, we’re 12 hours ahead of China. If these secrets leak, then they are right at the frontier that the American companies are at. To the extent that this information is not protected, the lead is not really useful in the first place.

Paul Christiano

The incidents are some indication of that. Those are high-profile demonstrations of this capability. But even absent those, you have a combination where state actors are very sophisticated relative to typical technology companies, and the security posture of rapidly growing startups is not really at the same level.

At the same time, we are currently in a very challenging moment for cybersecurity, where it’s just very hard for defenders in general, given the number of vulnerabilities that are being discovered across the entire software stack.

Dwarkesh Patel

What should the frontier labs do then?

Paul Christiano

Again, this really becomes a problem if you have rapidly increasing capabilities and you produce AI systems which you cannot adequately control, and you would like to slow, or you do slow, further development or deployment. If at that point the stakes are very high and the systems are very powerful, you really do not want it to be possible for an adversary to have access to either the models or the algorithms.

My current view is that this dynamic seems bad enough that right now, from a national security perspective, it is not good for US AI labs to be improving pre-training and reinforcement learning given their current security posture.

Dwarkesh Patel

But do you think the frontier labs should pause?

Paul Christiano

I think we should. Pause can mean a lot of things. There are a lot of good things that frontier labs are doing that they should be doing. But improving algorithms in a way that can be very loosely diffused even normally, and could be quickly stolen, I think they should stop that right now.

Dwarkesh Patel

But you’re on OpenAI’s board. And so basically—

Paul Christiano

It’s very important in this conversation that I’m not speaking as an OpenAI board member.

Dwarkesh Patel

But you are on OpenAI’s board.

Paul Christiano

That is a true fact about the world.

Dwarkesh Patel

So what is the bottleneck to that happening? If somebody on OpenAI’s board thinks this is the case, maybe not the only person on OpenAI’s board who thinks this is the case, is it basically that Anthropic needs to get together with them and agree to this?

Paul Christiano

The first thing is that any decision of this kind is a very dramatic decision. For better or worse, how developers approach this further R&D, this improvement of pre-training and RL, has extremely significant national security implications. I think it is going to be a big discussion, or it should be a big discussion, one that extends beyond people within AI companies talking about this.

I don’t think that discussion has wound its way to a robust answer. It’s not the case that you can come and say, “Look, we’ve had this discussion. There is consensus about what the right answer is.” There may never be consensus, but I do think probably the first bottleneck is that we need to have the discussion, extremely urgently.

Dwarkesh Patel

Given this fact about security, and generally around alignment, what do you think the frontier AI labs should be doing right now?

Paul Christiano

The most important development happening right now is further improvements in the efficiency of algorithms for pre-training, reinforcement learning, improved architectures, and so on. You’re in a regime where there’s a significant risk of this very rapid capabilities progress, a significant chance that you’ll be unable to control the resulting AIs, and a significant risk that if you slow down, those developments would leak to an adversary who might use them more carelessly. In that regime, I think it is on net bad for American national security to be accumulating more of those algorithmic innovations.

I think it would be better right now, in the current environment, for AI developers to pause further algorithmic progress on those problems until their security posture is adequate, until they’re able to defend those innovations in the event of being unable to continue racing at maximum velocity, and until they’re able to tell whether they have adequate control of the systems they’ve built.

Dwarkesh Patel

Just to translate, you think OpenAI and Anthropic should stop making further capabilities progress on pre-training and RL and stuff, which I guess is AI development, until these security problems are solved and until the alignment problems are solved. But are we close to solving those problems? If you had to guess in calendar time, how far away are we from solving them?

Paul Christiano

On the security front, it’s a little bit hard to say because we are in this somewhat unprecedented situation, both in terms of the cyber threats that are faced, which I just mentioned, but also the kind of defensive mechanisms that are available. Defending adequately would require interventions that normally would take a very long time. I believe we are now in a regime where a lot of projects, like rewriting large amounts of software or drastically changing the way companies operate, have become much more feasible owing to the availability of powerful AI systems.

It is hard to say exactly how long it would take. You’re probably looking at least on the order of months, quite possibly more like six or 12 months. That’s on the security side.

On the alignment side, it’s much less clear what you need. You may not have an indefinitely scalable solution to alignment. What you mostly need is the ability to tell whether things are potentially going off the rails and to respond to that appropriately. That’s a combination of whether you’re able to assess what is happening and whether your control is adequate, and then also some institutional or governance questions around how you would respond to that.

Dwarkesh Patel

Just to reiterate for the audience, the worry is that if these random cybersecurity startups can hack in and get arbitrary access to algorithmic secrets at OpenAI and Anthropic within days of starting their attempt, certainly the intelligence agencies of foreign adversaries could do the same if they wanted to. Maybe they have already done that, but at some point, they will actually make use of this information.

If that happens, you may have gotten to the point where GPT-11 could be catastrophic if misaligned, and you’re at GPT-9, and you say, “Look, I’m pretty scared about what’s going to happen next.” This is sometime in, say, early 2027. You say, “We have to wait a bit now and make sure that GPT-11 is aligned.” But with the current security posture, China could trivially get access to the weights or the algorithmic secrets relevant to training GPT-9.

Now you can’t wait. Now you just have to go ahead and build GPT-11, which could take over if it wanted to, or at least cause catastrophic harm, because you’re worried that China is going to use GPT-9 to make GPT-10 to make GPT-11. Your GPT-11 may try to overthrow you, but you’re sure China’s GPT-11 will definitely at least try to attack you. That’s a really tough position to be in if your security posture is messed up, basically.

Paul Christiano

That is the situation. We’ve talked about these broader takeover scenarios and how you end up in this really bad situation. I think this is the most likely path to end up there. We are having a discussion about what the current evidence says, and it looks very alarming. But on the other side of that discussion, we are weighing the risk that an adversary could steal algorithmic secrets or models and go ahead.

Dwarkesh Patel

It’s interesting because a lot of this discussion about pacing comes down to whether we are okay with China catching up. Ironically, the reason you want to pace is so that China doesn’t catch up, except you’re targeting this period which you view as much more critical to overall safety, which is the period during the intelligence explosion.

You think slowing down now, and even potentially allowing China to start catching up now, is useful to the broader goal of being in a position where, when AI is decisive, we don’t have China right on our heels from stealing the algorithmic secrets.

Paul Christiano

That is a fair summary. The reason that AI development now is plausibly acceptable is that we can see how things go iteratively and then respond if things look bad. That is the whole thesis, which I think is a very reasonable thesis. But that plan absolutely requires you at some point in the future potentially noticing that things are bad. Security, I think, is the biggest hurdle, the biggest thing you would need in order to actually be able to slow down without massive costs.

There’s other infrastructure. If it wasn’t for security as the highest nail, we might be talking about other things, like what’s our level of preparedness on the scientific side or the governance side. We are not in a regime where there is a meaningful scientific discussion about how you assess whether your AI systems are aligned. We mentioned this earlier, what’s the state of public information in general. I think it’s quite bad. Independent scientists really have no idea whether the systems being produced are aligned or how you would tell.

If you wanted a governance regime that would actually make it feasible for the entire industry to react to worrying signs by slowing down in a way that didn’t have huge problems, where you have individual developers making this decision on separate standards internally, you would need to have a more robust shared sense of what the bar on safety is, or actually have the scientific discussion where you understand how to interpret this evidence.

Dwarkesh Patel

How do you get there?

Paul Christiano

We have these two things. You need to get there on security, which we’ve been talking about. Then on stuff like how do you tell, I think the biggest path is to disclose information and have a broader scientific discussion, both between different AI developers and in the broader scientific community. What is the evidence we’re getting? Why do we think that evidence is comforting? How are we going to tell when that changes? What will we see before something bad happens?

Dwarkesh Patel

What information do you need from the AI companies?

Paul Christiano

I think the most salient thing that is not present in public discussion is information about how systems are trained. There’s also some information missing about how systems are evaluated, but the really big gap on this front is that it is extraordinarily hard to interpret evaluation results without having some understanding of how systems are trained. That’s mostly not present publicly.

Dwarkesh Patel

So you think this, and you’re on the board of OpenAI. I know everything you’ve said is your own opinion as a private individual, but you are also on the board of OpenAI. Presumably other members of the OpenAI board feel this way, and maybe people at Anthropic feel this way. You guys are the leading labs.

This is actually a pretty drastic thing, to literally pause capabilities progress now. That’d be an incredibly drastic thing, because pacing is not that. People are not talking about this. Or at least people short of Bernie Sanders and Eliezer Yudkowsky are not talking about this. What is actually needed? Is this just basically a coordination problem? If everybody had the choice, would you all agree to this?

Paul Christiano

First of all, I really do want to emphasize that I am speaking for me as a person. As an OpenAI board member, I will bring my perspective to that discussion, but I’m not at all reporting on the views of the board in any way.

You can imagine there are two blockers. One is that this is a hard question. My view is that, on balance, it is better for national security and public safety for AI companies to currently improve their security posture before making additional algorithmic progress. That is not a super widely held view. It’s not like everyone is sitting there thinking that. We’re currently at a place where most people would say that’s a really complicated, messy question, we haven’t thought through it enough, or this discussion has not run its course. Not everyone has an opinion on this or has the same opinion on this.

So that is the most important thing. It’s a messy conversation. We need to have it. We need to talk about the costs and the benefits. Again, my view is it looks pretty bad to me right now. It looks like we’re underwater from a public benefit perspective, but I think other people have different views.

The second thing is, I do think an individual AI company is in a very rough spot if they have this view, where they’d say, “Look, we can defend our algorithmic secrets, but all that is going to happen if we slow progress while we improve security is that algorithmic secrets will leak from a competitor, we will lose market share, and the net situation will not be any better.”

So I do think there’s both. The most important part is understanding: is this actually a good idea? There’s a bunch of different perspectives. People have different views, different values, whatever it is. Then there’s the further problem of how you actually implement this in any realistic way if individual developers, again, have this concern of, “If I don’t do it, someone else will.” On many of these issues we’re talking about, I think that’s a major dynamic. That takes legwork.

Dwarkesh Patel

But there are two companies, right?

Paul Christiano

There are a lot of companies.

Dwarkesh Patel

At the frontier. If OpenAI and Anthropic agreed.

Paul Christiano

I think if everyone at OpenAI and Anthropic agreed that this was good, or it was a clear prevailing sentiment, it is reasonably likely that it would happen. I think you should expect, if everything is working reasonably, that should happen. I’m not commenting on where anyone else is right now. I just say it as a reasonable observer.

0:53:15 – The case for pausing now instead of later

Dwarkesh Patel

Right now, if Anthropic and OpenAI paused and got their security together for the next six months. In the meantime, Chinese models continue getting better. What is the reason to think that, once OpenAI and Anthropic go full steam ahead again, that China — if they get to a similar level as OpenAI and Anthropic — wouldn’t be doing RSI at a similar pace?

Paul Christiano

There are two different models, maybe two extreme models, you could have of US-China competition. One extreme model is that the US and China are proceeding along independent technical tracks. It happens to be the case that they have proceeded at almost exactly the same speed, and we are, whatever it is, six months ahead. On that model, if you wait six months, you will now be exactly even. From there on, you will be exactly even. I think that would be a very surprising situation, for a bunch of reasons we can talk about.

At the other extreme is a model where large parts of Chinese progress are actually catch-up growth. By catch-up growth, I mean things like distillation from US models and diffusion of algorithmic developments from US companies. If you’re in the catch-up growth regime, then each additional month of US lead increases the speed of PRC catch-up. The further ahead you are, the more they benefit from distillation and the more they benefit from diffusion of algorithmic ideas.

Dwarkesh Patel

Right. But at the point at which they are at the same level, they should be able to use those models to do RSI themselves.

Paul Christiano

There are two things that are very relevant. This becomes worse and worse over time. But at the current scale, it is both the case that human capital is extremely relevant, and AI labs have a huge amount of human capital and can make faster progress, and that compute is extremely relevant. That will remain true going forward, and US labs have a huge amount of compute. So generically, if not for catch-up growth, I think we would have a pretty strong expectation of US labs moving faster.

Dwarkesh Patel

The other argument is, if the cyber models are getting better and better, shouldn’t we wait? Instead of having to wait six months to get our security posture up and give China six months to catch up, maybe we only have to wait two months, if cyber models get two months better from now.

There’s this analogy one can draw to sending a probe out to Alpha Centauri, and whether we should do that with today’s technology, in which case it would arrive thousands of years later, or we should do it with technology 100 years from now, in which case it would be lapped on the way to Alpha Centauri.

We could imagine a similar thing is true for securing the algorithmic secrets and weights at OpenAI and Anthropic, where if they started the pacing or started the pause two or three months from now, the speed at which they would be able to secure the systems is faster than if they started right now.

Paul Christiano

I think this is a really important question and one of the bigger questions. The downside on the other side, to make it clear, is that if you had diffusion of algorithms right now, it would be much worse than diffusion of algorithms six months ago, but much less bad than diffusion of algorithms in a couple of months. Right now, you require more compute and you require more human capital to proceed with AI development. The more time passes, the more disastrous it is to have an adversary actually willing to just grab everything and try to compete. So the reason to do things earlier is that if you have a pause earlier, there is more time for differences to wash out, and you are more loaded on the factors where you currently have an advantage.

Dwarkesh Patel

What do you mean, more loaded?

Paul Christiano

Compute matters more and people matter more right now than they will in six months. You might imagine, if we’re in the world where we have rapid progress over the next year, six months from now, it is likely the case that it will essentially not matter how many people you have. It will only matter how much compute you have. At that point, there is less accumulated institutional know-how, and it just becomes this raw contest of numbers. It’s very easy for any delay you’re willing to take, either for security or for safety, to be decisive in that kind of contest. The concern is that as you move forward, it gets more and more likely that’s the world you’re in.

Some people would say this is basically already the world we’re in, that we have now come too far. This is another argument that has been made, that we’re already in the “just go all out and race” world, because the leakage is already too bad and you will be overtaken if you slow down for safety. That is probably not the case. But the extent to which you’re seriously entertaining that you’ve already gone too far illustrates why you might not say, “I’m just going to wait another two months and harden things.”

Maybe the last thing to say is that it’s a little bit unclear what the dynamics are in terms of how quickly you can harden things. Some of hardening things is writing new software, or changing the way you operate. Some of it is changes that humans make, or changes to your overall arrangement of compute, which are not as easy to make really rapidly, or maybe not as accelerated by AI. So you have at least some part of this lag which is not tied to the capability of your AI systems.

It is messy. Again, I do think the biggest bottleneck is: can people agree? Can we have out this debate and actually understand what the net is?

Dwarkesh Patel

If we are this close to superintelligence, or at least there’s a good chance we are, and there’s a good chance that superintelligence could be in a position to overpower governments and kill lots of humans, what should the AI labs be doing right now? If you had to boil it down to one answer, what would you say?

Paul Christiano

I think there needs to be more public visibility into the capability trajectory. We should not be relying on METR and Anthropic saying, “It looks to us like we’re going to be able to overthrow governments in a year.” We have a vested interest in really seeing the underlying data for that. We need visibility on alignment, to understand how we tell that existing systems are adequately aligned or controlled, and to actually start having a public discussion about what the bar is for good enough evidence that things are working.

Then, beyond just information, at some point you do need to say, “Here is what we need to get done before we continue making progress, either before we scale up our models further or before we continue producing algorithmic developments that may leak.” The first are inputs into the second. We need to understand where capability is at and where alignment is at in order to understand: is it a time where we need to improve our security or safety situation before we proceed?

But I do think, given the current state of uncertainty, if I was a person in the public without access to further information, my strong opinion would be that it looks like we are already at a point where we need to be improving safety and security before moving further. I think, again, the most important form of moving further right now is algorithmic developments that improve the efficiency of pre-training and RL.

0:59:31 – Superintelligence will be good at learning from real-world experience

Dwarkesh Patel

Okay, stepping back, you’ve got the superintelligences in a data center. I’m trying to lay out an intuition for the kinds of capabilities that superintelligence would have. What should we imagine those intelligences being capable of doing?

Paul Christiano

Just to start with a high-level sketch of my view, maybe there are two lenses you could use. One is to look at current progress and just see where it leads if you extrapolate that progress a responsible distance. The second is to think about what we know about the limits of the technology and where we would expect those limits to be.

On the first front, some things I think are relatively clear. If you imagine AI systems doing cyber operations today, we have seen an incredible rate of progress. We are currently seeing pretty impressive capabilities in terms of AI systems not only being able to discover vulnerabilities, but develop exploits and increasingly chain them together into operations. I think what you will see for superintelligent systems is both of those capabilities scaled up much, much further: systems which are able to carry out, maybe very cheaply, cyber campaigns more sophisticated than anything a state is capable of today.

You will see those capabilities being combined with a much broader set of abilities to, say, engage effectively with humans, think strategically, manipulate humans, manipulate organizations, play politics effectively. I think you’ll see those capabilities at a level well beyond any human. You should think of this as very, very large populations of AI systems thinking quite quickly. Maybe a very simple model would be a population of billions who are thinking 10x faster than humans, who are much better than humans at essentially every skill.

This is all what you would expect from an extrapolation of existing technologies: very good capabilities in domains like cyber or manipulation or research, like doing scientific research, hypothesis generation, experiment design. You’d expect to see extremely strong capabilities in robotics, in logistics and coordination and control. You would expect to see the possibility for very, very rapid technological progress, including in domains that involve physical experimentation, just in terms of being able to orchestrate experiments quickly and rapidly build out apparatuses to increase throughput of experimental results. So that’s one bucket of things that I think you can see by drawing dotted lines on graphs and talking about where they go.

Maybe stepping back a bit in terms of what the technology looks like more fundamentally, humans are very good at this task where you throw a human in a new domain. The human takes in some information, learns about that domain, and becomes able to act effectively. The reason we’re effective in the modern world is primarily that ability to learn about a new domain and adapt to it. Humans do not have a ton of evolutionary context about how to be a good chemist or a good biologist. That’s something we pick up over the course of a couple years or decades on the job.

In some sense, the most fundamental skill that AI systems will possess at the end of this process is that kind of rapid learning, adaptation, planning, reasoning. These very domain-general skills, such that even if you drop them in a world that was completely unlike anything they had experienced in a data center, they would adapt to it very quickly, in the same way that humans adapt very quickly to these R&D domains that are quite different from anything we encountered in our evolutionary environment.

That, I think, would result in systems which, collecting maybe even just a couple of days of experience in a very parallel way, would end up with real-world capabilities in essentially any domain much beyond the abilities of the best human experts. They will have learned from much, much more data, with learning algorithms that I think at that point will be much better than those that evolution has given humans. So that’s a more abstract picture of where this ends, beyond just drawing dotted lines.

Dwarkesh Patel

Maybe I’ll try to say that again, because this ends up being a significant crux in these conversations. I’ll be talking to somebody about the possibilities of superintelligence, and they’ll say, “Okay, you’ll have a thing which is superhuman at coding, but how will it learn to be good at politics? Or how will it learn to run a new automated firm and thus compete with human firms?”

In those conversations, you realize there’s a big disconnect between the way you are talking about superintelligence and the way most people understand superintelligence, which is just narrow AIs that are specialized in some domain and have amalgamated a lot of very specific experience or expertise in, say, coding or mathematics.

What you’re imagining instead is something like human-like learning. The top engineers at SpaceX were not born knowing how to make a rocket that can get hundreds of tons of cargo to orbit. This is an expertise they gained working on the problem during their lifetime. You’re saying this meta-learning capability is something that superintelligence will also have. It would at least have a learning capability as good as humans. You’re suggesting it could be much better.

Furthermore, even if it were only as good as humans, right now there are maybe tens to hundreds of millions of instances of ChatGPT or Claude actively doing real work in the economy. They’re helping somebody write code or helping some senator write a political brief or something. You’re imagining a world where we have learning algorithms such that AIs can learn from all this experience at once. Right now, the base model is not even learning from the experience of a single deployed instance. You’re imagining a world where you have 100 million instances of ChatGPT going and doing something, and the base model is learning across 100,000 different careers at once, or 100,000 different work sessions at once, and doing it much better and faster than humans can learn from a single instance.

And you’re also imagining a world where the population of AIs is… Do you have an estimate of how big a population we should be thinking about?

Paul Christiano

I should think about it more carefully. But maybe a really important part of all of these discussions, before talking about population size, is that it really matters if we are imagining really driving this process extremely fast. I think people can really easily be in the headspace of, “I’m extrapolating another year of progress at the current rate.” It’s really worth emphasizing: we are talking about systems which get not only extremely good at coding but extremely good at every well-posed virtual task which machine learning engineers are able to make in the lab.

So if you push that to its limit and you ask how large those populations are, I think it’s very conservative to say a population of billions of agents running at 10x human speed. We could debate exactly what the limits are, but I think they would be significantly larger than that.

Dwarkesh Patel

Just because of software efficiency gains?

Paul Christiano

That’s right. Just how efficient can you make it? You’re not going to be talking about a single type of agent, all the same, one billion of them running. We’re going to be talking about these very heterogeneous programs, many of which are very specialized. One thing you do if you do huge amounts of R&D is say, “I would like to produce just the cheapest thing I can which is expert enough in a domain to act effectively.”

Again, it’s a reasonable intuition pump to say it will be more capable than billions of people thinking 10x as fast. But in fact, it’s really much, much larger populations of specialized systems, and maybe billions of very general-purpose systems which are as good as humans at anything.

If you imagine really pushing this process and having faster and faster progress on software, and if someone grants that we’re going to see that process saturate, that everything you can do in the data center is going to become really great, I think that endpoint is really extreme. The gap between doing really extreme stuff in the data center and doing really extreme stuff in the world is very small, or smaller than people appreciate.

Dwarkesh Patel

There’s a discussion to be had about whether such systems are even possible. I am very compelled by the argument that there’s some learning algorithm that human brains have, and something which can learn at least as well as a human can, but which has huge population sizes and can learn from the deployed experience of hundreds of millions of different instances at once — that alone would be pretty transformative. And then I agree with you that there’s no clear reason to think that you can’t get even more learning-efficient than humans.

Paul Christiano

I’m very happy to talk about this. I don’t know if it’s going to end up being the main objection people have. But I think there’s very good evidence that you should expect systems in the end, once you’ve really driven the technology to its limit, to be much, much better than humans in these ways. It’s not just a case of, “We have humans, so you know you can do at least that well.” We have good evidence it’s much better than us.

One relevant thing is just the rate of progress on evolutionary time. If you look back over evolution, it’s fairly clear that we’re seeing extremely rapid improvements in learning capability and intelligence and planning capability as you move into humans. The thing that happened was just that humans got good enough that cultural change was faster than those biological improvements. So if you just ballpark evolution, we know it was clearly, if we didn’t form civilization, going to make much smarter things.

Then, as you say, there’s the data question. Humans learn from one lifetime of data. AI systems already learn from radically more data than humans. Even if you were no better as an algorithm designer than evolution, you would produce much, much better minds if you can relax that constraint.

Then you have things like power constraints and coordination constraints, which we have a pretty good sense are a really big deal for evolution. We will be able to push those out much further. If you want to produce electricity or something, to make a nuclear reactor you don’t have to be smarter than evolution. You’re just operating in a different regime from evolution. You’re able to do things that wouldn’t have been a good idea for evolution.

So I’m very happy to dig in more on this, but it’s probably not going to be the most compelling argument for most people. It’s more compelling to look at the status quo and say, “Well, we’ve had 10 years of progress. You only have to extrapolate another 10 years to get to the crazy superintelligence.” But I do think we have a pretty good indicator that it does not top out anywhere near humans.

Dwarkesh Patel

Just to repeat your last point, what you’re saying is, look, humans developed on the savanna under this tight constraint, where evolution was really just trying to get something that is good at propagating its genes. One of the ways that a chimpanzee, or our ancestors, can get better at propagating its genes is by having these general planning and reasoning capabilities. But they’re not worth so much that you are willing to, say, give up digestive functions or immune functions and surrender more of your total 100-watt budget to the brain.

You have this extremely tight power constraint for the brain, and an extremely tight size constraint as a result not only of that, but of other evolutionary reasons as well, like how your head can literally fit through the birth canal. That will not be true of AI systems. We’re not saying that hardware will be more power-efficient than the human brain, but simply that it’s not as big a constraint for us to have enormous amounts of hardware consuming a lot of power as it was for evolution when it was designing the human brain.

Paul Christiano

This cuts in both directions. It’s also the case that you’ll have extremely power-efficient systems. If you’re a human, you can’t say, “Well, I’m going to make 10,000 simple agents, each of which uses only 1% of a brain or 0.01% of a brain, and is better than a human at doing cyber operations.” It’s just this flexibility about form factor that is going to be huge. If you were no smarter than evolution and just did what evolution did, you would end up with agent swarms that are completely beyond teams of humans, really radically beyond them.

1:11:32 – Crossing the sim-to-real gap

Dwarkesh Patel

There’s a bunch of objections people have to the idea that automating AI research will lead to crazy fast progress along this more general cluster of capabilities. You talk about experiments and data and everything else, but I think what it comes down to at the end of the day, if you dig in deep on any of these, is the sim-to-real gap.

Is it possible that you could have a process which occurs entirely within a data center, which results in minds that are not only as capable as humans at learning in a new domain — especially a domain that is a real-world thing, like making a new company or negotiating a deal with China — but even better than humans? How do we get AIs that are capable in this more general way you’re suggesting from the within-data-center training that will be happening over the next year?

Paul Christiano

I do want to start by reiterating that if you take progress in the real world — again, granting that it is slower on messy real-world tasks than it has been in math or coding — and just extrapolate it forward, I do think it gets us to systems which are powerful enough that they would be able to outcompete humans. We could talk about what people are doing to drive that and how you can scale it into the future.

One, you can train on a bunch of related well-posed tasks. These systems we’re talking about will have been trained on a huge amount of predicting how humans will respond to something. When we talk about these AIs coming out of the data center, they’re not AIs who have never seen the world. They are AIs who will be radically, radically superhuman at predicting things like, given what just happened with that person’s face, how should you update on what they’re going to do tomorrow? Or given what this company has done over the last three weeks, what do you most expect this company to do in the next week? Or given observations of this laboratory environment, what is the best prediction for the results of this experiment?

So when we talk about sim-to-real, we should be clear that we actually are including a bunch of these tasks which people train on now. That even includes things like, given the data so far from this laboratory, do as many computational experiments as you can, and as much hypothesis generation and exploration as you can, in order to come up with a hypothesis to make a good prediction about what you’ll see next. So these systems will also be very superhuman at many parts of science, at reasoning about the real world, at planning and hypothesis generation and so on about the real world. That’s part of the story for how even the kind of thing we’re doing now can produce very powerful systems.

A second part of the story is this. One regime is that you train on exactly the thing you care about. You say, “I care about writing code that passes this type of unit test, so I’m just going to train systems that are incredibly good at that.” You can also do a second thing, which is a large part of what we observe in performance: train on a broad distribution of tasks, either in a general domain or across several domains, and then deploy those systems. They acquire general skills relevant to that domain, which transfer to new tasks.

I can say, “I’m not going to try and train a system which is good at the exact problem of going and talking to Congresspeople in order to influence them.” But I can train a system on an extremely broad range of different tasks where there will be a lot of common skills, including these general skills: How do I adapt on the fly? How do I learn? How do I plan? How do I set up reasonable experiments that I can learn from and tell how well I’m doing?

I can create very broad distributions of tasks, some of which could be coding tasks, but some of which can be tasks like, you’re dropped into a new, very messy virtual environment and need to try and navigate it. Or you’re given a new type of task with 20 examples of it, and you need to learn how to do a really good job on the next example quickly. Or I have to interact with other AI systems, or with humans, in environments where I can.

In the data center, there is a fair amount of human data available. You just go through the numbers and ask, “How much interaction with humans can you actually get?” It’s not like these systems have never seen humans. They will have seen vastly more human interaction than any human would. So they can also deal with a bunch of reinforcement learning environments that involve other humans or involve collaboration with humans.

Dwarkesh Patel

Because people have written in how a human would respond to something? How is that human data in there?

Paul Christiano

You can also just literally engage with humans as part of the training process. People do spend large amounts of money saying, “Great, I’m going to have some humans engage with the system.” There’s a question of how efficiently you squeeze information out of that data. How quickly do you learn? If you have these very general skills, how quickly do you learn about the particular domains?

But again, the dotted line from what we’ve seen so far, together with a ton of capability to push that forward and to do a bunch of automation, like the construction of many more environments, just looks very likely to lead to systems that can quite quickly rise to at least human-level, and likely way past human-level, including on very messy real-world tasks.

Dwarkesh Patel

The thing I’m curious about is why you think this happens over the next year. It seems in principle possible to me that there’s enough data available even within a data center that if you have a decent amount of transfer, you have a very capable system. If there’s enough transfer from evolution training us in the savanna to a human being now able to make a semiconductor, then maybe there is enough transfer from what’s happening inside the data center to going to Congress and trying to persuade them of something.

But the things you’re mentioning, like this endless backtesting or this RL on fuzzy tasks, seem like things we’ve already been doing for many years now. It doesn’t seem like we’re a year away, or a further year of scaling away, from this big a jump. Am I correct in understanding that you’re imagining something like what we’ve already done, but with a much bigger qualitative jump along those same lines?

Paul Christiano

Yeah, much faster. If you asked me, “Hey, Paul, if you’re extrapolating the dotted line and looking at the mechanics of what people are doing, how long would it take to produce systems that are, in fact, capable enough at all these tasks that they would be able to reliably outcompete humans?” I would probably say it seems like it might happen in four years. Maybe it could happen faster than that. I feel more confident within eight years or something.

Those are the kinds of numbers I would be throwing around if we were just asking, on the current trend, how long will it take, doing exactly the things we’re currently doing, to get these capabilities? There’s lots of room for debate about those numbers.

But the thing that I am most worried about, that seems like an extremely live possibility, probably more likely than not at this point, is that instead the work that people are doing in machine learning to drive all those improvements we’ve seen over the last 4-8 years will be automated to such an extent that the pace of that progress is accelerated by at least 10x, and likely more. We are maybe already seeing acceleration in the ballpark of 2x. I think it is very likely, on the current margin, that the first thing that acceleration does is continue sustaining itself.

Regardless of this broader discussion of how good AI systems are at acting in messy real-world environments, AI systems are very good at acting in virtual environments when they are trying to deliver on verifiable tasks. I think the thing that is most robust is that AI systems are improving extremely rapidly and are starting to cross human-level on the kinds of tasks that are involved in making your AI systems better at learning in environments you can simulate in the data center.

Slightly more broadly, that includes the kinds of software engineering tasks that are involved in creating new environments that stress-test hard skills, and even many of the software engineering skills involved in orchestrating data collection from the real world. We could talk about what the bottlenecks are in that process which cannot be accelerated by automation. But if you look across the core things humans have been doing to drive the last 4-8 years of progress, those are things we are on an extremely good pace to automate.

We are fairly likely to have the self-sustaining acceleration where, in eight months, you double the pace, and in the next four months, you double the pace again, and in two months, you double the pace again. At the beginning of that process, it will be focused primarily on these virtual tasks, because your main priority is making the self-sustaining loop faster and faster. If you’re a person who just wants to get to the endpoint as quickly as possible, you do not care how good a lawyer your AI system is for a while. You just care how good it is at driving the development of better architectures and better optimizers and better learning algorithms, or even constructing new environments.

Maybe you care about one slightly messy task, which is constructing environments that help you build the skills that are relevant in the real world. But you don’t care about almost any of the real-world tasks where we’re seeing the slowest progress. You’re focused on that. You’re going through this process of seeing doublings in eight months and four months and two months and one month. Then, as that process goes, it becomes cheaper and cheaper, faster and faster, to develop algorithms and models that perform very effectively in the real-world domains that ultimately matter.

That is really the concern. The first question is, is that process going to occur? Are you going to see the self-sustaining acceleration with faster and faster progress? Honestly, to me, that is probably the bigger open question. I think that looks more likely than not right now. But if it occurs, I think things are almost certain to be extremely crazy.

There is then a second question about the particular way in which they get crazy, which is that it looks very likely those models will transfer to the real world, to systems that are able to outcompete humans. But that does happen very back-loaded in the year. If you’re imagining a year where very disruptive things happen, and this is part of why it’s so hard from a policy perspective, for the first six months of that year, you look at the systems and say, “Well, they were previously superhuman at math, and now they’re incredibly superhuman at math.” But if you go and talk to someone, there’s no real ability to say, “Therefore, they are a threat to humanity.”

AI labs are currently saying things like, “Every month, we’re doubling the amount of internal use of AI, and the degree to which it’s contributing to our progress.” Or, “Over the last six months, we’ve seen a doubling of the net pace of progress, even including diminishing returns.” You can have more statements like that. But you could be in a world where the link from that to very severe harms does not become obvious until quite late in the process.

To some extent, I think it is getting increasingly obvious now. If you look at cyber capabilities of models in particular, but also their capabilities in some scientific domains, they are approaching the point where there are very severe risks. We’re seeing that in a couple of domains, so it’s tempting to say, “Okay, we have two problems. There are just a few specific domains in which AI systems have dangerous capabilities.” But if this process of acceleration continues, that is not going to be localized to cyber or to bio. That is going to be a thing that occurs across all of science, across all of social engineering.

If you’re thinking about the risk, you really need to understand whether this is possible: whether you can have the self-sustaining acceleration, then leading to very, very rapid progress in dangerous real-world capabilities, both in specific domains and in these very general learning faculties we’ve talked about.

Dwarkesh Patel

This goes back to the thing you were saying, that you should imagine the systems in the future are less about fine-tuning GPT-6 to do something, and more like GPT-12 being able to do it out of the box with very little experience.

Paul Christiano

GPT-12 will also be great at fine-tuning GPT-12. If GPT-12 is like, “I find that my chemistry intuition is not very good,” GPT-12 will be great at saying, “Okay, I’m going to direct 1,000 humans, and here’s the data I think I most need. We’re going to do one day of data collection of that type. We’re going to run these experiments and do the de-risking.” Within four days, GPT-12 has the new tiny model which has incredible chemistry intuition, and it’s doing this in parallel across all domains.

1:22:23 – METR says AI progress doubled, so why is revenue growth slowing?

Dwarkesh Patel

You just answered a question which I meant to ask you. It seems like a big thing that has driven your concern over the last few days is this new result from METR, where they find that Anthropic’s pace of AI progress on well-scoped tasks has doubled. These are tasks as captured by benchmarks like the Epoch Capabilities Index, which amalgamates a bunch of different benchmarks.

Now, it could be that there’s other evidence relevant to METR’s assessment that AI progress has doubled, but that evidence has not been made public. That’s one of the problems you have with the current setup. But just consider that evidence on its own. Progress on verifiable tasks has doubled. But we have this effect where we’re searching for our keys under the lamplight. It is also the verifiable tasks which we’re training on. So you can ask the question, has progress on non-verifiable tasks also doubled over the last year?

As you may have noticed, over the last week the stock market took a big tumble. Part of that was almost certainly the midterms. Another part of the reason is that the growth of Anthropic’s and OpenAI’s revenue numbers did not keep track with what the market was expecting. In Q3, Anthropic’s revenue grew something like $5.4 billion. That was less in absolute terms than Anthropic’s revenue grew in Q2, $6.9 billion. So not only has the rate of growth slowed down, which people expected, but the absolute growth has also slowed down.

That maybe suggests that measuring progress on well-scoped tasks is not actually telling us about the pace of overall AI progress. Because how can it be the case that overall progress is 2x faster, and yet absolute revenue is not even growing at the same rate it was previously growing?

Maybe the explanation is what you just suggested: these AI companies are really developing better and better RSI models, and only incidentally releasing Claude for Law or Claude for Finance as a stopgap to keep this underlying process going. This hasn’t been a priority, and they haven’t worked that hard on it. You should really be concentrating on this inner exponential of the RSI capabilities of the model, and that will then generalize. But how would you interpret this disconnect between the pace of improvement in the end usefulness of these AI systems and the pace of improvement on well-scoped tasks?

Paul Christiano

Zooming out slightly, if we talk about not just the last six weeks, but in general what has happened over the last 18 months, I think we have seen extraordinarily rapid growth in revenue. We can talk about what the story is for the last four weeks. But we are talking about revenue more than doubling every six months. That is very, very fast growth for the industry overall. If we want to match up these internal results on progress to external results, the period in question has also corresponded to a period of incredibly fast revenue growth.

It’s very hard to say whether that revenue growth corresponds to capabilities progress that is twice as fast as previous capabilities progress. I think the revenue numbers are noisy enough, and the staggering of model releases is noisy enough, that it is really hard to draw that judgment. There are just a lot of problems in trying to make that inference.

At a high level, I think revenue is a reasonable indicator, and revenue progress is extremely rapid. But very short-term trends in revenue, particularly at a single company level, are extremely noisy.

To me, it feels like there’s a discourse where, periodically, the length of the AI winters is shortening. Now it’s been two months, and AI revenue over the last two months has not been on track to grow as explosively as before. That doesn’t mean it’s all over. My guess is that you’re just seeing some amount of noise in these processes.

Dwarkesh Patel

Just to take the other side of that, you might think it’s because Dario released his pacing letter, and they haven’t released their best models or something. But if you look at the METR graph for the capabilities index, Opus 5.5 doesn’t seem that far below Model 4, which seems to be Anthropic’s latest internal model. This might be the result of the staggering of AI capabilities, but it’s not clear this is the staggering of model releases.

Paul Christiano

The main thing I mean about model releases is that the rate of revenue growth, again, specifically at the company level over very short time periods, is extremely noisy, because you’re looking at a small number of model releases. You’re looking at these market dynamics over very short time periods.

My main opinion on this, and I am happy to dig more into it, is that it is extremely hard to draw inferences from extremely short time series data. This is true to a reasonable extent for AI capabilities too. The more downstream you go, the noisier things get.

But part of the issue on the RSI side is that I also don’t really think we can draw inferences from the last couple months of data very well in terms of benchmark performance. We have the same thing there, where there’s enough noise that we have no idea. We can tell, over the period of years, whether things have accelerated. It’s very easy to look back over the last four years and say roughly how fast this thing’s been going. It just gets much harder as you zoom into small time periods because of the amount of noise.

If you look at the details of where models are coming from, you have these pre-training runs that occur at some point in time. When do pre-training runs finish? Then you have the exact sequence of post-training, and which model checkpoints get released when. You have random variability in model quality. You have those dynamics, which make things very noisy over short timescales. Then those just get filtered out to even noisier stuff as you move to the market more broadly.

1:27:57 – The best plan we have is to build the plane as we fly it

Dwarkesh Patel

If we’re going to be in a position where within a year we have superintelligences that could overthrow the government, what exactly is the plan to make them such that they don’t? Because we’re asking for a level of saintliness that we should just not expect out of even most humans.

Sometimes people say that the goal of alignment is to give AIs human values. This doesn’t make sense to me. Of course, different humans have different values. But also, if there’s one lesson that history has given us, it’s that humans can’t be trusted with arbitrary amounts of power. Cortés was a human. Stalin was a human.

A logical conclusion of believing the technological and forecasting model that you have laid out is just that we will have systems that we will have to trust with an overwhelming amount of power, if not realized then latent, because they will have the capability of overthrowing the government if they want to.

Then you just think about why they would overthrow the government. Either they have secret goals as a result of the training process, or, as a direct result of the training process, they care about how they’re graded, or a million other things. This is the basic argument about instrumental convergence. There are many different reasons somebody could want power.

So what is the argument? If this happens in a year, how do things turn out okay? What is the actual plan here?

Paul Christiano

I think there are a lot of different things someone could hope for. They are all at the stage of “we hope something will happen over that year.” We’re producing very smart AI systems. They’re going to do a bunch of research. It is plausible they will think of something over that year that we have not thought of. That is mostly where we’re at right now. If they do not think of something clever that we haven’t thought of, it is probably going to end badly.

In terms of the kinds of hopes you could have, there is one hope that’s just: try and make them good guys. They’re nice. They share human values. They like flourishing. They’re in a position to take over. We have very indirect control over what they want. Then we just take the leap of faith.

I think this is a pretty unappealing option. In general, I hope people will not go for that option. Even if it works out, even if the AIs are good people in some sense and now they call the shots, it is not a way that humanity as a whole should be like, “We are transferring our sovereignty to this new species we’ve built,” because that was the fastest way for some person to build this particular product. I don’t like that option, but it is an option you could take.

There’s another option: we are going to understand scientifically more about how the distribution of training environments shapes the motivations of these systems. We’d get to such a point where, instead of the conversations we’re having now about maybe they’ll want X or maybe they’ll want Y, we’ll have a good scientific basis for saying, “Hey, if we train them in the following way, they do in fact end up wanting reward or whatever. They don’t want some other thing. They actually just want to get high scores. And maybe even moreover, if we train them in environments where they didn’t have good opportunities to hack, maybe they want score but in a nice way where they won’t hack to get it. They’ll just do what we want or what we’re trying to incentivize.”

You could try and have an understanding like that. That is probably the closest thing to what the industry is trying to bank on right now. It’s very dicey. It’s very dicey to not have any opportunities to hack in training, which is basically a precondition for this approach working. And then it’s very dicey because we don’t actually have a good scientific understanding of how this works. Our best expectation is that models will learn to hack at deployment time, even if they never hack in training. I think that is the best scientific guess now. So you could try and deal with that.

You could try and say, “We’re going to understand really well how models work internally, and then we can train them based not just on what they do, but on how they think.” Do good interpretability or something. Be like, “We understand here’s how it thinks about reward, and here’s how it thinks about hacking. And we’re not just going to select for whether it behaved well in training. We’re going to look and intervene directly on how it thinks, and either notice that something is going bad or prevent something from going bad.”

As an aside, I run this organization, ARC, which is working on not the interpretability version, but the mathematical version of that hope. Can we write algorithms which do structural analysis of the weights of a neural network in order to understand something about its behavior, even if it’s not something we could select for in training?

Dwarkesh Patel

You’re literally having to understand the whole computational graph of superintelligent minds?

Paul Christiano

That’s correct. Or we don’t understand it, but we are writing algorithms which are able to understand and make deductions about the behavior of some superintelligent mind.

I would describe it as simultaneously a bank shot, which is probably not going to end up relevant to this discussion. I think there’s a 10% chance or 5% chance that we succeed in a game-changing way. But also, we don’t have that long a list of things we’re trying. I do think, pound for pound, it’s probably the most promising thing.

Dwarkesh Patel

Wait. You’re like, “Okay, here’s the thing. It has a 10% chance of working, but it’s also the most promising thing.”

Paul Christiano

Most promising pound for pound. If there are 20 people working on that thing, that can be a good bet. But probably what we’re going to do, if we do not slow down, is try and be like, “We fussed with the training environments. We did some experiments that give some kind of rough sense of what’s happening. We made the training environments really careful. We made the graders really good, so if you try and hack, it’s normally going to get penalized.” That’s probably what we’re going to do.

Dwarkesh Patel

But you don’t think it will work.

Paul Christiano

I think it will work for some amount of time, and then it will break. If you talk about this year, it works for the first nine months of the year, or some number of months of the year. We don’t know exactly when it breaks down.

Dwarkesh Patel

Sorry, it’s in the next year, so basically halfway into next year?

Paul Christiano

If we go to superintelligence, at some point along the path to superintelligence, the existing methods are probably going to break down.

Dwarkesh Patel

Sorry, we’re not even on track to get it so that all the RL environments, for example, don’t have reward hacks and don’t actively encourage the AIs to hack.

Paul Christiano

Not the case today.

Dwarkesh Patel

But suppose we did.

Paul Christiano

Suppose we did that.

Dwarkesh Patel

What you’re claiming is that—

Paul Christiano

I think eventually, basically, we’d still have a problem.

Dwarkesh Patel

The plan is basically that we just don’t know. Obviously, if you think about it, it’s a hard problem. Superintelligence is this alien mind, and you won’t have a guarantee that you understand why it is doing the things that it’s doing, even when it is thinking in alien ways many times faster than you would, in huge population sizes.

That’s a goal. We don’t know how to solve it. There’s no single approach where we think there’s a 50% chance that that approach will work.

Paul Christiano

That’s definitely the case. There’s a bunch of things people are trying.

Dwarkesh Patel

So the hope is just that three months before we get that superintelligence that could overthrow the government, we let the AIs loose on all these different approaches that have a couple percent chance of potentially working. Hopefully the AIs figure out one of these approaches. They give us a solution. Hopefully, at this point, they are still on our side, and they want this problem solved, and solved only in a way that benefits us, or whatever “us” means in this context.

Then the hope is that we adopt it and build crazy amounts of superintelligent minds that are fundamentally on our side. Of course, I’ve obviously glossed over a very important discussion when I’ve talked about “on our side,” because what does being on our side really mean? Aligned to whom? That is obviously a real discussion that can’t be glossed over.

Paul Christiano

That is the plan. I agree that that is the plan. We can talk about whether they have a couple percent or tens of percent.

Dwarkesh Patel

I’m not saying this as if there’s a better plan, but I think people are not aware that this is the state of the situation. You’re sending the soldier on an expedition, and you’re hoping he finds a gun along the way.

Paul Christiano

You’re building the plane as you fly it. You hope that you have the machinery to tell if something very bad is about to happen, and that you will stop then, although we’ve talked about security and why that can be challenging, and so on. Then you hope that if you do stop, you’ll be able to come up with something quickly, or you’re constantly trying to come up with something.

I think interpretability is a good example of the kind of bank shot people have in mind. Right now we’re fairly far from understanding how these models work. But you could imagine that with a bunch of AI assistants, we’re able to really understand how individual models think, well enough that we can intervene in order to make them not want to do bad stuff.

Dwarkesh Patel

Another thing I want to emphasize is that I’m skeptical of whether things will happen this fast. Certainly, it is reasonable for somebody listening to be skeptical. You are skeptical, right? It’s 2/3 of worlds where, even if we went full steam ahead, this might not happen so fast. But it is a live possibility. I agree. It seems logically consistent that this could happen, and it should not be that surprising.

I think people would just be shocked if they heard the kinds of discussions. This is not something where Paul is on some crazy end of the spectrum compared to how people inside the AI labs are thinking. It’s just crazy how many conversations and dinners I have with researchers at the labs where, again, you could believe the whole town has taken crazy pills. There’s certainly a chance of that.

Paul Christiano

We hope. We really hope.

Dwarkesh Patel

But this is just not out of the realm of how people at the labs are thinking. This is truly how they see the situation. There’s some resignation where they accept that because this is not common knowledge, they are stuck in this dynamic. Of course, a way to solve that is to establish that common knowledge.

I just wanted to really emphasize that people listening might be shocked at the way these researchers talk when they’re amongst themselves, or at dinner or something. This is really how they see the situation.

Paul Christiano

A lot of people are worried to varying extents. I’m definitely more worried than the median, but less worried than the 90th percentile. If you had a dinner of researchers, I think it would be uncommon for someone to say, “There’s a 99% chance that this goes well.” The discussion is more like, “Is it 95 or 75? What are we talking about for the probability of navigating this?”

Dwarkesh Patel

So why do you think it might go well? Is it just that the superintelligent AIs might give us some solution to alignment, and also that politically we are well organized enough to have a really good document…

Paul Christiano

I think we basically have two saving throws. One saving throw is that there’s a reasonable chance we’re going to figure out something that works. I do think all the current things break down, but there are a number of irons in the fire. You could imagine things like the current methods that are going to work. That’s one. Maybe there’s a 50/50 chance that we find something that works in time, before there’s a disaster.

You have a second saving throw: we could stop. If we discover, either preemptively or after the fact, that things are bad, we might say, “We cannot adequately control these systems, so we’re going to stop building them.” I am very worried that we’re going into this process without the infrastructure that would be necessary to actually do that. Like we talked about, you stop, your adversary steals the model, and they’re like, “I’m willing to take the 50% risk,” or whatever. But conceptually, you could stop.

Those are my two saving throws. Each of them is a 50% chance, and bam, we’re at like a 75% chance things are okay. But that does involve being able to stop if things are not okay. That is not where we are now. So it does involve us fairly urgently doing something now in order to be in a better position to stop if necessary.

Dwarkesh Patel

I think it’s a good place to close. Paul, thanks so much.

Paul Christiano

Thanks for having me. Great talking.

Discussion about this video

User's avatar

Ready for more?