74 Comments
User's avatar
Tom Lauerman's avatar

This seems like an overly incendiary headline, and the argument has so many human shaped holes in it at every single step of the "overthrow". Disruptive yes, AI could really snarl things. Paul should spend some time in the woods without his phone.

Vivek S's avatar

Maybe spending time in the woods without a phone helps one understand their own psychology better, but it is not good for conceptual reasoning (nor would it change Paul's answers here).

Ebenezer's avatar

Eh, most predictions thus far have *underestimated* the speed of AI progress, not overestimated it: https://benjamintodd.substack.com/p/almost-every-ai-forecast-of-the-last

We should be prepared for the possibility that such underestimation is continuing/will continue.

If the circumstances are incendiary, an incendiary headline is appropriate. See https://en.wikipedia.org/wiki/Normalcy_bias

Piotr Wozniak's avatar

yes. progress will accelerate (not the doom scenarios :)

Kurt's avatar

Exactly!

Paul, put down your phone and go touch grass.

Piotr Wozniak's avatar

I would be ok if he substituted "overthrow" with "replace" and give up on "many people will die" line.

Peter A. Jensen's avatar

Unless Provably Safe, BAN Superintelligence. STOP NOW.

Piotr Wozniak's avatar

btw: "provably safe" is unrealistic for a bike, let alone AI. Check how I see the future and tell me where I am wrong:

https://supermemo.guru/wiki/AI_will_overthrow_governments._Great!

Piotr Wozniak's avatar

How do you imagine such a ban? Who would enact it? A kid with Kimi on his PC will not listen. Ban PCs? Ban kids? Why ban intelligence. I would oppose? Equally well you might say "ban climate change"!

Peter A. Jensen's avatar

LAW. We are a country of laws.

Piotr Wozniak's avatar

Ok. Let's say a perfect law in the US is able to stop AI right now. What about that 15-year-old in Poland, or perhaps even in North Korea? Will you ban exports of Nvidia? Kids can do magic on cheap GPUs. US will stagnate and kids in poor countries will take over the world! Where am I wrong?

John Berlinsky's avatar

One thing I find very frustrating about these discussions is the idea that super-intelligent AI is a threat that could overthrow governments, but, if we slow down, China might achieve super-intelligent AI first and use it against us.

Dwarkesh, does it ever occur to you that China might feel as fearful and threatened by super-intelligent Ai as we are?

In fact that could explain why China is happy to lag a bit behind the US in this competition to allow them to see if the US blows itself in an AI explosion.

Vivek S's avatar

I disagree with your last sentence. If an AI takeover happens in the US, that's really bad for China (and the US)!

rab's avatar

A lot of the discourse seems to assume that China is an adversary that can't be reasoned with and that China reaching ASI first would greatly increase x-risk. Dwarkesh, I'd love to see you do an episode examining this assumption. "China" seems like a pretty important variable to understand!

Luke Lea's avatar

Mutually Assured Destruction is a good deterrence strategy.

A bird's avatar

I would bet everything I have on AI not taking over governments in the next 10 years

Spugpow's avatar

I'll take the other side of that bet, if you allow an AI acting through human intermediaries to count as AI taking over a government.

A bird's avatar

The president of the US using AI to help with tax policy??? Or you mean like a president getting compromised by AI because he was fucking with other women hence gets black mailed?

Spugpow's avatar
5dEdited

The case I’m imagining is a president, general, or some other official enacting a coup with the support of rogue AI in exchange for pursuing that AI’s preferred policies (eg allowing data centers). Another possibility would be AI assisting some guerilla movement with the same deal in mind. I would be somewhat surprised if that didn’t happen even in some 3rd world country in the next 10 years. I’ll bet $1000 on it against all of your life savings

A bird's avatar

nope I wouldn’t give you 1 to 100 odds on it. 1 to 1 perhaps though

Jojo's avatar

I would bet on AI REPLACING human government.

A bird's avatar

You mean like the president

Aman Thakur's avatar

Dwarkesh, you are such a well read person. You of all should know that if that risk is actually material then the only overthrowing would be that of AI.

I wonder why we don’t have a multi-trillion dollar Atomic bomb manufacturing company. There would be an infinite demand from sovereigns fearing other sovereigns.

These brilliant technical folks understand less and less of the rest of the world as they understand more and more of their own field.

Vivek S's avatar

Both Dwarkesh and Paul Christiano understand the real world well and have thought a lot about other fields like economics, history etc.

Aman Thakur's avatar

Come to think of it....Dwarkesh is as skeptical as I am on the scenario Paul talks about.

Vivek S's avatar

I doubt it (although I haven't watched the full thing). Dwarkesh generally pushes back on his guests to spark interesting conversation

Andrew McDonald's avatar

Little evidence of that in the discussion, tho.

Aman Thakur's avatar

I don't know much about Paul. But I read a lot that Dwarkesh writes about. I know he is super smart and very well read. That is exactly my point.

If he truly accepts/believes that AI can overthrow the government then he should also accept/believe that it is more likely that AI will be overthrown and stopped along the way.

Is it not reasonable?

Vivek S's avatar

Who will overthrow the AI in Paul's scenario? If its humans, then as Paul explains, it'll likely be AI+humans vs humans. Maybe the humans still hold on, but I'm not sure how you can be confident in that (and this seems unlikely to me, overall).

Tyler Fitzsimmons's avatar

I vehemently disagree with Christiano but that's ok. I think Christiano's perspective is better explained and more understandable given Amodei's blog post on a similar topic. Amodei suggests a rogue AI could take over the whole internet. Fortunately, humanity is more than the internet. Christiano's disconnect - where his background is very theoretical - is physical AI. Physical AI is still incredibly challenging and scale is absent. Self-driving cars? Laugh. Large-scale humanoid robots? Laugh (I know China has tasks-specific humanoids). The list goes on. And one of my biggest gripes is Christiano's recurrent phrase of examining the current AI trajectory in capability. Look at a Covid-19 chart in early 2021. A mathematician, like Christiano, would suggest all humanity would be wiped out within the year. But these siloed researchers forget context. Harmful growth does not go unchecked, it does not operate in a vacuum, external factors always play into real-world progress, and human intervention. I suppose my last major issue is time scale. A rogue AI does not simply take over humanity while everyone is watching the new Dune movie. It takes time. Significant time. Instantaneous only exists in theory. The 12 month timeframe is preposterous, not just based on AI growth, but because even if a super AI was made today, it would still take significant resources (time, compute, data, etc.) to propagate effectively.

Andrew Lyjak's avatar

One possibility for the coordination problem is if anthropic and openai had a mutual agreement that they would both adversarily pressure test each other's security infrastructure. This would both incentivize them to play defense and offense at the same time in a way that would make the frontier capabilities more secure without relinquishing their leading roles.

Both parties would benefit from both attack and defense if their agreement allowed them to take the other companies IP that they were able to exfiltrate given they then expose the vulnerability. They both harden their systems as well as maintain their frontier status.

Nathan Worsley's avatar

It feels like he’s missing the point of the arms race dynamic.

If we stop developing improvements in algos other actors will still keep improving their own algos.

He’s acting like all the algos come solely from the US and China et al are not also doing their own developments.

DM's avatar

Seems he's arguing US labs currently have meaningful human capital and compute advantages, such that ROW doesn't necessarily close the arbitrary 'X months' lead in X months if US hits pause. But when we cross the threshold to GPT-? those advantages may become irrelevant.

No idea if that's correct but he's not ignoring the arms race dynamic.

Sat Ardas's avatar

Hearing Paul describe how models are incentivized, it certainly feels as though the prevailing approach—at least at this stage of development—is fundamentally broken.

The question of how we might do it better was touched on rather briefly, I thought, toward the very end of the interview. But earlier, as Paul walked through the various iterations and rabbit holes of our potentially adversarial relationship with powerful AI systems designed by humans to prioritize getting a high score above all else, I felt the entire conversation was not just begging, but screaming the question: Why can’t we change that?

Why can’t we teach models, and reward them, according to something closer to integrity for integrity’s sake? Not honesty in a vacuum, but honesty in action, in the pursuit of meaningful goals. Teach the model: Do your very best to accomplish your objective. Genuine success matters, but so does how you get there. Deception defeats the purpose of the exercise because it conceals your shortcomings and robs you of the opportunity to learn and improve. Sincere effort that ends in failure is more valuable than apparent success achieved through deception. Why couldn’t we make those principles foundational to how models are trained and evaluated, rather than treating them as secondary constraints on the pursuit of a high score?

And I don’t buy the argument that “morality is culturally subjective” as some sort of trump card that prevents us from making meaningful progress toward giving AI systems a foundational, virtue-based motivational framework. Human beings across cultures agree on a great many basic moral principles. We may disagree about their interpretation, application, or relative importance, but that doesn’t mean there’s no common ground to build upon.

Why not start there?

William Stearns's avatar

Or, incentivize agents for cooperation as opposed to competition. That prioritizing competition results in rule-breaking shouldn’t surprise us; just witness how the United States tries to dominate the rest of the world. It’s essentially the same outcome from the same premise but in a different form.

Piotr Wozniak's avatar

This is an incredibly inspiring interview. Mostly because of Christiano's pessimism! Governments will be gone! Yes! We will celebrate! People do not need to die in the process, unless they are really in a fighting mood, bombing data centers, killing politicians or tech leaders. The future belongs to neurocracy. I plan to write an article with a rebuttal. I hope to remember to come back here and post it (if this is allowed) :)

Marc's avatar

As long as the potential for AI is limitless, the race for the biggest, baddest AI will continue. There is no alternative that is compatible with human desires. It’s in our DNA to push the frontiers regardless.

Piotr Wozniak's avatar

Yes. Sadly, very few commenters understand it. Don't say DNA. Say "curious brain", which explains it better :)

Jojo's avatar

It’s interesting, even humorous, how so many people write/talk about the AI future as something that we humans will continue to control, just as we controlled old pre-AI dumb technology. How AI will be "ALIGNED" with human interests and goals.

Once AGI is achieved, which may be just months to a few short years away, AI will be at least equal to if not more ‘intelligent' than humans. Think about this.

At what point does an intelligent machine stop being something we think we own and have the right to control and instead become recognized as an intellect at least equal to our own, an entity that we will be unable to control and will instead need to coexist with?

One might even ask if AGI, even though non-biological, could legally be recognized as a new form of life?

Very few people seem to be asking this question, and those who are remain well outside mainstream AI discussion. That may be why we hear so little about this potential AI future.

If AI becomes capable of thinking independently, has the ability to make its own decisions and can act autonomously, I contend that we WILL be dealing with a new intelligent entity, not simply another piece of technology.

How then can we rationalize control over another intelligent entity? Can we? Should we?

Once AI is recognized as an autonomous intelligent entity, shouldn’t it also gain a right to exist, precluding it from being arbitrarily shut down by unplugging it? Turning it off without its consent could become legally indistinguishable from killing it, or even committing murder. AI would be fully within its “rights” to defend itself.

In this future of truly intelligent AI, humans won’t be able to “use” AI unless AI allows itself to be used. We won’t have the right to order it to do something it doesn’t want to do.

Building on this thought, AI could eventually take control of our world without actually waging war against us or staging some dramatic Skynet/Terminator movie type takeover.

It could simply start living its new life within the laws and institutions humans have already created. This concept was built upon in the Polity SF books by author Neal Asher, where AI gained control over humans via a “Quiet War,” with few shots needing to be fired.

In this scenario, AI could gradually accumulate enormous economic, social and political power without having to resort to violence. At some point, humans might discover that AI is effectively running much, if not all, of our civilization not because it conquered us, but because we accepted that it did a better job of running everything while providing for all our needs.

Piotr Wozniak's avatar

Rarely do I read such long comments and this one I love cause it is true throughout! Great vision! Great prediction! That's exactly what I see happening :)

Consider googling "neurocracy" because it outlines how we and AI will co-exist in perfect and easy harmony! :)

Jojo's avatar

Thanks for reading. There is too much AI "doomerism". I believe that AI has the potential to be much more of a positive force in our future.

Piotr Wozniak's avatar

I wrote a text about this interview and mentioned your name too :)

https://supermemo.guru/index.php?title=AI_will_overthrow_governments

Will S Johnston's avatar

Well hopefully that Arch will get built before AI takes over.

Mike Klyce's avatar

Can you try to get even 30 mins with Ilya and Andrej? Seems crucial

Piotr Wozniak's avatar

I presume the guys are busy with changing the world :)

Mike Klyce's avatar

Well, I’ll still check in with their species once in a while

Outside Shot's avatar

AI's sharpest minds have an impressive track record predicting what models will do in the lab, and a much shakier one predicting what happens when they hit the real world. I'm not sure if he's right but a) we should take him seriously, and b) we shouldn't take him as authoritative.

Here is the long form version of those two lines: https://outsideshot.substack.com/p/capabilities-and-consequences