General relativity is said to be the most beautiful idea the human mind has ever produced. Most of us will never get to fully appreciate its elegance by taking the 20-lecture graduate course Adam taught on it at Stanford. But in this episode, Adam distills the key idea at its heart so clearly and compellingly that even I could keep up lol.

At the core of general relativity, Einstein is trying to figure out the principle behind a particular coincidence: that the mass that resists acceleration and the mass that gravity pulls on just happen to be exactly the same. Adam then leads us through the path of insight which Einstein called his “happiest thought.”

Then Adam lectures on black holes. First, by showing how even under special relativity you could create a perpetual motion machine if black holes weren’t truly black. And then, by explaining why the observations of an infalling observer and a distant bystander to the black hole would be so radically different

Adam leads Blueshift, the team at Google DeepMind cracking science and reasoning, which gave us the opportunity to discuss at the very end how close we are to AIs that could rediscover general relativity from scratch. Stay till the close for some philosophy of science.

Watch on YouTube.

Sponsors

• Jane Street has traders from all sorts of different backgrounds. For example, I recently got to speak with Jed Thompson, a trader who started his career in particle physics. Jed told me how the habits he built as a physicist (like never running a calculation without first having a good guess at the answer) helped him build good trading intuition. So no matter what field you’re working in right now, your experience may be more applicable than you think. Check out open positions at janestreet.com/dwarkesh

• Crusoe gave me early access to their new serverless fine-tuning product, so I decided to try fine-tuning a Dwarkesh-style question generator. Crusoe made this really easy: I just turned my interview transcripts into training data and then kicked off a run – I never had to touch infra or tweak hyperparameters. After training was done, I ran a blind eval with my team: they preferred the fine-tuned model’s proposed questions over my own suggestions about 30% of the time. Serverless fine-tuning goes live next week. Learn more at crusoe.ai/dwarkesh

• Cursor’s iOS app lets me kick off real work no matter where I am. For example, recently I was at dinner with friends when I had an idea about how to investigate the past few years of progress in sample efficiency. I pulled out the Cursor app, dumped my thoughts into a voice note, and 15 minutes later, Cursor had cloned the relevant repo, done the necessary analysis, and written up its findings. And now I’m expanding that work into a full write-up. Without the Cursor app, the idea would’ve floated away. Check out the app now at cursor.com/dwarkesh

Timestamps

(00:00:00) – The coincidence that led Einstein to general relativity

(00:16:42) – Gravity is a consequence of curved spacetime, not a force

(00:31:46) – Why black holes prevent unlimited energy extraction

(00:47:12) – Black holes are the ultimate power plants

(01:13:50) – What falling into a black hole would actually feel like

(01:18:51) – The three ways we know black holes are real

(01:24:21) – The first time we saw gravity bend light

(01:29:33) – How far can AI get without experimental evidence?

Transcript

00:00:00 – The coincidence that led Einstein to general relativity

Dwarkesh Patel

I’m back with Adam Brown. You currently lead BlueShift at Google DeepMind, which is cracking science and reasoning. In a previous life, Adam was a prolific physicist, taught at Stanford, and did research on everything from cosmology to string theory to general relativity.

It’s said that general relativity is the most beautiful thing the human mind has ever conceived or seen. I was curious if there’s a way that ordinary people like me could understand what is happening, or have some vantage on why it’s beautiful, without taking your 20-lecture graduate course. That was the prompt for this lecture. I appreciate you being willing to do it.

Adam Brown

Super exciting to be here. Yes, I think the answer is yes, we can.

General relativity, Einstein’s theory of gravity, is, as you say, the most beautiful product of a single mind that we’ve ever created. It’s one of the two great theories of 20th century physics, along with quantum mechanics. Unlike quantum mechanics, it was basically Einstein. He had a little help, but basically it was one person doggedly pursuing this idea for 10 years and then he wrote down this theory that ends up describing the motion of planets in the solar system and also the origin and fate of the universe. It’s pretty extraordinary.

It took Einstein, one of the most famous minds in history, about a decade to figure it out. But when I teach it I’ll do a 10-week course, and in 10 weeks people will get a better idea of general relativity than Einstein really had in 10 years. That’s because we have an advantage that Einstein didn’t have. We have Einstein, and many others like him going before us, who’ve been able to take these super complicated ideas—understood at the time as being totally incomprehensible by anybody with a sub-Einstein level of intelligence—and boil them down to their essentials, and not make many of the same mistakes that were made by our forebears.

In 10 or 20 minutes, I can’t give you a better idea of general relativity than Einstein had, but we can get to the core insight—what Einstein said was his most beautiful idea—and push through it to try and understand what the central idea of this theory is. Ok, let’s go.

Before general relativity, there was special relativity. Special, meaning it doesn’t apply everywhere. That was also invented by Einstein, 10 years earlier, in 1905, during his annus mirabilis. If you want to sloganize special relativity, you would start with the observation, or the hypothesis, that nothing can go faster than light. Special relativity takes that observation, promotes it to a principle, takes that principle extremely seriously as the central observation of our understanding of spacetime, and you arrive at special relativity.

Special relativity applies to electromagnetism. It applies—though Einstein didn’t even know about these at the time—straightforwardly to the strong and weak nuclear forces, two of the other fundamental forces that we know about. It does not obviously apply to gravity. That was corrected 10 years later by Einstein in his general theory of relativity, a theory more general because it includes gravity. It completes the set of fundamental forces. Again, it was invented by Einstein after 10 years of dogged pursuit, in 1915. If you wanted to sloganize general relativity, you might say, “Not even gravity.” Nothing can go faster than light, not even gravity. There’s much more to it than that, but it’s going to complete this arc of the centrality of nothing being able to go faster than the speed of light.

To see some background here, we’re going to have to rewind all the way back to the theory of gravity that existed before Einstein. The reigning theory of gravity at the time of Einstein stretches all the way back to Newton in the late 17th century, Newton’s laws, in his Principia in 1687. Well, he had a few. Maybe the one we could most talk about today is two of them. His famous second law says the acceleration, a, caused by a force is given by the formula ma=F. If you have a force F, it’ll cause an acceleration on an object given by a, where the mass tells you how much an object resists being accelerated. The bigger the mass, the bigger the force you need to cause a given acceleration.

This law, his second law, will turn out to be still true once we come to general relativity. We’ll have to have a more sophisticated understanding of what we mean by force and acceleration, but this will be preserved by general relativity.

A special case of the second law is Newton’s first law. Newton’s first law says that if the force is zero, then the acceleration is zero. If the force is zero, then objects continue to move on a straight line at all times. That will also continue to be true in general relativity. That if not subject to an external force, objects move along straight lines. However, we’ll have to upgrade our understanding of what we mean by force and what we mean indeed by straight line. That’s going to keep being true.

The one that’s not going to keep being true is Newton’s law of gravity. Newtonian gravity tells you what the acceleration is in response to a force, but you need to know what the force is to be able to do anything with that. Newton’s law of gravity says that the force caused by the gravitational interaction of two bodies is what’s called Newton’s constant—just some constant of nature—times the mass of one body times the mass of the other body—the mass of the sun times the mass of the Earth—divided by the distance between them squared. It’s the famous inverse-square law. It’s a vector that points in the direction of separation, and it’s attractive, so there’s a minus sign there. This will not be true in general relativity.

In fact, you immediately see that there’s a tension between this gravitational force law and the claim that nothing can go faster than the speed of light. If this were literally true, then by jiggling the sun, a straightforward interpretation of this law would just say that the force at the Earth varies immediately. I’ve changed the distance of the Earth and the sun, and so I can immediately detect it at the Earth, not eight minutes later but immediately. That would imply that you could send an influence faster than the speed of light. Newton’s force law is inconsistent with this principle.

One option, of course, could be that this is true for non-gravitational forces, but not true once you have gravity, and that indeed, using gravity, you could perhaps build a faster-than-light telephone using gravitational effects. That’s a possibility, but not a possibility that Einstein really wanted to embrace. He’d spent many years chasing out any possibility of going faster than light or any superluminal influences. So Einstein, and in fact many people at the time, thought that this is the one that has to give. Indeed, that is what’s going to turn out to be true.

Okay, so where are we? There’s actually a precedent here for an inverse-square law getting modified in such a way that it ends up being consistent with special relativity, and that precedent is the other force of nature, the electric force. There’s also the electrostatic force law—not written down by Newton, but written down a century or so later—which says that the force caused, not by the gravitational interaction of two objects, but by the electrostatic interaction of two charged objects, has a very similar form to the gravitational force. It tells you that the force is equal to some constant times the charge of one object times the charge of the other object, pointing also in the direction of separation between the two objects, divided by the distance squared. It’s another inverse-square law.

Again, for exactly the same reason, electrostatics looks to be inconsistent with special relativity. But ultimately it’s not. Or ultimately, this is not the full story. Electrostatics is just one limit of the true theory of electromagnetism, which is Maxwell’s laws, which has not just electric forces, but it also has magnetic forces. The electric forces only look exactly like this when nothing is moving. When things do start to move, there are additional corrections to this, all of which conspire to make the electrostatic force law fully consistent with special relativity.

In fact, the historical direction of understanding ran the opposite way. First of all, you have Maxwell in the middle of the nineteenth century writing down Maxwell’s equations. Only later do people notice, “Hey, Maxwell’s equations actually are fully consistent with nothing going faster than the speed of light.” That consistency is reflected in a symmetry called the Lorentz symmetry of the Maxwell field equations, only noticed later after they were written down, that eventually led Einstein to formulate his special theory of relativity.

So we have a precedent for starting with an inverse-square law and then dressing it up in a full relativistically invariant theory. So you might say, well, let’s just take gravity and do exactly the same thing to gravity that we did to electrostatics, in order to make some gravito-magnetic theory that makes Newton’s second law an approximation that’s ultimately consistent with special relativity. In some grand sense, that is what we’re going to end up doing. That is what Einstein’s going to end up doing. But it’s going to be a much more radical departure than the Maxwell generalization of electrostatics. There’s really two hints, both of which are visible in this formula, that we’re going to have to do something slightly different than we did for electrostatics.

The first difference between the electrostatic force law and Newton’s law of gravity is this sign difference. There is a big difference, which is that here it is a minus sign, and here it is a plus sign. That is reflected in the fact that if you have two positive masses—the Earth and the Sun—they gravitationally attract each other. Conversely, if you have two like charges, they electrostatically repel each other, which is why that’s a minus sign and that’s a plus sign. That means that you cannot do literally the same thing for gravity that you did for electromagnetism, because otherwise, if you did mathematically the same trick, you’d end up with mathematically the same result, which is that you would find that like masses would repel rather than attract.

Not to get ahead of ourselves, but ultimately that’s because electrostatics is mediated by a spin-1 particle, the photon, and gravity is going to be mediated by a spin-2 particle, and that’s responsible for the change in that sign there.

That’s why you can’t do exactly the same thing as electrostatics. So Einstein had to look for something else. He had to look for some other way to try and lift this to a relativistically invariant theory. In doing that, he had one clue. There’s lots of stuff going on. It’s part of Einstein’s central genius to focus on this as a highly significant clue of where he should look. It’s sometimes described as his most beautiful thought, that’s how he would describe it.

The clue is this. There is another difference between the gravitational force law and the electrostatics, and that is the object that plays the analog of the charge in electrostatics, for gravity. It’s the fact that it’s the mass sitting here. That’s a strange coincidence from Newtonian physics.

Mass, in electrostatic forces and accelerations, plays exactly one role. It’s sitting here. It’s the inertia of the object, and it’s what is resisting being accelerated. This is sometimes called the inertial mass. And then the charge is completely different and unrelated to the mass. You can have heavy objects that have no charge, like the neutron. You can have light objects, like the electron, that have high charge. There is no necessary relation between the charge of a particle and its mass. They’re just two entirely separate things.

Not true in gravity. In gravity, this mass that’s sitting here in Newton’s second law—the inertial mass that’s resisting the force—is exactly equal to the mass that’s sitting here in Newton’s gravitational law, that’s telling you how much you’re pulled along. It’s the same mass. So this is sometimes called the gravitational mass, and this is sometimes called the inertial mass. Unlike in electrostatics, the gravitational mass that appears in this formula is equal to what’s sometimes called the inertial mass that sits in this formula.

This equation is already true in Newtonian physics. Newton noticed it, in fact, and did a number of experiments to confirm that this was true to one part in 1,000 or so. By the time of Einstein, we knew it was true to one part in a billion, and now we know it’s true to one part in 10¹⁵. It’s striking that these two—which in Newtonian physics is just a complete coincidence, essentially, that they’re the same thing—nevertheless were observed to be exactly the same thing. Einstein honed in on this fact, and it was his central clue for what to do next.

This is sometimes called the equivalence principle. It’s responsible for the fact that if you take a feather and a brick in a vacuum chamber and drop them both, they will both fall and hit the ground at the same time. They’ll fall and hit the ground at the same time because even though the force on the brick is much stronger than the force on the feather because it’s heavier, that exactly cancels out the fact that the resistance to acceleration of the brick is larger than the resistance to acceleration of the feather, and they fall exactly at the same rate. The equality of those two is responsible for that exact equality.

Einstein’s genius was to hone in on this as a central clue for how he is going to end up replacing Newton’s law. A reason it’s a central clue is because there is, in fact, another class of forces—not fundamental forces like electromagnetism or gravity, but a set of emergent forces—that exactly have this property baked into them, that’s guaranteed in those theories. To explain that, we’re now going to move over to the experimental section of this discussion.

00:16:42 – Gravity is a consequence of curved spacetime, not a force

Adam Brown

So here’s a bucket. Here’s some water filling the bucket.

Dwarkesh Patel

You better know your physics, Adam. Otherwise you’ll destroy the studio.

Adam Brown

No tricks. Will you put your finger in that and confirm that it’s wet? Here is the bucket. At the bottom, there’s no mystery why the water is not falling out of the bucket. It’s not falling out because the force of gravity, as we’d understand it, is pointing down to the bottom of the bucket. But now what we’re going to do is go a little bit faster and loop the loop. Now there is what you might find superficially surprising, which is that the water doesn’t fall out of the bucket even when the bucket is upside down.

There are two ways to understand that. One way is just the straightforward way. You would say the water wants to fall out of the bucket when it’s at the top of its arc, but by the time it’s got itself together to accelerate enough to fall out of the bucket, the bucket’s moved on and is now below, and it just didn’t have time to fall out of the bucket. Or you might say the same reason that astronauts don’t end up falling to Earth.

The second perspective, which is an equally valid perspective, is imagining that you’re riding along with the water in the bucket. And from that point of view, there’s another explanation for why the water doesn’t fall out of the bucket, and that is the centrifugal force. From the perspective of somebody moving along with the bucket, there is a force pushing them towards the bottom of the bucket, and that force is known as the centrifugal force. It’s what’s known as a fictitious or inertial force.

The centrifugal force just says that there is a force caused by being in a rotating reference frame, given by your speed divided by the radius of the circle you’re going round in, pointing positively outwards. So this is the centrifugal force that pins you to the bottom of the bucket, or pins you to the outside of the car as you go around a bend.

What do we notice? What we notice is that your charge under the centrifugal force, if you will—how intensely you feel a centrifugal force—is once again, just like with gravity but unlike with electrostatics, given by your mass. The mass that tells you how much centrifugal force you get is given by your inertial mass.

But of course, here it’s absolutely no mystery whatsoever why the mass that’s sitting here on the right-hand side is given by your inertial mass. It is given by your inertial mass precisely because the reason you’re experiencing this force is precisely the tendency of masses to wish to move along straight lines. The fact that you’re not moving along a straight line, you’re moving in a circle, it is precisely that inertial tendency that causes the mass to begin with.

Another way to say it: any time you have one of these inertial forces, caused just by your inertia, it is guaranteed to be the case that the charge under that force is given by the inertial mass. So inertial forces always have a charge given by the inertial mass. Gravity has a charge, and the charge of gravity is given by the inertial mass.

So Einstein leapt: could it be the case, and this was his central idea, that gravity itself is an inertial force? That’s permitted because the gravitational mass is equal to the inertial mass. It would be totally impossible for something like electromagnetism, because it would require that the electromagnetic charge was equal to the inertial mass, which is simply false for electromagnetism. Could it be the case, Einstein asked, that it’s true for gravity? It’s permitted by this fact. It would also explain this fact as now not an accidental truth like in Newton’s laws, but a necessary fact about the world.

So this was Einstein’s central idea in 1907, his most beautiful thought. But it sounds totally crazy. It sounds totally crazy because it requires us to be wrong about what straight lines are. It is an extremely radical proposition for the reason that I will describe right now. Inertial forces—like the centrifugal force or the Coriolis force or any of these other ones that we’re familiar with—are forces you experience when you are not moving on a straight line. When you are moving on a straight line, you don’t experience any inertial forces.

So in order for this to be true, we’d have to say that astronauts who are free-floating and free-falling are moving along a straight line. We’d have to say that you, who are just sitting there, seemingly not moving, are experiencing the force of gravity pushing you into your chair. We’d have to say that you’re not moving along a straight line. So we’d have to be pretty wrong about who’s moving along a straight line and who’s not.

So this is a radical idea because it requires us to be wrong about what a straight line is. In particular, you sitting here, just sitting in your chair… Here is your height above the center of the Earth as a function of time. Here is Dwarkesh just sitting here at constant height. Because you are experiencing the force of gravity, if gravity is an inertial force—because you’re experiencing a force down—that means that you have to be moving along a not straight line. So that’s you.

By contrast, this piece of chalk, as it goes up and down, executes something that’s well approximated by a parabola. The chalk is in free fall until I catch it, which means that if gravity is an inertial force, this has to be straight. Now, certainly the way I’ve plotted it, this looks straight and that one does not. So if gravity is to be an inertial force, we have to be wrong about what is a straight line and what is not a straight line.

However, this is actually a situation with which you should be familiar if you’ve sat in an airplane seat and looked at the screen in front of you. Imagine the map that you see on an airplane, and imagine you are flying from San Francisco to London. Now, I’m not good at drawing the Earth, but here is my version of it. Here we are in San Francisco. Here is Greenland, coming in from the North Pole. And here is England. Here’s London.

Dwarkesh Patel

I can tell you’re a physicist because of the very idealized forms of the continents.

Adam Brown

Sometimes it can be quite frustrating sitting there in the backseat of the airplane, because obviously the plane should be flying like this, moving along the shortest distance from one place to another. But instead they take this massive detour that clips Greenland and heads on down. You know that in fact that’s not what’s going on. You know that in fact, despite what it looks like on the graph, this is not a straight line. This rhumb line, it’s sometimes called, is not straight, and would certainly not be the shortest path from San Francisco to London. And this is in fact, to a good approximation, a straight line.

So in fact, the straight line from San Francisco to London does indeed go over Greenland, as I will now demonstrate. Here is San Francisco, here is London. You can see that the straight line that goes straight from one to the other would go in this direction over Greenland and hit London. That’s obvious on this map, because this map reflects the curvature of the Earth.

This map is getting confused, and it’s getting confused because it’s trying to pretend that the Earth is flat. It is trying to ignore the curvature of the Earth, and because it’s trying to map a round Earth onto a flat panel, there have to be distortions. Whenever you try and take something that is curved and pretend it’s not curved, you will inevitably end up being wrong about what is and is not a straight line.

You see this on the Earth, where this line that goes up and then comes down is in fact the straight line. You see it also in spacetime with general relativity, where this parabolic arc of the chalk as it’s thrown up, it is in free fall, it is the straight line in general relativity. And just like in general relativity, the reason you are confused about what’s straight and what’s not straight is that you are trying to pretend with this graph that you are in a flat spacetime. In fact, you are in a curved spacetime.

So in Einstein’s theory, the effect of matter is going to be to curve spacetime. Through curving spacetime, it’s going to change what’s a straight line and what’s not a straight line. People who are going along what they incorrectly think of as straight lines are going to experience the gravitational force, whereas astronauts are going to not experience the gravitational force.

The only missing piece here is to mathematically characterize the way in which spacetime is curved. In Newtonian physics, the Newtonian force is caused by the presence of mass. In Einstein’s general theory of relativity, it will be the curvature of spacetime that is caused by the mass.

He struggled for eight years between 1907, when he had this picture approximately mapped out, and 1915, when he wrote down in its finished form his general theory of relativity. The final output of those eight years was his famous formula, that I will not explain but will write down, that exactly captures his intuition.

I will walk you through this formula. This is just a beautiful formula. The left-hand side is some mathematics invented by some Eastern Europeans that characterizes the curvature of spacetime. This says how much spacetime is curved. This is some tensor, and the tensor will be zero if spacetime were flat, and non-zero when spacetime is curved in a particular way.

On the right-hand side is not spacetime anymore. On the right-hand side is matter. There are some constants: our old friend Newton’s constant, π an even older friend, and the speed of light. And then this quantity T_μν. T_μν is like a relativistic generalization of the mass that sits on the right-hand side of Newton’s force equation. So it is saying that the presence of mass—and in fact not just mass, but all forms of mass and energy—on the right-hand side causes the curvature of spacetime on the left-hand side. Or in a slogan: matter tells spacetime how to curve.

Once matter’s told spacetime how to curve, the curvature of spacetime tells matter how to move, in the second half of the slogan. The curvature of spacetime tells matter to move along straight lines of the curved space, and so experience fictitious forces if you try and pretend that spacetime is flat. That is Einstein’s general theory of relativity in a nutshell.

Backing up: an amazing thing about Newtonian gravity is that he invented it, allegedly due to some thought experiment to do with an apple falling off a tree. It describes not only an apple falling off a tree, but the motion of the objects in the heavens. It’s a massive cross hit that it describes planetary motion and also an apple falling off a tree. This was this amazing thing that Newton unified the heavens and the Earth and had one formula that applied to both. General relativity does all of that and goes one step further. It describes the motion of apples falling off trees, it describes the motion of Mercury and the planets in the solar system, and it describes the expansion of the entire universe. That’s a crazy, huge number of orders of magnitude that it hits.

00:31:46 – Why black holes prevent unlimited energy extraction

Dwarkesh Patel

You were saying a moment ago that one of the beautiful things about this theory is that it has reach in all these interesting ways that were not originally anticipated, to solve this original observation that Einstein had. One of them, obviously, is the black hole. I would love to get more insight than the high school version—that light falls into it and can’t get out—of why black holes work the way they do.

Adam Brown

Black holes are fascinating objects in general relativity, really the quintessential object in general relativity that doesn’t exist in the same way in Newtonian physics. The story is kind of wild. Einstein wrote down his field equations, the field equations we wrote on the board, describing the relationship between curvature and the amount of energy in the system. He thought that those equations are so complicated, no one would ever come up with exact solutions to them, that we’d just always be having to do approximations.

That turned out not to be correct. Schwarzschild was a Prussian artillery officer in the First World War. In between calculating the trajectories of artillery they were lobbing in the direction of their enemy, he figured out that Einstein’s equations—pretty much immediately after Einstein had written them down, within a matter of months—in fact have an exact solution, a solution now known as the Schwarzschild equation, and that we now understand describes a black hole. It is a solution in which there is no matter, except possibly at the very center in a way we’ll not describe. It’s a central point-like amount of matter. It describes what the spacetime around that looks like. It’s called the Schwarzschild solution, and it describes a black hole.

They were not called black holes at the time. In fact, people were extremely confused about what this solution even meant. People wrote down wrong things for about half a century about what this meant. Perhaps the worst offender was Einstein, who got extremely confused about it. He got particularly confused about what I will describe as the event horizon, and wrote all sorts of wrong things about how objects would maybe bounce off the event horizon. He was just totally confused, but from a modern perspective it’s extremely simple to understand what’s going on.

So let me tell you what a black hole is. General relativity, as we set it up, is about a collision between gravity and the finite speed of light. The simplest collision you could do was actually noticed by people even in the 18th century, before we had special relativity or anything like that. They just asked a very simple question. If you want to shoot something off the Earth, you know you need to shoot it with a certain velocity, the escape velocity. You need to shoot it fast enough if you want to escape far away from the Earth, so that the kinetic energy of the object you’re shooting is equal to the gravitational binding energy of the Earth’s surface. M is the mass of the Earth and r is the surface of the Earth. For Earth, the escape velocity turns out to be about 11 kilometers per second.

But for objects that are heavier or more compact, the escape velocity is larger. For example, for Jupiter it would be hundreds of kilometers a second. You can imagine objects that are so heavy or so compact that in fact the escape velocity becomes equal to the speed of light. So people idly wondered what would happen then. They didn’t have the tools to address it, but in the 18th century they wondered what would happen. You can calculate what the critical value of the velocity is. Just putting the velocity equal to the speed of light, this gives a critical radius of 2GM—the mass of the object cancels—divided by c².

So that’s somewhat suggestive. If you had an object that was this compact and had this mass, the escape velocity would be given by c, the speed of light.

Dwarkesh Patel

The connection you’re pointing out is a Newtonian one. Did anyone make this connection before GR?

Adam Brown

Absolutely. People in the late 18th century wrote this formula down. I think both Michell and Laplace had this. And they said that if you had an object that was that massive and compact, light would not be able to escape. That reason is not particularly compelling by modern standards but it turns out to be particularly correct, including crazily this factor of 2, which is correct for completely coincidental reasons.

But let me give you a more compelling argument that something funny is going to happen around this radius. And to do that, let’s think about trying to extract energy from objects by lowering the objects down towards a central mass. Let’s start off perhaps with the Earth. Here it is. We’re going to start off a long way away from the Earth, with a brick of mass m. I’m going to take this brick, attach it to a pulley system, and then slowly lower the brick down towards the surface of the Earth and deposit it with zero velocity on the surface of the Earth down there.

In doing so, I can extract energy from the brick. I’ve extracted energy from the brick because there’s a force pulling the brick down. That force I’m doing through a certain distance, and that gives you an energy. We know, at least in Newtonian physics, what the formula for the amount of energy you can extract from the brick is: G times the mass of the Earth times the mass of the brick, divided by r, the radius away. It’s an energy, not force, so it’s an inverse distance law, not an inverse square distance law.

That’s the energy you can extract from the brick by lowering it down to a distance r away from the Earth. Of course, if you try to lower it beyond the surface of the Earth, this formula changes. But let’s just put it on the surface of the Earth.

So this is the amount of energy I have extracted from the brick, out here a long way away. You can ask, what fraction of the rest mass energy of the brick have I extracted? This is a question that you would only naturally ask once you’ve invented special relativity and know that the rest mass energy is given by mc². We can straightforwardly calculate, at least in this approximation, that the fraction of the energy that you’ve extracted is that divided by the rest mass energy you started with. The mass of the brick, of course, is going to cancel, but not the mass of the Earth. This is going to be given by G times the mass of the Earth, divided by c² times the radius away from the Earth at which you stop, the radius of the Earth.

So what fraction have I got out of it? If you lower down to the Earth’s surface, the answer is you haven’t really extracted that much from the brick. You’ve extracted a fraction 7x10^-10 of the original rest mass energy of the brick, doing useful work a long way away.

First observation: this is small. In other words, the gravitational binding energy of something on the Earth’s surface is quite small in natural units. That’s why we didn’t really notice general relativity on the Earth’s surface until we did very sensitive experiments, because general relativity is in some sense a Taylor expansion in this number, where the relativistic effects, where the first order term is just Newtonian, and then the next order terms will give you the GR corrections to the Newtonian answer.

Observation number two, and this is something of a digression, is that by essentially sheer coincidence, this number here is very close to the chemical binding energy of rocket fuel. So if you take a rocket fuel like an oxygen-hydrogen mix, the chemical energy binding the rocket together, which is the energy that you’re going to extract when you burn it to make your rocket go, divided by the mc² of the oxygen and hydrogen you’re going to mix together, is given by 1.5x10^-10.

First observation: these two are close to each other, even though they came from completely different calculations. This was a gravitational calculation that was something to do with the Earth. This is a chemical property of hydrogen and oxygen. This is also very small. The reason it’s very small is that almost all of the energy in hydrogen and oxygen is not stored in the chemical binding energy of these things going together.

The vast majority of it is stored in just the rest mass energy of the protons and the neutrons, which chemical burning doesn’t affect at all. The second largest amount is stored in the nuclear binding energy of the protons and the neutrons to each other, given by the strong force and the weak force, which again chemical reactions don’t touch at all. This is a small number because chemical bonds are very weak compared to the rest mass of the things we’re considering.

These two small numbers are almost exactly equal to each other, which is why we can use chemical rockets to get to space, but it’s hard. In particular, this number is a few times bigger than this number, which means that your payload fraction is quite small when trying to use chemical rockets to get to space, because most of your fuel cannot get to orbit. You have to pay a rocket factor that’s going to tell you that most of what’s sitting there on the launch pad is going to have to be burnt up before you get to space, in order to get a small fraction of the rocket up to space. In other words, we can use chemical rockets to get to space in a way that would be totally impossible if we tried to do it from the surface of the sun, but it’s hard.

Okay, that’s the fraction on the Earth. But this formula tells you that if you have an object that’s heavier or more compact, the fraction of energy that you extract by lowering the object down to the surface is going to be larger. For example, if you lower it not down to the Earth’s surface but down to the sun’s surface, this would be larger. It’d be a million times larger, because the sun is a few million times the mass of the Earth, but then it’s also bigger, so that takes it away a little bit. You end up with 2x10^-6, the famous redshift from the sun’s surface.

You can escalate from there. You can imagine cramming a sun-like mass into an Earth-like radius to make this formula even bigger. Sun mass, Earth radius. That’s pretty much exactly what happens in a white dwarf like Sirius B. And this would get even bigger again. A larger fraction of the mass of the object you’d be extracting by lowering it down to the surface.

But it really feels like something has to give before we make an object that is too massive and too compact. In particular, if you look at this formula, what happens for r less than or equal to GM over c²? If this object were so compact and so heavy that it had a radius less than the mass of the object divided by c², it sure looks like you could get more than a hundred percent. The fraction would be bigger than one. You could get more than a hundred percent of the mass of your brick back by lowering it down to the surface of this object.

And that feels wrong. That feels, in fact, more wrong than what’s going on here, because now you’ve got all this energy a long way away. You could perhaps use it to make a whole new brick. You’ve got all this more than mc² out there. Lower that one down, and it feels like we’ve figured out a way to make a huge amount of energy where there was no energy before.

This argument is pretty suggestive that something has to go wrong by the time you get down to that radius. Indeed, when you do the calculation—this is a Newtonian calculation, so it’s only suggestive —in full general relativity, indeed something does go wrong. The thing that goes wrong is that you form a black hole.

You can imagine two ways that you could avoid this conclusion. One would be somehow that gravity becomes very weak when you get close to a massive object, weaker than the Newtonian law would predict. That’s sort of what saves you if you try and repeat this same trick in electromagnetism, lowering a charge down towards another charge and trying to extract the electrostatic energy between them. What happens is, essentially due to quantum effects, when one gets too close to the other, they start to fuzz out. The energy going like inverse r gets softened, and you can’t extract more energy because they stop attracting each other so hard. So that’s one possibility, the force gets weaker than Newtonian law would predict as you approach the other object.

That’s actually the opposite of how general relativity resolves this. General relativity resolves this paradox by the force getting stronger than Newtonian law would predict. In particular, the force gets so strong when you try to get within this radius, that in fact you cannot slowly lower the brick down towards the surface because you’ve formed a black hole. The gravitational force becomes infinite at a finite distance away—not at r=0, but at some finite value of r—and the brick simply gets ripped out of your hand and you’re unable to extract any more energy out of it. That’s the resolution that general relativity provides to this paradox. In particular, you will find that you’ve formed a black hole.

00:47:12 – Black holes are the ultimate power plants

So far, everything we’ve written down on the board is Newtonian. It’s just Newtonian, and you start plugging in the speed of light, and you start getting confused. To actually answer some of these questions that we’re asking, you need to go to general relativity, the theory that correctly unifies the speed of light with gravity. This was first done in the context of black holes by Schwarzschild, who wrote down the Schwarzschild metric that describes the gravitational field around a central mass, including potentially around a black hole.

Let me write down some of the formulas that emerge. In fact, I think I’m going to write down three formulas, three direct consequences of the Schwarzschild metric. They’re going to give us intuition for what it’s like outside and indeed inside a black hole.

The first formula I’m going to write down is the formula for the gravitational field that you would experience if you were trying to remain static outside a central mass. So let’s just talk about static observers. I can discuss how these will get upgraded for observers who are moving around. But for now, I’m just going to imagine that you’re trying to sit here at some fixed radius r away from the black hole. The reason you don’t fall in, maybe I’m lowering you down on a pulley. You’re just sitting here holding the pulley. The question is, how strong a force do you need to stop you falling down? You’re abseiling down very slowly. You’re static. What is the local force of gravity that you experience?

Or you can imagine that you’re sitting here, and the reason you’re static is that you’re firing a rocket very hard. The question is, how much acceleration do you locally feel? So by whatever mechanism, you’re remaining static. What is the local force of gravity that you feel? In Newtonian physics, you know what the answer to that question would be. The force of gravity is GM/r², which is Newton’s famous inverse-square law. But this gets a correction from general relativity. The correction is 1/√(1-2GM/(c²r)), this same 2GM/c² that we find all over the place.

What this tells you: first of all, if you’re a very long way away from the black hole, this here is essentially one. r is very big, and you get Newton’s force law back again. For the Earth, this is very small. As we discussed, it’s down by a factor of 10^-10, and then you take the square root.

So you don’t really notice it, but you can Taylor-expand this at large r, and you find that you get corrections. You get an inverse-square law plus an inverse-cube law correction plus an inverse-fourth law correction. You find that gravity at short distances is stronger than it would have been in Newtonian physics. This is the general relativity correction and it’s making the gravitational field stronger. You have to accelerate harder to not fall into the black hole.

In particular, once r is equal to 2GM/c², what’s called the Schwarzschild radius, you have to accelerate infinitely. The proper acceleration required to not move in r goes to infinity. In fact, if we now convert this Earth to a black hole, this is a very significant radius over here, 2GM/c². It’s called the event horizon.

It’s called the event horizon because if you want to remain static outside the event horizon, further away from the event horizon, you just need to accelerate with some finite velocity in order to remain static. You need to have a finite gravitational field. But the gravitational field, as you approach the event horizon, becomes infinite. So once you’re at or beyond the event horizon, it is impossible to remain static. You will inevitably get sucked into the black hole no matter how hard you fire your rocket.

Now, this is just a static formula. You might imagine, “Okay, it’s impossible to remain static closer than that, but maybe I could avoid falling into the black hole by orbiting really, really fast. If I orbit really fast, I have a huge centrifugal force that pushes me away from the black hole, and I can stay out of the black hole that way.” That actually doesn’t work. The reason it doesn’t work is somewhat instructive for the way gravitational attraction happens in general relativity.

Of course, if you think about the International Space Station, why doesn’t it fall towards the Earth? It is precisely the fact that it’s orbiting. The fact that it’s orbiting gives it a centrifugal force that shoots the astronauts away from the Earth and precisely balances the gravitational field on the astronauts, which is why they feel weightless there. So orbital angular momentum, if you’re a long way away from the black hole, helps you stay away from the black hole, stops you falling in.

There is this kind of sci-fi notion that black holes just suck in everything around them. Not true. You are perfectly able to orbit around a black hole if you’re a long way away from it, just like you would orbit around any central mass. You are not inevitably falling into the black hole. You can orbit just fine. But orbiting stops helping when you get too close to the black hole.

We said that the event horizon is 2GM/c². In fact, once you’re already within 3GM/c², orbiting is counterproductive if you’re trying to stay away from the black hole. That’s because there are two effects of orbiting. One effect helps you stay away from the black hole. That’s the centrifugal effect. Orbital angular momentum pushes you away from the black hole due to the centrifugal effect. It’s not too hard to write down the version of this formula that applies when you have angular momentum, and you would see that pushing you away from the black hole.

But there’s another effect which drags you towards the black hole, and that is the fact that in general relativity, all energy gravitates, not just rest mass energy. Kinetic energy also gravitates. So the effect of orbiting is that you have an additional pull down towards the black hole from the coupling between the gravitational attraction between the mass of the black hole and your orbital angular energy.

When you’re far away from the black hole, the centrifugal force is the more important term. When you’re close to the black hole, that coupling is the more important term. In fact, once you get within 3GM, orbital angular momentum stops helping and starts hurting. There are no ballistic orbits that go within 3GM and manage to escape again.

So that’s formula number one. It tells you what the gravitational field is a distance r away from a black hole. In particular, it shows you that once you get to this critical radius, the gravitational field becomes infinite. If you cross that, you must proceed to the center of the black hole, no matter how hard you fire a rocket. That’s called the event horizon. At the event horizon, you are not yet dead. You are, however, doomed if you cross the event horizon. You will never be able to escape, not if you convert yourself to light and try to shoot yourself out, not if you fire your rocket infinitely hard.

The other place, of course, is r=0, which is where you actually die. That’s at the singularity, and we’ll describe that a little bit in a moment. In Newtonian physics, the gravitational force only becomes infinite there. In general relativity, it becomes infinite already at the event horizon if you try to resist the force of gravity. That’s formula number one.

Now let’s do formula number two. All three formulas I’m going to write down are heavily related to each other. They’re really going to be reformulations of each other. Formula number two asks about gravitational time dilation. Let’s again imagine that you’re sitting here, Dwarkesh sitting here some radius r away from the black hole. I’m sitting out here, way off at infinity, just watching you. We’re static relative to each other. There’s no relative motion. You’re just suspended here by your pulley system. The question is, how fast does your watch go relative to mine?

Of course, as far as you’re concerned, your watch is ticking at one second per second. As far as I’m concerned, my watch is ticking at one second per second. But if I look at you, I see your watch as running slow. If you look at me, you see my watch as running fast. The second formula makes that quantitative. How much slower does your wristwatch—which is closer to the black hole—run than mine? It says that the time interval, as measured by your wristwatch, is given by the time interval as measured by my wristwatch a long way away, times this exact same square root factor that’s showing up all over the place: the square root of 1-2GM/(rc²).

This factor here is less than one. So if I think one second has passed, you think less than one second has passed. In other words, if I slowly lower you down towards the black hole—you hang out some finite distance away from the black hole for what feels to you like a year, and then I raise you back up a long way away from the black hole—you will return to a world that has aged a lot more than you have. This formula makes that precise. I observe your wristwatch to be running slow. You observe my wristwatch to be running fast. Time passes slower down here than it does up here.

This is a fact that has by now been extremely well observed experimentally. In the 1950s, in the Harvard physics department, they put two atomic clocks at two different heights in the building, and noticed the one that was higher was running faster than the one that was lower. This is an effect that is now considerably within the precision of, for example, GPS. It just has to subtract that effect, otherwise everything would drift all over the place. GPS clocks that are sitting on the Earth’s surface are running slow compared to the atomic clocks that are in orbit sending out the signal. You have to account for that difference and subtract it off in order to get an accurate read.

This is known as gravitational time dilation. Notice it’s quite different from the relativistic time dilation you see in special relativity, which is caused by two objects being in motion relative to each other. Here, we’re not in motion relative to each other. We’re both static. We’re fixed. This is caused by us being at a different place in the gravitational potential, you deeper in the gravitational potential than me.

So those are two different sources of time dilation, and they stack. Let’s say instead of being static here, you’re in orbit. You’re far enough away that you can orbit the black hole. How slow do I see you as moving? There are now two contributions, both of which make you look slow relative to me. One contribution is the gravitational time dilation given by this formula. A second contribution is the good old special relativity correction where moving observers look like they’re going slow, and we’ll have both of those effects. So you’ll look like you’re going even slower than you would have done otherwise as you go around the black hole.

Dwarkesh Patel

One thing that seems different between this and special relativity is that there’s no symmetry. In special relativity, both observers will feel that the other one is aging slower than they are, because they’re both moving relative to each other at the same rate, and there’s no true inertial path. But here, it actually does seem like there’s a global sense in which one is a more relevant inertial frame than the other one.

Adam Brown

You’re exactly right. In special relativity, if you and I are moving relative to each other, I think your watch is moving slow, you think my watch is moving slow. Neither of us is more correct than the other. The principle of relativity tells you that both of our perspectives are equally valid.

Here, both of our perspectives are not equally valid, because there is not the symmetry that there was in special relativity. In particular, the symmetry is broken by the black hole. We both agree that you are deeper in the gravitational well than I am, and your clock runs slower than mine does. You do not see my clock reciprocally running slow. You, in fact, see me sped up. If you were observing me, you see me living my life in fast-forward. So this is the second formula. It says how fast our wristwatches move relative to each other.

Now let’s imagine that you’re here with your slow-moving wristwatch, and you shine a light towards me. Let’s say the light has a particular frequency. You made it with a sodium transition, for example. As that light travels upwards, by the time it reaches me, I’m going to think that it is lower frequency than you thought it was when you sent it. Why? Because frequency is about how rapidly it oscillates. I just think that everything you do is moving slow relative to me. You think it’s oscillating slower. It has lower frequency, which means it gets shifted towards the red part of the spectrum.

The word that we use is redshift, gravitational redshift. It’s redshifted: lower frequency, and therefore less energy. If you send one photon up, the energy of the photon is given by the frequency. It’ll arrive at me more redshifted and with lower energy than it had when it left you.

Conversely, if I am up here, and I send you a photon generated by the sodium transition, as observed by you, by the time the light reaches you—you see me moving in fast-forward—you think that it has a higher frequency than I thought it had when it left me. It’s moved towards the blue part of the spectrum. We say that it is blueshifted.

So this thought experiment tells you that knowing the exchange rate for how time passes at different altitudes directly gives you the exchange rate for how much energy is worth at different altitudes. If you try to send me some energy, by the time it reaches me, it’s worth less to me than you perceived it as being worth to you. The amount it’s less by is going to be precisely given by the same square root formula that’s controlling everything else.

So that gives us our third equation. The third formula says: suppose that you, Dwarkesh, have an object of mass, mc², sitting with you at this fixed radius down there. How much energy, as measured by me a long way away from the black hole, is that worth to me? Of course, if I had it with me, it would be worth mc² worth of energy. But I don’t have it with me. It’s unfortunately sitting with you deep in a gravitational potential. So it’s worth less than mc² to me. In fact, it’s just the exact same formula. The amount of energy that it’s worth to me, by the time it reaches me, is GM/(rc²).

There are a couple of ways to see that. One is the way that we just said. Suppose you take your object of mass m. It’s just half an Avogadro’s number of carbon atoms and half an Avogadro’s number of anti-carbon atoms. One way you could send me the energy is by smashing them together, a violent explosion. You convert all of that energy to light and you try to beam that light energy up to me.

But what you find, precisely because of this gravitational time dilation, is that by the time it reaches me, I’m not getting mc² worth out. I’m getting, by the argument we just gave, less than mc² worth out. I’m getting 1-GM/(rc²) out. Mass down here suffers this redshifting as it goes up and has less energy by the time it reaches infinity than it did to begin with.

There is another way that you could have got the energy to me, not by beaming it up as light, but by just taking your mass object, attaching it to the pulley, and having me pull the object out. By the time I’ve pulled it out, I’ve now got mc² sitting out here, a long way away from the black hole. So I do have the full mc² worth of energy. But to get it, I needed to pay. What I needed to pay was precisely pulling it out of the gravitational potential. So from that way of thinking about it, that’s why I have less than mc² worth of energy left, because I had to pay to pull it out of the potential in order to accrue that mass.

So this formula tells you: if I have a brick of mass, mc², sitting at some radius r away from the black hole, how much energy can I extract from that brick if I’m a long way from the black hole? If we know that formula, then we can in fact calculate exactly this formula: how much energy have I extracted from the brick by lowering it down to a radius r?

Well, we know the answer to that question. The energy it started with is mc². The energy it now has is this. So the energy I’ve extracted from the brick while slowly lowering it down using my pulley system must be the energy I started with, mc², minus the energy it now has. In other words, the fraction of the energy that I’ve extracted by lowering it down to a radius r is mc² minus this, all divided by mc²: 1 - √(1 - 2GM/(c²r)).

This is the exactly correct answer for the fraction of the energy extracted. It doesn’t look exactly like this, because this is only correct in the Newtonian limit. We derived this using Newtonian physics. This is exactly correct, not just in the Newtonian limit, but all the way to where the effects of general relativity are important.

Now, if you’re a very, very long way away from the black hole—r is much, much bigger than 2GM/c²—then you can Taylor-expand this formula. The first order term is just the old Newtonian formula. It better be. It better be that the long-distance limit of general relativity recovers the Newtonian physics that we originally discovered. But as you get closer and closer to the black hole, this starts to deviate from the Newtonian answer, in a way that exactly is going to end up resolving our original thought experiment to do with lowering a brick down towards a black hole.

So how much, then, looking at this formula, have I extracted from the brick as I lower it down towards the black hole? If r equals infinity—if the brick’s still a long way from the black hole—then I’ve extracted 1-1=0. I haven’t extracted any energy. As I lower it closer and closer to the black hole, initially I just get the Newtonian formula. So in fact, these are pretty close to correct in general relativity as well, because the corrections are only going to start getting large when this term becomes order one, and it’s still very small here. So these are all essentially correct. But once I get closer and closer to the black hole, they stop being correct. What I see is that as r approaches the black hole event horizon, as this formula goes to zero, I have extracted exactly all of the energy from the brick.

So I start off with a brick a very, very long way from the black hole, attach it to a rope, slowly lower the brick down towards the event horizon. Of course, I can’t lower it past the event horizon, otherwise I’ll lose control of the brick. But I lower it right above the event horizon—the last possible place I can lower it to—and then just let go of it with zero velocity. The brick falls into the black hole and I have extracted the entire mc² that used to be in the brick in my pulley system out there. So it exactly resolves this question we had. Is it possible to extract more than mc² from the brick? No. Is it possible to extract the full mc² from the brick using a black hole? Yes, it is.

That’s actually pretty neat, and why people talk about using black holes as power plants. Most power plants today operate by burning chemical energy. That is not very efficient. You have to pay a factor of 10^-10, because chemical bonds are super weak compared to the rest masses of objects. You’re really only extracting a tiny fraction of the rest mass of the fuel that you’re considering. You can level up from there by going to nuclear energy, which instead of dealing with the feeble electromagnetic bonds between atoms, starts to concern itself with the nuclear forces between the protons and the neutrons within the nucleus. So you can go up from about 10^-10 to about 10^-3 for fission, or 10^-2 for fusion.

But that’s about as good as you can go, even with fission and fusion. Because even though you can extract energy from the strong nuclear force, neither fission nor fusion changes the total number of protons plus neutrons in your process. The bulk of the energy—99% of the energy—is stored not in the electromagnetic interaction, not in the strong interaction, but in the rest mass energy of the protons and neutrons, something that neither chemical reactions nor nuclear reactions can touch.

But gravity can touch them. If I start off with a mass object of m, I can extract, up to quantum corrections, essentially 100% of the rest mass energy that I’ve gone in with. It is the most efficient possible power plant, because by building an apparatus like this, in principle, I could extract 100% of the energy of whatever I started with.

Dwarkesh Patel

I intuitively get how energy equals mass. There’s these chemical bonds. Those get dissolved, they release energy. The thing weighs less if those bonds are released. I even get that if the bonds between the protons and the neutrons are broken, that releases energy and makes the thing have less mass. But if something with protons and neutrons is just slightly above the event horizon, is the interpretation that those protons and neutrons stop existing right at that point? What does it even mean for them to have 1% or 2% or 5% of their original mass?

Adam Brown

That’s a great question. It really becomes relevant once you turn on quantum mechanics, which is beyond the scope of today’s discussion. Classically, the black hole just sits there forever. So you can just say, “Well, what happened to the protons and neutrons?” You say, “Well, they now live inside the black hole.” The number of protons plus neutrons is still conserved out there in the universe. It’s just you need to assign what’s called a nucleon number to the black hole itself. That’s fine as far as it goes classically.

Quantum mechanically—way beyond the scope of today’s lecture—Hawking and Bekenstein discovered that black holes radiate away energy, and eventually the black hole will be gone. All of the energy, if you calculate it, ends up in gravitons and photons and perhaps some neutrinos. None of it, or almost none of it, ends up in protons and neutrons. So it is a very interesting fact, once you turn on quantum gravity, that black holes eat nucleon number. This thing that seems like it’s conserved, at least perturbatively, both by electromagnetism and by the nuclear forces, ends up being eaten by gravity.

People like to promote this—we’re talking about quantum gravity now—to a general principle that quantum gravity doesn’t respect any global symmetries. It doesn’t respect nucleon number symmetry. It doesn’t respect any of these symmetries. That’s a whole other can of worms that we can open some other day.

01:13:50 – What falling into a black hole would actually feel like

Dwarkesh Patel

Okay, Adam. I like to think on this podcast we impart not only theoretical but practical knowledge as well. So suppose one learns all these equations, but then finds themself in the unfortunate position of falling into a black hole. What would they see?

Adam Brown

Great question. There are actually two different perspectives you could take. One is the perspective of me watching you falling into the black hole. The other is the perspective of you falling into the black hole. Those two perspectives are consistent with each other but interestingly different, so maybe I should describe them both.

First, let’s ask the question: what do I see as you fall into the black hole? This is how hard you need to fire your rocket to not fall into the black hole but you’re not going to do this. You’re just going to sit here a long, long way away from the black hole, turn off your rocket, and accept what comes. What comes is you’ll slowly accelerate towards the black hole, at a rate first given by the Newtonian formula and then, when you get close to the black hole, start picking up general relativity corrections to the Newtonian inverse-square law.

What I will see as I watch you fall towards the black hole is that first you’ll go faster and faster and faster as you fall down the gravitational potential of the black hole. But then something strange will happen. You’ll stop going faster, and you’ll start going slower. The reason you’re going slower is that, as I watch you, you start to get gravitational time dilation as you fall down, and I start to see your clock running slow. The static formula doesn’t apply exactly since you’re moving, but the formula has the same effect, which is that as you get closer and closer to the black hole, your wristwatch starts running slower and slower and slower.

In fact, if you do the appropriate integral, I never see you cross the event horizon. I just see you getting closer and closer to the event horizon, but slowing and slowing as you approach it. As I watch you—I’m presumably using light to watch you—that light gets more and more redshifted. The wavelength gets longer and longer, and the longer the wavelength of light, the harder it is to even really see you. You start getting delocalized by the wavelength of the light, and eventually I just stop seeing you entirely. There’s a final photon that you emit, and then you just fade to black, fade through red to black. I never see you cross the event horizon.

This was noticed by people in the early days of general relativity and greatly confused them. They started to think that you would experience something funny yourself as you fell across the event horizon. That is not true. If I instead adopt your perspective, from your point of view, your clock isn’t running slow. It’s running at one second per second. If you look back at me, there’s some funny stuff going on to do with me running fast perhaps. But as far as you’re concerned, everything’s totally normal. You accelerate towards the black hole, getting faster and faster as you approach it. You just sail across the event horizon totally as normal.

The event horizon is not a particularly violent place for you. You can calculate the tidal forces as you approach and then cross the event horizon. They’re not particularly big, or rather, for large black holes, they’re not particularly big. For a solar mass black hole, they would be pretty big and would be pretty painful. You’d find that your feet are being attracted to the black hole much more vigorously than your head is, because they’re closer, and you end up getting stretched. But if I take a big enough black hole, you wouldn’t notice anything funny happening whatsoever. The bigger the black hole, the smaller the tidal effects. If I took a black hole the mass of the galaxy, you’d be basically fine as you cross the event horizon. If I took an even bigger black hole than that, you could live out your entire life having crossed the event horizon, before you hit the singularity, which is fatal.

When you cross the event horizon, you are doomed. You are doomed because once you cross the event horizon, you must proceed to the singularity. There’s no way you can fire a rocket to stop yourself hitting the singularity. You are doomed, but you are not dead. You are only for sure dead once you hit the singularity and get spaghettified, mangled by the tidal forces. But for a large enough black hole, you can be doomed and not even know it.

The event horizon is really a not locally measurable quantity. It is a teleological fact. It says that once you have crossed the event horizon, you must proceed to the singularity. But it can take a long time to get there for a large enough black hole. In principle, for a black hole that was many light centuries across, you could live out your entire life. You could have descendants, all of whom live inside the black hole. Only once you really approach the singularity do the tidal forces get strong and kill you.

01:18:51 – The three ways we know black holes are real

Dwarkesh Patel

As you were saying, GR explains or predicts a lot of phenomena. Some we think are correct, some we don’t know are correct. Why do we think black holes are correct but not wormholes?

Adam Brown

That’s a great question. People did not believe it to begin with. Schwarzschild wrote down his solution almost immediately after Einstein wrote his field equations. People thought that that equation was sick in some way, that it was a measure zero thing that would never happen. It was some mathematical monstrosity, but it was impossible to make black holes naturally in the real universe. They were wrong, because black holes do exist. We’re extremely confident now.

There were theoretical developments, and there was experimental evidence that black holes exist. The biggest theoretical development was Penrose, and then later Hawking and Penrose—for which he won the Nobel Prize—who showed theoretically that the formation of black holes is a generic feature of general relativity. It’s not just some sick thing that happens if you fine-tune the initial conditions. If you start off with generic initial conditions, the development of black holes is a generic feature. That was a huge development.

Then there was the experimental side. The pieces of experimental evidence we have for black holes are now huge. They did not exist in Einstein’s day, and for 50 years after Einstein, people were extremely confused about black holes and thought they didn’t exist. But there are numerous pieces of evidence.

I think the most visually appealing piece of evidence is just observing the center of our galaxy. If you look at the center of the galaxy—spoiler alert—there is a black hole there. We call it Sagittarius A*. It’s a huge black hole, weighing many millions of times the mass of the sun. You can’t see the black hole directly, because it’s black.

What you can see is the stars around it. If you watch these stars over the course of decades—and we now have a number of decades of observations of them—you will see the stars not moving along what we would call straight lines, but instead moving in nice little ellipses, or precessing ellipses. Those ellipses look like they are orbiting something. You cannot see the something, but you can see the stars that are orbiting it. You can calculate how big it is, how massive it is.

What you find is that it’s very massive indeed, and it’s also very small indeed. You know it’s small because the stars get super close to it but don’t seem to collide with it. So by tracing these orbits, you can tell that there is something super heavy, super dark, and super compact at the center of the galaxy. That is Sagittarius A*, the black hole at the center of our galaxy. That’s one compelling piece of evidence.

Another piece of compelling evidence: about a decade ago, we not only saw black holes, we felt them. LIGO is this huge laser interferometer that we built at a number of different sites, that is super attuned to vibrations in spacetime itself. There’s a famous event pretty much immediately after we turned it on in late 2015, where we felt spacetime shaking.

You knew it was spacetime shaking, not just the Earth shaking, because we had a bunch of these detectors—then two, now four—at different points on the Earth, and they all shook in exactly the same way. So it couldn’t just be explained by a truck passing one and not the other, or a seismic event on one and not the other. They all shook in exactly the same way, and we were able to back-calculate that the thing causing them to shake was the collision of two ginormous black holes. Two black holes, both of which weighed about 30 times as much as the sun, on the other side of the universe, about 1.6 billion light-years away. That collision happened about 1.6 billion years ago, and just happened to reach the Earth within weeks of us turning on the LIGO detectors. We’ve now felt thousands of such shakings corresponding to thousands of black hole mergers.

And then there’s more evidence. Later, we had what’s called the Event Horizon Telescope, which is a ginormous conglomeration of radio telescopes all over the Earth, that were able to look very closely at the black hole at the center of our galaxy, Sagittarius A*, and the even bigger black hole at the center of our neighboring galaxy, and see, very faintly, the radio emissions of matter falling into these black holes, which shines super brightly as it does so.

So we felt them, we’ve seen them, and we’ve seen their gravitational effects on orbiting stars. We’re extremely confident at this stage that black holes exist.

Dwarkesh Patel

It’s so beautiful that not only can a single mind come up with this theory, but the theory has so much reach, and that we can come up with the machinery to evaluate and perturb and understand its implications in so many different wild ways.

Adam Brown

It’s crazy the number of degrees of freedom, the number of orders of magnitude that it covers. You first start thinking about it by doing thought experiments to do with jumping up and down in elevators, and then it reaches out to describe the orbit of Mercury and detectable perturbations of orbital dynamics within the Solar System, and then the bending of light, and then it describes the rotation of the entire galaxy, and then it describes the expansion and potential fate of the entire universe. That’s many orders of magnitude indeed, and it’s pretty impressive that it was the work of almost a single mind. Frankly, our universe should be honored to be described by such a beautiful theory.

01:24:21 – The first time we saw gravity bend light

Dwarkesh Patel

Can you tell the story of how GR went from a theory that Einstein had to something that the world came to believe is true?

Adam Brown

That would be the bending of light. There were known anomalies with Newton’s physics beforehand, like we couldn’t get the orbit of Mercury exactly right. One of the very nice early tests of general relativity is that it did get the orbit of Mercury exactly right. So that was a pretty good confirmation. But at the same time, that’s not quite so satisfying because it was a number that’s already known, as opposed to one where you invent a theory and then it correctly predicts. It’s considered more impressive if you get the right answer without knowing what the right answer is in advance. So that would be the bending of light. Certainly, historically, that was the most influential.

According to general relativity, all energy gravitates and all energy is affected by gravity. So light, as it’s passing a massive object like the Sun, will get bent in the direction of the Sun. Actually, the same will happen in Newtonian physics. Suppose you have a particle going along. You know how much it gets bent as it passes the Sun, depending on its impact parameter, but also its velocity. The faster it’s going, the less it gets bent. So you just take that Newtonian formula, plug in velocity equal to speed of light, and see what answer you get. You can get a certain amount of bending through Newtonian physics. In general relativity, you can do the same calculation, and you actually get double the Newtonian answer.

The slightly strange history of it. Before he had finished writing down general relativity, Einstein had a prediction based on his understanding of the equivalence principle for what this answer should be. He wrote down the answer, and then in response to him and a number of other people being interested in this, people were sending out expeditions to go and try and measure it.

I think the very first thing that he did was he phoned up the observatory and said, “Can you look at distant stars behind the Sun and measure how light bends as it passes the Sun?” This was the true theorist move, because I think the director of the Mount Wilson Observatory said, “Absolutely not. We cannot do that. If you point a telescope at the Sun, you’ll go blind. If you point it just next to the Sun, you’ll just get washed out by the corona of the Sun and you won’t see anything.”

Except there’s one time when you won’t get washed out by the Sun, and that’s during a total solar eclipse, when the Moon blocks the Sun and you’re able to see stars very close to the Sun and measure the bending of the light behind them. So during the 1910s, there were a whole bunch of expeditions sent to measure the deflection of light. They’d park out in the path of totality and look through telescopes at the stars right next to the Sun and see if they moved in the sky, and if they moved, how much they moved.

I think the first one was in 1911. They went to Argentina for an eclipse, and everything’s set up. This is the problem with this thing. You get there, all the way to Argentina, a very long way in those days, and then it’s just washed out by the clouds. You don’t see anything, and it’s very frustrating. The next one was a German expedition sponsored by the arms manufacturer Krupp, who went to the Crimea and tried to measure it there. Just before the solar eclipse happens, World War I breaks out, and now Germany and Russia are at war. They’re all arrested and turned for the rest of the war, and so that also fails.

It actually turns out to be a good thing for Einstein that they all failed. It turned out that Einstein’s original equivalence principle argument—before he had full general relativity—was wrong and led him to predict that the bending of light in general relativity would be the same as it was in Newtonian physics. During the war, while everything is shut down and no one is thinking about eclipse expeditions, he corrects this mistake and comes up with a new prediction that actually it’ll be double the Newtonian prediction.

And then in 1919, Sir Arthur Eddington launches a British expedition to go and observe the eclipses all over the world and successfully comes back and declares that indeed it was the Einstein prediction. It was double the Newtonian prediction. That’s really what launches Einstein as a global celebrity. This British experiment confirming a German-origin theory was part of the post-war reconciliation, and Einstein had figured out everything. That is, I’d say, the point at which general relativity became the consensus view, and people were super convinced by this very impressive test.

Nowadays, we’ve done hugely more tests than that, very precise orbital dynamics. You can see it in the orbit of Mercury and indeed even in the other planets. You can just measure the redshifting of light as it goes, the gravitational effect on the propagation of light or the energy of light, all over the place. But historically, that was the most impressive confirmation of general relativity.

01:29:33 – How far can AI get without experimental evidence?

Dwarkesh Patel

One question you could ask is, we are maybe spending, as a society, billions, maybe tens of billions of dollars on building these huge physics experiments. If you look at maybe the most beautiful, the most important theory of physics ever conjured, it seems like a guy who’s just thinking in a cave. It seems like the empirical basis for that theory is maybe knowing that light has a speed, and maybe you need to measure G experimentally.

Adam Brown

Not really. G is a free parameter in general relativity. It’s not required. You’re right, the empirical basis is pretty thin. You don’t need much, and theoretical physicists are pretty cheap. There’s a great temptation. Why don’t we just spend it all on theoretical physicists and not build these vastly expensive experiments?

Dwarkesh Patel

Well, these AI companies are really increasing the demand curve for theoretical physicists.

Adam Brown

That’s right, not so cheap anymore. But how far can that get you? I would say that general relativity is perhaps one extreme of that. That is not how it usually works in the history of physics. This really is closer to some Ayn Rand hero just sitting alone, the product of a single mind. He got a lot of help in various ways, but it really was a singular vision that he pursued for years.

He wrote it down, and a lot of people were very impressed almost immediately. It did require launching a somewhat expensive eclipse expedition to go confirm it before he really achieved global celebrity and most people were sold on it. But it’s perhaps one of the most extreme examples of this, where somebody just sits down and thinks very hard and writes down a true theory.

In some sense, physics has been chasing that high ever since. People love that romantic vision of themselves just sitting down with very few empirical insights and thinking very, very hard and doing thought experiments. It’s typically not worked out quite as well for everybody else as it worked out for Einstein. In fact, it didn’t even work out that well for Einstein in the later part of his career.

How far could you get just by thinking? What do you need to do general relativity? You need the finiteness of the speed of light. You need to convince yourself not just that the speed of light is finite, but that there’s the symmetry that protects that, which Einstein came up with in special relativity. Then you probably want the equivalence principle: it’s an empirical fact that the inertial mass and the gravitational mass are the same for everything. But that’s pretty sparse. From just those two things…

There’s still a few options. But if you have lots and lots of large language models and there’s only a limited number of options, you can just explore the entire tree and say, “Okay, focus on this. The equivalence principle is something that’s super significant, and this other thing. Now abandon simultaneity and see how far that takes you.” There’s only a finite number of things to explore there.

I think they got very, very lucky with general relativity, that it’s quite so powerful under those circumstances. But if you just had lots and lots of Einsteins and you gave each of them various options, you could presumably see them in parallel.

Dwarkesh Patel

At the frontiers of physics today, in your experience, does it feel like if you just have millions of them running autonomously you could have enormous discoveries, or are we in a different era now, and really there’s limited usefulness of that parallelism?

Adam Brown

I think there is usefulness. I do think that different parts of science have different branching fractions, and how much experiment you need to cut off that branch.

I talked about chasing the high of Einstein. Arguably, string theory has really been going all in on that. Einstein’s theory is just general relativity. There’s also quantum mechanics. Trying to marry those two in a consistent way—which general relativity doesn’t do at all, there’s no quantum mechanics in general relativity—has motivated a lot of people. The problem is that in order to see that in experiments, if you just do the dimensional analysis, you need ginormous particle colliders, absolutely galactic-sized particle colliders. It’s just very hard to see any of that stuff.

But that doesn’t stop people. I mean, it stopped many people, but many people didn’t stop and keep trying to do it. So there you just have to hope that it works out sort of like it did with general relativity, where just by thinking very, very hard with minimal input from experiment, you can feel your way to the right answer.

For that to be true, the tool you have at your disposal is mathematical consistency and whether it reduces correctly in the known limits. So you better hope that there’s only one or a very small number of possible consistent theories if you were going to do that. If it turns out that there’s an unlimited number of consistent theories, you’re never going to feel your way to the correct answer, because they’re all consistent, and the only tool you have is consistency, and perhaps some notion of aesthetics. But if there’s only a few, then maybe you could do it all the way.

So string theory has kind of gone all in on that, I would say. Trying, with minimal experimental input, believing that there’s only one consistent theory of gravity and that just by doing sufficient consistency checks you can find it. For other examples it’s much harder. In condensed matter physics, often you simply need to go and do an experiment to find out which one is correct.

Dwarkesh Patel

Whenever our future AI civilization does come up with more and more unified theories of physics, or deeper theories that make better predictions, do you think that humans will be able to keep up? Once this step is taken, will we actually be in a position to understand what our AI civilization understands?

Adam Brown

I don’t know that we’re going to be able to keep up entirely, but I think we’ll keep up much better than pessimistic forecasts would suggest.

Let’s take mathematics as a simpler example than physics. Many mathematicians are worried that these LLMs are just going to turn into proof machines. Terry Tao has this phrase, “indigestion”, he uses, in which these LLMs will produce billion-line inscrutable Lean code that will serve as a certificate that a particular theorem is true without providing any insight as to why that might be true. Wouldn’t that be a depressing world, say the mathematicians.

I think that is a possible future, but I actually don’t find that to be a very likely future. Because as well as being superhuman provers, we also expect these large language models to be superhuman explainers. Maybe they’ll do the exact opposite of that. Maybe they’ll take proofs that are very hard to understand, and by doggedly trying and trying and trying, they will be able to come up with ways that are human comprehensible. They will take proofs that are difficult to understand and make them easy to understand.

I think the empirical evidence, it’s early days, but it’s pretty supportive of that more positive vision. There was an Erdős problem that was proved a few months ago now, and it wasn’t just an incomprehensible set of Lean. In fact, it wasn’t even proved in Lean at all. It was proved informally.

There was a follow-up paper by some human mathematicians that took these new, human interpretable ideas that the machine had come up with to prove this Erdős conjecture, and used them to prove new theorems. So that was the exact opposite of that case. It came up with a very human interpretable idea, and then humans were able to fully comprehend it, comprehend it so well they were able to deploy it in a new scenario.

We’ve seen that throughout. The unit distance conjecture, I think, is a good example here for a number of the themes you’ve been discussing. One is that it’s totally comprehensible, the disproof of the unit distance conjecture that it came up with.

Dwarkesh Patel

To you, perhaps.

Adam Brown

To some mathematicians. I’m not a mathematician. Perhaps the reason that humans haven’t disproved the unit distance conjecture is because they erroneously believe the conjecture to be true. The good thing about large language models is that they’re willing to push through that barrier and just waste their time, as a human would understand it, trying to disprove a presumed true conjecture and reach the other end. So that’s another aspect of large language models that makes me pretty optimistic. They just have extreme patience, even for doing things that perhaps look like a low probability of success.

Dwarkesh Patel

Adam, thanks so much for coming back on and doing this while you could be building superintelligence. You’re explaining 100-year-old physics, but it was very interesting.

Adam Brown

It’s a super fun subject. I’m super happy to share it.

First Place - Jassi Pannu

Jassi Pannu is an Assistant Professor at Johns Hopkins University, where she focuses on biosecurity and pandemic preparedness. She serves on the board of Blueprint Biosecurity.

Jassi answered the question about what the OpenAI Foundation should do. She persuasively argues that we can live in a post-disease world, and gave very concrete and well thought out ideas about how to dedicate 10s of billions of dollars to that project.

Second Place - Ege Erdil

Ege Erdil is a co-founder of Mechanize, a startup building environments and evals for frontier coding agents. He was previously a researcher at Epoch AI.

Ege answered the question about what countries outside the AI supply chain should do to avoid increase their odds of not being totally sidestepped by transformative growth.

He argues that these countries should concentrate on enacting the kinds of policies that already work well in increasing growth and improving productivity. These strategies (strong property rights, low capital taxes, and an open regulatory regime) will be even more important in a world where enacting them can drive a much higher growth differential than is possible today.

What I love about Ege’s essay is that, in one sense, he’s giving very common-sense advice (as opposed to much more galaxy-brain schemes some other applicants proposed - one application suggested middle countries blackmail China and American by threatening to nuke their fabs and datacenters). But it’s actually this much more grounded and timeless advice that felt the most contrarian. And it’s also more likely to work.

Third Place - Michael Li

Michael Li is a Master of Public Policy candidate at Harvard Kennedy School. He writes Ceteris Paribus — a blog at the intersection of emerging tech, econ and policy.

Michael wrote about how the labs will actually make money. His was selected for the unique analogy he drew between AI labs and Hong Kong’s Mass Transit Railway business model - even if your main product consumes crazy CapEx and doesn’t directly earn it back, maybe you can make up for it by buying out all the complementary assets. In the case of Hong Kong MTR, that would be the adjacent properties - I don’t know what it looks like for the AI labs, but it was a interesting analogy to think about.

Essay #1 - Jassi Pannu on how she would run the OpenAI Foundation

I’d run the Foundation as a state-scale operation to end airborne transmission.

AI’s largest welfare upsides (curing diseases) and deadliest tail risks (engineered pandemics) both run through biology. By radically suppressing airborne pathogen transmission, we’d unlock >$1T in annual global GDP (through ending seasonal flu and the like, chronic diseases increasingly linked to viral infections, productivity losses, healthcare costs, etc.) and would take the possibility of catastrophic pandemics entirely off the table.

The dual-payoff principle: Most “make AI go well” interventions are insurance against bad outcomes, especially tail risks. My meta-level argument is that the best way of converting money into impact is to identify interventions that have the property of paying off big in both worlds: by producing step-changes in welfare in the everyday world as well as significantly reducing tail-risks in the emergency world. The bio resilience interventions I describe below are the best example of this.

AI for biology is on the critical path to cures, but destabilizing capabilities will arise early

Using AI to automate and scale every step in the biological research process, including managing the process itself (something I’ll call autonomous biological discovery), will bring humanity closer to a post-disease world. Over 4 billion years, life has been doing a random walk on an astronomically tiny subset of viable, connected, fitness-positive paths. Multi-component AI feedback loops (that include bio foundation models and systematic wet-lab experimentation at scale) for autonomous discovery will enable us to explore much more of possible biological design space. While we’re most interested in predicting and designing multicellular systems, it’s likely that the destabilizing capability of manipulating simpler pathogens will emerge first. The challenge this poses is that AI-enabled offense (seeding an outbreak) will be much easier than defense, which will remain constrained by physical-world deployment; I argue this advantages pre-positioned defensive technologies already embedded in our infrastructure.

There’s a clear path to ending airborne transmission, using physical infrastructure.

Regardless of what you think about the above, though, ending airborne transmission can be more than justified based on everyday benefits. Respiratory infections cause acute illness and productivity losses, but are increasingly linked to dementia, cardiovascular disease, and more; even “normal” childhood respiratory infections are being linked to long-term neurodevelopmental outcomes.

After evaluating many approaches, I’d argue ending airborne transmission is more achievable than most realize, through a specific, under-appreciated approach. I’m currently sitting in a building that provides me with pathogen-free water, keeps my food cold and pathogen-free, helps me heat my food to eliminate pathogens, and pipes away sewage. We have already embedded technologies all around us that enable a post-cholera, post-typhoid, post-dysentery world.

There is passive, pathogen-agnostic (works against any pathogen), physical infrastructure tech capable of making our buildings entirely free of respiratory pathogens, such as lamps that emit wavelengths safe for humans but are deadly for bugs (called far-UVC). Researchers have suspected these would work at scale for decades; the reasons we haven’t deployed them are primarily non-technological. Consider this analogical case.

We now live in a post-smallpox world. This is one of humanity’s greatest accomplishments. How long did it take for us to do this? Jenner demonstrated vaccination could prevent smallpox in 1796. 171 years later, in 1967, D.A. Henderson launched the campaign that would successfully eradicate smallpox. In that period of time, humanity discovered electromagnetism, thermodynamics, general relativity, and we were 2 years from landing on the moon. Eradication was accomplished within a mere 10 years (with limited tech advances). Delays in smallpox eradication, clean water, and pasteurized milk were not due to lack of tech advancement; they were primarily market and coordination failures exacerbated by lack of political will. This is why this problem is so philanthropy-shaped.

4 steps to ending airborne transmission

Total: ~$40-$60B over 10 years for physical infrastructure to end airborne transmission; the rest of OAIF’s stake remaining for other interventions meeting the dual-payoff principle. By year 10, every primary school and major transport hub in OECD countries operates with passive pathogen-reduction infrastructure as default. Seasonal flu mortality is reduced by 60%. The probability of a respiratory pathogen achieving pandemic-scale spread is reduced by an order of magnitude.

1. Push-funding to resolve the target product profile ($5B, Years 1-3)→ Hire Jacob Swett, director of Blueprint Biosecurity, to lead a DARPA-style program office focused on: a) pathogen inactivation data from human aerosols, b) computational modeling for deployment, c) safety studies beyond conventional UV effects, d) gold standard cluster-randomized trials powered to detect plausible effect sizes. By the end of year 3, deliver a validated TPP for far-UVC lamps and real-world efficacy data demonstrating >30% transmission reduction.

2. AMCs to guarantee demand and pull private capital ($15B, Years 1-5)→ Create laddered purchase commitments for (a) 100K far-UVC fixtures that meet an interim TPP, (b) 1M for fixtures meeting the full TPP including safety and efficacy validation, (c) 10M commitment to retrofit specific buildings (see step 3). Modeled on Kremer’s pneumococcal vaccine AMC and Ransohoff/Frontier’s carbon removal commitments. By year 5, expectation is $30-50B in private capital mobilized, and supply chain capacity built to retrofit ~10% of global building stock.

3. Large-scale deployments to generate evidence ($15-25B, Years 2-7)→ Years 2-4: Deploy in all hospitals and long-term care facilities in 50 largest metro areas globally. Years 3-6: Primary and secondary schools in the same metro areas. Years 2-7: Major airports and high-density workplaces. By year 7, substantial real-world evidence base.

4. Political infrastructure and state handoff ($3-5B, Years 1-10)→ Smallpox eradication was a genuinely contingent historical event predicated on political will. Fund a memetic campaign à la the Rockefeller Foundation’s IHD yellow fever playbook which would transform respiratory transmission from “normal” to undesirable/unnecessary, and the training of a cadre of thousands that move into governments to build the political constituency and institutional infrastructure needed for global deployment and standards-setting. When pilot deployments demonstrate ≥40% reduction across 3 OECD countries, the Foundation transitions to catalyst rather than principal funder role, activating state procurement at orders-of-magnitude scale.

Essay #2 - Ege Erdil on what countries outside the AI production chain should do

I think a good baseline which very few countries, in the AI supply chain or not, will beat is “do nothing and ignore populist pressures to take radical actions”.

This is because people are naturally technologically conservative, and they hate economic disruption that causes job losses. Full automation of human labor by AI would bring about rapid technological and economic progress that would result in all humans losing their jobs. So the default expectation should be that policymaking in an era of AI automation will be profoundly irrational and counterproductive.

Resisting this political pressure will be hard enough for even the most functional governments. Expecting more from governments with poor track records such as India and Nigeria is unreasonable.

What does good policy in the AI era look like?

Having given this basic but uninspiring answer, I’ll now flesh out what I actually think will be important for good policymaking in the era of AI automation, though in practice it’s unlikely to have much relevance for how policy decisions will be taken.

Today, the economic output of a country depends on its endowment of natural resources, on how much physical and human capital it has, and how efficiently it’s able to make use of these resources. The major shift with AI will be that human capital will drop out of this equation. If a country wants to do well in a world after full automation of labor; they need more natural resources, more capital, or more ability to make use of these inputs effectively, i.e. more total factor productivity.

While the capital elasticity of output will dominate any other factor of production, how much capital a country ends up with is itself endogenous. Capital moves across borders more easily than labor, and it will likely flow to the places where factors complementary to it – total factor productivity and natural resources – are abundant.

So I think the pillars of good policy in the AI era involve going in the following directions, to whatever extent possible:

• Get out of the way of AI automation. Repeal or revise occupational licensing laws, liability laws, data protection laws, and intellectual property. Abolish price and wage controls. Dismantle cartels and unions of human workers who will try to protect themselves from AI competition. Apply safety and security standards consistently between humans and AIs, instead of discriminating in favor of humans. Make it much easier to start new businesses, as AIs will need organizational structures just as much as humans do in order to bring out their productive potential.

• Provide political and legal stability. Investment will be anemic under instability, and if a country can stand out as an island of stability in what will likely be tumultuous decades for the world, it will attract enormous amounts of investment. The lowest bar is avoiding civil wars, revolutions or coups (not trivial for Nigeria to clear, since there was a serious military conspiracy to overthrow Tinubu as recently as September 2025); but this is not enough by itself. People must be confident that their investments will not be expropriated, and that the core operation of their businesses won’t suddenly be declared illegal.

• Increase capital formation in your country. Reduce or eliminate taxes on capital gains and corporate income, don’t let important projects be held up by permitting, don’t hamper construction by requirements of “having to pay prevailing wages”. Remove exchange, interest rate, and capital controls.

• If industries necessary for AI deployment are broken, urgently fix them. For example, Nigeria’s grid sucks and needs to be fixed before anything else can happen, and this is downstream of the entire economy’s price system being messed up by government restrictions. If this isn’t fixed, the country will never be able to accumulate the kind of capital it needs to be productive in an era of AI automation.

These recommendations feel extreme because humans instinctively like policies that favor labor over capital. This already causes problems in our world where capital’s output share is only around 30%, but it will likely become a critical obstacle in a world where labor has been made obsolete and capital’s output share is much higher.

If a country that’s currently in bad shape succeeds in doing these, they would attract a ridiculous amount of outside investment and be transformed in the decades in which AI automates the world economy. The fact that they had no previous AI labs or fab companies with trillion dollar valuations would be irrelevant, because those very companies would at that moment be in the process of automating their moat – their human capital and organizational knowledge – away.

Will this happen?

Probably not. As I’ve said, the relevant countries are far too dysfunctional for these radical reforms to be adopted, and the countries currently in the AI supply chain (e.g. the US and China) already perform well relative to other countries on the metrics I’ve listed.

However, there’s still some hope for countries outside the AI supply chain: the incumbents can screw up. This is not that implausible, because policy quality in the industrial era will not be that strongly correlated with policy quality in the AI era, just like how the best foraging economies in the world weren’t the best agrarian ones and the best agrarian economies weren’t the best industrial ones.

Concretely, the incumbents can outlaw AI automation in key industries, slow down datacenter buildouts, expropriate capital holders, impose strict safety standards for rolling out AI agents in critical industries like medicine and law, et cetera. There’s virtually no limit to the mistakes that they can make in the coming period of acute irrationality. If a mediocre country simply holds on and doesn’t make any big blunders, they will probably come out the other side of this in a much better relative position than they had been in at the start.

Essay #3 - Michael Li on how the labs will make money

A subway company solved AI’s business model

There is an industry with the following economics: billions in upfront capital before earning a dollar. Core service priced near marginal cost. Enormous value created for users, almost none captured by the builder. Relentless pressure to keep investing in the next generation of infrastructure. I’m not talking about AI labs. I’m talking about Hong Kong’s Mass Transit Railway.

Many have reached for the railroads analogy when discussing the business of AI. Most conclude that the lesson is commercial viability requires state subsidy for a general purpose technology with public good properties.

I want to challenge that, because Hong Kong’s MTR actually solved the problem. It’s one of the only mass transit systems in the world that is commercially self-sustaining, publicly listed, paying dividends with no government operating subsidy.

The financials are structurally identical

MTR’s core rail service has never funded its own expansion. In 2018, its best pre-covid year, transport operations earned HK$2.0 billion in EBIT. Estimated capital expenditure for 2024–2026 is HK$87.9 billion, nearly all rail-related. Three years of peak rail earnings would cover 8% of that. The rail has never paid for itself through fares. It was never designed to.

MTR fares are kept affordable through a government fare adjustment mechanism. You can’t price transit to recover full construction costs because it would be unaffordable and defeat the purpose. Each rail line can maybe cover its operating costs, but fare revenue never stretches to fund the next line. AI API pricing faces the same constraint from the other direction. Distillation and open source alternatives deflate API prices roughly 10x per year, and any lab that prices above marginal cost loses volume to rivals. Each model can be operationally profitable on inference, but the margin never stretches to fund the next training run.

The standard global solution is subsidy. London Underground requires billions in TfL grants. China’s national HSR carries a trillion dollars in debt with 94% of routes unprofitable. AI is on the same trajectory: CHIPS Act, Stargate, sovereign wealth fund investments, Pentagon contracts. The default endpoint is subsidy-dependent quasi-public infrastructure.

MTR found another way.

Rail plus property

When MTR was built in 1979, its designers understood that fares alone would never recover construction costs. So they structured the corporation around a different premise: the rail line would make surrounding land valuable. So own the land.

MTR develops residential towers, offices and shopping malls above and adjacent to its stations, capturing the value appreciation that its own infrastructure creates. Property profits cross-subsidize rail operations and fund the next line. Today MTR owns 13 shopping malls, manages 47 developments above its stations, and property generates the majority of actual profit. Don’t try to capture value through the rail service itself. Own the assets that appreciate because of the rail service.

The AI parallel

“When do labs make money?” has the same structure as “when does rail pay for itself through fares?” It doesn’t, and that’s the wrong question.

A biotech startup uses a frontier model to screen drug compounds, shaving two years off a clinical trial. A logistics firm uses it to optimize routing, saving $40 million in fuel costs. A solo developer ships in a weekend what used to take a five person team three months. In each case the model provider captures a fraction of a percent through API fees. The provider can’t charge more, because four other labs and a dozen open source alternatives offer comparable capability. The surplus flows to the users and the broader economy. This is what general purpose technologies do. The steam engine, electricity and TCP/IP generated zero revenue for their creators.

MTR’s lesson: stop trying to make fares cover construction. Find the property.

Four candidates, ranked by defensibility

Government-granted deployment rights come first. A government grants a lab exclusive deployment access to national health records, tax systems or defense logistics. The lab accumulates domain data, integration depth and regulatory clearance that takes years to replicate. This is MTR’s own mechanism: development rights granted by the state, justified by natural monopoly properties.

Accumulated RL reward data is second. Billions of interaction signals that train the next model generation. Unlike weights (which depreciate via distillation), RL data is practically non-replicable and compounds across generations. It doesn’t convert to revenue directly, but it’s a land bank. Appreciating, undeveloped.

Forward-deployed integration is third. Instead of selling model access to a consulting firm that captures the productivity surplus, own the service delivery end to end. The way Palantir embeds engineers inside government agencies rather than licensing software. The lab doesn’t charge the law firm an API fee. The lab becomes the legal research service, priced against the outcome it delivers rather than the tokens it serves. Switching costs compound with accumulated domain data and institutional knowledge. This is MTR’s shopping mall: capture the foot traffic the rail creates rather than charging passengers more for the ride.

Data trusteeship over national datasets is fourth. Governments sit on enormous under-leveraged datasets (patient records, tax filings). A frontier lab designated as trustee gets exclusive access to train on and build products against this data. The scarcity is genuine, justified by the same logic as MTR’s land. You don’t want five competing labs accessing fifty million patient records any more than you want five developers building on the same plot. But this creates a public-private data monopoly and would require careful governance: clear boundaries on usage, benefits flowing back to the public, independent monitoring and real sanctions for misuse.

The reframe

The labs that survive won’t be the ones that make the API profitable. They’ll be the ones that identify their property above the station and build toward it now. The API is the rail. It will never be profitable enough. The money is in what appreciates around it.

The policy question follows: instead of subsidizing training runs, governments should design institutional mechanisms (deployment rights frameworks, data trusteeship structures, productivity measurement standards) that let labs capture the surplus their infrastructure creates.

There’s a final irony. The AI policy conversation is dominated by the US-China frame. American free market labs versus Chinese state-funded champions. The most relevant institutional model may be neither. It may be Hong Kong’s: a 45 year old public-private hybrid, commercially operated, self-financing through institutional design rather than ideology. The model that makes frontier AI sustainable might already exist. It just runs trains.

AI has been making much faster progress in math than in other fields. As a result, mathematics is showing us, very concretely, what AI progress in other fields will look like. Even within mathematics, there’s a jagged landscape. What does it look like?

What is the nature of the most important conceptual breakthroughs in the history of mathematics, and how different are they from what AIs are currently able to do?

Does AI (on net) increase or decrease human understanding of the field?

How big is the overhang from having AIs systematically try to connect ideas already in the literature?

And what advice does Grant have for aspiring mathematicians, coders, and other students who are passionate about fields that are being most transformed upon by AI?

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Gemini 3.5 Live Translate is what I wished I’d had on my last trip to China. It detects more than 70 languages and translates them in near real-time… and it preserves your original pacing and intonation. If you’re building an app that needs live translation, you should check out Gemini 3.5 Live Translate. Get started at ai.studio/live

• Cursor’s harness lets me use models for a huge range of tasks at the podcast. For example, Cursor cuts out the ads from each episode I produce so I can post them on Bilibili. It also helps me prep for interviews — I have a repo full of books and papers that Cursor sorts through to find the exact right file for any given question. Try Cursor yourself at cursor.com/dwarkesh

• Jane Street sponsors 3Blue1Brown, so Grant has gotten to spend a lot of time with various Jane Streeters. He actually just recorded an interview with a few of them, so when we sat down for this episode, he told me about some of the things he learned, like how Jane Street keeps their role definitions fuzzy to make sure their people keep learning and growing. Go check out Grant’s full interview at 3b1b.co/janestreet

Timestamps

(00:00:00) – AI is discovering new proofs. Is that AGI?

(00:11:32) – The verification loop on conceptual breakthroughs can be a century long

(00:26:12) – Will we understand an AI proof of the Riemann hypothesis?

(00:38:08) – Can AI find the hidden bridges between fields?

(00:53:48) – Why real-world tasks don’t fit into RL environments

(01:07:07) – Good writing requires theory of mind that AI still lacks

(01:16:02) – Why learning will still depend on human curation

Transcript

00:00:00 – AI is discovering new proofs. Is that AGI?

Dwarkesh Patel

Today, I’m chatting with Grant Sanderson, who runs 3Blue1Brown and is now working on a new project documenting the progress AI is making in math. I wanted to talk to you about this because AI has been making the fastest progress in mathematics out of any other field. Whatever is happening here, and whatever way we’re seeing AI progress happen or not happen, will tell us about what will happen to the rest of the world as AI gets better and better.

I wanted to start with this question I asked you when I first interviewed you three years ago. I asked you, once we have AIs that can get gold in the International Math Olympiad, wouldn’t that just be AGI? Wouldn’t this just be able to do anything any human can do, given how hard these problems are?

You had an answer, which in retrospect turned out to be very wise and correct. You said it’ll be another benchmark, like all these other benchmarks that AI are passing. Obviously, AI has gotten better in a general way since then, but there won’t be some “aha” moment when this happens.

First, I’d be curious to get your heuristics on why that turned out to be true. Second, I’m curious how long you think this narrowness can continue to be true. By the point that AI has solved a Millennium Prize problem, do you think it’s still possible that there are lots of tasks humans are doing that AI still can’t automate in the economy?

Grant Sanderson

It’s an interesting question because it’s hard to answer without knowing what the solution looks like ahead of time. If we take the IMO, the spirit of your question three years ago was in looking at how some of the solutions to these problems really seem to require creativity. The designers of these problems try to come up with things that you can’t train for as easily.

The dirty secret with the IMO is that you really can train for a lot of them. With the whole AI and math project underway, as you point out, one of the reasons it’s interesting at all is that there’s a spiky frontier to AI, and math is just right there in one of the spikes.

But there’s a fractal nature to that spikiness, because when you zoom into the specific progress within math, you have some things that are a lot easier than others. If we just think about IMO, which is old news at this point. It’s been two years since they’re really doing quite well. They would have gotten a gold in 2024 if not for the following reason. They’re very good. They just cold-solved geometry basically. The IMO has these four categories of problems: geometry, number theory, algebra, and combinatorics. Geometry, it just solves it in nineteen seconds since 2024 because it’s a brute force solver.

The dirty secret is that for students, there’s also a brute force way you can go at it. Combinatorics is the wild card: much more playful, puzzly-seeming problems. There were two combinatorics problems on that year’s test, and there’s not always. There are four categories and six different problems, so it’s a toss-up which one is going to have two questions. Had it been more geometry questions, they would have gotten a gold that year.

But it struggles on those combinatorics ones. Someone who’s trying to keep that torch of the last holdout of math for humanity might say those are the ones that require more creativity. Even then, the spirit of your question—if they’re solving a Millennium Prize problem, does that also service a lot of white-collar work?—suggests that whatever the rate limiter is between where we are now and that is the same as the rate limiter for making things better at white-collar work.

We could paint a couple of different ways. If we focus on the Riemann hypothesis, what would it look like to solve that? These things are extremely good at a specific domain of knowledge, knowing it very deeply, and then knowing another domain, and another. You’ve pointed this out. It’s bizarre to have something with this superhuman breadth that knows all the fields so well, and yet isn’t finding those lightning bolts that connect them.

I think we’re starting to see sparks of it actually finding connections between the things it’s an expert at. I’m sure we’ll talk about it. If the nature of the solution to the Riemann hypothesis was something like that, that feels pretty distinct to me from what’s necessary to get good at white-collar work.

And there’s a reason to believe that might be the nature of the solution. I don’t know if you know the story of Hugh Montgomery and Freeman Dyson at the IAS. This is a side tangent, but it’s a fun story. I don’t know if it was over lunch or something like that, but you have this number theorist who is just trying to understand the statistical correlation between pairs of zeros of the Riemann zeta function.

The Riemann hypothesis is all about whether all these zeros sit on a straight line. He finds this quantitative question you could ask, and he writes down a formula. It looks like one over sine squared or something like that. Freeman Dyson, a physicist, is like, “I know that expression. That expression comes up in studying the eigenvalues for random Hermitian matrices,” which was something that comes up in studying the energy levels of a nucleus.

The idea that the statistics of those two seemingly different things were the same prompted an exploration of whether there are aspects of random matrix theory that might be relevant to the Riemann zeta function. I think it’s a little bit of an open question whether there is fruit to be had there. But that bridging together of two different fields—if it turned out that the solution to the Riemann hypothesis was exploring an idea like that even further—has the character of how you expect LLMs to be good at math. They’re experts at quantum physics. They’re experts at analytic number theory. They should be able to see that similarity in a way that doesn’t require Montgomery and Dyson to be having lunch and happen to talk about it. That’s totally different from white-collar work. To the extent that you have a hard time using an AI as an editor, it’s not because they know everything and you just need them to find that lightning bolt in between.

A different possibility would be… What’s the right analogy? Maybe if we think of Fermat’s Last Theorem, between the moment of Fermat phrasing the question and what the solution itself looks like, where the solution ultimately involves such heavy machinery in math. The beauty of that problem is you can phrase it so simply. You ask about xⁿ + yⁿ = zⁿ. Do you have integer solutions for this when n is bigger than three?

It’s something you might expect there to be an elementary number theory approach to, but as far as we can tell, there’s just not. Whereas the actual solution, maybe there is something simpler, but this might be what it has to be. There’s such a complicated set of ideas that build on centuries of work centered around elliptic curves. Then there’s this other mountain of ideas centered around these things called modular forms. Both of those mountains have to be built before you can ask the right question that connects them.

If the solution to the Riemann hypothesis involved building a new mountain, that’s a kind of skill—the ability to come up with the right new ideas—that feels sufficiently different from the character of how they’re intelligent right now. It’s not like that’s what you need from your hired video editor per se. But if it’s capable of building mountains that are the correct new theory crystallizing how we should be thinking about a subject, that’s just such a level of intelligence that it would be surprising if it didn’t permeate into other aspects of the economy besides just the mountain-building for math itself.

Dwarkesh Patel

Or at the very least, even if it couldn’t literally do every single thing white-collar humans can do, it would just have transformative effects in the way that getting gold in the IMO did not have transformative effects on the world.

First of all, I do want to point out that I’m totally moving the goalpost here. When I interviewed Dario two or three years ago, I asked this question about why they haven’t been able to use their vast knowledge to connect ideas together and come up with a new discovery that way. That seems like the kind of thing where even a moderately intelligent person, if they knew this much information, would be able to come up with a medical diagnosis from the fact that this drug causes migraines, and this other thing does this, and maybe it’s the same drug that can cure both things.

From an outsider’s perspective, mathematics seems clearly like a field where finding the counterexample to the unit distance problem conjecture was an example of this kind of thing. So it’s total goalpost moving. But then we can ask, what is the next benchmark? Now that AIs can do this thing we should have thought they’d be able to do, what is the next thing that would be quite impressive?

There are a couple of candidate ideas here. One could be coming up with interesting problems in the first place, and the other is coming up with new kinds of objects or conceptualizations that create or unify fields. On the first one, right now we have these Millennium Prize problems because mathematicians have noted them. Riemann came up with this idea of the Riemann zeta function because he thought the zeros of this function would have some connection to the density of prime numbers.

Figuring out why we think this is an interesting thing to study in the first place, why we are building this object and trying to answer questions about it—and answer this particular question about it—seems like the kind of thing that would be the next benchmark.

Grant Sanderson

You highlight two pretty good examples there. For anyone curious about the unit distance conjecture, there’s this really nice video by a math channel called Polylog where they talk about it.

All of these discussions cause people to reflect on the process of doing math. They’re like, “Oh, this thing can do this impressive stuff. What does that mean for us?” One of the people in that video highlights this quote: “good mathematicians prove theorems, great mathematicians come up with conjectures, and the greatest mathematicians come up with definitions.” That’s more or less exactly your framing here. We need the conjecture generator and then the definition generator. That’s the premium-tier mathematician.

I don’t understand how exactly you’d make that a benchmark. Usually, when I think of the word benchmark, I’m thinking of something that is a goalpost. The ball is through the goal or it’s not. You can clearly say, “Yes, this is done.” Partly that’s to be able to do things like RLVR, but also partly just to know that you haven’t moved the goalpost in answering. OpenAI can have their headline on disproving the unit distance conjecture because it’s a clear, distinct thing. It did it. Whereas imagine trying to have a headline on GPT-5.4 coming up with a really good conjecture. “We promise, everyone thinks it’s a good conjecture.” It just doesn’t land the same way.

But maybe that doesn’t negate the fact that it’s the right thing to be thinking about. I would be surprised if it ever took the form of looking like a benchmark, where we have a score saying it’s passed because we can quantify how good a conjecture is. The nature of what it would take is probably that you’d feel a tone shift in conversations with mathematicians about the way it’s useful to work with.

This series you referenced, which is not at all produced yet and probably won’t be for a couple of months, takes the form of us interviewing a lot of mathematicians. What’s interesting is that we started doing this over a year ago, and it’s fun to see a little bit of a tone shift in the way they talk about AI between mid-2025 and where we are now in 2026. In the real world, that’s a very short amount of time. In the AI world, that’s eons. We’re able to see this tone shift over those eons.

I think the way you’d measure conjecture-generating ability is going to be more subjective, based on that tone shift. It will be mathematicians saying they’re not just using it to solve their problems, but that as they step back and decide what their research field should even be, a conversation with such-and-such model was genuinely helpful for that. I don’t think it’s likely you’d see it in the form of a headline saying this was yet another benchmark knocked down.

00:11:32 – The verification loop on conceptual breakthroughs can be a century long

Dwarkesh Patel

It’s very interesting. The kinds of things you can’t make benchmarks for are also the kinds of things, at least in the current paradigm, you can’t easily train for. There’s really no fundamental difference between a benchmark and a training environment.

It’s very easy to come up with some dichotomy of, “here’s a deep reason why AI can’t do a certain thing”, and then it turns out you’re just thinking about it the wrong way, and actually it can do it pretty soon thereafter. But I’m going to come up with—

Grant Sanderson

You’re going to come up with a couple anyway.

Dwarkesh Patel

It’ll probably turn out that there are ways we can train AIs to do these kinds of things in the relatively near term. But it seems like it would have to be different from current RLVR training.

The thing I’m curious about—and the thing that seems to me to drive a lot of the big progress in mathematics and in science generally—is coming up with a new way to think about a problem or a new way to understand the world that unifies different fields, spawns entire new fields, and solves problems we weren’t even trying to solve in the first place. The reason Einstein was thinking about GR is not because he wanted to explain why light bends or why black holes exist. These are phenomena he didn’t even need explained in the first place.

In mathematics, as a total outsider who doesn’t even know what he’s talking about here, it seems like there are often ways to prove a specific problem that can motivate a new conceptualization—one which results in a whole new field, a whole new way of thinking, which is immensely productive—and ways which don’t. I’d be curious to hear you talk about Galois coming up with group theory, distinguishing his solution to the quintic having no formula for the roots, and Abel coming up with a different proof a few years earlier that didn’t come up with group theory.

If you wanted to do a verification loop on whether group theory is an interesting concept—was something useful done here, or why is this proof better?—potentially that verification loop is a hundred years long. It involves cryptography coming around and physics making progress, and the ideas in group theory being relevant to understanding symmetries in physics. There’s a hundred-year verification loop on why this is a productive concept in the first place.

Grant Sanderson

You struck a nerve, because I had this project about Galois I was going to do in 2022 that I put on the shelf, but I spent a year of my life thinking a lot about what he did. There’s a risk of me accidentally talking too long on the specifics, which you can hold me back on.

It’s a perfect example for your case, because describing why it was a valuable insight does not come from immediate utility. Certainly, if you’re thinking about RLVR environments, this is going to be really hard to do. But it’s interesting to note that even with human verifiers at the time, it took a really long time to recognize it as being useful.

With Einstein and GR, people could feel this was a good theory right away. What makes Galois theory such an interesting example is that you literally have this hundred-year segment of an idea that flows through many different people’s heads before it settles into something the math community agrees is good. To back up a little bit… Do you want the background on the problem at all? We all learn about the quadratic formula in school.

Dwarkesh Patel

I thought you were going to say we all learn about group theory in school, but I missed that class.

Grant Sanderson

We all learn about group theory… No, the quadratic formula. This was known. In some sense, the Greeks could solve quadratics, but they didn’t really write things in algebra. It’s really the Arabs who wrote down that formula.

There’s this delightful story about dueling Italian mathematicians—not real duels, just intellectual challenges—who secretly found a formula for the cubic, and then very shortly thereafter found a formula for degree-four polynomials. So a natural open question for mathematicians is, can you find a formula that solves degree-five equations?

The degree-four formula is a monster. It would be wild to write it down. You usually don’t write it down in full. You break it up as a procedural thing. You might believe these things have this exponentially increasing complexity. So for many hundreds of years, nobody was really answering that question.

Usually, we say Abel was the first to prove it. He was this young, precocious Norwegian mathematician. He showed it’s simply impossible. It’s not that you can find a quintic formula. He thought he found one initially, but he showed it’s impossible.

I think the real credit though, you have to back up a bit and talk about Lagrange. He found the right kind of question to ask about this. I’ll give it at a very high level. He was studying the question and recognized that being able to solve these polynomials is very related to understanding the way certain algebraic expressions are symmetric.

If I write down a + b + c + d, just adding four variables, and I permute those, it doesn’t change the value of the expression. Whereas if I write a + b * c + d, some of the permutations don’t change it, but some of them do. He had this really nice insight about how if you can find expressions that have four free variables, but all the permutations take on three distinct values, that has this unexpected relationship with being able to reduce degree four into degree three.

He started approaching the question of whether we can find a quintic polynomial by wondering if he could extend that method. To extend that method, you would have to have an expression that has five free variables such that as you permute them over all the five factorial permutations, it takes on only four values or fewer. You could put that in a puzzle book. You could put that in a brain teaser that a twelve-year-old could engage with. It’s not too hard to find yourself feeling like that’s an impossible task.

Lagrange is sitting there saying, “Here is a strategy to solve this problem of finding a quintic polynomial. It seems like it might be impossible, at least from this strategy.” But that was the first time in history that people had the instinct that some kind of question about symmetry was the right way to study these polynomials. In his mind, it was just a way. It had yet to be discovered that there was actually a tighter connection. Also maybe rather than searching for the formula, we should be asking the opposite question: can you prove that it’s impossible? He sort of planted that seed.

Around fifty years later, Abel definitely read Lagrange and was influenced by it. We know that Galois loved Lagrange when he was falling in love with math. It’s very hard to imagine that these two young geniuses coming up with pretty similar insights around that problem wasn’t born from Lagrange.

But to your question on whether you are able to verify that this was a good idea, there wasn’t any result that Lagrange came to. He didn’t solve the problem, so it wasn’t a case of knowing it was the right question to ask based on a solution. He just asked it. There’s some intrinsically interesting thing about it. It also wasn’t very important for math at the time. Most people were more interested in the applications to physics. This was almost a side, recreational, hobbyist-type thing.

Abel started working on quintic stuff, but then he was advised to spend more of his efforts studying elliptic functions, so more of his work was on that before he died young. He died at twenty-six from tuberculosis. And then Galois pushed both of those ideas in the right direction, where he really understood the nature of abstraction. He had this really nice piece that he wrote while he was in prison. We could talk all about his life story. It’s pretty wild. But he’s this teenager, he’s in prison, and he had tried to submit his math papers and they had been rejected.

So again, thinking about verifiable reward, the verifier function that is the academy at that time is rejecting what he wrote. Frankly, it was not very coherent. It wasn’t a complete proof. He wasn’t giving a clear thought of what the theory actually was. He was just a young fledgling mathematician getting his bearings. The verified reward there is, “No good.” But he has some instinct that there’s something there.

So he’s writing this diatribe on the nature of math being something that undergoes these shifts over time. He talks about the advent of algebra itself and going from just thinking in terms of numbers to having a certain fluency with pure algebraic expressions, where you’re not tied to interpreting those expressions. He has this instinct that there seems to be another layer of abstraction that we should be doing, where rather than thinking about the formulas themselves, we’re thinking about what symmetries underlie those formulas. But it was still a pretty ill-defined theory.

If you’re trying to say the verified reward is that he solved a problem that other people haven’t, well, Abel already proved that quintics are unsolvable. So what was Galois doing? In principle, Galois theory lets you take a specific polynomial, and it gives you the rules to say whether that specific polynomial has roots that you could write down. For example, with x⁵ - 1, you know that a solution is 1. Or x⁵ - 2, you can write down the fifth root of two.

So it’s not that you can’t write down the solution for every quintic polynomial, but could you find a specific one where you prove you can’t write the solution using radicals? He also didn’t even solve that exactly. He didn’t show for a specific example that he couldn’t. Even describing what problem he solved is very tricky.

He then dies. It’s this very romantic story of him having this duel. There’s a lot of myth around how he supposedly writes up all his ideas the night before the duel, but really, he tried to get them published five times before.

Dwarkesh Patel

Working on the quintic doesn’t seem to be good for your health.

Grant Sanderson

It’s very bad. If you’re a young genius, don’t work on the quintic. He asks his brother and his close friend to get his notes to Gauss, to get these notes to the important mathematicians of the day, because he thinks there’s something there. Even then, it didn’t really take. His brother and his friend tried to get them out, but it was another twenty years until Liouville sees these notes, sees that maybe there’s something in them, and tries to clean them up and understand what Galois was getting at. Even then, it was another twenty years or so until Jordan actually puts together something like a modern treatment of group theory that they attributed to Galois.

You could easily imagine history turning differently, where these ideas were coming about from other points in math, and Galois could have been forgotten in history if he was a less florid character. But between the time of Lagrange having this inkling that maybe symmetries of roots is the right way to go, to where it all looks like modern group theory, you’ve got this long span. A lot of the time, it’s not even passing the verified reward of human reviewers. It gets on someone’s desk and they say, “I don’t really know if there’s anything here.” You have to have this one person recognize it.

Even then, it’s not really solving practical problems at that point. You pointed out cryptography and physics and things like that. You have to get into the twentieth century before you have Gell-Mann thinking that maybe understanding the nature of how certain groups break down has a relationship with what particles are made out of. He anticipates quarks based on a purely group-theoretic question. That’s one of the more interesting applications of group theory: to even predict the existence of quarks is a group-theoretic question. That’s so long after Lagrange before you have anything like that.

So you have to ask, what is the way of measuring progress that’s not based on solving a problem, but that is somehow capturing the instinct inside Galois’s mind when he says, “I think there’s something here”? What’s the instinct inside Lagrange’s mind when he says, “I think this is the right way to think about it”? What’s the instinct inside Liouville’s mind when he says, “These scattered notes from this long-dead youngster might have something to them”? It’s so hard to put a finger on that.

A different series of videos I’m making right now is about the whole “compression is intelligence” idea. Even though this isn’t really the angle I’m taking, there is something to the idea that the smaller expression that’s more predictive feels more intelligent. So I wonder the extent to which you can give some kind of verifiable reward around not just whether you solved it or what it is solving, but around the smallness of the concepts required to do it.

Going back to Riemann hypothesis solutions, what would that look like if an AI solves it? I think a third way it could happen is it just straight-up works harder. In the same way, you could maybe have an elementary proof of Fermat’s Last Theorem that’s just spelled out over thousands of pages that would be incoherent. But the cleaner way to view it is with elliptic curves and all that. Maybe there’s some thousand-page proof of the Riemann hypothesis that no one’s really getting anything out of, and what you actually want are the succinct, compressed versions of those ideas that would then lend themselves to human understanding.

Maybe you throw Kolmogorov complexity into your attempt to quantify what you mean by elegance. I don’t think it’s easy, but I do think it’s something you would have to do in order to reward the Galois-like instinct, rather than just rewarding whether you solved a problem.

Dwarkesh Patel

It’s very hard to come up with the heuristic for science. But it’s clear humans have been doing this somehow, and obviously, AIs will do it at some point.

Grant Sanderson

It’s relevant also not just in terms of verified reward, but presumably, the end goal is understanding, human understanding. Even if you do have some thousand-page proof of some math thing or some grand new physical theory, the goal is understanding.

Maybe if the goal is predictiveness, you can just have automated engineers go off and build rocket ships where we have no idea how they work, but we can get between stars. But there are going to be a lot of people who want to understand. You’re still going to want whatever the concision function is that distills down this complicated way of thinking into the right one, like the equivalent of the universal law of gravitation for Newton. You would still want to train AIs to be able to do that and find the compressed representation.

00:26:12 – Will we understand an AI proof of the Riemann hypothesis?

Dwarkesh Patel

People have this worry about mathematics in particular that AIs will prove the Riemann hypothesis, and our understanding of mathematics won’t be any the better for it. I have a couple of questions about this. The first one is whether this is something you should expect.

Isn’t the reason humans come up with general, natural objects and subgoals when we’re working on a big problem that this is just useful when you’re trying to work on a complicated, important problem? Theoretically, would this even be a simpler way to solve the Riemann hypothesis, as opposed to just coming up with the natural abstractions that are relevant to thinking about the problem?

And then two, empirically, is this what we observe when AIs make progress on problems today? When the AI came up with that counterexample to the unit distance conjecture, you can just read its chain of thought. It’s not understandable to me, because I don’t know anything about mathematics, but it seems that to other mathematicians it was understandable. It made use of known concepts of mathematics and proved relationships between them, all in natural language. As a result, it accelerated our understanding of the connection between this object and this conjecture.

Empirically, is this a thing we should be worried about?

Grant Sanderson

I think it depends on the nature… If we break down the three possible ways of solving the Riemann hypothesis… The other big one from this year was a certain Erdős problem numbered 1196, about these things called primitive sets. It had that character of bringing an idea from a seemingly different field. As soon as you present the basic idea to a mathematician… You say, “What if we try the Markov chain process where we show that this thing is one from the bottom up probabilistically rather than the top down, and use the von Mangoldt function?”

If you say that to someone in the know, they’d know how to run with it. You have this very small idea that has the form of expertise in one field and expertise in another, drawing a little lightning bolt between them. Those are going to be very human-parsable, because all you have to do is show the start and end point of what those connections are.

If the character of it is mountain building, you have to put in a lot more time to understand that new mountain that was built, because it’s a new thread, not just a lightning bolt between them. And if the nature of the progress was just raw hustle—a super long chain of reasoning with no new theories—then you would have that worry of this whole digestion process. So I don’t think there’s one clear answer. It depends on what the solution would look like.

On the mountain building side, that would actually be really interesting to see. Is it by default very human-understandable, the way we see new theories from great mathematicians? Or is it an alien, different kind of mountain being built where we have to reprocess the kinds of abstractions we engage with?

The closest example here would be the attempted solution of the abc conjecture. We maybe shouldn’t get into that one, but it probably is not a correct solution. Basically it’s this whole new way of thinking that this otherwise reputable mathematician in Japan had come up with. It took mathematicians a long time to even parse what he was saying, but it had the feeling of an alien bit of mathematics that’s theory building, not just a long chain of reasoning. He called it inter-universal geometry.

The biggest fear would be that an AI does that, and then much like the abc conjecture, people work for years to go up the mountain, and they’re like, “Dang it. This just isn’t right.” If it turns out to be wrong, but it really looked right. Even if it was right, there’s just a lot of effort to hike up a new mountain.

Dwarkesh Patel

If we end up in that situation, David Bessis had a really great blog post called “The Fall of the Theorem Economy”. He’s talking about how historically, as you were saying, mathematics is coming up with these definitions and problems, and it’s about proving theorems about them. The theorem-proving stuff is what gets all the credit, but it’s really a parasite on the coming-up-with-the-definition stuff.

Historically, this has not been a problem in terms of credit apportionment, because if you come up with a definition, you’re probably going to be the guy who comes up with a theorem. But now we’re in a situation where if the valuable work is coming up with the insight and AI automates the latter part…

Imagine a scenario where an AI comes up with Abel-like direct arguments about a bunch of important conjectures in the world, and then we just have these proofs. Now it’s up to humans or future AIs to consolidate. Again, having no object-level understanding of this argument whatsoever, I’m sure that if you had access to it, it would make it easier for you to think about what’s going on. Is there some deeper way in which we can understand why this proof works that would make it easier to come up with the ideas behind group theory?

Grant Sanderson

I think it would be hugely helpful. So much of trying to discover new math is mostly being wrong. You’re trying to solve a problem, and it doesn’t feel like constantly taking the correct step up the mountain. Mostly it feels like a random drunken walk, where you’re doing a thing and then you’re wrong and constantly discovering that. If at the very least you know that trying to digest what you have is ultimately leading to a correct solution, that feels like progress, simply because of the sense of knowing it leads to a solution.

There are plenty of instances in the recent history of math where it feels like the reach has exceeded the grasp, where things are proven long before they’re understood. One of my favorite openings to a paper—it’s not even a research paper, it’s more like an expository one—is from a mathematician named Timothy Chow, who was trying to understand a concept called forcing. There’s this problem called the continuum hypothesis that more or less asks: you have a size of infinity for the natural numbers, and a size of infinity for the real numbers. Is there something in between?

The answer is both yes and no. It depends on your axioms. It’s outside the scope of our usual axiom systems, which is an interesting answer. But the method to describe it is really hard to understand. It’s this thing called forcing. In the beginning of this paper, he writes, everyone knows the idea of an unsolved research problem. I want to propose the idea of an unsolved expository problem. Sure, we’ve proven it, but we don’t really know why it’s true.

Then he proposes a partial solution to that expository problem. You can imagine why I loved that framing, because this is my whole life. I don’t do research math. It’s wholly about what’s the most clear way to understand this, even if it’s proven. There is a difference between proof and explanation, and I think you’re getting at the importance of that distinction.

Dwarkesh Patel

Yeah. That will be the main incentive. Or the incentive would have to change, not just in mathematics but in other areas of science, from proving things about the world to consolidating proofs into problems or higher-level insights.

We were having a discussion earlier at lunch about a recent talk you were giving on design and how it helps us understand things. In the limit, is there really a difference between the conceptualization of an idea and the idea itself? If you think about special relativity and spacetime diagrams, and Minkowski spacetime, this is a way in which we illustrate why there’s length contraction and time dilation. But that is the reality… So, the exposition does seem to be the explanation in some sense here.

Grant Sanderson

There’s a couple of interesting things there. One is that there seems to be a really strong correlation between the people who come up with genuinely novel insights and the people who are actually quite clear in their communication of it. You might imagine the opposite, given that the experience of a university student is often that the expert teaching them is not necessarily the best explainer of that topic, because they’re so spoiled by their expertise. But what seems, at least in some cases, to be the case is that the people who are really coming up with something quite novel—you’ve got Einstein or Claude Shannon or someone—you read their papers, and they’re really lucid.

It doesn’t feel like this is just for the experts and you have to chop through it with a machete. They’re very good expositors. Feynman has this characteristic too, he’s a very good expositor. Maybe the same part of the brain that comes up with the correct new way of thinking about it at a research level also has this knack for good explanation.

I think this is pertinent to AI. I used to think that AIs would become these automated theorem provers, but the role of mathematicians was going to shift towards my job, explaining these things. Now I suspect that actually they’ll also be quite good at doing that, probably better than most humans are at explaining and distilling. So digesting and explaining what was going on is probably actually not what’s left for mathematicians, by the nature of how these things are going. We can talk about ways this might not be it, but probably the same thing that comes up with the really good new idea that solves some new problem is also just good at explaining it. That’s a way my beliefs have changed.

Dwarkesh Patel

What’s the last thing you think you’ll be doing? Both you and also what the human mathematical community will be doing.

Grant Sanderson

I will probably be doing something like what I am until I die.

Dwarkesh Patel

If the doomers are right, maybe it’ll be for the same reason.

Grant Sanderson

Yeah. You build a man a fire, and he’s warm for one night. But set a man on fire, and he’s warm for the rest of his life. So that’s where I am with AI.

Some of the function of an explainer or a teacher is to add clarity to a thing that someone’s curious about. That’s one thing. But some of it is more relational, providing motivation and a sense of curation. One interesting take that I’ve heard about what mathematicians will end up being is that it’s actually more analogous to art museum curators than anything else.

The AI solved the thing, so the art exists. They even know how to explain it really well. But you still want someone to help you navigate this nearly infinite space of what ideas are worth engaging with. Even if AIs were in some sense better at that, I think we would always still prefer a human that we had a relationship with, because the way we get motivated to be interested in things is a social phenomenon.

If you have some specific technology you’re trying to build, that might be different. But the people listening to this podcast trust your curation on what’s an interesting topic in the first place. It’s not that they’re landing here because whatever your next topic is, that’s what they wanted to understand in a prior sense. They’re trusting you as a curator.

So my role, and arguably that of other mathematicians, might actually just shift subtly into that curation direction of what ideas are worth pursuing. That’s a lot of my job right now. I think people assume a lot of the time for a video goes into the visuals. Sure, it does. It’s not immediate. But actually a lot of it is just deciding what’s worth saying in the first place, what’s worth putting there.

I want to engage with that, and I think I have a trust with certain people, and they’re curious what I would choose to put forward even if the AIs are better than that. It’s the same reason human musicians are always going to have a role: that social function of the story behind them, even if the objective quality of the MP3 file coming out of some model is better. That’s what I see happening to my job.

00:38:08 – Can AI find the hidden bridges between fields?

Dwarkesh Patel

I want to go back to a question from earlier. Just as AI has crossed this threshold, this important benchmark of being able to connect existing ideas to come up with a new discovery or prove or disprove something, we’re like, “Okay, but what’s the next thing?”

Grant Sanderson

There’s a lot more to do on that one, by the way. Just because a couple lightning bolts have been thrown… I think there’s this flourishing future over the next couple years of really connecting.

Dwarkesh Patel

Right. So in the limit, you could even say—I don’t know if this is accurate, but potentially—a lot of the biggest breakthroughs look like this at some level. With general relativity, you’re just connecting together Riemannian geometry and special relativity. So as AIs keep getting better and better at this connection thing, maybe a lot of big breakthroughs are not really of a different qualitative nature. I don’t know if you have a take on that.

Grant Sanderson

A lot of the conversation has focused on problem-solving and that nature of math, ticking off Erdős problems or something. But I’d say it’s not even a majority of mathematicians who would characterize their work as really targeting the next problem to tick down. Are you familiar with the Langlands program?

Dwarkesh Patel

No.

Grant Sanderson

It’s not even a field of math so much as it is a research ethos. Fermat’s Last Theorem is one inkling of this. You had these two seemingly disparate things, and a connection between them led to a solution.

Langlands was a mathematician. He has this famous letter essentially spelling out how it seems likely that there’s a lot more connections like that. He even got a little bit more specific about the nature of the connections, such that you might imagine this large map, and you’ve got this valley over here and this mountain over here and this set of plains over there. There’s a lot of mathematicians who would characterize their work as being part of trying to understand the threads on this map.

The progress there, it’s not even “Here’s this one specific problem that we know will be solved by that connection.” It’s more that time and time again, there have been cases where big problems were knocked down by finding connections, such that it’s almost preemptively finding the connections. It’s actually very interesting. Anytime you run into a mathematician, ask them whether the character of their work is more akin to the Langlands program or to targeting one particular problem. You get a certain bifurcated split there.

The possibility of AIs being supercharged connectors feels like it might be an amplifying tool in that pursuit. It’s hard to measure, though. This cuts to what we were saying earlier: How do you assign a score to say, “Yes, you’ve done it”? If it’s knocking down a problem, you have a clear way of saying, “Yes, you’ve done it.” You can write the headline. You can have your PR move as the AI company to say, “We did it.”

Whereas if it feels like that was the right connection to draw, you can write theorems around it. That’s the nature of what the papers in that field look like. But I think it will require a lot more “human in the loop” to say, “What was the kind of connection that we were going for?” That’s my guess on what most of the useful progress from these models will look like in the next five years. It’s just really filling in that landscape of connections that you can draw if you’re an expert in multiple fields. As you’ve pointed out, it’s kind of surprising we haven’t already had this.

I would be curious to know at a technical level what causes the unlock there. On the one hand, you can paint an explanation in your head for why you could be an expert in all of these things and not be drawing those connections. When the method of reasoning is this autoregressive chain-of-thought phenomenon… Autoregression is actually a really weird way to produce stuff, if you think about it.

You’re an intelligent person. Imagine I’ve locked you in a box, and the only way you have of interacting with the world is that you receive a slip of paper, and someone says, “Can you predict what will come next?” You predict what will come next, and then your memory’s wiped. You get another slip of paper. Imagine that was done a whole bunch of times, and then what comes out on the other end. They say, “Look at this essay that you wrote.” You might look at that and say, “This is awful. That’s not the essay that I would’ve written.” The process of repeatedly predicting something is just pretty different from how you would think as a writer to compose it and think it through.

In particular, what would probably happen is that you’re a slave to your context. You might be answering some question about a particular field, so you draw on all the context around that. But the connection where all the substance is going to come from is, by its nature, a very unlikely one. You can do all the RL that you want to try to get better in some way, but what’s the thing that’s specifically upweighting and incentivizing making these unlikely connections when the vast majority of them aren’t the predictable next token that would come in there?

So it might be the case that you just have this intelligence locked inside that box, but it’s a weird way of interacting with it. The thing I’m curious about is: do you ever get any fruit by questioning the premise of how tokens are generated? I don’t think it would be as simple as manipulating the temperature, but are there any things that you can do that take the existing level of intelligence but find the right ways of sparking those connections that unlock these sorts of things that we’ve seen? Or do you just need a little bit more intelligence, such that at the level of prediction, it’s predicting that it should be making that lightning bolt to another field?

Dwarkesh Patel

I think it’s more productive to reason, instead of architecture or even loss function, about data. We have diffusion models that do text, and the kinds of things they produce are not of a wholly different character. They’ve just not been explored as much. I think the more relevant thing is: what is the data on which whatever architecture or loss function you have is incentivizing you to produce? It does seem like they’re getting better.

Forget about math. We did have a couple of examples of this kind of thing, but if you just look at why they’re getting better at being autonomous agents… They’re in an environment where they’re autoregressively producing the step that says “Let’s step back and do a search over the whole codebase,” and then “Let’s step back and assess my mistake,” is the thing that works.

I assume what happened in the case of progress in science or maybe in math is you have frontier math-like problems. Mathematicians have specifically designed them because they require connecting together two different fields. I’m guessing there’s all kinds of clever, partially synthetic ways to make harder and harder problems like that that require these kinds of connections—for example, by eliminating assumptions and still requiring the AI to get to the answer—and then it doesn’t really end up mattering what the loss function is. It’s really about, can you come up with an environment that incentivizes this ability?

Grant Sanderson

It feels like you should be able to. I certainly can’t speak to the correct ways of doing that to unlock all this, but it would just be pretty surprising. Don’t you think it would be surprising if, over the next three years, there weren’t a lot more of those lightning bolts?

Dwarkesh Patel

I think this is an important thing to think about. We often think about how smart a single system is. And we don’t think about AIs having advantages that are more the result of other facts about them. So in this context, the key fact about them is that we can just parallelize and arbitrarily scale them. Whatever level of capability they have, it’s not just one idiosyncratic genius in the history of mathematics who makes a few connections and then dies in a duel.

It’s universally applying that waterline across all problems that are accessible at that level of capability. This is among the many advantages that digital minds inherently have that we don’t think enough about. The other ones being that they can merge all their knowledge together—or at least that there will be techniques that allow this to happen—and that you can spawn off copies with identical levels of knowledge. This parallelization is quite an important property.

I’d be curious about your predictions. Even if they’re not as smart as human mathematicians, the fact that for PR reasons the AI companies are just throwing billions and billions of dollars at this means that quantity has a quality all of its own.

Grant Sanderson

That seems in the right direction. If we take that conversation between Montgomery and Dyson at the IAS that suggests some connection between the Riemann hypothesis—or the Riemann zeta-function zeros—and random matrices, that feels like the kind of thing that you could try to automate. You have agents representing expertise in all these fields. We all know that an institute is smarter than an individual.

The reason for having people all in the same geographic location is that you want those serendipitous conversations to happen. What does it look like to engineer those between agents? It’s interesting, because you point out that you can pool all your knowledge, but I really wonder if one of the advantages is that you can do the opposite of that.

Sometimes when an AI is failing, it’s because it gets into a bad chain of thought and it’s really hard to get it out. So you say, “I’ll just start again.” Same deal with humans. Sometimes you start thinking about it in a certain way, and what’s required is to just back up. There are stories about people trying to prove something for a long time, and then at some point they say, “Hang on a second. What if I tried to prove that it’s impossible, or prove the opposite?”

Unwinding your own context and going at it with a fresh mind… You could imagine systematizing that, or having multiple different agents deliberately given different pieces of context and trying to compare and contrast there. We don’t have the same level of manipulation on our own context.

In this AI and math series, the first episode we’ll do will be about when they solved the IMO. I want to focus on one specific IMO problem that they failed on, which is one that a lot of very smart students failed on. Terry Tao also failed on it. People were very mad at the problem because they called it a troll problem. I almost don’t want to spoil it, because I want to construct the episode around leading someone in without their knowing that it turns out to have a simple solution. You can really empathize with what it’s like to be a student solving this.

Basically, there’s a really elegant way of going down what you really feel like is going to be the solution based on the context of it being an International Math Olympiad problem. The character of the solution is really enticing, but it’s hard to prove that it’s the best. The reason is that it’s not. There’s this almost brain-dead solution that is the best.

The relevance of that to the whole AI story is that for a human, what’s required to answer that question is to escape your context. Escape the context of being in the IMO. Escape the context of the way you’ve been trained to solve these contest math problems. If you just approached it like a brain teaser that I throw at someone off the street, they’d probably answer it well.

You want the same sometimes for human research in other contexts, just being able to refresh your thinking and come at it completely differently. Of all the advantages that digital minds have, that might actually be one of them: a more systematic approach to refreshing your thinking. Spin off two agents, one who’s trying to prove it and one who’s trying to disprove it, one who tries it this way and one who tries it another. They deliberately have different contexts.

I would be curious to see, if we’re having this conversation three years from now, how many of the significant results that make headlines have that character of basically erasing the context previously, trying a bunch of different things as opposed to merging the results of a bunch of different agents.

Dwarkesh Patel

It is incredibly interesting, because a common concern people have about AIs is this entropy collapse where they all think the same way, because they’re trained in similar ways. This is why they’re bad at writing. They go down the same path and have similar patterns of speaking and so forth.

But maybe the key advantage AIs have is that you can systematically… It sounded like one of the reasons the unit distance problem conjecture took so long to be disproven was that people assumed the conjecture was actually true, so they were mostly trying to figure out ways to prove it. Maybe one of the key advantages the AIs will have is to increase the entropy by systematically trying out both the negation and trying to prove the positive of any given statement, or being able to systematically give different agents different biases.

It seems like an important thing in the history of human science is that Einstein was really motivated by this bias that things should look the same in different reference frames. He had multiple other biases like these, but that one was very formative in his thinking. You can systematically survey a bunch of heuristics and see which ones are being productive on a given problem.

Grant Sanderson

So you would suggest systematically increasing entropy at the prompt level even though you have this inevitable collapse at the autoregression level? Einstein would be an interesting example, because he’s got this bias toward things being relative. He also has a bias toward “God should not play dice.” You want to make sure you don’t accidentally have all your LLMs be Einstein, because you might halt progress on quantum mechanics.

Dwarkesh Patel

Which goes to show you that there’s not a correct heuristic for science. You just need multiple independent research programs with their own heuristics.

Grant Sanderson

That feels like old-school software. As long as you’re able to describe that in some way. You have old-school software that amplifies that entropy. If you’re able to put a clear ontology to the distinct ways of thinking that you want to prompt, you explore that full ontology, and then each individual one runs off doing what it is.

There’s a certain design question there about how exactly you describe the different approaches. The easy one is: are you trying to prove it or disprove it? The harder one would be to say, what are all the tactics you could take to prove this, and make sure you’re applying sufficient breadth to exploring them.

00:53:48 – Why real-world tasks don’t fit into RL environments

Dwarkesh Patel

Obviously, AI in math is making much faster progress than everything else, and people point to the verifiability of the domain as the key reason this is happening. I think that’s one of the two important reasons, but people really neglect the other one. I’m outside the labs, so I don’t know what’s actually going on. This is a totally naive theory.

A tangential question to why AI is making so much progress in math: why has it been so slow at computer use? A computer is very verifiable. Is my Etsy package coming? Is my event booked? These are extremely verifiable things to survey. What computer use lacks is grindability.

Because websites have bot detectors—and it takes a tremendous amount of compute to run parallel rollouts—it’s very hard to run a thousand parallel rollouts of the same checkout flow on Amazon. You’ll get shut down by Andy Jassy.

Grant Sanderson

Him personally. He presses the red X on Dwarkesh button.

Dwarkesh Patel

Exactly. You could try to build clones of every single website, but that’s very labor-intensive and slows you down. The reason you currently need to do so many parallel rollouts to learn a skill with deep learning is that we haven’t solved sample efficiency.

Grant Sanderson

Sucking supervision through a straw, as Karpathy says?

Dwarkesh Patel

Exactly. Of course people are working on many different techniques, but fundamentally there’s this big constraint in the way we train AIs. With code, you can containerize a given level of progress in a repository and then spin out hundreds of parallel containers and say, “Try to implement this feature,” and it’s totally deterministic. Because it’s deterministic, you can solve the credit assignment problem because you know that whatever caused this rollout to succeed and this one to fail, the diff is the thing that worked. If you have situations that are starting off at different starting points, this credit assignment problem becomes much harder to solve.

Most things in the real world are very hard to containerize in the same way. Coding and math are exceptions to this rule. But if you’re trying to figure out how to build a new business that succeeds, or how to go trade in the markets for a day and make money, the fact that you have to interact with the real world and things change day after day means that you can’t keep replaying and grinding and farming the simulator.

Math, of course, is the exception, and I feel like this is an important driver of progress in this domain and also in coding. It’s not just verifiability; it has to be grindable.

The third reason people point out that AI is making fast progress is they focus a lot on Lean and formalization. Again, I have literally no idea what’s going on in the labs. I feel like Lean just doesn’t matter that much for the current level of progress in AI. Why is AI able to disprove the conjecture about the unit distance problem? They released the chain of thought, or at least a rewrite of the chain of thought. It didn’t have any Lean in it. I think the process-based supervision that Lean provides, where you know each step is correct, seems less relevant than just having this grindable outcome that is verifiable.

Grant Sanderson

It’s an interesting point about grindability mattering more. Naively you might think Lean provides something unique for math because you’re able to see if it can prove it. You have old-school software that can tell you yes or no, and you use that as your VR. What would corroborate your point is the initial attempts. Again, I’ll circle back to the IMO. Initially, DeepMind basically does that. Everything is in Lean, and then the next year it’s all in natural language. So to your point, it’s not needed.

I do think there’s a yet-to-be-explored benefit of that formalization domain, which is that at the moment you still need a human reviewing that counterexample to the unit distance conjecture to say, “Looks good.” That provides a certain bound on how endlessly explorable things are. If you consider AlphaGo or AlphaZero-style systems, they’re off in their own universe playing a bunch of Go and exploring themselves, potentially going off the rails of what any human needs to look at, but they still have this automated verifiable reward. It’s not just that you can do RL on that. It’s also that you basically never have to check in, and you can just pour compute at them exploring the universe of Go.

What stands to be interesting—maybe this won’t pan out, but the jury should still be out on whether it’ll yield anything—is that with Lean, you could imagine having a basically endlessly running program that’s constantly trying to extend Mathlib. Mathlib is this GitHub repository that’s basically all of math written in code. It’s very far from all of math, but they want it to be all of math. It’s written in code where you can ask, “Is this proof correct?” It’s very labor-intensive to write these proofs. There’s a whole subcommunity around it.

But you could imagine having an AI where you say, “Simply try to extend Mathlib.” Maybe it’s a fork of it so that it doesn’t have trash in it, because people have a certain taste for what they want to be in there. So you have your fork of the pure AI Mathlib, and it just goes and it doesn’t stop. It doesn’t need anybody to check in on it. It could just keep going. It might come up with its own conjectures. It might come up with its own theories and different definitions. Maybe many of them are useless, but it just has this infinite tree that it can grow out.

That’s a very unique thing that math has that nothing else has, where you could press go and just pour compute at it, look away for ten years, and then come back and say, “What do you have?” There’s going to be something. Then there’s a question: is it useful or not? How do you suss that out? That’s just an interesting thing to be able to do. It would be very surprising if that didn’t yield some sort of interesting mathematical insight from it.

There are two different ways that Lean is important in this story. The first one is how you could let go, not even check in, and progress will be made. You can do that with Go. I don’t think you can do that with natural language math.

Dwarkesh Patel

That’s very interesting. Did you see Karpathy’s auto research idea? He wrote this one Python file that does basic LLM training, and then had a repo where LLM agents would try to make modifications to the file, and if it sped up the speed run, the modification stays.

Eric Jang, who came on to explain how AlphaGo works, did a similar thing when he was trying to build a very strong Go bot. He had interesting observations. It’s really good at running an experiment and going down that path, but it’s bad at stopping at dead ends and doing extremely parallel things. Anyway, this will probably change in the future.

It’s very interesting to think about what it looks like in the limit. This is fundamentally what the human institution of mathematical research is. It’s a library extended in interesting and useful ways. This way you don’t have any outcome-based supervision. There’s no outcome that you’re trying to incentivize, but you have a process. You know the steps are correct, you just don’t know if it’s going in an interesting direction.

Grant Sanderson

If you were doing that, you don’t want to completely go off the rails and do a random walk through the space of logic. You’d probably want some supervisor model that’s trying to provide heuristics on whether it’s useful or not. You know people are working on it. That’s one of those “five years from now” things where I’d be curious to get the future version of us talking about it.

Maybe that goes nowhere, but Terry Tao was talking about one research project that tries to exhaustively search the space of possible algebras. You could imagine different axioms that you apply to algebraic systems. When we come up with group theory, there’s a certain axiom system that looks like arbitrary rules unless you know the motivation. What if you tried all of them? Do any of them yield useful things? The vast majority of them are just trash in some way. It all collapses to no interesting results.

But every now and then, there would be this little island of a completely different type of axiom system that at the very least seems rich in terms of the number of theorems that can come out of it. That’s bread and butter for what you would imagine automated provers being good for, exploring that space and seeing which one of them turns out to be something. Maybe one of those islands actually turns out to be something you can retroactively put motivation on, to say this is the kind of structure it’s trying to get at.

In the same way that you could imagine looking at the axioms for a group, not knowing that it’s about symmetry, but you retroactively realize this is very relevant to studying symmetry. You could imagine results of that flavor, but instead of just exploring possible algebra systems, it’s exploring all possible logical consequences of any kind of axiom.

Dwarkesh Patel

On the point about whether you can provide process-based supervision without Lean, DeepSeek had their DeepSeek Math model. They released a paper on how they trained it, and it was quite interesting.

The problem with natural language proofs is you don’t know if it’s correct or not. They have a verifier, and the verifier is trained by a meta-verifier that makes sure that for all the problems they’re training this model to solve in the art of problem-solving, the verifier is giving good feedback. It works. It’s interesting that natural language verification with some sort of meta-verification seems to work so far in the published literature.

It also seems to work in the published products that we’re using. If you look at coding agents, they’re getting better and better at writing clean code and refactoring code. I’m sure there are process-based “LLM-as-a-judge” systems providing taste and saying, “Is this a clean way to write this function? Are there duplicates of the same kind of modular forms?” That should also work for mathematics, right?

Grant Sanderson

It seems more plausible for math than anything else, even if you’re only working in natural language, that you could trust a verifier. You and I were talking earlier about why they’re bad at writing. They seem to be good judges. If I give them two essays that students wrote, they’d be able to say which one is more accurate and insightful.  So why can’t you just have a verifier saying, “Is this a good piece of writing or not?”

Maybe the ultimate failure there is that even if they’re good at discriminating between a B essay and an A essay, they’re not actually good at discriminating between an A essay and a thing you actually want to read, something that would be followable on Substack and insightful. They actually end up preferring uninsightful pieces of writing.

On the math front, the step to simply know if a proof is correct or not lends itself to an automated verifier, even in natural language. You could probably still make a ton of progress. I still like the tree of logic out of Lean, just in that you can really go off the rails. There’s no constraint on the previous way things had been phrased before. Everyone talks about move 37 in AlphaGo. What is the thing that lends itself to going outside the prior heuristics? It seems productive to have a disconnection from the rest of the world in that exploration, as a complementary research pursuit to the natural language math front.

The other relevance of Lean would be, let’s say you have your pure natural language RL environments and a pure natural language set of proofs. People say, “Proceed, AI mathematicians,” and they generate ten papers a day. If there’s any error rate to that at all… Alex Kontorovich has talked about this. It becomes insufferable as a mathematician. Every single time you see one of these, you don’t know if it’s worth your time. Even if 99 out of 100 are right, I don’t know if it’s worth my time because it’s really labor-intensive to find what that error would be. It’s really frustrating to spend all your time on a paper that was trash.

Having something that’s able to give you that green checkmark that says, “Even if this is going to be complicated to understand, even if it’s going to be a pain, you at the very least know it is correct,” every other field would kill for that. Math has that. If the models are also able to take their natural language proofs and formalize them, that seems huge. Every field would love to have something like that. So I think you’re right that Lean is maybe overrated regarding its importance as a VR environment for progress in math generally. But I definitely wouldn’t write it out of the story.

Dwarkesh Patel

I also love this extension of Mathlib as a metaphor for what’s going to happen to our civilization pretty soon. For millennia, humanity has built this corpus of knowledge and understanding, and everything that we have is now distilled into these models. At some point, the models will just extend that arbitrarily.

01:07:07 – Good writing requires theory of mind that AI still lacks

Dwarkesh Patel

By the way, on the writing front, I have a theory of why writing is making worse progress than these other domains. One reason is what you said, that they’re bad at judging not only A versus B, but they get totally derailed by B*, which is this shitty essay that hits all the bells and whistles that A is supposed to hit. The reward hacking thing just goes off the rails.

But the other important thing is that writing is not modular in the same way that code and math are. You can write a function many different ways, and they do the same thing. Of course you want it to be clean, but at the end of the day, if it works, it works. Same with lemmas in mathematics. You can have some end product that’s different from the way it’s produced. Code is the thing that produces some end product, and you want a functional end product.

Whereas in writing, the end product is directly the thing the AI is producing. Each paragraph, sentence, and word matters because that is the substance. It’s not some separate thing produced out of the writing. It can’t be slop in the way that code can be slop and still produce the outcome you want.

Grant Sanderson

But you were just pointing out how we’ve actually gotten much better at agents writing not just functional code, but clean code. Why is it not the case that the same progress that lets you go from merely functional to a clean and mergeable PR also results in clearer writing?

Dwarkesh Patel

That’s a good point. Also, has it not? I agree there are many ways in which they’re terrible writers. But for a lot of writing I consume, I find it’s better to just copy-paste it into an LLM and say, “Explain this to me.” The explanation will be better than the thing produced by the human. It’s funny that we say these are such terrible writers, and yet my revealed preference is to have an LLM explain it.

Even when I’m talking to a human expert live on a call, if it’s a piece of knowledge that only they have that’s not encoded in the distribution, I want them to explain it to me. But if in order to understand that, I need to understand a more basic concept, I would prefer if it were socially acceptable for me to just say, “Let’s pause here. I’m just going to ask an LLM how that works, and then we can come back to your special piece of knowledge.”

Grant Sanderson

That’s distillation, an explanation. If I’m thinking of your quality as an essay writer—if I give you a book to read and I want a book report—I might believe that the LLM gives me a better book report. But what people are really getting at when they say it’s bad is, what is writing? It’s not just the distillation of preexisting ideas. It’s not just how you explain clearly, because they are good explainers. It’s about what the insight is.

This is where autoregression is a very weird way to generate things. When you’re writing, you sort of know that in order for it to be good, you have to have an element of the unpredictable. It’s not just increasing the temperature in your mind. It’s knowing exactly the correct point when you want to make an unpredictable move, and that that’s going to be what’s more insightful. Even if it’s better at explaining a preexisting thing, what generated that book that you wanted distilled in the first place?

It wasn’t an LLM that generated it and you just needed it. It was some author who, through a lot of exploration of ideas in the world, decided what aspects were interesting and what ways of presenting it formed a coherent, well-motivated narrative. They put that all together in some way. If they’re a good author, you would probably err on the side of reading their book instead of the distillation.

Still, what makes it worthwhile to explore at all in the first place and want to upload it at all? It’s that side of it that people cite when they say LLMs are bad at writing. It’s that element of unpredictability, of deliberately choosing something novel that is very directly contradictory to the way things are typically produced.

Dwarkesh Patel

That’s a good point. I think they’re also really bad at building really good mental models of people, which is a very important skill in writing. Andy Matuschak and another collaborator, whose name I’m forgetting right now, did an interesting report where they tried to teach LLMs to write good spaced-repetition prompts. I really like this because even though it seems like a totally random skill… It’s just like, people are talking about recursive self-improvement in a year, and we can’t get these things to write good flashcards. What’s going on there?

They tried many different kinds of techniques, and they’re sophisticated people. They tried to RL open source models. They tried all kinds of things, including chain of thought and a big prompt they sent to the best closed source model. The key constraint, it seemed to me, was that writing a good card is about projecting somebody’s mind in three months. What is the way in which they’ll associate the question? What kind of answer will they be thinking at that moment? Is the elicitation that inspires the detail you actually want to take away from the passage you’re trying to make cards about?

I think writing is similar to this. If you’re writing something, the reason it’s such an enervating process that takes so long is that with each word or each sentence, you have to be thinking: what is happening in my reader’s mind right now?

Even if I flip the phrasing around so the end phrase goes to the beginning and this is the first image that comes to your mind before you read the rest of the sentence… Maybe autoregression is bad at that. This is maybe a more diffusion-like property of considering the whole rather than going sentence by sentence. But also I think that requires a lot of mentalizing, which these models weirdly struggle at.

Grant Sanderson

It’s an interesting question. Is it weird that they struggle at that? I might butcher this. You know how you cite studies that you once read and maybe the study wasn’t real? There’s one very memorable one. Let’s say you want to quiz people’s EQ. You show a flashcard of someone’s facial expression and someone is trying to describe that emotion. There are really good tests online that have a face and then four possible emotions. It’s surprisingly hard to describe exactly the correct emotion, but you also get the sense there really is a correct answer. If you try this with people in your life, you’ll notice that the ones who are pretty plugged in socially do really well on it, and the ones who are a little bit more left-brain don’t. That is a kind of test you can do.

I vaguely remember an experiment to this effect where they took people who had freshly gotten Botox, and they did a pretest and a post-test. Post-test, they were just much worse at reading people’s expressions. That feels weird.

Dwarkesh Patel

Wait, they got Botox?

Grant Sanderson

The person taking the test. You do the test, and then you go and get Botox and your face is all frozen, and now you’re worse at understanding the emotions of what you see. The thought is that part of understanding the emotion you’re looking at is doing it yourself. At a facial level, you’re moving your face muscles. You see that, you mimic that, and you’re like, “Oh yeah, that’s anxiety,” at some very subconscious level.

So in that sense, if it is the case that models have bad theory of mind, sure, they know everything because they’ve read what everyone wrote. But at the level of actually being able to put themselves in your shoes in the same way that my face muscles are mimicking your face muscles—that’s what helps me understand how you feel—it’s not surprising at all. They don’t have face muscles. Their brain works completely differently. It’s like an alien trying to empathize. How could it have theory of mind? It would be this very emergent thing to have.

Whereas we can just plug it into our own minds. We’ve got the ready-made hardware to just place it in. From that lens, it’s not that surprising.

01:16:02 – Why learning will still depend on human curation

Dwarkesh Patel

What advice do you have about using LLMs to learn? As I was describing, for a lot of well-known concepts, I find them very helpful. But often, just a couple of messages further down, I’m trying to understand something, and they’re so confused themselves that they’re confusing me. They don’t explain it the right way. I know that talking to the right human could clear up my confusion in three minutes.

More and more, we’re going to want to use these things to learn. People talk a lot about education and representation stuff. Have you noticed ways to use them more productively to understand concepts?

Grant Sanderson

I’m curious to hear your take on this. I’ll give mine. Even pre-LLM, I feel like a relevant insight in learning was recognizing who matters more than what.

My advice to any college student when they’re choosing what courses to take: care a little bit less about your preexisting interests, because they’re kind of arbitrary right now, and care a little bit more about whether the person teaching it is a good educator and someone you resonate with. In choosing what books to read, who the author is maybe matters more than whether it’s a prior interest. If there’s a book you’ve liked before, read what else that author has written rather than reading another thing on that subject.

I’m getting to LLMs on this. There’s a difference in feel for trying to learn something from a Wikipedia page versus, if it’s a philosophy topic, going to the Stanford Encyclopedia of Philosophy. Or if it’s a math topic, you go to the Princeton Companion to Mathematics. The difference there is the articles are deliberately written by one individual who tries to actually craft a motivation around it.

Whereas on Wikipedia, it’s this local minimum that’s reached where every sentence has to be correct. In a good exposition, you care a little bit less about correctness on the way. You can deliberately craft things that are a little bit wrong that you correct along the way, which gets edited out in a crowdsourced environment. LLM explanations feel to me at the moment a lot like Wikipedia, which is to say, amazing. Imagine a world before Wikipedia, how long it would take to find and suss everything.

But nevertheless, what’s the most useful part of a Wikipedia page? It’s often just the references at the bottom. You look at the key references, and you go to them, and you read them. Sometimes that gives a much better overview. So often I like to just ask an LLM, “Who should I read?” Maybe I can even give some specifics on ways I want to learn.

I actually got gaslit by this once when I was trying to learn about semiconductors or something. I felt it was a very visual topic, but all the resources were text. I asked, “Is there a well-visualized video explaining the concepts you’re getting at?” And Claude said, “Yeah, here’s a couple,” and the top one was like, “Here’s one from 3Blue1Brown”. I’m like, “I can guarantee that there’s not.”

It was an actual video, an actual link, but it had just misattributed someone else’s. It was good. I had a much better experience clicking over and watching it to learn rather than trying to proceed forward with questions there. In that sense, I’m basically using it like a very souped-up version of Google to zero in on the right human-written resource. What about you? You engage with these a lot. What’s the best way to use them?

Dwarkesh Patel

I think you put your finger on it. The most productive learning sessions I’ve had are when there’s some artifact that a human has produced—whether it’s an article, a book, or a video—that organizes the relevant concepts in the correct way. It builds up the motivation for why the next idea would be relevant to solving the next problem you’d encounter, and the next idea, and the next idea. Then you use the LLMs to just do a little bit of pruning around this branch that the book has identified. I was actually going through—I think you might have recommended it—Steven Strogatz‘s textbook on…

Grant Sanderson

The chaos one? Nonlinear Dynamics and Chaos? I love that book.

Dwarkesh Patel

Yeah, I was going through it, and it was bliss. It was like your videos in book form. It was super fun. The way I was learning it, I’d have his university lecture on one-third of the screen, that part of the textbook on another third, and an LLM on the last third. I was actually thinking, if I were back in college and watching this lecture live, it would totally go over my head. These kids must be really smart, because I’m pausing, reading the textbook, talking to LLMs, and then restarting again. But with him curating the right order to understand concepts and the right problems to motivate understanding them…

Another thing LLMs are really bad at. Something a really good human can do, when you ask a question, a human can say, “Actually, you’re not really thinking about this topic the correct way. The question you want to be asking, the correct way to organize these concepts, is X.” An LLM just can’t really do that.

Grant Sanderson

It’s a little too placating. This is ultimately that sycophantic behavior where it’s very, “Oh, what an insightful question.” You want to strip that down. That’s a good point, and I think it cuts to theory of mind a little bit, recognizing that asking a certain kind of question reveals that the student’s mental structures are not the same as the explainer’s.

Sometimes people do this to a fault. With a really good teacher, let’s say you have a middle school math classroom. If a student asks a question that suggests they’re thinking about it in a different way, it’s actually really hard to take that seriously in the moment and ask, “Hang on, could you get to a right answer with that?” before you say, “Instead of that, let’s do this.” The really good teachers are able to jujitsu the creative way the student was thinking about it and bring it in. LLMs aren’t doing that. They aren’t reframing your question. Instead, they kind of run off.

At the very least, it feels like there are three levels here. An LLM is at one, a good explainer is at another, but the A+ explainer is the one who can jujitsu your way of thinking and say, “That’s where that’s useful.” Maybe there is a cycle all the way around where, five years from now, the LLMs will be doing that, but in a better way.

Dwarkesh Patel

What is your recommendation to students who I’m sure email you this question all the time: “I was curious about doing mathematics. I’m really passionate about the subject, but seeing all the progress AIs are making, I don’t know if it makes sense for me to pursue this as a career.” This is relevant not only to people in mathematics, but to anyone noticing that their field is getting productivity gains from AI. Coding is very adjacent to this. What advice do you have for people?

Grant Sanderson

I wouldn’t trust any advice that I give. That’s how I’d couch it. But even pre-AI, it feels very important for any job you’re going to go into to really understand… If we’re talking about a job—not being a gentleman-scientist engaging with the math world or something—you should understand where the money is coming from, what value you’re actually adding, and the connection between those two. A surprisingly small amount of thought is put towards that, especially by students.

They’re in this environment where they probably want to go into math because they’ve always been good at it. They’ve been rewarded in life for proceeding through the next hoop correctly. When they think they want to be a mathematician, it’s because they think it’s a way to continue engaging with that. They think, “Where do people get to do this?” rather than thinking, “What value am I adding to other people, and to what extent is that the reason a salary is flowing in my direction?”

It’s actually quite different in different cases. In some cases, it’s a very prestigious mathematician, and their presence at a university lends a certain brand value, which is why the university wants them. In some cases, an NSF grant is given because of the public good belief we have around basic science. You’ve got an institution around that, and a whole bureaucracy acting as a proxy for what we think that public good is, with a whole song and dance around how to make them correctly predict that your progress will be in the spirit of that funding. Sometimes it’s just straight-up teaching. People like to send their kids to an institute that has experts teaching them. You provide brand value by being an expert, and direct value by being a teacher.

Regardless of whether AIs are proving theorems or not, or whether we’re talking about 2016 or 2026, that is something not enough students thinking “I want to be a mathematician” consider. I think it’s worth thinking about. For me, I wasn’t necessarily thinking about it, and I stumbled into a career path where math exploration can be monetized as entertainment. I stumbled into that and I’m very grateful I did, but it was an accident. It wasn’t deliberate. I could have avoided relying on serendipity and done it a little bit more by design had I been thinking critically about it.

To your question—if we have almost-automated theorem proving, and let’s say they’re also really good explainers so you even get the human understanding—I think a lot of the social role that mathematicians serve actually doesn’t change that much. As a public, we still feel there’s value to basic science, and we trust the judgment of mathematicians to determine where their time is best spent. The prestige comes from within that community. It’s other members saying that a result is really good, more than the grant writer really understanding algebraic number theory to understand it was a good result.

There’s going to be an inner culture of what constitutes valuable contributions. Maybe it shifts away from theorem proving and towards good definition writing. Maybe it’s that museum curator idea. But you’re going to have that same community as long as society as a whole is still valuing the premise of basic science. And if we’re in the abundance world that AI brings, there’s probably more funding in that direction in some sense.

On the side of prestige to institutions for who their lecturers are, I actually think teaching is one of the most stable post-AGI jobs that there is, because it’s so relational. This is where parents want to spend their money if they have an abundance of wealth: on good teaching and good educating. It goes so far beyond explanations. Even if LLMs are good explainers, the thing that a teacher is doing is such a social, coaching, mentor-type thing that that’s probably one of the most stable careers that’s going to exist over the next fifty years.

Insofar as a lot of mathematicians’ roles overlap with that, as the prospective student going into it, you could lean into that. I actually think a lot more students should think about and give credence to the idea of being just a math educator and the value that can serve towards the next generation.

I’ll couch again that I don’t think I’m the one to say, “Here, prospective young mathematician, here’s how you should think about the future,” because I’m a YouTuber. I’m not in the institution that they’re thinking of going into, so I’m speaking as an outsider looking in. But it feels like generally good, universal advice: know where the money is coming from, know where you plug into that. And if you’re just asking those questions, you’re actually already steps ahead of all the other fledgling prospective mathematicians.

Dwarkesh Patel

In fact, think about the crazy world where, within five or ten years, the AIs are coming up with not only solutions to the Millennium Prize problems, but totally novel problems to be solving in the first place, novel mathematical fields and objects and stuff. It is in that world where, first of all, there’s a ton of abundance. Two, the thing AI minds will have gone furthest in, where they will have seen furthest beyond our horizons, will be mathematics. There will be so much demand for, “What have the AIs seen? Can you explain it to us?”

In that world, if there are any jobs whatsoever, surely distilling what the AIs have learned will be one of them.

Grant Sanderson

Also, it’s funny because all of this presumes that it’s useless. We’re not talking about the actual practical applications of what math is being done. Insofar as there’s any economic utility to it, you would imagine that the people who understand it and are able to make the decision of where it should point actually have a lot more economic value by being able to make that judgment as a curator and point this behemoth of new math in a useful direction. Suddenly, that’s a much more levered move to make than it had been previously.

Dwarkesh Patel

Can I ask you about that? Obviously, one question for AI for math is not only can it do it, but is it any good? Or is it good for anything?

You were describing all the ways in which, with group theory, we’re trying to figure out random facts about the roots of different kinds of functions, and now there are all these different applications that are practical across many different fields. Do you have some sense of whether, if we just totally get to a place where the field of human mathematics is accelerated 10X or 100X and some crazy shit happens, or are we just going to be bottlenecked by other fields?

Grant Sanderson

I think there are some fields that probably will. It’s super spiky. With progress in algebraic number theory, it feels unlikely that that then unlocks something. But I remember talking to this mathematician who does more dynamics and PDE-solving type stuff. He was referencing that his group had some ideas. Let me see if I summarize this right. It’s like the way Boeing would make planes is that they’d make it, do a bunch of tests, and they had to disassemble it and reassemble it based on those tests. His group essentially had some insights on how to do more in simulation such that you don’t have to deconstruct and rebuild it. It saved Boeing billions of dollars or something, and then they just started funding that group.

That’s much more obviously application-adjacent, because PDEs just are that. Progress in that domain, you would imagine actually does unlock some things. I don’t know if it’s these step changes, but maybe it’s more on the side of engine design becoming a little bit more fluid, or coming up with the right wing shape instead of running a whole bunch of complicated CFD. Maybe you’re able to speed up your CFD simulations because certain pure math insights make those more efficient. I bet you’d just see a lot of great incremental improvement there.

It seems less likely that the massive breakthroughs in math immediately turn into this massive economic breakthrough, like you solve the Navier-Stokes problems, and then that unlocks an ability to simulate more things. But you probably will see, at those fringes, some meaningful leakage out of the pure math insights into other things. There’s a ton of people working on things like AI engineering, physical engineering, and material science. You have to imagine they’d be in a good position to look at the AI math insights and decide whether they’re relevant in some way or not.

It’s another one of these things where I’m not going to sit here and put a flag in the sand predicting that there will be. But it’d be a little bit disappointing and a little bit surprising if there weren’t, over the next five years, economically valuable improvements made that were directly referable to the AI progress in math. It would just be disappointing if it was just taking down a bunch of Erdős problems and none of them were doing any of the math that actually directly touches the physical world.

Dwarkesh Patel

To your point about how a lot of the history of mathematics was about building up these piles of concepts and connections. Sometimes the piles connect with each other, or you discover an application somewhere else. At the very least, you just build up this huge pile.

Then as broader progress in society happens during the singularity, when we get to the industrial part of the singularity, you just have all these different ideas that hopefully are useful in other parts of the world.

Grant Sanderson

As I said, one of the interesting things about what’s happening is it causes people to step back and ask, “What is math?” Maybe one of the awkward conclusions will be revealing that it’s just become wholly useless. The kind of questions being asked have become so divorced from things that are physically applicable that that’s one of the things mathematicians have to come to terms with.

Everyone will look and say, “Hang on a second, weren’t you guys supposed to… If there’s 10X progress there, why aren’t we seeing it over here?” And then mathematicians are like, “Ugh.” Every time we wrote those grant proposals and said, “Trust us, the elliptic curve progress is going to help with cryptography,” it shines a light on the fact that maybe it doesn’t. So that’s one possibility.

Dwarkesh Patel

Grant, this was super fun. Thanks so much for doing it.

Grant Sanderson

Absolutely. My pleasure.

The people optimistic about this vision would say that anything we might consider a fundamental deficits with the current learning paradigm—for example, data inefficiency and lack of continual learning—can be steamrolled by just scaling training more, just as all the supposed “fundamental” research problems in natural language processing collapsed against the flood of compute thrown into LLMs.

Yes, these models are 1/1-millionth as sample efficient as humans during training. But training a one-time cost amortized across billions of user sessions. What matters is how smart, general, and sample efficient the model is within a session, and that’s clearly been improving as we do more RL training. AIs are able to solve more and more ambitious problems across longer and longer time spans - anybody who’s been using these models for coding knows that.

Similarly, continual learning—as defined as the model’s weights getting updated from deployment—may simply not be necessary. Again, because if in-context learning gets so good across longer and longer horizons, then we don’t need to distill back to weights to get on-the-job learning. People often say that their employees are not net productive until six months or more on the job, so clearly online learning is necessary for competence. But what if you could just fit those six months into the context window? There’s been tons of architectural innovations on the transformer which dramatically increase the length of context you can store. With a couple more years of progress, why couldn’t we have arbitrarily large context windows?

Grindability is just as important as verifiability

To address whether this will work, I want to first take a detour and ask a question about the current nature of AI progress that I find confusing and interesting. Why has progress on computer use been so slow?

Computer use is so clearly verifiable (did the desired Etsy item get ordered, is everything I need corporate for my event booked, have my taxes been submitted). So isn’t it weird that computer use has been making much slower progress than coding and math and other verifiable domains? There’s many reasons for this, I’m sure, among them the fact that the models are exposed to far less high quality multimodal data during pretraining, and that video consumes the context window far faster.^[1]

But one reason that I think it quite underrated, and also which reveals the canyon walls against which the river of AI progress will only slowly chip away at, is that it is not enough for a domain to be verifiable. It also has to be very grindable—in the sense that you can run lots of parallel rollouts against a deterministic and replayable simulator. If you’re trying to make a model better at coding, you can create an environment that has a software repo with some missing feature that you’ve tasked the AIs with creating, and then you have a thousand parallel agents just go at the problem, each with their identical copy of the container.^[2]

But this doesn’t work with computer use—at least not trivially. You can’t have a thousand agents go try the same checkout flow on Amazon.com. Because Andy Jassy will find and detect your bots and shut your ass down.

You can solve this by making clones of Slack, Gmail, and all the other common applications and websites. But at least currently, this is a very labor-intensive and unscalable way to build environments. Of course, once AIs get good enough at coding to themselves build these clones with extremely high fidelity, then I’m sure computer use will make a ton of progress. And you’re also killing two birds with one stone with this kind of procedure, because getting AIs to rebuild whole complex applications from scratch is a great RL objective for coding as well.

But while computer use itself may soon be solved, its current lethargy tells us the following: that unless you can build a very replayable training target for a domain, the models will struggle to make much progress. The reason this is true is, of course, that the models are incredibly sample inefficient during training. This is the point I was making in my last monologue.

In computer use, we might be able to make up for this sample efficiency deficit by building these farmable deterministic simulators. But for so many different other kinds of skills an AGI would need to learn, we simply can’t do this.

How would we train an AI to build a business? How would you make an AI that’s really good at winning court cases? Or having a profitable day trading in the markets? Or helping a candidate win an election? The rollout requires interacting with the world and cannot be recreated simply within the datacenter. And the outer loop verification may take months or years of real world actions to elicit, and cannot be re-observed by perturbing the model’s actions thousands of times in parallel so that you can isolate what exactly the model did that actually worked.

Dealing with such reset-free non-stationary environments is a known open problem in RL. I’m not pointing out anything new. But I really do want to emphasize that because of the idiosyncratic and sparse nature of the data in most domains in the world, you need sample efficiency in order to get proficient.

If AIs are to develop all the skills that humans have, and even skills that no humans have, then they need to be able to learn from information revealed in unstructured, unverifiable, and ambiguous ways from scarce amounts of real world interaction. Because in many domains, the relevant training information simply doesn’t exist in any other way.

What is the RL environment to make an AI as good at politics as Lyndon Johnson, or as good at building a space launch business as Elon Musk?

Will RLVR alone generalize?

The labs are betting that RLVR will generalize to all these other domains. If you train in enough containerized, reproducible environments, you will develop a very general agent that can make and execute plans, and learn rapidly from new information, and even pick up new skills, all within a session.

If you drop this endlessly RLVRed AI into Texas politics in 1948, it could give you better advice than LBJ about winning the Senate seat; And if you gave it 100 million dollars in 2002 and let it cook, it could build SpaceX for you.

Whether RLVR generalizes that well is an open empirical question: if labs went from spending billions of dollars on RL environments to a trillion dollars, would you get a fully general, human-like intelligence operating within the context window?

Dario gave a telling quote during our podcast together, which I think hints that RLVR generalization is not this infinitely strong. When he was explaining why model performance tends to degrade at long context, he said:

There’s the context length you train at and there’s a context length that you serve at. If you train at a small context length and then try to serve at a long context length, maybe you get these degradations.

Maybe I’m reading too much into it but he seems to be saying that short-horizon RL training doesn’t necessarily generalize to long-horizon RL performance. And if we can’t generalize from short to long horizon, how are agents supposed to generalize from lots of white collar task training, to, say, getting dropped into the real world and building a business from scratch as well as Sam Walton?

And even if after enough in-context experience, the AIs could become Albert Einsteins and Henry Fords, all that would be ephemeral and wasted if you can’t get those learnings back into the weights. Around 30-50% of a lab’s compute goes to inference, and that compute is currently not really doing anything productive in helping improve the model. What a waste! It’s even worse than it sounds. Because it is only in deployment that the most valuable bits of information which your model could learn from are revealed (What’s actually happening in the organizations I’m being used at? What are they using me for? And what kind of mistakes do I tend to make in the real world?)

We’ve got some genius grad student who has never been allowed to take an internship. And we keep giving it more and more classroom case studies in the form of RL training on environments. It’s bizarre and wasteful that we don’t train the AIs against all this experience could be accumulating thanks to being so broadly deployed through the economy and getting to practice against millions of different assignments given to them and being privy to so much tacit organization- and domain-specific knowledge

Getting the learning back to the weights

But this kind of continual learning requires going back to the weights. AIs can’t just keep building up a KV cache that grows in size as you keep learning from more and more users. That’s just not scalable, and it’s also not how humans learn. We don’t have some separation between parameters and activations. And there’s not some lump of these fast-weight representations that juts out further and further from our skull as we learn more things throughout our lifetime. When we learn stuff, there’s clearly some kind of compression, which actually aids generalization and grokking. There are in fact some humans who have this autistic savant type recall of random tables of numbers or nonsense syllables years later—basically the kind of fidelity of information that models have in context. And such sheer volume cripples these humans’ ability to understand abstractions and metaphors. Human continual learning is less about having all your observations at the tip of your tongue, and more about chiseling the right intuitions and big picture knowledge back into the weights.

But the moment you move into the weights, you have to give up on in-context learning’s sample efficiency. Because gradient updates are super sample-inefficient, all the successfully shipped online learning models have had to learn the same thing across millions of users. For example, the Cursor Tab model online-learns by predicting the same exact objective for over 400M+ requests a day (that objective is which edits got accepted). At least so far, we haven’t seen models online-learn different kinds of things for different users, because while a single session may generate more than enough data for a human to learn from, it’s not enough to train a more capable AI.

Current online learning can work for a very limited number of use cases. But the whole point of continual learning is that the world is very complicated, and each job and company and problem is different, and you need your intelligence to be able to learn the specific information related to a particular deployment, which simply can’t be stuffed into a shared training run. Things like how everything in your organization works and fits together, how to cooperate with the infrastructure and the other people around you to make progress on some larger project, what common failure modes are, etc.

This is the way in which sample efficiency and continual learning are actually deeply connected problems. Relatively little data is available to the model “on-the-job”. To learn from that data requires sample efficiency. Models can do that in context, but the “fast weights” built on the fly by attention^[3] which allow for this sample efficiency scale very poorly in terms of memory. So we need architectural innovations which allow for some kind of intermediate representation. I talked before about how there are already many different working ideas for this kind of thing, from sparse attention and KV cache compaction. It doesn’t seem to me that architecture is fundamentally the bottleneck to continual learning.

Perhaps the bottleneck is the loss function. How do you update the weights (aka improve the model itself) based on information that was learned from one particular session? Even here naively it seems like there are many ideas that oughta work. Lots of people have been talking about on-policy self-distillation recently. If you want to learn more about how it works, check out this little impromptu blackboard lecture that Sasha Rush gave me a couple of weeks ago. But to summarize the explanation a bit, the idea is that we encourage the base model to make the same predictions when trying to solve some real world problem as the model with all the context accumulated after a long session would have made. The whole point of this procedure is to distill what the model learned in a session back into the weights themselves.

This is better than RLVR for two reasons. One, OPSD doesn’t require an outer loop verifiable reward. We just need a model that can learn the right things within the context window. As long as we have that, we can train the base model to match our veteran teacher model which has built up all this experience during the session. And two, OPSD provides a much denser supervision signal than naive RL—instead of projecting a single reward through the whole trajectory, you can train on the per token probability discrepancy between the teacher and student^[4].

For continual learning, OPSD is also superior to supervised fine tuning. The most naive version of SFT for this application you can imagine is to train the base model to predict all the tokens observed during the session. But this makes no sense as a learning target - the way you get better at your job is not by recalling the transcript of what happened through every single day with perfect fidelity. Rather, it’s by consolidating the handful of insights and pieces of knowledge that are relevant to doing your job better.

RL training doesn’t suffer from this failure mode, and it’s great at concentrating the gradient update to only what is relevant to getting the outcome right—that’s why the updates from RL are incredibly sparse. And this is a very important property for continual learning, because as you’re learning on the job, you don’t want to overwrite and forget all the other things the base model knows.

I wrote a post a few months earlier arguing that RL learns much less information per sample than supervised learning. But this may be a good thing rather than a bad thing—you only change the model as much as is absolutely necessary to achieve the outcome, and no more. OPSD preserves this property of RL where instead of slingshotting towards the teacher distribution like supervised learning would have you do, you only extract the knowledge that is necessary to achieve the same results on real world tasks.

Dreaming

So OPSD is one way to attack the sample-efficiency problem: you can take this scarce real world experience and squeeze all the signal into a tiny, well-targeted update. But there’s also another much more speculative idea. Let’s call it dreaming^[5]. If the AI can build a good simulation of reality against which to rehearse new skills, or try alternative strategies and reinforce what works, then it could experience orders of magnitude more simulated samples in the same wall clock time.

A couple years after DeepMind released AlphaZero, a group of researchers trained a model called EfficientZero. If this model and a human both got 2 hours total to play against a simulator of an Atari game they hadn’t seen before, this model would likely beat the novice human. Does that mean this model was more sample efficient than humans? Well it depends on how you measure sample efficiency. Because for each step in the real game, EfficientZero is playing dozens of simulated games in its head. In a similar way, future LLMs might be able to consume far less real-world data while practicing endlessly against environments they build for themselves. The big difference, of course, is that it’s much harder to build a simulation of the whole world than it is to emulate the game of Go. That’s why I said this is much more speculative.

If it works, it would become a fourth axis of scaling, alongside pretraining, RL, and inference-time compute. You can call it test-time training or dreaming. The model spends compute writing up RL environments in which rehearse the skills that will actually be used in production for a specific user. Instead of hitting /compact on Codex or Cursor or Claude, which kindles a small amount of compute to write up a summary, and which gives you a simulacrum of continual learning, you hit /dream, which incinerates huge amounts of compute to build and train against a video game version of what the model is witnessing in the world.

What 2027 looks like

So what might continual learning look like at the end of 2027, and how do we get there? All this RLVR training is producing an agent that can get its bearings when it’s thrown at an unfamiliar problem, and try different strategies, and iterate when it hits a roadblock. This is the crucial thing that RLVR has given you: an AI that is at least competent enough to start getting some real-world experience. Once you have that, you send it out into the world to do real work, even on projects off the training distribution.

By this point, effective context lengths may have expanded such that this AI can cowork with you for a full week of wall clock time. At the end of the week you give it a thumbs up or a thumbs down. If you give it a thumbs up, the base model distills everything the AI learned during the session, and it may use OPSD, or dreaming, or some other technique we aren’t even aware of, or a combination of all of the above, to do so. And AI can get better at domains that are adjacent to what it was explicitly trained for beforehand with RLVR. And in the next round it gets better at the thing adjacent to what it was previously online learned. The gamut of AI skills and knowledge and capability expands far beyond the verifiable domains against which the model was trained before it was deployed. Just as pre-training created a base intelligence that was smart enough to become a competent agent with further RLVR training, so RLVR has created an agent that is competent enough to actually be deployed in the world and thus take advantage of the future paradigm of continual learning.

By this point, the main way that AI gets better is not through the training received before the model is released to the public. Rather, it’s from all this experience that they are accumulating from being broadly deployed through the world and engaging in so many different kinds of tasks. Every time you interact with AI, it’ll be smarter. Not only because it has been learning from all your previous sessions, but also from all its interactions with all the other users in the world. And that’s scary and exciting and very different from the way that AI improves right now.

Sponsor

Mercury has automated basically my entire bill pay process for my business. I just give contractors a dedicated email address, and when they send an invoice, Mercury automatically creates a draft payment for me to review. I no longer have to hunt through my inbox for invoices or deal with messy spreadsheets to track my bills. Mercury handles it all. Learn more at mercury.com.

1. Just one hour of video consumes around 1 million tokens of text.

2. I’ve heard that AI agents are especially good at Go, because it has an excellent standardized package manager whereas Python and Typescript have a “massive combinatorial space of frameworks, typing approaches, and utility libraries.” Such spaces are less amenable to clean, high throughput, parallel search via gradient descent.

3. Let’s use Llama 3 70B as a reference. The KV cache (aka the representation that is built up from learning the context) grows 320 KB with each token. Whereas in training, the model only stores 0.075 bits per token (it’s a 70B model with 16 bit parameters trained on 15 trillion tokens). So between in-context learning and pre-training there’s a 35 million fold difference in the amount of information you’re storing per token.

4. One obvious issue you might anticipate with OPSD is this: you get dense supervision up to and at the point where the student makes an error, but the rest of the trajectory follows from that error - it continues down an already-mistaken path that the teacher was never going to visit anyway. So past that point, you’re no longer getting useful feedback from the teacher for the rest of the rollout. This seems fixable by a technique called Trajectory-Refined Distillation, where the teacher rewrites the trajectory from the error onward into a complete, correct continuation.

5. For what it’s worth, I’m not talking about the upcoming dreaming feature in the leaked Claude Code source code, which I’m guessing will be more about the model writing lots of Markdown files for itself. I mean actually updating the weights themselves. I just don’t think you can accumulate new skills by passing yourself notes. The analogy I used in a previous blog post: imagine if the way students learned how to play the saxophone is by giving this new instrument you’d never tried before a go, taking some notes about what went wrong, and then giving it to the next student who is also playing it for the first time.

The main way that AIs have been getting better is from adding more and better data, and scaling the compute to develop that data in the first place. Obviously RL is the main way that has happened. You can think of RL as a kind of synthetic data generation - you dump a lot of compute against a verifier in order to find the “good” data. Then you train your model to predict these correct rollouts, much in the same way that you might train it to predict the next word in internet text.

For this process to work, the model must have at least prior some probability to anticipate the correct solution, which is why you also need mind-stretching amounts of human expert trajectories in every single field and skill you want the model to be competent at.

It’s hard to overstate how task specific and bespoke this human expert data is. If you want to get some intuition, go read some job descriptions at Mercor or Surge’s websites. There are listings for a word specialists who will convert legacy documents into polished Word files, and legal experts who will write realistic M&A diligences or securities filings, and management consultants who will write up template market research, and dozens more other particular categories.

And it is not only that the data have to be so domain specific, but there has to be so much of it! Each skill corresponds to at least hundreds of human experts who are generating example completions, writing rubrics, and explaining their chain of thought. There’s a reason that the data industry producing these expert labels (and the RL environments in which their meticulously catalogued skills can congeal) is earning billions a year in revenue, soon deca-billions.

Imagine if it took a couple decades worth of courses with hundreds of concurrent professors and millions of practice tasks for you to learn how to polish a word file. Even the task count difference understates the gap - the models have to grind their far more numerous tasks each far harder. Whereas a human student might practice a textbook problem once or twice, GRPO has the model generate hundreds to thousands of rollouts per task. We are building some Frankenstein’s monster, with a billion grafts of carefully constructed examples sewn together.

Epoch recently reported that open models only lag state-of-the-art closed models by 4 months. I think the reason it is relatively easy for open source and previous laggards to catch up to within months of the frontier is that data is the real driver of progress. And data can be easily distilled from public APIs, whereas hyper-parameters and training tricks and architectural micro-optimizations cannot - if the latter were driving most of progress, then catching up would be harder than we are observing it to be.

It is easy to forget how much data these models are trained on, and how much more it is than what we humans see in our lifetimes. We see these AIs as a galaxy glittering with capabilities, but at their center, invisible to the naked eye, holding all the constellations together, is an unimaginably massive black hole of data.

Intermission: Comparing human and AI sample efficiency

• If a person hears and sees on average ~2,000 words an hour, then from birth to adulthood, they’ll see ~200 millions tokens. By contrast, frontier models are trained on somewhere between 10s to 100s of trillions of tokens. That is close to a million fold difference.

• A person can learn to teleoperate any random humanoid or robot arm within hours. The reason robotics isn’t already a deca-trillion dollar industry, with a endless army of Unitree G1s doing all kinds of useful work in world, is that our AIs learn so much less efficiently than humans, and even the millions of hours of demonstrations we’ve collected is not enough to allow them to perform complex, open ended tasks.

• A teenager can learn to drive a car with about 20 hours of practice. Even if you include their ~16 years of accumulated physical intuition as relevant training data, that is at least 3-4 orders of magnitude less than the amount of data Waymo and Tesla have needed to train their self-driving car models.

I wanna deal with some common objections to this kind of comparison:

• Many billions of years of evolution is our pre-training, so it’s unfair to compare how little data we see simply within our lifetime to what these cold-started LLMs have to learn from.

  • Our genome is 3GB, about 1-2% protein coding. That is just not enough space to store the model parameters that are supposedly pretrained (frontier models are terabytes sized). The closer analogy is probably that evolution has found the right hyperparameters and loss functions (Sidenote: I had an interesting podcast with Adam Marblestone where he argued that the loss functions were the more significant find from evolution), but that the equivalent of parameter training is still happening within lifetime, and is encoded in the map of neural connections in the brain built up over a lifetime.

  • Even if it were the case that we can explain away the trillions of tokens required to pretrain a base model as catching up to evolution, it doesn’t explain why the marginal capabilities take so much data - once you have been educated, you don’t need 100 different professors to learn a new programming language, but the AIs (even once pretrained) do.

• These comparisons are not including the multimodal data we see in our lifetimes. If you include all this sensory information, we’re probably in the 10s to 100s of billions of tokens range from birth to adulthood

  • Blind/deaf people who are cut off from this kind of sensory information might lack faculty with the relevant sense, but still have the same general intelligence as everyone else. Which suggests that all these billions of sensory tokens are not really the thing making humans smart.

  • In fact, deaf people who can only communicate via sign language and reading (and not from hearing) are ingesting far less than the 200 million language tokens we calculated earlier, and even this is sufficient for them to be fully general intelligences.

• Scaling laws tell us that bigger models are more sample efficient. The human brain is 100T synapses - if each synapse is ~1 parameter, and frontier models are currently roughly ~5T parameters, then maybe we could achieve human-level sample efficiency with another order of magnitude or two of parameter scaling.

  • The way the scaling law equations work is that parameter and data terms are added to the loss independently. If you have a model that is trained compute optimally, and suppose you ask, well what if I just wanna maximize sample efficiency and use less data - and I’ll throw in as many parameters as it takes to make that happen. With the constants from the Chinchilla scaling laws paper (and the nature of the result wouldn’t change even with different constants), even if you increased the number of parameters by infinity, that would only decrease by a factor of ~10 the amount of data you need in order to keep the same loss. Humans are somewhere between thousands to millions of times more sample efficient than these models. Scaling of current models simply can’t make up for that discrepancy. This really does suggest that humans are on a different scaling curve altogether.

Does sample efficiency matter?

But you might ask, why does sample efficiency matter? The labs have two overarching objectives: automate white collar work, and automate AI research itself. Is human level sample efficiency necessary for either?

The bet with white collar work is the common tasks that a software engineer or analyst or accountant does are, well common. And we can bring common tasks into distribution quite easily through RL and SFT. The revenue curves of these AI labs suggest that there is enormous value from bringing tasks into distribution, even if we don’t replicate human sample efficiency.

Yes it is far more inefficient to train AIs to do these tasks than it is to train humans. But so what? Human lifespan does not allow for the quantity and breath of training these models experience. If you as a human had some weird learning disability where you needed to read through every public repository on Github before you could be a competent developer, it would not make sense to train you up. You’d be on Social Security by the early stages of your education, and even once you were trained, you could work on only one project at a time. But AIs can learn these skills by firehosing gigawatts of training at a time. And what they learn can be amortized across billions of sessions, so we can be ludicrously inefficient in training them and still be wildly in the green.

How much “out-of-distribution” thinking do white collar employees need to do that you simply can’t train for in advance? Well this is more a question about the nature of different jobs rather than a question about AI research. And also depends on the job - some jobs are mechanical and predictable enough that they were automated long before the modern era of AI, for example bank tellers or travel agents. And there are other jobs which require dealing on a daily basis with problems that are quite distant from the data distribution. Even software engineering (the jobs AIs are supposed to take first) is one such. I would be willing to bet that there’s overall more demand for human software engineers in 2028 than there is now, largely due to the complementary input of AI.

The labs’ plan for these later kinds of jobs is to first automate AI research, and then have the automated AI researchers solve this sample efficiency problem. So then the question is, can AIs, which do not have human-level sample efficiency, nonetheless solve the remaining research problems on the way to human-like intelligence and learning.

That question I’ll address in a future blog post - I think the way that people currently think about an intelligence explosion is pretty clumsy. Either people dismiss the possibility of AIs speeding up AI progress altogether, or they just assume that God pops out the other end. People are not reasoning about what extremely rapid progress, but starting with LLMs, looks like.

Sponsor

Thanks to Mercury for sponsoring this essay! Mercury is my banking platform, and they just released a new AI feature called Command. Since I already use Mercury to run basically my entire business, Command has access to all the info it needs to get real work done. I can ask it to send invoices, or categorize expenses, or even transfer money… and Command just handles it. Learn more at mercury.com/command

Machiavelli cut his teeth as a high-level diplomat for Florence, a position from which he got to closely observe the most important rulers in Europe at the time, including the ones who were on the path to destroying his dearly beloved Florence.

In 1513 the Medici retook control of Florence and, wrongly suspecting Machiavelli of participating in a coup attempt, fired, tortured, and exiled him.

Machiavelli could have fled his exile and worked for any number of different principalities that would have been eager to make use of his talents.

Instead, he decided to rot in the countryside and compile his career’s lessons about power, politics, and human nature into a book he dedicated to the very man whose new regime had tortured and exiled him, Lorenzo di Piero de’ Medici.

But at least the Medici were in a position to use his insights to defend Florence. Machiavelli the patriot did not want any other hands to touch this book, because those hands, armed with these lessons, might pose an existential danger to Florence.

The closest modern analogy, at least as Machiavelli would have seen it, would be Szilard’s letter warning FDR about the possibility of a nuclear fission bomb.

What were those insights? And how were they inspired by Machiavelli’s dangerous diplomatic missions all across Europe, and his extensive reading of antiquity? Watch this episode with Ada Palmer to find out!

By the way, Ada is launching a new podcast which I’m very excited about. The first season will be about Machiavelli – a perfect way to dive deeper into the topics we discussed in this episode. Subscribe at Beforecast’s website to be notified of the first episode, subscribe on YouTube, follow her on Patreon, and if you want even more Ada, check out her FixTheNews Podcast episode, and check out her books and more.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Cursor recently saved one of my podcast recordings. When a video file from a shoot came out corrupted, I pointed Cursor at it: it recovered the footage on its own, tracking down the right reference file from the file’s metadata and realigning the out-of-sync audio. My whole team now uses Cursor for everyday tasks, not just coding. Get started at cursor.com/dwarkesh

• Jane Street’s hiring process has been going viral on Twitter lately. The memes are pretty funny, but I wanted to see what their interviews were actually like. So I had Ricson, one of Jane Street’s ML researchers, walk me through a retired puzzle: he gave me an image dataset where 50% of the files had been corrupted – I had to figure out how to recover them. If you’re interested in these sorts of puzzles, you can find Jane Street’s open roles at janestreet.com/dwarkesh

• Crusoe is turning the AI datacenter buildout into an industrial process. At their massive Colorado factory, they assemble Spark units, modular datacenters with power, cooling, and fire suppression built in. They also manufacture specific components in-house to skip the longest lead times. Crusoe has experience running these Spark units on a range of energy sources, including solar and used EV batteries, ensuring they don’t get bottlenecked by grid availability. Learn more at crusoe.ai/dwarkesh

Timestamps

(00:00:00) – How Florence bargained with Cesare Borgia for survival

(00:15:08) – Machiavelli’s analytical innovations

(00:23:58) – Why popes became warlords

(00:36:13) – Why the common people demanded nepotism

(00:47:57) – Cesare Borgia brought terror to rulers and justice to the people

(00:57:55) – Art as a proxy for war

(01:06:41) – Florence, a city famous in hell

(01:15:57) – The Prince was a job application to Machiavelli’s torturers

(01:41:39) – During the Renaissance, original ideas had to be couched in antiquity

(01:50:44) – Why copyright began with the Inquisition

(02:02:12) – Machiavelli wasn’t Machiavellian

Transcript

00:00:00 – How Florence bargained with Cesare Borgia for survival

Dwarkesh Patel

Okay, I’m back with Ada Palmer, who is a science fiction author, composer, and historian at the University of Chicago. Today, I want to talk to you about Machiavelli. He writes The Prince. He dedicates it to Lorenzo di Piero de’ Medici and gives it to him in 1513. He says in the final chapter, “You’re the only person who can bring Italy from its current place of ruin and ravage.” Why were things so bad? What is the historical context in which he’s writing The Prince?

Ada Palmer

I’m going to give a two-part answer to that, although of course with any granular history there can be many parts. The papacy is part of it, and then the city-state structure of Italy is another part of it.

I’ll start with the city-state structure. There’s a principle in politics that when there’s long continuity of a government, and the government has been in power a long time, that government has a lot of legitimacy. People believe in its institutions. People are used to it. Even if you complain about it, it’s the government. When you break that—when you overthrow the ruler, when you dissolve the republic, when you put in a new thing—it doesn’t have that same staying power. So it’s very common when there’s one regime change for there to then be five regime changes, rapid fire, over and over.

We see this with how many iterations the French Republic goes through: the French Republic, and then restored monarchy, and then republic, and then monarchy. When a long thing cracks—boom, boom, boom, boom, boom—you get chaos. England’s Wars of the Roses are similar. There was one stable dynasty for a long time. The moment that a king is overthrown, then you have overthrow, overthrow, overthrow, overthrow for a long time, because the thread of continuity was cut.

In Machiavelli’s lifetime, that thread of continuity is cut for the majority of cities in Italy. And that guarantees, from his perspective, that there are going to be more, and more, and more overthrows in those governments. When Machiavelli was born, there were six or seven city-states in Italy that had had their governments uprooted recently. By the time he’s writing The Prince, it’s dozens, in fact, the majority of these places. So it’s volatile. Almost no government has staying power. Almost every government is ripe for yet another replacement, yet another replacement, yet another replacement. That’s half the answer of why he perceives there to be this urgency and this guarantee that there cannot be stability.

The other half is the papacy. The papacy, of course, is a long and evolving organism. The papacy is one of the oldest institutions in the world now. It was one of the oldest institutions in the world even then, even though this is 500 years ago. As we all know, when you have power centralized in an authority, especially an executive, there can be changes in how that executive uses that power. Each one sets norms for the next one.

Over the course of Machiavelli’s lifetime and just before, a bunch of consecutive popes expanded executive power, especially in the military side, and launched more wars, or did more arbitrary overthrow of governments. You have a number of city-states that are directly ruled by the papacy, and in theory, the pope can appoint anybody to be ruler of that city. Here is a pope. He has an illegitimate son. He wants his illegitimate son to be ruler of something, so he overthrows the government of a city and puts in his son. The next pope does it to three cities. The next pope does it to five. Soon we have a precedent that every new pope feels he has the authority to knock down every pawn upon the chessboard if he feels like it.

Once that is the norm, even a fairly nice pope still inherits the idea that the pope is going to overthrow and replace governments. This creates a unique instability within Italy that no other part of Europe is subject to, because there is no predictability to who’s going to be pope next. It isn’t hereditary. You can’t plan for it. The next pope is elected. As is often the case with elections, very frequently the next pope will be elected by a coalition of all the people who hate the current pope.

One of the things that electoral politics does is that it tends to swing, in which those outside of power work hard to get into power with the next regime. Let’s assume the average length of a papacy is ten years in this period. So every ten years, you suddenly have a completely unpredictable new monarch who’s almost guaranteed to be one of the enemies of the old monarch, and will therefore rip up and replace all of the things that that monarch tried to do with new things.

So Machiavelli, when he’s writing the last chapter of The Prince, is looking around and saying, “Okay, we have a perfect storm. Practically every polity in this region has just had the thread of legitimacy cut. Its institutions have no traditions. Its people have no investment in its current rulers. These are all pawns that have been knocked over before and barely stood up again. They’re ready to fall.” Meanwhile, nothing will stop the turnover of popes. The only thing that could stop the turnover of popes would be one person gaining enough power and ascendancy near this region, who has staying power, who has sons and heredity, that he can do what Cesare Borgia tried to do: have enough power near the papacy to strongly influence the next pope to create a kind of stability that’s otherwise impossible.

Dwarkesh Patel

So he wants the Medicis to not unify Italy, but stabilize Italy at the very least.

Ada Palmer

Exactly, by having conquered enough of a chunk that the papacy fears them and must negotiate with them, as opposed to the papacy being surrounded by small, weakened powers that will constantly be turned over and turned over and turned over.

Dwarkesh Patel

Right, and the pope now is a Medici, right?

Ada Palmer

At that moment, yes.

Dwarkesh Patel

So it makes it even more plausible. Let’s lay down a little more historical context. Before Machiavelli writes The Prince, he’s a bureaucratic diplomat. He meets through his career a lot of these famous figures. I want to know what he makes, for example, of King Louis of France, Maximilian of Germany, the Holy Roman Empire. I want to know what he made of Cesare Borgia.

Ada Palmer

He spends a lot of The Prince, in fact, trying to veil how much more he cares about Cesare Borgia than everyone else. It’s so interesting. He tries to be balanced. He tries to talk about this example, and this example, and this example, and Valentino, and this example. Sometimes he just can’t.

There’s that incredible, magical moment when he’s discussing Valentino’s fall. It’s the moment when he has amassed all this power, he’s successfully conquered almost everything within Italy. Suddenly both his father, the Pope, and him fall ill at once. When Machiavelli describes this, he’s saying, “Everything Cesare Borgia did, he did right. He conquered this kingdom. He would’ve kept it. The only reason he lost it was fortune.”

What Machiavelli should say is, “Valentino had planned for every contingency at his father’s death, except the possibility that he would also be on death’s door.” But that’s not what Machiavelli says. What Machiavelli says is, “He told me that he had planned.” The first person breaks in. Our historian cannot veil himself anymore. He cares too much. “He told me”, first person, that he had prepared for everything in the event of his father’s death, except the possibility that he himself would also be incapacitated at the moment.

It’s such a magical moment where the veil between the author and the reader breaks for just that moment. We realize that all of these others, he observed from a distance. But Machiavelli was in the room next to Valentino, at Valentino’s side through this. He had the most incredible, life-changing, first-person view of this man so unique, and charismatic, and terrifying, that when you read accounts of him, they range from “This was the most incredible, charismatic leader I’ve ever met,” to “This man was supernaturally charismatic to the degree that he must be literally the Antichrist or an incarnation of the angel of death on Earth, because I have no other explanation of how he could be so persuasive and charismatic.” Machiavelli was in the room. And every so often you just feel that he’s still in the spell of this incredible figure at whose side he had the scariest job in the world.

Machiavelli’s job dealing with Cesare Borgia… It’s very clear that the Borgia plan is to conquer the Papal States in the middle of Italy. Tuscany, Florence’s dominion, is this little notch, like a puzzle piece out of the side of the Papal States. Anybody with a map looking at it is like, “You’ve got to conquer that. You just have to conquer this. You can’t have a kingdom without it.” There is no way to stop it. So what do you do?

Machiavelli’s advice to his polity is: this time we’re not going to succeed in persuading this conqueror to pass us by. We can’t bribe him into doing something else permanently. But we can buy time. We can absolutely and abjectly swear to do anything he wants. We can give him our forces, and we can give him our money. We can pay him and help him conquer the rest of it, and betray our allies. Betray Bologna. Florence had had a 300-year alliance to defend Bologna. He said, “We have to break it. The whole world is broken right now. We have to break every promise and every hereditary alliance we had. We must be at the side of this man.”

The only possible survival mechanism is to win from him through loyalty, through support, and through Machiavelli being at his ear whispering forever, “Florence is loyal. Florence is loyal.” By that, we buy the boon of Polyphemus, the terrifying promise of the conqueror: “I like you, my guest. I’ll eat you last.” That’s the republic’s only hope. That’s Machiavelli’s job: to stand next to the scariest man who has lived in Europe since Frederick Barbarossa and whisper constantly in his ear, “The Florentine Republic will support you and will give your grace anything you ask. Just eat us last.”

Dwarkesh Patel

Doesn’t it contradict what he was saying in The Prince about how you should never rise with the help of great powers, for even in success you have empowered somebody who is stronger than you and at whose mercy you are?

Ada Palmer

This is not Florence aiming to rise. This is not Florence expecting that it will gain anything by this. This is Florence knowing it will lose. Machiavelli’s very open about the fact that if Alexander had lived another year, Valentino would have finished his conquests, and taken Florence at last, and it would’ve been over. But popes are mortal. Buying time is sometimes the survival mechanism.

So Machiavelli has this incredible firsthand experience of being with Valentino through all of these decisions, being with him at the massacre at Senigallia when rumor had reached Valentino that some of his people were terrified of him and plotting to overthrow him. They were so scared of him, they decided to abandon the plot, and he heard. He met with them and told them, “I forgive you. It’s okay. You’ve renewed your loyalty to me. You’ve passed the test. I trust you. All is well.” He invites them to the banquet, and then massacres them all. The forgiveness is false. The betrayals are punished.

There’s this amazing letter a couple of months afterward where Machiavelli’s loved ones are writing from Florence because they’ve received a letter from him after the massacre at Senigallia. They say, “Oh, thank God, you’re alive. We had no idea. All we heard was that he had massacred a large number of the people who were with him. We didn’t know if you were alive.” It took months in the chaos, the postal system had completely broken down. It took months for them to get word that Machiavelli was still alive. They didn’t know whether he had been caught up in the conspiracy. He easily could’ve been on a list of names of people the conspirators intended to recruit, and been gone. So his wife and his loved ones back at home, his children, had to wait months to find out whether he too had been slaughtered. It felt to them like a miracle that he hadn’t.

But it meant that he watched these incredible deeds: you encounter them, you forgive them, you renew vows of amity—sacred vows taken in the cathedral—and then you slaughter them at dinner, violating the laws of hospitality. Dante would say if you do that, you’ve committed such a grave sin that you’re not just regular damned. A devil comes up out of Hell and takes your soul out of your body and inhabits you. You’re actually already in Hell even though your body is still alive on Earth, because that’s how heinous a sin this is. And yet, it works, and all the rest of Valentino’s men are more loyal to him afterward than ever before, and won’t even whisper to each other about dissatisfaction, because even the faintest whiff of conspiracy might result in death.

So why does Valentino’s kingdom, for which he did everything right, ultimately fall apart? Because he happens to eat the same thing that gives him food poisoning as his father, and happens to be ill at the wrong moment. Also the puppet that he manages to get in power, Pius III, dies too fast, and then he’s outmaneuvered by Julius. If all those things hadn’t gone wrong in a row, the kingdom would’ve stood, and indeed, he would’ve conquered Florence.

Machiavelli is constantly reminding us that, yes, we have all of these things we can try to do. We can remember it’s better to be feared than loved. We can remember not to be hated. We have power over, at maximum, half of what causes outcomes. The other half is always going to be fortune. We look at Machiavelli. We know he’s the origin of utilitarian thought, and that he says we need to evaluate people’s deeds based on outcome. But he doesn’t just say we need to evaluate their deeds based on outcome. He says we need to evaluate their deeds based on what the most probable outcome was before fortune intervened.

So he says people look at Valentino Borgia and say, “But the Borgias fell. They were feared, and then they were hated, and then they fell, and then their enemies took power and chiseled their coats of arms off of every surface in Rome, so that to this day you’re walking through Rome and you sit down at a pizzeria and there’s a weird scar on the wall, and that scar is where the Borgia bull is no more.” People want to make the moral of that be, “Don’t do what the Borgias did. They fell.” Machiavelli’s like, “No, they did not fall because of their choices. They fell because half of what happens in the world is never in our control. You can do everything right, and it’s out of your control. But we have to evaluate what would have happened, and therefore we should imitate them, because everything they did was right.”

00:15:08 – Machiavelli’s analytical innovations

Dwarkesh Patel

I think one misconception of Machiavelli that I had, because I had not read these books before, is that he says the means don’t matter, the end matters. There’s a virtue ethic sense in which maybe he doesn’t think the means matter, but he is way more concerned about the means than I would’ve naively thought. He thinks the means are incredibly important, because the means by which you achieve power determine how stable and how fruitful that power will be.

In the context of military conflicts, he says if you achieve some power with the help of mercenaries or with the help of great powers—people who become stronger than you as a result of you achieving power—that is a very precarious spot to be in. But speaking of Julius, he makes another point that if you achieve your power by lying, by breaking oaths, by being unfaithful, that this is okay, because his view is that people will forget that you are not faithful. They will just take you at your word the next time they encounter you.

It’s actually a very interesting meditation on by what means you can achieve power that will make that power stable versus not. The fact that he thinks breaking your word is totally fine…

Ada Palmer

It’s even subtler than that. Because if you are someone who breaks your word and you break it this way, it’ll bite you in the ass. If you break it these ways, it’ll be okay.

He also does analysis of figures like Savonarola, who would make prophecies and promises, and then some of them would happen and some of them wouldn’t. He would then make new ones and correct what he said yesterday. He handled his manipulation and untruths badly, in Machiavelli’s analysis, in a way that did turn people against him and make him lose power. Partly because Savonarola, as a religious demagogue, the core of his power was people believing that he was divinely inspired and that he wouldn’t make mistakes and wouldn’t err. So his power base was fragile vis-à-vis untruth. For him, because of the specific shape of his power and then the specific way he handled his contradictions, that did hurt him.

Whereas if it’s somebody like Cesare Borgia, who will make an alliance and work with that ally for a while and then betray them—because meanwhile he was such an effective conqueror and he was so scary and everyone was so afraid of him—even when he would betray an ally, his other allies would say, “I’ve got to be more faithful to him so that the next person to be betrayed isn’t me, and try to work hard to be in the good graces of the prince so that I’m not next”, as opposed to turning on him, because he was so scary.

Savonarola was not scary. Savonarola was charismatic and persuasive and had one of these voices that made crowds thrill and women swoon. Decades later, when people asked Michelangelo what Savonarola had been like—when Savonarola had been dead for decades—Michelangelo’s answer was, “I still hear his voice.” He had one of those charismatic presences. That wasn’t enough when he started flip-flopping on policy and truth. Whereas Valentino was so scary that he could betray his top general and seize his lands and overthrow his city, and all of his other generals would say, “Better step even further into line.”

So it’s not just that lying is okay, it’s that lying is sometimes okay if you check these other boxes, and it’s not if you don’t. So this is even more reinforcement of how much he zooms in on the means. If you do A and B, you’re okay, but if you do A and C, you’re not. He’s looking at the minutiae of different ways you can wield power and different reasons people can have to follow you. If you’re a prince who’s decided to invest in being loved, you have to keep it up, or cultivate being feared alongside it. If you’ve invested heavily in being feared, there are things you can then do that you can’t do if you’re a prince whose power depends on being loved.

Dwarkesh Patel

This actually gets to the famous quote in The Prince, “It is better to be feared than loved.” What he’s getting at there is, I think, that he’s very cynical about people’s nature. If people make you a promise, they’ll just go back on it. If your power base depends on people’s promises and loyalties, as soon as your rule seems to be tattering, they’ll go back on it. Whereas if your rule depends on people having the expectation that if they break their oath to you, they’ll be punished, that’s much more stable.

He basically thinks people will act as badly as they are allowed to, whether they’re the tyrant or the people or the nobles. This goes back to the thing in The Discourses. His whole justification of checks and balances is not dissimilar to the founders of the US and their reason for wanting checks and balances and wanting to put different factions against each other. He’s just cynical and thinks people will act as badly as you allow them to.

Ada Palmer

On that topic, Machiavelli is the first person that we have ever in the European tradition to suggest that it could be viable for there to be more than one political party in a state at the same time, and that they would compete against each other and vent the society’s tension through competition and vie to try to dominate an election and then the next one. This is what we’re used to, but this is innovative in Machiavelli. He talks about how competition within a city, if the parties are kind of stable—he’s observing Siena as one of the examples of this—can vent local tensions and allow interior adjustments of who has power, and be stable. I’m going to come back to interior adjustments of who has power in a second.

The standard attitude toward political parties is that if there are two political parties in a polity, it will not be stable until one of those political parties is dead, and their heads have been cut off and put on spikes, and their houses have been burned down and paved over. That has been Florence’s solution to political parties before. Florence massacred its Ghibellines and killed all of them, and rubbed salt into the earth where the houses used to be so nothing can grow there. Nothing still grows there. Then when the Black Guelphs and the White Guelphs split into two sub-parties, they immediately started slaughtering each other as well. The standard was that one party must wipe out the other party for there to be stability. There are comparatively few examples, although Florence’s neighbor Siena is one, where political parties managed to not only coexist side by side but be politically helpful.

Dwarkesh Patel

One element of governance, or of being a good prince at the time, that I didn’t appreciate but Machiavelli makes a huge point of, is how formidable and reputable people consider you to be. That’s relevant both for preventing others from invading you and for extracting concessions from other people. In his diplomatic career, he is sent out to a bunch of different foreign polities to basically determine, “Hey, is this a serious person?” So Maximilian is trying to extract a bribe from Florence to not invade it on its way down to Italy. Florence says, “Go check out if this guy’s for real.” He has to make some judgment about this person.

Ada Palmer

It’s useful to remember, Florence has paid these bribes a lot. Florence’s tactic is: if someone is invading the area, can we bribe them? Because paying somebody to not attack you is a much more surefire thing than preparing to actually fight against them. Your family’s lands get trampled by soldiers. You suffer economically. So it’s an old Florentine tactic. It’s not a new thing that Maximilian is threatening to invade Italy and trying to extract a bribe.

Florence basically every year is like, “Okay, who do we need to bribe this year to not invade us? Here’s this year’s bribing-a-king budget. To whom does it go? Is Maximilian a serious threat, or are we saving this money in case there’s a threat from somebody more serious like the King of Naples or the King of France or Milan or the Venetians?”

00:23:58 – Why popes became warlords

Dwarkesh Patel

At this time, the pope is not just a spiritual leader but a temporal power.

Ada Palmer

Very much so.

Dwarkesh Patel

He and his son are literally fighting wars against other Catholics, but the other Catholics are fighting them back. What does it mean to be a Catholic who is fighting a war against the pope?

Ada Palmer

Here is where geographic proximity is everything. If you’re far from Rome—when you’re Denmark or Iceland, and the pope is all the way over there—the way you interact with him is that occasionally an incredibly impressive papal legate will visit. There’ll be vast pomp and circumstance, and the city will rename a street in honor of the fact that somebody sent by the pope has visited. He has this great power to say yes or no to petitions, and different countries have been trying to petition for specific things for ages, and the pope’s legate is here to interview the emperor, to judge whether the queen can be queen or not, and it feels like a big deal, and the pope is very abstract.

It’s easy to have a lot of respect for that pope, because what do you see that pope do? You see that pope in pomp and circumstance. You see that pope make judgments about fates of popes and kings. You see that pope put out papal bulls and edicts that give theological answers to questions. You see that pope exercise judgment of life or death over people at a distance. He’s very abstract, and the difference between one pope and the next pope is kind of small from your perspective. You don’t see their policy differences.

If you’re in Italy, the pope is that asshole who went to college with your brother and beat him up when they were at college, and then was drunken and irresponsible at middle age, and you’ve been negotiating with him in these other jobs, and you know this jerk. You know his family. You know the other jerks who are also competitors for this. You’re allied with him, you’re not allied with him. His ancestors are allies of yours or not allies of yours. He’s a specific dude.

You’re much more likely to judge a pope based on, “He’s that guy.” This is not Pope Julius II. This is Giuliano della Rovere, and I judge him based on his uncle who put him in power, and the actions of his friends, and the actions of the city he’s from. You know all of his dirty laundry. You are subject to the fact that when he moves into power, everyone who’s related to him is going to get promoted within Italy, and everybody who’s not is going to get removed from Italy. So it’s much easier for an Italian to see this pope, and it’s actually quite hard to see the papacy. That’s how you have these fascinating wars where even the cities that are hereditarily incredibly loyal to the papacy will sometimes be fighting a war against the papacy.

All Italy is divided into these two factions, the Guelphs and Ghibellines. Theoretically, what these two factions mean is that Guelph powers, Guelph families, Guelph cities believe the correct successor to the Roman emperors is the pope. The pope is the emperor. He has the right to be the ruler of Italy, and indeed of everything that was once Rome’s. He is the ultimate political and military power, and he is the rightful and only rightful overlord of Italy.

The Ghibellines believe that in 800 AD, when Charlemagne conquered a bunch of stuff and made the empire that we now refer to as the Holy Roman Empire, when the Pope crowned Charlemagne, he delegated the political and military side of his authority to that emperor, and made himself the spiritual authority, but the emperor the political and temporal authority. Therefore, the rightful ruler of Italy is the emperor, the successors of Charlemagne.

These are the two factions for which these parties fought originally, 300 years ago. These days, what these factions actually mean is, “Those jerks murdered Uncle Tybalt, and we will never forgive them.” They are the team that is our enemy, and we are this team. They are that team, and we hate them. We want to crush them because they want to crush us.

This means that sometimes a pope will be elected who’s from a hereditarily Ghibelline family, and the pope will start promoting people from the anti-papal faction, and the pro-papal faction will unite against the pope. It makes no rational sense until we remember that they are serving the pope abstractly. So you get multiple situations where there’s a war between Rome and Florence over the fact that Florence wants to defend papal authority in papal lands against the pope itself, because that individual pope was from the anti-papal faction.

Dwarkesh Patel

Do they not believe that he is the vicar of Christ on Earth? It makes sense in a normal political state for you to think, “I believe in America, but I don’t like the president,” or something. But isn’t the pope supposed to be…?

Ada Palmer

Yes and no. Again, when you’re far away, yes. When you’re close up, you know too much of the dirty laundry of these people. So let me use a fun example: the most passive-aggressive letter ever written in the entire history of time, in my opinion.

There’s a type of ceremony that happens when a new pope is elected, which is the giving of oaths of obedience. A major ambassador from every polity in Christendom comes to Rome. They wait in line for a long time, and then they give a long-winded speech about how great the monarch is that they’re there to represent, and how vast his power is, blah, blah, blah, and how pious he is, and how glad he is, Your Holiness, that you’re the pope now. Congratulations on behalf of my wonderful king.

And you’re supposed to send the highest-status possible person who can leave your polity without it falling down. You might send a younger son of the king. You might send a lord chancellor. In the case of Florence, you’re going to send the most prominent citizen you can.

So when Pope Sixtus was elected, it was Lorenzo de’ Medici himself—not the dedicatee of The Prince, the grandfather and namesake of the dedicatee of The Prince—who went to deliver this oration of obedience, which means literally prostrating yourself in front of the pope, literally kissing his feet, and giving this oath. Lorenzo did this for Pope Sixtus, with whom he was negotiating to try desperately to get a cardinalship for his brother. Pope Sixtus instead organized the Pazzi conspiracy to try to butcher the Medici family, killed Lorenzo’s brother, killed a number of his allies as well, and attempted to have a coup to take over Florence.

Then the next pope was elected after Sixtus, Pope Innocent, who was as everyone knew, a puppet of the same faction that Sixtus was from. So we go from this very dangerous pope who had tried to wipe out Lorenzo’s family to a puppet of the same faction. Lorenzo sent his son, instead of himself, to give this oath. He had his son deliver the message, apologizing to His Holiness that, “I could not come myself, but the last time this duty fell upon me, I had a brother upon whom I could leave the burden of the state in my absence. Since now I have no brother, I cannot come in person.” It’s a very respectful letter, but it’s also very overt about the fact that he does not trust and will not again trust this faction.

So they negotiate very carefully how to deal with the fact that the popes have this great spiritual power, but sometimes the popes are acting as horrifically selfish warlords. That’s also something which has worsened over time, and it’s important for us to remember that the papacy becomes gradually more corrupted over time. This is because with every generation, more people leave donations of wealth to the Church. A widow who has no son and has property decides to piously leave this to a monastery. The Church gets wealthier and wealthier. As the Church gets wealthier, with wealth comes power. More and more power is in the state. This makes a stronger and stronger incentive for every ambitious family to send their second son into the Church.

And this goes all the way down. We have personal letters of Machiavelli writing to and from relatives of his, where they’re debating the correct-sized bribe to offer to buy a priesthood for his little brother, Totto. They don’t want to offer too big a bribe, because it would impoverish the family. They don’t want to offer too small a bribe. They’ve heard that another family that’s after this priesthood offered an extra big bribe. That’s kind of not fair. How do they respond to being out-bribed? They just write about this as the most everyday, normal thing in the world. This is a wealthy merchant-prince-level family. They are in the top 5% of wealth and power in Florence, but not in the top 1%. But for them, too, it’s normal to talk about paying a bribe to get a priesthood. That’s just how it works.

Every generation sees the Church get wealthier and have more power. Therefore the incentives to corrupt it are even greater. It even becomes a kind of prisoner’s dilemma system. If you’re the duke and you don’t manipulate the papacy, if you don’t bribe the pope, if you don’t work hard to get your brother to be bishop, and your enemies do, you’re screwed. So you even see it as defensive: “I must manipulate the Church. It’s the only way my people will be safe. If I don’t manipulate the Church, my enemies may manipulate the Church, and then there’s danger.”

This happens all the way up to the scale of kings, where popes can make your enemy the most powerful bishop in your kingdom or can deny you the right to marry, because inevitably the person you want to marry is a cousin, and you’re going to need a special dispensation to marry them. The pope can prevent that and mess with your marriage alliances. You need the pope very desperately if you’re a king. You also need the pope all the way down. That means bribes and other kinds of incentives make the papacy more corrupt with each generation.

So the papacy is worse in everyone’s lived experience than it used to be even a few popes ago. You see every generation for 100 years say, “Popes are much worse now than popes used to be when I was young.” Everybody says that. Dante says that in 1300. Machiavelli’s grandparents’ generation is saying that in 1400. Machiavelli is saying that in 1500. In everybody’s lived experience, the popes are getting more secular, more military, and more corrupt over time. It’s a gradual accumulation, and it comes to a peak, as such things do, triggering the reformation, when it becomes so bad that there has to be a massive move against it.

Machiavelli, in an interesting way, anticipates this, because Machiavelli says, “All institutions are gradually corrupted and need to be reformed and returned to their foundations, or they will collapse under the weight of their corruption.” He thinks that the papacy has been undergoing this, and that Christianity has been undergoing this. And that, if not for the fact that St. Francis of Assisi—and also to some extent St. Dominic a couple of centuries before his time—reformed the Church and brought in a lot more popular support, Christianity would already have cracked under the weight of its own corruption 200 years before, and that it will need such a restoration again—as any institution needs, as city governments need, as republics need—as corruption accumulates over time.

00:36:13 – Why the common people demanded nepotism

Dwarkesh Patel

One big way in which our world is different from 500 years ago is the focus on patronage and it being the basis of political power. It was much more prominent, right? So that’s something that would be interesting to understand.

Ada Palmer

It’s not just that it was more prominent, but that it was the fundamental glue of the society, as opposed to one of several glues of the society. Patronage, which was also familial and therefore entangled with nepotism, was so fundamental. For example, when Alessandro Farnese was elected Pope Paul III in the middle of the 1500s, he didn’t corruptly make one of his kinsmen commander of the papal armies. He instead appointed a really competent, experienced general instead of his own not very competent, illegitimate son.

And there were riots in Rome. “Your Holiness, the people demand more nepotism. You must appoint your illegitimate son to command your armies, because your illegitimate son will never betray you, and we will know we can trust the papal armies not to turn on Rome if the Pope’s son is the commander. We don’t know that about this other commander. He might turn against Your Holiness. There might be a rift between the Pope and the papal armies if he’s not somebody who rises and falls with you the way your son does. Therefore, by popular demand, the people want more nepotism, because the system depends on it.”

That’s how you see how the system depends on it. There are levels of trust that the patronage system creates, because it involves multi-generational entanglement of families. If these families rise, they rise together. If they fall, they fall together. That creates levels of trust that can sustain things like this world where the oath of a soldier is to his commander, not to the polity that he serves.

In modernity, we realize another solution to that. The oath of the soldier is not to the commander, but to the Constitution, or to the country, or to the people. But in this period, the oath of the soldier is to the commander, mostly because communications are so slow that the commander has to be able to give speedy field commands. But it means you’re creating an army and handing it to a man. If you cannot trust that man, then the people will be terrified that there could be a rift between Rome and its own armies, or between Rome and its treasurer, or between Rome and its other allies. Patronage is the glue that makes things work all the way down.

All the way down to the level of, if you need a defense attorney, that’s done through patronage. The outcomes of trials are a really great way to see patronage. We’re all familiar with the fact that law codes in the Middle Ages are really cruel: death for everything. Death for theft, death for adultery, death for homosexuality, death for setting fire to the prince’s beehive. Whatever it is, that’s the sentence on the books. You look at the actual trial records and maybe one in 100 convictions for that crime actually ends in a capital sentence. Almost all of the other ones end in a fine or a public flogging, but not in the sentence that’s on the books. We say, “Why and how is that happening?” Patronage is the answer.

So say it’s the Middle Ages or the Renaissance, and you’re a carpenter, and your teenage son gets drunk and punches somebody in a brawl, breaks the guy’s nose in a way that makes him die. He accidentally kills a guy in a drunken brawl and your son is now on trial for murder. You’re a carpenter. You have worked for the rich family whose family carpenter you are. Let’s say it’s the Medici family. Whenever they need new pews for the family church or new furniture or repairs for the family gates, they go to you.

So you go to them and say, “My son is in danger. He’s on trial. Please put in a good word.” Your patron has the ability to influence the judges, and they will put in a good word for you, and you will get a lighter sentence. This is an ancestor of having a character witness to say, “So-and-so is such a good person, they should have the milder punishment, not the more severe one.”

The norm is: you’re accused of a severe crime, you’re put on trial for your life, your patron intervenes, and you get a lighter sentence. This is how justice is supposed to work. This is a very severe line that changes in the 18th century with the Enlightenment. Because we now think of proportional justice: the sentence for the crime should be this, and ideal justice is that everyone who is guilty of the crime gets that sentence. That is fair. It doesn’t matter who you know. It doesn’t matter whether you’re rich or poor. The sentence should be the same. This is the ideal of Enlightenment justice.

The ideal of this period’s justice, which is much more shaped by Christianity, is that the purpose of the trial is the spiritual interior correction of the soul of the sinner. Therefore the ideal outcome is for them to fear for their life. They’re before a terrifying judge who is the earthly representation of God. They know that they’re guilty, and they deserve to be thrown into the pits of hell. But miraculously, they are given grace, and they are pardoned. The process of being put on trial, fearing for your life, begging to the patron, and then receiving mercy is supposed to be an earthly preview of the process your soul will undergo when you are before divine judgment. Therefore it should make you come out the other end a good person.

The goal of the justice system is the spiritual improvement of the sinner and the hope that they will come out the other end better and more likely to go to heaven. Even when people are being sentenced to death, there are religious organizations who sit with them overnight, having a final prayer group, and walk with them to the gallows, holding their hand and holding a painting of the Virgin Mary in front of their face, so that to the very last moment, the person who’s about to be executed is thinking about heaven. The ideal outcome of the execution is that the soul goes to heaven.

So the whole structure of the justice system expects the intervention of a patron, who represents the intervention of a patron saint, persuading the judge, who is God, to give you mercy. So when we see 100 trials end with 99 where the person paid a small fine, and one where the person was executed, what that actually means is that in 99, their patron stepped in. Somebody persuaded somebody who put in a good word, and they got the light sentence. In one, that person had fallen out of the patronage network. That person had angered their boss, their protector. That’s why it went all the way to being a capital offense.

Probably a lot of people listening are familiar with Giordano Bruno, very famous as a martyr for science because he was burnt at the stake by the Inquisition. Fewer people know that that was not his first Inquisition trial. He was investigated a number of times by the Inquisition for doing various radical forms of thought. The earlier trials had the usual outcome for that kind of trial. He had a patron, there were rich people that he worked with or for, the university was hosting him. They put in a good word. He’s fine. The Inquisition tells him, “Be good,” and things continue as they are.

That time, he had angered the person he worked for. He pissed off his patron. It’s his patron who turns him into the Inquisition and says, “This guy is a charlatan. He promised he could teach me these things, and he can’t. I don’t trust him. He’s no good. Throw the book at him.” The reason that trial goes all the way to a capital sentence is that he doesn’t have a patron. He’s the 99th case that time.

If he had had a patron protecting him, despite how radical his stuff was, he would’ve been okay. We see that in the trial of Giovanni Pico della Mirandola, who was candidly substantially more radical than Giordano Bruno. But when Pico is on trial, Lorenzo de’ Medici and other powerful people really care about Pico, and they pull out all the stops. Lorenzo talks to his brother-in-law, who’s an Orsini. The Orsini have enormous influence in Rome. They get permission for Pico to be let go and sent home to Lorenzo to sort of live under house arrest, under Lorenzo’s promise that he’ll be good from then on.

Or you have Marsilio Ficino who is this radical Platonist who publishes a book on how to project your soul outside of time and summon angels, arguing for the existence of reincarnation, and is very clearly being extremely theologically weird. This is the man who wrote the best letter of recommendation ever written in the history of time when he was recommending a young scholar for a job with the King of Hungary. He writes in the recommendation letter, “This young man is the reincarnation of St. Thomas Aquinas, so you should give him a job.” Now, that is a letter of recommendation. But you think, “The reincarnation of St. Thomas Aquinas, huh?”

And the Inquisition comes knocking on Ficino’s door and is like, “Hmm, reincarnation?” Ficino’s like, “Oh, no. Help. Talk to Lorenzo.” Lorenzo talks to his brother-in-law, Cardinal Orsini. Cardinal Orsini shuts it down, and Ficino is told, “Maybe lay off talking quite so overtly about the reincarnation.” Ficino says, “Yes, of course, and I will only teach very pious people how to summon angels and project their souls out of their bodies. I promise I won’t teach it to anybody who will use these powers irresponsibly.” The Inquisition is like, “Okay,” and goes home. Because patronage kicked in.

Patronage is the glue that makes everything work. You can’t even stay in a hotel or buy an apple—I’m not kidding—without a patron. You arrive at a city. Nobody knows you. You’re a stranger. What you have is a letter of recommendation from your patron who’s friends with some important person there. You present that at the hotel. That’s why they let you stay.

00:47:57 – Cesare Borgia brought terror to rulers and justice to the people

Dwarkesh Patel

To tie together a couple of threads you were talking about, The Prince is painting a picture of regimes being incredibly unstable. You have to worry about foreign powers. You have to worry about rival factions within your own country. You have to worry about mercenaries. You have to worry about lots of different things. So any given regime is very unstable. So what had to happen for things to get more stable? We’re talking about a couple of the ways in which people owed their loyalties not to the regime, but to others within the regime, which created instability.

In Discourses on Livy, Machiavelli talks about how one of the reasons the Roman Empire’s fall was instigated is that these generals were months away on the frontier fighting these wars because the empire was so big, they had to amass for periods of years—or in some cases, for Caesar of course, decades—the command of so many men who have for decades just been listening to this guy tell them what to do, who to fight next. This is the person they’re loyal to. As opposed to, say, if the consuls could be giving dictates every single day, then the loyalty could be to the regime in Rome.

Same with patronage, if there’s not a system of deterministic justice like we have in the modern world today. A lot of The Prince and Discourses on Livy is dedicated to: how do you make sure that a family is not pissed off that their son got killed and it wasn’t avenged? If you just have a reliable criminal justice system, that problem goes away. It’s the same with the welfare state and getting rid of the patronage system. If you don’t have to rely on this family, then this disintermediates them, and the state can have your loyalty.

So it’s interesting to connect all these threads together—communication time, impartial justice system, impartial welfare state—as being what is required for the regime to have enough legitimacy and then, as a result, enough stability to have modern nation states.

Ada Palmer

Yeah. One thing that everyone is surprised by is that when Cesare Borgia—Valentino is much more what he’s called in the period—conquers these cities in Central Italy, he goes in, and he massacres the ruling family. He works hard to kill every member of them that he can so that there isn’t a potential rival claimant to come displace him. He implements neutral justice, because he and his cronies have no side in that city. They aren’t connected with one group of families against another. When they implement justice, they do so neutrally because they aren’t interested in the local backstory of factions.

As a result, to everyone’s surprise, he moves into a city, he massacres the rulers, he implements an authoritarian regime, and he’s incredibly popular and beloved by the people. Everyone says, “Why are they liking this man? He is a cruel, murdering tyrant.” The answer is, for the first time in generations, they have something close to fair justice.

Meaning, it used to be that there was one faction in power and another faction out of power. In our scenario where a carpenter’s son gets drunk and kills someone in a drunken brawl, if that carpenter’s son is the carpenter of the power that’s in power, then there will be no justice and no consequences for this murder. It’ll be maybe the smallest of fines. If that carpenter works for the families that are out of power, then throw the book at him. His son will be executed for that death. There will be no fair justice. The outcome of the sentence will be entirely who’s in power and out of power, not the fairness of the case.

But when both of those ruling families have been wiped out, and an outside power is here, and a homicide takes place, the neutral judge hears this neutrally and gives the same answer regardless of whose family’s carpenter that is. The people who have lived in generations of “justice for some and injustice for others,” suddenly having equitable justice, are delighted by this and find that wrongs are finally being punished. The people that they’ve resented and hated for so long who are in power are finally being punished for the crimes that they commit.

This makes Valentino’s conquering and violent regime incredibly popular with the everyday people of these cities, who are therefore willing to sign up for his armies and help defend his conquests and keep them in power and man his fortresses. So Machiavelli and others are startled by this. They had expected that if a conqueror moves in and massacres the rulers of a city, everyone in the city will hate and fear that conqueror. But if the conqueror is feared and not hated, because he wiped them out but then was fair toward the people, then it works.

Dwarkesh Patel

So why would it have been so bad if Valentino took over Florence and he had survived? He would’ve massacred maybe the ruling regime at the time, the republic, but I don’t know what Machiavelli is especially concerned about. Would the cultural treasures of Florence and everything have survived?

Ada Palmer

The cultural treasures of Florence would potentially have been okay. There’s two answers to that. One of them is that Machiavelli is very adamant that if you live such that there is somebody who can have you summarily executed—he can walk by you in the street and point at you and say, “Him, kill him,” and it happens—then you are not free. In his vocabulary in the text, if you live in a state where there is an arbitrary power who can have you put to death, you are a slave. If instead you live in a system where there must be a trial, and there must be a process, and this must be examined and public, if there is a system, then you have liberty. That system may be unfair. It may be biased. It may be, in Machiavelli’s case, the very system that tortured and exiled him. But there was a system. He considers that difference to be enormously important.

So if Valentino conquers Florence, it’s not going to be that system anymore. There will be a man who can walk down the street and point at a Florentine citizen and say, “Kill him,” and they will kill him. Will that tyrant be fair? Maybe. Will that tyrant exercise this power well? Perhaps. Will his successor be worse or better than him? We don’t know. We can’t predict. It’s a monarchy. It’s vulnerable to good successors and bad successors. But the people of Florence are not free if there exists a man who can say, “Execute him.”

That meant a lot to Machiavelli, and it meant a lot to the Florentine people. It’s kind of hard for us to see how few liberties and how little franchise they had and yet how much they cared. Florentines are constantly willing to go into the street and risk their lives flying the banner that says “Libertas” across it: liberty. The banner LIBERTAS is the coat of arms of the Signoria, the Senate, which is selected from the 1% super mega elite, tiny minority of the city that is eligible to be in government.

They aren’t rioting to defend their right to participate in the republic. They’re rioting to defend their boss’s boss’s right to be in the republic. Yet they care so deeply about it, and they consider it fundamentally different from the situation in which there is a man who can walk down the street and point at you and say, “Him, kill him.” That tradition of liberty means a lot and would be gone even if the most beneficent tyrant in the world took the city. So that’s half of the answer.

Dwarkesh Patel

Can I ask about that real quick? When  Lorenzo de’ Piero di Medici takes over, is he not that guy?

Ada Palmer

That’s the second half of the answer. There is a huge difference between when the conqueror is from your city, loves your city, and wants to take care of your city, and when the conqueror is from the outside. When the Medici take over Florence, they want Florence, and they want Florence to be Florence. They want all of its beauty and all of its treasures to still exist and be theirs. They would never consider razing important parts of it to the ground. They would never consider threatening the Florentines with, “We will destroy your city walls,” or, “We will destroy your cathedral if you rebel.” Any outsider would.

So Florence looks more like Florence under a Medici duke than Milan looks like Republican Milan under a Visconti or Sforza duke, or than Ferrara, which has no remnants of its republic, does under the dukes d’Este, who can do anything they want, including murderously gouging each other’s eyes out and the city will never take one step against them.

So Machiavelli is aware that if Florence has to fall, falling to the Medici is gentlest. It’s most volatile perhaps, because they aren’t going to be feared as much as an outside conqueror would be feared, but certainly gentlest. You preserve some important rights when you’re conquered from inside that you don’t when you’re conquered from outside.

00:57:55 – Art as a proxy for war

Dwarkesh Patel

Obviously, we remember this period for producing all these great cultural artifacts, all these amazing buildings, all this art. Then we’re talking about the precariousness of the prince, the constant wars, how they’re literally fighting all the time. How is there this surplus that is available for all these different projects?

You write in your book about how the older Lorenzo de’ Medici spends what would be today, because of the expense of building libraries and buying books, $30 million to build a library to educate his grandsons. How is there all the surplus available for education and arts and so forth in a period where everybody’s fighting everybody, and if you lose a war, your city will get, if not razed, at least the ruling faction of it will get killed?

Ada Palmer

Half of that answer is finance is incredibly profitable. If you’re the banking center, the amount of money that is flowing in is staggering. Big Wool, the big industry for Florence, is also incredibly wealthy. In the same way that Henry Ford becomes incredibly rich, in a period when a suit of clothing is something you save up for like buying a car, and everybody needs one, you can get very rich that way.

So, there’s lots of money. But, do you remember how it’s often said that the biggest impact per dollar for US defense spending is the Fulbright Program? Because diplomacy is cheaper than war. Sending a bright-eyed, bushy-tailed young graduate student out to a country to enthuse about its culture and make connections and make everyone feel positive does a lot more to avoid conflict, and also get help in conflict, than the same amount of spending on the actual army does. Dollar for dollar, diplomacy is cheaper than war.

They’re using the art to do diplomacy. So in one sense, if you’re not doing the art, you would have to spend more on the war. It’s not that the art is being made from a surplus of the war. It’s, “Oh, no, we can’t afford enough armies to actually defend us against France. Even if we spent every penny we have on armies, it would not defend us against France. But we sure can spend it on painting fleur-de-lis all over our seat of government and creating beautiful, expensive gifts for the King of France, so that when the King of France comes, he will feel like we are friends and we are giving him all of this cultural output. If we fought him, we would lose. But if we play the culture victory game, that’s cheaper, and we can try to win.”

Dwarkesh Patel

We talked about this last time, the experience of what it must have been like for a French diplomat to arrive at Florence and look at these people he considered to be nothing, not even descended from the Caesars, and they’re producing all this stuff.

When one goes to visit Florence now, the interest is in part because these are historical artifacts, because somebody made them 500 years ago. But if you’re seeing them at the time, this would be something either you thought only the Romans could have done that we can’t do anymore, or something that even the Romans couldn’t have done.

Ada Palmer

Right. They’re high-tech then. They’re like when we look at an incredibly impressive skyscraper that’s taller and more precarious and amazing than any past skyscraper.

Dwarkesh Patel

I think that is an underrated aspect of what it must have been like to be a foreign power evaluating Florence at the time.

Ada Palmer

Yeah. We have to remind ourselves that these are high-tech achievements as well as historic achievements. Also that this is a period in which backwards is forwards. That is to say, this is not a period that, like us, thinks of the future as where potential is, and that humanity might get better and better over time. The potential of humanity is recapturing Rome. Backwards is forwards. If we can get more and more like that, that’ll be better. That’s what we aspire to.

They do debate: can we surpass the Romans? Can we make things even better than the Romans? But it’s an “if,” it’s a debate. It’s not a “definitely, of course”. For us, it’s “definitely, of course.” We’re moving forward. We’re trying to build bigger and more impressive things. Even people who are cynical about progress will say, “Yeah, we will be more powerful in the future. We’ll be able to do more. We may use it to stab ourselves in the foot, but we will be more powerful in the future.”

For them, it’s very much: will we ever be as powerful as the Romans were? We don’t know. We can debate it. We hope so. We aspire to it. Will there be another Pax Romana? Will there be another universal peace someday? Will we ever achieve that again?

So when we look at something like Florence’s cathedral or Florence’s neoclassical buildings, we look at it and we know they’re imitating the past. So we don’t think of it as cutting-edge technology. But for them, cutting-edge technology is imitating the past.

Dwarkesh Patel

We talked about last time how both Machiavelli and the other umanisti, in the different ways they understood virtue, were trying to emulate the virtues that made Rome originally great. How much are they going off of just these random myths that Livy or whoever would write down about something that supposedly happened, where Brutus killed his own sons, or who was that guy who put his hand in a fire to show that the Roman people will be loyal and you shouldn’t fight us?

You look at actual Roman history, and it’s incredibly fucked up. We were just talking before we started recording about the life of Claudius and the period of the emperors and so forth, and surely this must have been known to them that actually…

Ada Palmer

Part of it is they’re zooming in on different emperors. When we want to make an HBO drama, we don’t make it about the boring, competent emperors who just do a really good job. Our society might be better off if we did. But the dramatic emperors where there’s lots of stabbing and lots of orgies make for good television.

Everybody curates their history. Often when you’re writing the history of your own culture, you pick the heroes. You look at a middle school history textbook, it’s going to celebrate the heroes of that country. If it’s trying hard to be unbiased, it will also acknowledge the faults, but the heroes are going to be in there.

When they are trying to create a handbook of what was, what stands out for them is what’s different from their present. Their present has plenty of tyrants. Their present has plenty of orgies. Their present has plenty of massacres. Their present does not have 70 years of peace. So that’s what stands out as different.

I think for us, some of the orgies and massacres stand out more because we don’t have as many orgies and massacres now, or at least not publicly that we know about. When we do expose that our leaders have been involved in scandalous orgies, we get very upset about it. But to them, they read about all of this, and they read about the successes and the stability from Trajan, Hadrian, and Marcus Aurelius, and they say, “That is alien to us. That we haven’t had in so long. That is what we want to have again.”

Dwarkesh Patel

So much so that when Gibbon is writing The History of the Decline and Fall of the Roman Empire, in multiple volumes, in the late 18th century he says that there’s never been a better time for humanity than during the era of the five good emperors.

Ada Palmer

Yeah and to the degree that medieval Europe can’t cope with the idea that these good emperors were also pagans and are therefore in hell… It’s where you get this gorgeous legend that Pope Gregory the Great summoned the ghost of Trajan and baptized his ghost so that he could go to heaven, even though Trajan is a during-the-persecution-of-the-Christians emperor. But they just love him so much, they can’t handle the idea that he would be in hell despite being a great Caesar.

So in the medieval world, it is canon that Emperor Trajan was posthumously baptized so that he could go to heaven, because he’s such a good emperor. Dante centers him in Paradiso as the ideal Christian ruler. But he wasn’t Christian. He was persecuting the Christians. But medieval and Renaissance Europe are very good at having their cake and eating it too—in terms of getting to pick and choose the best parts of the pagan world and the best parts of the Christian world when constructing their imagined antiquity—to have both and celebrate both at once.

01:06:41 – Florence, a city famous in hell

Dwarkesh Patel

Here’s something I’m confused about. Machiavelli makes a point of pointing out Cesare Borgia’s betrayals because of how remarkable they are. For example…

Ada Palmer

Ramiro d’Orco?

Dwarkesh Patel

Yes. Slaying the very deputy that he had tasked with being harsh—and as a result bringing peace to a region—for that harshness that he had delegated. Or inviting, as a gesture of goodwill, some people who are going to do a revolution against him, and then killing them all at the banquet.

But should we take the fact that he’s making a special point—”hey, take this kind of betrayal or this kind of deed as something you should consider doing”—as evidence that this was actually rare at the time? Maybe another way to ask the question is, to the extent that they are all Christians at the time, surely they really did believe they’re going to go to hell if they betray people or lie or break their oaths, right?

So just as you were saying a second ago about how capital punishment was actually less prominent at the time than we in retrospect think it to be, are these kinds of crazy political intrigues less common than these stories make them out to be?

Ada Palmer

Two halves to that answer, and I’ll do the second one first, the second one being about the religious one. They all believe in this religion that says, “If you do this, you’re going to go to hell,” and then they all do this. That’s something that this period really wrestles with.

Everybody is sinning and breaking their rules all the time: killing for honor, committing usury, lending money at interest. They’re all sinning all the time. They’re all doing these things that are against the rules all the time. People in the period do bring that up and say, “Hey, this is not okay.” This is one of the big focuses of Dante’s Commedia.

Dante in it says, “Look, when you do these things, you will go to hell for them”. He fills his hell with Florentines. There’s that wonderful line where he meets yet another group of Florentines, and he says, “Congratulations, Florence, a city famous in hell,” because he considers his Florentine peers to be particularly hypocritical.

As he goes through, we see Florentines especially in the sections for usury and for sodomy, but also heretics and unbelievers. All through he’s encountering his countrymen, including people he himself loves and respects. Because Dante is making this painful point of, “Guys, it says that if we do this, we go to hell. I’m going to make a book where that’s literally true.”

One of the chapters of Inferno that hits extra hard in his period and in ours is Canto V, where he’s encountering the lustful, and we see Paolo and Francesca. Paolo and Francesca is a story that was an incredibly popular love story at the time. There was a young, beautiful noblewoman who had an older, horrible husband. While he was away, there was also this wonderful, handsome, young nobleman who visited her, and they read the romantic stories about King Arthur and Guinevere and Lancelot. One thing led to another, and they committed adultery together. Then her husband came home and found them and murdered them both. Everyone loves this story. It is the ubiquitous love story. It’s their cultural equivalent to Romeo and Juliet, a touchstone story. People sing songs about it. Everyone knows this exciting love tragedy.

And he puts them in hell because they were guilty of adultery. It’s really shocking to everyone who has celebrated this love story. “No, if this is true, and this is our religion, then this is where they would be.” Dante is very stern and very strict and very unusual, and starts a lot of discussion of this question. “We’re breaking these rules all the time. Should we just take this more seriously than we have?” He says, “Repent, or you will all go to hell, my fellow citizens.” So they’re worried about that.

But another part of it is that Christianity as practiced then has much less of a focus on purity than the Christianity that especially America is used to, and also the Protestant-dominated parts of Europe. There was a big change in Christianity that comes in the course of the Reformation, primarily from Calvin, Calvinism, and then Puritanism, which has a greater focus on trying to live an unspotted and pure life. It’s the idea of, “we’re going to create a community of people who are all going to stick to the rules and live by them. And if you are a sinner and have broken these rules, you should be expelled from this community. You are impure, you are stained.”

That is not the way Christianity thinks in this period. The assumption is everybody sins all the time. There is no such thing as purity. Everybody sins every five minutes. Everybody is envious. Everybody is lustful. Everybody is slothful. Everybody will make these mistakes, and then you repent of them, and you feel sorry, and you do penance, and you make spiritual progress, and you are forgiven, and then you sin again. Everybody sins. St. Francis of Assisi sins. He had a big focus on himself as a sinner and was constantly self-flagellating despite being, in many ways, the most virtuous man in all of Europe, but stressing his own sin.

So one saint who’s super popular in the Renaissance who is not very popular today is St. Julian the Hospitaller, patron saint of murderers. He is the patron saint of murderers because his legend is an Oedipus-like legend. When he was born, he was cursed by a witch that when he grew up, he would slay his parents. He runs far away hoping that he will never encounter his parents and so not meet them. But eventually he feels homesick and comes home, and is tricked by the devil into slaughtering his parents, and he slaughters his parents. He spends the rest of his life trying to make up for it, going on pilgrimage, and then dedicating his life to running pilgrim hostels to help others be pilgrims. He is the patron saint for people who have committed murder and feel really sorry and need to live with it and repent of it.

That’s not the attitude we have toward murderers right now. Our cultural attitude toward murderers is, “That person is a murderer. They should be shunned. They should be locked in a box without the key, or they should be executed. They should be removed from society. There is no turning back from homicide.” But the Renaissance’s idea is sometimes you have to commit homicide, and then what’s important is that you feel sorry. You need to have a patron saint whose job it is to be a spiritual mentor for you, he too committed homicide. He committed a worse homicide than you did, because he killed his parents. If he went on a spiritual journey to recover from being a murderer, so can you.

There are dozens and dozens and dozens of icons of St. Julian all over Renaissance Florence. Everywhere you go and you see one, you’re like, “That was commissioned by somebody who committed a homicide and is trying to live with it.” This is a society that really thinks about sin as something you do, and then you pay for it afterward.

And people like Dante and Savonarola come to people and say, “No, this is not okay. You are perverting these things. No, you cannot put your family’s coat of arms all over the inside of a church, turning the church into an advertisement for your banking business when it should be a place of God. That’s inappropriate, and no, God will not forgive you for it.”

And society says, “Yeah, well, but God forgives maybe anything if we repent a lot.” So it’s a complicated, sophisticated hypocrisy that builds up a lot of apparatus to let the society’s actions be at odds with its religious precepts to that degree.

01:15:57 – The Prince was a job application to Machiavelli’s torturers

Dwarkesh Patel

I couldn’t get enough of Ada or of Machiavelli, and so there are a few more questions I wanted to ask you. Thanks for hopping on again.

Ada Palmer

Oh, my treat.

Dwarkesh Patel

We didn’t talk last time about the fact that Machiavelli was exiled. He’s writing these books in exile. We were talking about his diplomatic career, so maybe you can give a bit of context around how he ends up in exile, and what his plan is once he’s there.

Ada Palmer

Here we have to start with the fact that everybody who’s anybody in the intellectual tradition lives in exile for a while. Dante does. Voltaire does. Rousseau does. Thomas Hobbes does. Machiavelli does. More importantly, exile is a very common thing in Florence and doesn’t have the permanence that one expects. In Florence, exile means the people who are in charge of the regime distrust you right now. They want you out of the city, but they’re testing your loyalty. They’re testing whether you will stay true to them, and you’re told not, “Get out of the city,” like a Roman exile, but, “Go to a specific place. Go to London. Go to Bruges. Go here. Stay there, and we will send you instructions.”

You’re expected to act as a kind of unofficial official emissary for the government of Florence while in your exile. You’ll be asked to do diplomatic missions after a while. They’ll say, “Go talk to this person on our behalf,” or, “Go deliver this trusted letter.” If you’re good and you behave, then after some years of service to the republic, you’ll be recalled. So it’s a provisional exile. They pick a specific place to send you, and if you go and are good and do what they say, then after a while, they consider bringing you home. If you don’t–if you leave and you don’t stay where they said, if you run off to work for someone else—then you’re not allowed back in Florence anymore. You’re an exile at this point.

Machiavelli’s exile is unusual because they really don’t trust him. So they don’t send him to Bruges or London or Barcelona or the Germanies or any number of other places where he actually has political contacts. They send him to a middle-of-nowhere hamlet in the countryside outside of Florence in Tuscany, where there is nobody important and there is nothing to do. This isn’t a “go wait for instructions.” This is a “go rot and we’re testing whether you will faithfully stay and do basically nothing and be forbidden to talk to important people, be in isolation.”

When that exile is given, everybody expects that Machiavelli’s response will be, “Okay. They’re not giving me even a second chance. I’m going to run off and work for somebody else.” Because there are a jillion people in Europe who would love to employ a skillful classicist historian with military and diplomatic capacities who has political contacts in Rome and in France and has visited the court of the emperor. He could have worked for any number of cardinals. He could have gotten a very prestigious diplomatic job in any of a dozen courts. A Florentine historian especially is something that you absolutely want to hire to write a flattering history of your own family. For even a century before this, kings as far away as England had been trying to hire Florentine historians to come write about them.

So he could easily do this, and this is what is expected, and he doesn’t. Machiavelli says, “No. I’m going to stay, and I’m going to rot, and I’m going to write The Prince, which is my job application begging the new regime to bring me back and let me work for them and demonstrating my loyalty, and I’m going to send it to them and only them, them and my immediate friends. I’m not going to share it with anybody else.” Because Machiavelli is a patriot, and he will not serve any cause that is not his country.

No matter whether the pay at a royal court somewhere would be three times what he would ever get at home, that doesn’t matter to him. No matter whether this is the regime that just arrested, tortured, and exiled him despite him not having plotted against them, he wants to work for that. Because Machiavelli fundamentally is possibly one of the most patriotic patriots in Earth’s history. He will faithfully sit in the countryside and rot while begging to work for the people who ordered his torture, so long as they will recall him so that he can serve his country.

And this connects to the question we always ask about the target audience of The Prince, because his other work—his discourses, his histories, his comedic play—those were for public circulation. Those increased his fame. Those made important arguments. His history of Florence joined other important histories of Florence circulating, influencing the way people thought about politics. Not The Prince. The Prince is secret and proprietary, the secret sauce of how to maintain power.

He will not let any other power have that. It’s like a nuclear scientist with diplomatic secrets who is faithful to his country and will not sell out and let those secrets fall into other hands. Machiavelli knows that he has the beginnings of a new world of political science. He will only share that with the government of his country because he wants it to protect his country, and he will not serve any other cause.

This is why it’s so weirdly ironic to me that the reputation—the word “Machiavellian”—means “self-serving”, when Machiavelli himself is one of the most selfless men I’ve ever read about in the history of the Earth. He will give up and sacrifice career, diplomacy, fame, friends, the opportunity to even be in a city and have a nice day, to rot in the countryside to be faithful to his country. He would rather serve nothing and no one than give an hour of his time to advancing anything that is not Florence.

Dwarkesh Patel

You’re making the point that he is advocating a viciousness and a realism and a cynicism, but in service of protecting Florence, not in service of a generic prince of any generic principality.

Ada Palmer

Exactly, and he doesn’t let copies of it circulate to anybody but the rulers of Florence and his immediate scholarly, social, intimate circle of friends, people that he’s known for decades who are scholar peers who have discussed his ideas with him. That’s the audience of The Prince during his lifetime.

Dwarkesh Patel

Does he expect that at some point it will be more widely distributed? Is he writing in a way that suggests that? It is a literary masterpiece as well. I’ve only read, obviously, the translation, so I don’t know what it’s like in the original Italian. But somebody putting in that much literary effort into something that is just supposed to be a very pragmatic manual for a particular person seems a bit weird.

Ada Palmer

We have to remember this is a moment of transition from the manuscript to the print period, and also therefore an important moment of transition in what makes a written work important and how that written work is important to the career of someone who’s written it. It’s a normal thing in Machiavelli’s youth for a major important scholar like, say, Pontano, one of the greatest scholars of the previous generation, to be hired to write a handbook of princes that will exist in just one copy or three or four copies that are written for a specific prince.

For example, you have King Alfonso of Naples, the Spanish king who conquered Naples, Alfonso the Magnanimous, made famous for his vast patronage of arts and letters and for carefully cultivated personal anecdotes. There’s a moment when he was in the middle of fighting a war and a messenger rushed into his room, sweaty and covered with things, to interrupt the king’s morning time with his scholar friends discussing Plato. The king turned angrily on the messenger and said, “Get out. This is a place for men in togas, not for men in armor,” and refused to listen to the urgent message until he’d finished his hour of scholarly contemplation of the soul. As a result of which, he lost that battle but actually won the war. His reputation cultivated by anecdotes like that make him beloved.

He will pay a salary five times what the Republic of Florence will pay to hire somebody like Machiavelli. What does he hire them to do? He has a lot of children, princes and princesses, and he commissions a scholar to write a unique bespoke handbook of how to rule and use power for each of his children. These exist in manuscript only in one copy or three copies, and the addressee is the Duchess of Ferrara, who is a daughter of King Alfonso. That book is never intended to circulate. It’s intended to be private guidance for her and for her to perhaps pass on to her sons and daughters.

Meanwhile, the author’s fame is magnified by being told the special bespoke handbook of princes cultivated secretly for this important princess was written by so and so. That’s so cool, and letters circulate and let you know that it’s happening. In the same way that a scientist might become famous because we know he’s developing cool proprietary technology that only his government has, but we know that it’s happening, we have to think of these books as proprietary technology.

In that sense, it’s not an unusual thing to write a book with an audience of one, or an audience of one and her immediate circle. This is also one of the moments where the handbook of princes also means for women. The title of that book for the princess who becomes Duchess of Ferrara addresses her as a prince, because prince is a gender-neutral word at this point. It’s lexically masculine in terms of ending, the same way a table is feminine, but prince is used for men and women. Even Queen Elizabeth is Prince Elizabeth at this period of her life.

Dwarkesh Patel

That is so fascinating.

Ada Palmer

We have trouble wrapping our heads around the idea of writing a book for an audience of one. It’s just not what a book is to us.

Dwarkesh Patel

The funny thing is, I think we are entering a new era where that might be once again possible. It already is somewhat true, where at least half of the words I read on a given day are generated specifically for me and nobody else, because of AI. Obviously, AI is not capable of writing something which I think would be a literary masterpiece that everybody would want to read if they had access to it just yet. But eventually it will be. So it’s interesting to consider that as this knowledge progresses, it would bring us back to this era of bespoke scholars dedicated to a particular prince.

Ada Palmer

It’s important to remember that that never went away. Two halves of that. One, for ages it’s been true that half of the words we read every day are bespoke only for us, because they’re email. They’re letters. They’re the correspondence back and forth which has the audience of one, the addressee, and that’s the majority of what all of us read and write in our lives.

It’s also always been the case that in the halls of power, there are book-long things with an audience of one or an audience of five. There are historians and other scholars and scientists whose job is to provide that 100-page report on the history of Syria, to be given to a committee of Congress, where these nine senators or these nine congresspeople need the background on what’s happening so that they can understand a current events thing.

There are historian friends of mine who work for the Department of Defense Intelligence, who produce these book-length research projects with an audience of five or an audience of eight or an audience of a couple dozen, because it is the bespoke proprietary knowledge needed by the government at that moment. Sometimes it’s technological knowledge, but just as often it’s going to be historical knowledge of, “Here are the important rivers where military things are likely to happen,” says the historian who knows the history of this stuff.

Dwarkesh Patel

That actually brings up the question of how many such tracts through history, which are of the quality of The Prince—as original at their time as The Prince is and as wonderfully crafted and so on—have been lost to history? Maybe one way to answer this question or think about it is to talk about how The Prince itself went into mass publishing.

At some point in 1532, the Medici pope allows for its publishing, and then 27 years later, it is censored by that same papacy. So how does this book that Machiavelli himself did not want out in wide circulation end up in wide circulation and then stop ending up in wide circulation and then end up in wide circulation again?

Ada Palmer

It goes in and out and in and out, like a lot of important works. I’ll give the zoomed out answer and then the zoomed in answer to that question. It is often the case that a work which contains radically unusual ideas will drift along being not particularly zoomed in on by society and not widely read, until it hits a moment that the new questions being asked in that century or that decade are answered by something in that text. Then suddenly everyone will start reading it.

A different example of this, probably well known to the audience, because everyone here is a cool, smart, learned person, is Lucretius’ On the Nature of Things, De rerum natura, which is our best capsule of ancient atomism and the atoms and vacuum theory of matter. It’s written around the BC/AD turn, and drifts along being not very important for ages, until the 1600s when we’re getting the first ideas of germ theory of disease, very interested in new science. Suddenly it gets 30 print editions and is all over the place and influences science, and is even more influential in the 19th century when we’re interested in atoms and cells. So a book can exist for literally 2,000 years or close to it, and then suddenly answer the questions of that decade.

In that sense, The Prince will drift along and be not very important for a while. Why is it first published? It’s first published when Machiavelli’s still surviving relatives want fame for the family and fame for their beloved now dead kinsman. Here is a work of his that hasn’t been published yet. They ask for permission because this can spread his fame. It’s also dedicated to members of the Medici family, so the Medici are like, “Yeah, we get fame for publishing this thing too.” They don’t think as seriously about the power of its contents as its author did.

And so it’s one more book that can spread the fame both of the Machiavelli family and of the Medici family, and it goes around, and people are like, “Oh, that’s actually full of fairly scandalous ideas. Hmm.” That’s how it then ends up on the index as book censorship kicks up as a result of the printing press. Mini thesis: every time there’s a new information technology, there’s a subsequent wave of censorship to try to censor the new technology, and a bajillion books get banned all at once. Machiavelli’s is not a particularly prominent example among this. The index of banned books that contains his work carefully differentiates between the dangerous books by arch-heretics and the slightly dangerous books by meh people, and arch-heretics are in all caps.

I remember when I was first reading through one of these indexes, I was so excited to flip through and find Machiavelli, and there he was, not in all caps, and I was so angry. I was like, “What’s wrong with that? He should be in all caps.” But all the all caps people are Luther, Calvin, Zwingli, a bajillion Protestant theologians you’ve never heard of. All-caps arch-heretic status is reserved for Protestantism in this period. Machiavelli doesn’t catch on until later. So he’s censored in a wave of censoring everything, when there’s a big censorious wave, and then it diminishes and goes up and down.

The second zoomed-in half of that: if we say Lucretius becomes exciting when people want to know about the germ theory of disease, when does Machiavelli become exciting? Machiavelli becomes exciting first in the aftermath of the publication of Hobbes’s Leviathan, because Hobbes’s Leviathan hits European thought like a truck full of bricks. It has this incredibly persuasive, gorgeous reasoning that lands you on a terrifying vision of what humanity is and a terrifying vision of what God is that people find very scary, but also incredibly persuasive. It’s no exaggeration to say that in the aftermath of publishing Leviathan, there’s a 40-year period where the sole goal of Western European philosophy is coming up with a good way to refute Hobbes.

At that moment, they say, “Okay, Hobbes is using a lot of logics about politics and about history that sound like Machiavelli.” He’s doing these utilitarian consequentialist analyses of “if we do this, there’s that result”. He’s analyzing the origins of government as if there’s no divinity setting it up. He has this man in a state of nature inventing government instead of God from on high telling Adam, “Here is how you should organize the world.” So they say, “Okay, Hobbes is the monster. Hobbes is Leviathan the great, or the beast of Malmesbury,” as newspapers call him during his lifetime.

How do we refute the monster? Let’s look at the daddy monster that spawned the baby monster. If we can read Machiavelli and find holes in Machiavelli, maybe we can use those to refute Hobbes. So Machiavelli is suddenly useful not to people who sympathize with him, but to people who see him as an enemy and want to use him to try to defeat what to them is the greater enemy. So he surges in popularity at that point.

A different surge happens in the 19th century, and it’s not until the 19th century that Machiavelli’s Prince becomes a major global staple that you would put in a great book series. In the 19th century, in the aftermath of the Enlightenment and the Enlightenment’s revolutions—the American Republic, the French Republic, the transformations and democratic movements that are happening in lots of other governments—people want new ways to think about politics, and they want to think about politics in separation of church and state.

If you want to think about separation of church and state, which is a new Enlightenment-era value, what do you need? You need an apparatus for thinking about politics and ethics that doesn’t depend on God being part of it. The vast majority of political treatises available to humanity at that point have some sort of entanglement of religion with politics at their root, but Machiavelli doesn’t. Machiavelli is this early foundational “what if we think about government in a box without plugging into religion? What if we just think about government operating by itself and its earthly consequences?” It’s incredibly useful in the 19th century for developing a statecraft for separation of church and state.

It’s also useful for Italian nationalism to celebrate and claim, “Hey, we invented separation of church and state. Here’s Machiavelli, the first modern man. He’s our bid at ‘Italian culture invented modernity’ via Machiavelli.” At the same time, England is saying Francis Bacon is the first modern man because he invented the scientific method. At the same time, France is saying René Descartes was the first modern man because he invented logical reasoning and modern principles of logical deduction. There’s a competition in the 19th century, a nationalist one, of different countries that want to claim their cool thinker as the first modern man. Machiavelli becomes one of Italy’s big bids for the first modern man because he came up with separation of church and state. It’s a phrase that Machiavelli would not have recognized if you said it to him, but he would have thought about it for a long time, decided it was cool, and then written letters about it.

Dwarkesh Patel

Can I try out a counter-thesis just so you can dispel my confusion? One reason why Machiavelli might have gained a special significance in the 19th century is that now that you have these republics in the world, there’s a question of how you make sure that they are maintained. That is really the question that at least the first third of the Discourses is obsessed with. But one of the ways it says that you do this is by having a religion that people take into very strong consideration.

I think he says at some point early in the Discourses that more significant than Romulus in the founding of Rome was Numa, or whoever it was who was the prophet who gave the Roman gods and the Roman religion some legitimacy. Then it is this legitimacy and the fear of offending virtues, because you believe in some god that will punish you, that motivates people to act in a way that defends the republic.

He gives the example of Scipio after the battle in which Hannibal absolutely destroys the Roman armies. The people are about to flee Rome as a result because they think Hannibal’s coming. Scipio himself, with his sword, goes down and says, “Swear to our gods that you will stay and defend our homeland.” Just having them give the oath in that moment is enough to convince them, “Hannibal can’t be worse than the gods, so I have to stay here and defend our republic.”

It seems like he thinks that religion is super important to the legitimacy of the state.

Ada Palmer

Agreed. He’s thinking about it in a way parallel to the way late 18th-century, 19th-century figures are also thinking about it. We have to separate the institution of religion from the psychological effect of religion on the populace. The useful example here is Thomas Paine. We all know Thomas Paine’s “Common Sense”. Thomas Paine does a lot of thinking about the foundation of the institutions of the US. Thomas Paine is a deist and a radical. He has lots of treatises about how the most destructive force in the world is institutional religion. Whether it’s Catholicism or the Church of England, these institutions are giant, centuries-old or millennia-old conspiracies to control your mind and steal your money, and are incredibly pernicious to everything.

However, he says, religion is vital to citizenship because it is what makes people be good and is what makes people fear laws and want to obey the laws. So, says Thomas Paine, every country must have religion, and religious education must be mandatory in schools, but it doesn’t matter which religion. Thomas Paine advocated mandatory religion in total indifference to what religion it is, with the idea that fearing God and posthumous punishment is necessary to make a citizen, in a practical sense, willing to obey law.

Notice how that is Paine thinking in a utilitarian way about the psychological effects of religion being there. It’s very different from the older view that the state and a state religion are entangled with each other. The state promotes this state religion because it believes it to be true, and we’re going to have a Christian nationalist or Catholic nationalist or Roman paganism nationalist religion that advances X against others.

So Machiavelli is absolutely thinking about the psychological effects of a religion on the people. He has that wonderful analysis in the Discourses of the utility of Roman religion. He talks in one really striking and memorable passage about how Roman religion says that your ghost depends on being remembered. This is out of the Homeric tradition. Your ghost only retains its identity to the degree you are still remembered on Earth. If on Earth your name is forgotten, your ghost forgets its name.

This is not a “your ghost is okay forever,” like in Christianity. It is a “your ghost depends on being honored by your descendants on Earth.” When you are forgotten, your soul becomes an empty, mindless, wandering shade. Therefore, you have an incredibly strong incentive to be remembered by doing great deeds, especially sacrificing yourself for your country, because then your name will be honored for as long as your country lasts. Machiavelli says this is one of the big motivators that makes people sacrifice themselves for the state in ancient Rome, because then they’re guaranteeing their good afterlife.

While Christianity, he points out, says all that matters for a good afterlife is being pious and then ideally being martyred. You have no incentive to sacrifice yourself for your state. The safety of your afterlife is guaranteed by your interiority. This is going to encourage a citizen to sit in a box and be a monk, not to sign up for the military and defend his country. So, says Machiavelli, Roman religion was much better for patriotism and political stability than Christianity. But he says at the end of the chapter, “Christianity has the advantage of being true,” period, end of chapter. You’d say, “Hello, Machiavelli, we know that you had the mandatory subscript there.”

So think about Thomas Paine and Machiavelli in parallel. They’re thinking about the utility of religion for forming a citizen, but they’re not thinking about “this religion is true, we are doing God’s work, we need to craft our state to match the values of our religion,” which is what a theocrat would argue.

What you have is separation of church and state with the expectation that religiosity will be there. It will affect the people, it will affect the citizenry and their behavior. You need to think about it. You need to decide whether to cultivate it, but you need to think about it in the same neutral way you think about cultivating literacy skills or math skills in your citizenry. What skills do we want our citizenry to have for them to be well-informed citizens? What do we need? We need religion and we need good newspapers so that people are up on the news and can vote prudently. You are evaluating those things side by side from a utilitarian standpoint instead of “this religion is true, it is the obligation of our government to advance it, and our government expects to receive divine blessings if we advance the correct religion, and divine curses if we don’t.” It’s a radically different way of thinking about religion, while still recognizing it as a powerful factor affecting the psychology of the populace.

01:41:39 – During the Renaissance, original ideas had to be couched in antiquity

Dwarkesh Patel

That makes sense. Last episode we were talking about the psychological impact on scholarship of having books be so expensive and having to meditate on the same copies that are available in one library. Maybe Machiavelli’s the strongest example of this, where maybe through his life we’re seeing the impact of the printing press diffusing and making printing cheaper. But early on in his life, it’s still not been that long since Gutenberg came up with the first printing press. As a result—correct this story for me—his dad has to do months of drudge work indexing Livy in order to get a copy of Livy.

Ada Palmer

In the infancy of printing, books are scarce and few. For example, one of my favorite manuscripts ever that I’ve worked with is a copy of Lucretius in Machiavelli’s hand. He copied out the entire poem. This is in the Vatican library.

But what’s really neat is he copied the text from a printed copy. But as he copied it, he integrated into it corrections and improvements of errors in that one, taken from a manuscript copy, so that what he produced was better than either the printed version or the manuscript version. And then he made his marginal comments as he went.

But notice this is somebody who, even though print copies of this book exist, is so much in the manuscript world that he’s happy to spend months probably copying out and making his own custom improved version of this text that he can then work from, even though inevitably new print copies will come out in a few years that may have the very corrections that he’s working with. But he isn’t going to wait for that, and he’s not sure, so he makes his version. So he’s from this moment when print and manuscript are parallel technologies being used at the same time. The very people who are buying the first printed books are also producing manuscripts imitating those printed books and influenced by those printed books.

Dwarkesh Patel

I want to think about the impact that having this copy of Livy—which presumably is one of the very few books that young Machiavelli had access to—has on his intellectual development. We have this mode of scholarship at the time. Why does he spend two decades writing Discourses on Livy? Unlike us—where we can go through an audiobook a week or read our Kindle at night— he’s presumably just reading this book again and again and again, and is trying to connect it to the events he’s seeing in his own life on his 10th reread. So that I feel is very interesting psychologically in understanding how scholarship and intellectual thought must have been different at that time as compared to now.

Ada Palmer

Machiavelli can easily access other books by visiting friends, by asking to go to the library of his Medici patrons when he’s working for the Medici, of his Soderini patrons when he’s working for the Soderini. But that’s different from having it at home and being able to have it at your bedside and look at it at all hours and have this intimacy with it, and it’s your father’s copy and it’s your copy.

There’s another part of that though, and this is weird for modern people to understand. In the Renaissance, there is so much enthusiasm for antiquity. Antiquity is the cutting-edge thing. Antiquity is where it’s at. Antiquity is how we’re going to end the chaos of the previous world and have this new world where we’re basing everything on ancient Rome. There’s going to be peace. There’s going to be a golden age. It’s all coming from and imitating antiquity.

Therefore, if your book is a comment on an ancient, it is going to be way more popular and sell way better, and people will care more and think more of you than if your ideas are original. Nobody wants original ideas. Original ideas are out of vogue. Original ideas are dead. All ideas need to be from the ancients.

So a Renaissance scholar will bend over backwards to pretend that his beautiful original ideas are actually Livy or are actually Plato, or to couch them as a commentary on these things. That’s going to have a way bigger audience and be more popular and taken more seriously than if it’s original. So there are points where Giordano Bruno, in his commentaries on Aristotle, claims that Aristotle says things absolutely Aristotle does not, the opposite of what Aristotle says. But if he claims it’s Aristotle, people will take it more seriously.

The most extreme version of this is the brilliant and fascinating figure of Annius of Viterbo, who Tony Grafton has a great book about. Annius of Viterbo had this radical vision of how he wanted to rethink history and faked ancient texts. He made them up. He faked archaeological digs. He would secretly bury artifacts and then dig them up to great drama. And he forged antiquities to create this book that advanced his visionary original idea of ancient history, because if he pretended he got it from antiquity, people would take it more seriously than if it was an original book.

So Machiavelli’s Discourses on Livy are his big bid to have a popular, important, prestigious thing, because discourses on Livy are a bigger deal and more important and more interesting to everybody, and more likely to sell and get attention, than a Florentine history or a treatise of original thought on princes. Who wants that? That’s a very niche kind of thing. “Discourses on Livy, oh, exciting, we have to have this.”

This goes on for the next century. For example, huge amounts of radical political thought, including, believe it or not, commentaries on Machiavelli, happen in the footnotes in editions of Seneca and Livy. The text of Seneca will be a small square in the middle of the page, and then there’ll be these masses of footnotes and commentary. Huge original moments of political thought for the entirety of the 1600s are going on in wars, in footnotes, in editions of Seneca. But it’s not original thought. It’s all about Seneca, because that was what was in vogue then. The vogue of scholarly stuff shifts fast and is very interesting.

This is one of the weird reasons that Renaissance philosophy and Renaissance innovative thought—with the exception of a couple of oddball works like The Prince—gets pushed out of the history of philosophy, especially in the 19th century. Because when you get to the 19th century, the vogue is that everything has to be original. The philosopher’s ideas should be born like Athena, fully formed from the head of Zeus. The ideal philosopher lives in a cabin by the raging sea, contemplating in the wilderness. What they want is original treatises.

If you look at a 19th-century historian of philosophy, they’ll say, “in the Renaissance, there was almost no original thought.” There was Machiavelli’s Prince, and there was maybe a little bit of Giovanni Pico della Mirandola’s Oration on the Dignity of Man. (We have since proved it’s not an oration and it’s not about the dignity of man.) These things are the few lights in the darkness, and everything else in Renaissance philosophy is... Here’s a quote from a philosophy department person who actually said this to me: “The Renaissance is 200 years of people being wrong about Plato.”

A lot of people look at it, and you pick up Ficino, and he’s like, “Plato said these things.” You’re like, “No, Plato totally did not say those things at all. That’s absolute gibberish. No, Plato didn’t say that. What are you saying, Ficino?” If you think Ficino is what he says he is, a commentary on Plato, then indeed the Renaissance is 200 years of people being wrong about Plato, being wrong about Livy, being wrong about Aristotle. But if you realize that their style guide requires original thought to be presented in the form of a commentary on an ancient, what it is is 200 years of original thought using the ancients as the trellis up which the rose climbs in order to bloom.

When you restore that and recognize that in order to get at the real Renaissance, you need to not read the goofy outlier works like Machiavelli’s Prince, which present themselves as original—which is a weird thing to do—but read the commentaries on Livy. That’s where the original stuff is hidden, by pretending and claiming and sometimes sincerely convincing themselves that this is the secret coded true meaning of the ancient thing.

Like Ficino, the translator of Plato, definitely genuinely believes that all of the incredibly original cosmology and magic that he’s figured out is secretly coded in Plato, and he’s wrong. It’s not. It’s so adorable that he really, really believes it is. But what it is is an incredibly original vision of the universe that he got from reading Plato and thinking hard about it and combining it with other things. So he presents it as commentary on Plato, commentary on Dionysius the Areopagite.

That’s core to why this is a discourse on Livy, because a discourse on Livy is what a scholar is supposed to be doing. All the other things Machiavelli does are second-tier weird things for a scholar to be doing on the side of discourses on Livy.

01:50:44 – Why copyright began with the Inquisition

Dwarkesh Patel

So adult Machiavelli is now seeing some of his work start to get mass-produced. What is his reaction to this?

Ada Palmer

At first excitement, but also horror, because Machiavelli is facing this fascinating moment in the history of being an author when printing has come into being, but there isn’t copyright yet. In the manuscript period, there’s no such thing as copyright. If you find out that someone has made a copy of your book, you say, “Oh, thank God. There’s another copy of my book.” That reduces the chances of it being completely destroyed in a fire. Making one copy of a book is six months of incredibly difficult labor. You’re just grateful every time a text is reproduced.

But when it comes to printing, then you have this experience, which Machiavelli is one of the first men ever to have, of finding out that a local printer is printing a work of his without ever having asked him, without ever having talked to him. He looks at it, and it’s full of typos and minor errors. He’s panicking in these letters and saying, “Oh, no, everyone’s going to think I’m a bad scholar. There are all these little mistakes in the text, and they aren’t me. They’re the compositor having made typos when setting it up, and no one will know that. They’ll blame me, and it’ll destroy my reputation. What do I do? There’s nothing I can do because there’s no legal process and no legal recourse. Printing has just come into being.”

It’s neat seeing him and friends writing to each other about, “What can I do about the fact that this printer has printed my book without asking me?” There is no law. There is no apparatus. There is no anything. His friends are like, “Well, write letters to everybody who matters and tell them that the typos aren’t you. That’s all I can suggest,” because they don’t have the idea of authorial copyright yet. It’s going to come in the next couple decades.

The weird thing is how this gets entangled with censorship. Copyright and censorship are born together in Machiavelli’s world, counterintuitively, from the Inquisition. When the Inquisition begins book censorship after 1515, which is during Machiavelli’s lifetime, the policy that the Catholic Church promulgates is: before you may print any text, you must take it to an authority licensed by the church to do this—meaning an inquisitor or a bishop—and they must read it and give permission for it to be printed. This is so that they can make sure there isn’t heresy in it. So all books are effectively born pre-banned until you get permission for them to be printed.

In return for this, you get a monopoly license, and only the printer that took the book through the process can print it. You may now use the actual Inquisition record of you having gone through censorship as the document to prove that you and only you have the right to print the book. Therefore you can sue people for plagiarizing it or printing an unauthorized edition. So the very first version of copyright is the Inquisition.

Places outside the Catholic world then, like England, look at this. There’s actually popular demand in England for censorship, when they say, “Hey, we need what the Inquisition does, because the Inquisition is so cool. They let printers have a monopoly on printing a book, and they let authors deny print permission. We need something like that.” The very first version of what is not yet copyright passed in England—which is of course the ancestor of what applies in all Commonwealth nations and in the US—was originally an imitation of the Inquisition. It was: you need a license before you can print your thing, and then in return you get a monopoly.

Later, when there was a freedom of the press push—and by later, I mean this is happening over the course of the first half of the 1600s, so about a century after Machiavelli’s death, it takes a century for all this to get ironed out—the first version of copyright law is them basically saying, “Okay, we’re going to keep the copyright half of censorship while getting rid of the censorship half of censorship, or changing the censorship half of censorship.”

But it’s all born out of the Inquisition having met this weird demand that you feel in Machiavelli, where he’s like, “They printed my book. They did a bad job. There’s nothing I can do. Help. Authorities, give me some way to do something about this.” So that’s where you can feel Machiavelli as one of the first generation that needs copyright, which will then be born in the aftermath.

Dwarkesh Patel

Fascinating. And what was the Inquisition’s incentive to enforce the author’s prerogative on the text?

Ada Palmer

Partly the Inquisition does it because that encourages authors to come to them. It makes people much more willing to collaborate with their process. But also, think of an individual Inquisitor as an individual person who lives in a place and needs to have relationships in that place, and needs to have an income, and who is not usually getting enough to live on from the Inquisition itself.

If you’re working for the Inquisition, you’re an officer of the Inquisition, you’re probably a Dominican monk. You get some support from the monastery, but you have reason to want money, and you have family, they want money. You’re as pragmatic and self-serving as any other average human. So the fact that people want to have this positive relationship with you, they might gift you some bottles of wine in return for you being extra generous in your reading of their text.

They also have to negotiate with authorities. The Inquisition wants us to think of it as very centralized and very monopolar—the Inquisition, the Vatican, it controls everything—which is completely untrue and is propagandistic. The Inquisition is overseen by a whole bunch of isolated guys who are in isolated towns, and it takes weeks or months to even communicate with the Vatican. They’re making their own decisions.

For the most part, they don’t have their own large amount of funding. They don’t have their own officers to jail people. They don’t have their own jails. They don’t have their own authority to arrest directly. They get all of those from the local government. They collaborate with the local government, which means if the local government likes them and is pleased by them, and is like, “Ooh, the Inquisition, I can use this to scapegoat my enemies,” then the local government will drown the Inquisition in funding and give them all the guards and all the incentives they could want.

So when we hear about the infamous Spanish Inquisition, which everyone was expecting me to mention, the Spanish Inquisition is infamous because Ferdinand and Isabella of Spain really want to scapegoat the Jewish and Muslim populations that they’re anxious about. So they throw money at their Inquisition and really cultivate and make it big. That’s coming from them. It’s not coming from Rome.

Meanwhile, if you’re in somewhere like Florence, where the duke—if it’s early Medician ducal Florence, right when this is happening—is a Medici, he’s in deep with the weird Ficinian Platonic soul projection magic people. He’s an intellectual radical descended from intellectual radicals. His court is full of intellectual radicals. Here you are, the Inquisitor, and you’re like, “Your Grace, can I arrest this guy?” He’s like, “No, that guy works for me. You can’t touch him.”

You can only arrest as many people as the duke will give you funding for, or the local republic will give you funding for. So you need to please the local government if you’re the Inquisitor. We have letters of Inquisitors complaining, “This is a really liberal duke. He’s protecting all of these heretics around him, and there’s nothing I can do about it because I depend on the local authority for my ability to do stuff.”

So this is a really bizarre comparison, but think of the Inquisition operating kind of like Doctors Without Borders. It’s not the government. It’s an international organization that’s set up to try to achieve a goal that it believes is beneficial in different places. But it’s only as strong as, or as weak as, the government’s willingness to collaborate with it. If the government collaborates with it, it can be enormously powerful in an area and do a lot.

If the government is hostile to it and starves it of resources and doesn’t let its people in and insists on pushing it out, then you can get bubbles where the Inquisition is nearly impotent. Every time they want to arrest someone, they have to go to the duke’s agents, and if the duke’s agents keep saying no, they can’t do anything. What this really creates is bubbles of privileged access, where if you’re in with the government, you can be as heretical as you like, and the Inquisition can’t touch you.

This is also a lot of how homosexuality operates at the time. If you are in the protection of a powerful person, they can prevent the Inquisition or other officers of the church from getting at you. They just won’t do it, and they’re more powerful than those agents are, so they can’t touch you.

Machiavelli was, I would say, very definitely solidly bisexual, in that this is a man who recreationally had boyfriends and girlfriends throughout his life that he writes to. We have homoerotic poetry. We have heterosexual poetry. He’s definitely very excited by both sexes. He has a lot of gay friends.

He and his gay friends are writing back and forth about how at this particular moment in Rome, one of the agents in charge of Rome’s enforcement is really cracking down on homosexuality. Therefore all of their gay scholar and artist friends are rushing to get jobs working for cardinals, because if you work for a cardinal, nobody can touch you. Almost all of their friends have succeeded in getting jobs working for cardinals, except for one. So he has resorted to hiring two female prostitutes to hang out with him all the time and make him seem straight, by having him hang out with sexy courtesans, to defend himself against charges of homosexuality.

Heresy and homosexuality operate very similarly in this period. They’re both forbidden by the same things and policed by the same structures. So if you work for the cardinal or you work for the duke, you can be doing very radical magic, radical philosophy, radical politics, radical sexuality, and nobody in authority can touch you because authority’s trumped by a higher authority that is protecting you. This is part of the patronage system.

Dwarkesh Patel

How does that come back to the copyright?

Ada Palmer

The way that comes back to copyright stuff is that the Inquisition needs to please local authorities in order to get to operate at all. So the Inquisition will therefore try to figure out things that will please local authorities. If a book is being presented for publication that has a recommendation letter at the beginning written by an important political figure, the Inquisition will push it through. When printing presses and authors say, “Hey, can we have this be a monopoly license?”, figures like Machiavelli realize we could ask for, “Hey, you’re giving us permission. Can you deny everyone else permission?” The Inquisition immediately realized, this is a great way to get publishers on our side, to get authors on our side, and to get their bosses on our side, because we are protecting the book that is important to the duke because it’s dedicated to the duke, or it’s dedicated to his grandfather.

The Medici give permission to print The Prince partly because it’s dedicated to a member of the family, and it celebrates their fame. They want to be able to control its quality and make sure that it’s published in good quality and that it always has that dedicatory letter at the front. They have an incentive to control what we would now think of as copyright. The Inquisition, wanting to please them, has an incentive to give them that control.

02:02:12 – Machiavelli wasn’t Machiavellian

Dwarkesh Patel

To close off, do you have some sense of how to think about why Machiavelli’s remembered so differently from not only what he wrote, but why he was writing?

Ada Palmer

Sometimes in the history of thought, there are authors who become separated from their work. You have a parallel where there is the actual content of what the person did and said, and separately there is the idea of this person. In the case of Machiavelli, we have Machiavelli the patriot, Machiavelli who did all this work, and separately we have “Machiavellian” — “the murderous Machiavel”, as Shakespeare calls him. Old Nick, which is a nickname for the devil but became popular because of Niccolò Machiavelli. Old Nick, literally a synonym for the devil.

He splits, so that the idea of Machiavelli—the Machiavellian villainous figure that Shakespeare’s Richard III invokes as someone he’s modeling himself on—is useful to people as a character, as an idea. It’s the idea of the scheming politician who is probably atheistic, definitely self-serving, and who wants nothing but to advance himself in power. Of course that isn’t the real Machiavelli if you read the work. The real Machiavelli is not about advancing yourself. It’s not a manual for getting ahead. It shouldn’t be shelved next to How to Win Friends and Influence People, because it’s a manual not of how to gain power, but of how to keep power. If you have a government and want it to be stable and protect the people’s lives, do this.

But the idea of the murderous Machiavelli is very exciting, and this happens at other times to other intellectual figures. It happens to Thomas Hobbes in the phase that Thomas Hobbes is the Beast of Malmesbury, and the idea of Thomas Hobbes separates. It happens fascinatingly to Spinoza, an important radical Jewish thinker of the later 17th century. Spinoza is a neat one, because when you actually read Spinoza, he’s really warm and sweet. Like Machiavelli, he’s passionate and cares about people, and in his case is an incredibly pious theist. He’s a monist. He believes the entire universe is the body of God. You are a part of God. The table is part of God. The camera is part of God. Everything is God. Isn’t that great?

But a fact about Spinoza—and I know this feels tangential, but it’s not—was that he was the first person in ages and ages to be targeted with the Jewish equivalent of excommunication, the ceremonial, “Your radicalism is too radical. We are expelling you from the community of Jews.” It was such a rare ceremony that the Jews of his region actually had to send somebody traveling all around Europe to find a Jew who knew the ceremony, because it was so incredibly rarely done.

The fact of that spread around, and people had the idea that Spinoza must be even more weird and heretical than any heretic if even the Jews would expel him. The idea of Spinoza the arch-heretic becomes a character. Everyone talks about Spinoza the arch-heretic, and then you read him and it’s nothing like it.

But sometimes the character is useful. The thought experiment figure of Machiavelli the villain is useful for our philosophy. We like to talk about, “what is a Machiavellian self-serving politician?” What would they do? This has a separate life from Machiavelli’s real ideas, to the degree that all the way through the 16th century, there’s these amazing discussions of Machiavellianism in Spain. They’re talking about the Jews as Machiavellian and Machiavelli as the prince of the Jews. You’re like, “Machiavelli was in no way a Jew at all.”

But what they mean by Machiavellian and by Jewish is somehow the political thought that is undermining our good Catholic Spain. So Jewish and Machiavellian can become synonyms, mad as that is for us. Because for them, both of these are labels for the sinister underground of thought, and now we’re talking about the sinister underground of thought.

The idea of Machiavelli as the villain is itself enchanting and interesting. As we look at when Machiavelli is invoked in the modern day—when The Prince sits on the shelf and it feels like something exciting to buy and to read and to think of as a manual of getting ahead, when having it on your shelf makes it feel like you’re participating in the idea of strategic advancement and rationalism—that’s much more Machiavelli the character, Old Nick, than it is the Niccolò Machiavelli who faithfully sat in exile, willing to give up wealth, fame, society, the ability to visit his wife, anything, in order to serve his country.

To me, I think even more fascinating than looking at either Old Nick, the fictitious Machiavellian villain, or Machiavelli the patriot, is to look at how did we double-image this? What is the fascinating tendency of our society to take something real, powerful, exciting, intimate, and then say, “But we can also make the character,” and the character is itself interesting.

So if you take away a main message from this with Machiavelli, it’s that Machiavelli, the character of thought experiment, is an important backbone of our society. We use him as we think about politics. Machiavelli, the actual innovator, is a different backbone of our society and how we think about politics. If Machiavelli can be two such different things, Old Nick and Machiavelli the patriot, so many other things we encounter in life have actually been teased apart by our social utility and made into multiple things which are useful to us in different contexts.

If you have The Prince on your shelf, read it and remember it was written by somebody who was willing to give up anything to serve his country, and you’ll see a very different Machiavelli come through.

Dwarkesh Patel

I think that’s an excellent place to close. Ada, thanks so much for hopping on.

Ada Palmer

This was a pleasure, as always. I hope it won’t be the last time.

Dwarkesh Patel

I hope so, too.

There’s a bunch of important questions about how we deal with AI that only economics can answer.

What is the optimal way to tax and redistribute the wealth that will be generated? How should countries not in the AI supply chain index into the gains? Is there any world where inequality doesn’t explode?

It might seem like these questions have obvious answers, but the first thing economics teaches you is that your intuitions can often be entirely wrong.

It was very helpful to chat through these things with Alex and Phil.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Jane Street invests heavily in turning smart people into exceptional researchers and engineers. In addition to their apprenticeship model, Jane Street runs lectures and bootcamps in their in-office classrooms -- managers clear their teams’ schedules to encourage attendance. If you’d like to work at a place that takes learning this seriously, Jane Street is hiring. Check out their open roles at janestreet.com/dwarkesh

• Google’s Gemini Omni has incredible video editing capabilities -- you can upload a video and have Omni change the background, adjust lighting, or add specific elements. But Omni is also a preview of how future frontier models will be trained -- fully multimodal on both input and output. You can try it yourself in the Gemini app at gemini.google or in Flow at flow.google

• Cursor used targeted RL with textual feedback to help train their Composer 2.5 model. One of their researchers, Sasha Rush, gave me an impromptu blackboard lecture to explain how this form of on-policy self-distillation works -- I posted the full thing on X. If you want to try Composer 2.5, go to cursor.com/dwarkesh

Timestamps

(00:00:00) – Will capital share increase?

(00:19:36) – Messy Middle scenario

(00:25:57) – How to tax and redistribute AI wealth

(00:30:02) – Why demand collapse is unlikely

(00:39:26) – Human employees would be hard to integrate into the machine economy

(00:43:08) – What if some humans (or AIs) value wealth accumulation intrinsically?

(01:01:28) – What should developing countries do?

Transcript

00:00:00 – Will capital share increase?

Dwarkesh Patel

Today I’m chatting with Alex Imas, who is Director of AGI Economics at Google DeepMind and Professor of Economics at the University of Chicago, and Phil Trammell, who is Head of Economics at Epoch and research scholar at Stanford.

In general what I want to understand in this interview is what economics tells us about what we can expect in a world with more and more automation and more advanced AI. I want to understand what that tells us about what will happen to wages and the labor share, what the best way to tax and redistribute the wealth generated by AGI will be, and what kinds of things will be scarce. What is scarce tells you where the value will accrue.

I want to start there. What are some plausible candidates of what will be scarce?

Alex Imas

Something like the relational sector, which is defined as services and goods where the fact that a human was in the loop is part of the value of that product. Because humans are naturally scarce, if we have automation where a lot of other things stop being scarce, we will still have scarcity in the things that humans are involved in and in the loop for.

Dwarkesh Patel

I’m curious to understand whether humans doing services for other humans can ever be a big part of the economy. Here’s maybe one intuition pump. In a world where AI can physically do anything humans can do, there’s this whole machine economy where they’re building factories and doing research and coming up with new ideas. Humans may or may not be involved in the physical production of those things, but probably not in the ultimate limit, if robotics is solved. If you don’t care about humans being involved in that process, why would they be?

But then there are these other things you point out where we actually do want the ballerina or the barista to be a human. That’s part of the value of going to a cafe or a performance. But only humans have that preference. So there’s this human economy where humans are doing services for each other, and part of their wealth is flowing to other humans.

But part of their wealth is also flowing out, because they will want some of the automated goods this machine-only economy is creating. This is not a closed loop. A lot of things in the machine-only economy are a closed loop because the machines don’t care about getting the human barista to make them a coffee.

Within that model, isn’t it intrinsic that the human-only economy will become a smaller and smaller share?

Alex Imas

I would like to pitch a rephrasing of that question. My view is that the individual forecasts economists like us would make, as individual forecasts, are not necessarily very useful.

There was a blog post by Andrey Fradkin, Brian Jabarian, and Andrew Koh that came out yesterday looking at economists’ forecasts about the labor market. What they found is that there’s a ton of disagreement in every single direction.

What they advocate for, and I’m in agreement here, is that rather than thinking about individual forecasts, we should be generating prediction markets where you get aggregate forecasts and wisdom-of-the-crowd effects. The reason I think this is because we have been famously terrible at forecasting.

Let’s go all the way back to 1820. This debate we’ve been having is actually 200 years old. David Ricardo is one of the classical economists, not neoclassical. When the Industrial Revolution started happening, he wrote a bunch of stuff saying, “This is going to be great for everybody. Prices are going to come down.” But then he turned around and said, “Wait, I can see all these jobs that are creating value are going to be automated by these machines. This is going to be really bad. Everybody’s going to become unemployed, and there’s going to be political unrest.”

And if you look at Ricardo’s predictions, they’re actually right. All those jobs that made money in Ricardo’s time got automated. If David Ricardo woke up and somebody told him all those jobs did get automated, and then asked him, “What do you think the prime-age employment rate is in 2026?”, I think he’d be surprised to be told it was the highest it’s ever been other than 2000. We have the highest number of employed people that could potentially be employed since 2000. That was the peak and now it’s the second peak basically.

What David Ricardo ended up missing is that you have these economics of structural change, where everything that got automated became cheap. People had more money to spend, and then they started spending it on services. This is the lump-of-labor fallacy. David Ricardo didn’t consider that new jobs would be created.

But it’s not obvious that money would go to services. Why wouldn’t it go to more automated goods and something like that? I’m not using this anecdote to say this is what’s going to happen now and that we’re going to have full employment. I’m using it to say it’s really hard to make predictions.

What may be a really useful tool that economists have is to instead start with a premise. Maybe we start today: labor share is zero. Labor share has gone down. What could possibly explain this? Let’s write down an economic model of what happened. Phil will talk about this later today. Or you can write down a model that asks, “What if labor share just stays the same? What can make that happen?”

If you don’t take anything else out of this conversation from me: We don’t have any data. I’ve been saying we need a Manhattan Project for data. We don’t have data on consumer demand elasticities. We don’t know what they are. We’re not really tracking what jobs are getting created or destroyed. The O*NET database, with all of the tasks and different jobs, has been rarely updated and is super low quality.

What is really useful is to think about the potential scenarios, map them out, and say what dimension of scarcity will generate each scenario. If there’s full employment, we can talk about the relational sector. If the labor share collapses, we can talk about other sorts of scenarios. That will tell us what data we should be collecting.

Dwarkesh Patel

It’s probably worth defining labor share and capital share real quick. The whole economy, the total sum of goods and services sold, is either paid out to people in wages or it’s paid out to capital, which is to say, there’s rent on buildings and shareholders of companies that get paid out.

For many hundreds of years, ~60% of the economy basically gets paid out to humans in wages, and the other 30-40% gets paid out to people who own machines and land and claims on companies. The question is, if 60% is going to wages right now, does that shrink as AIs get smarter and better?

Alex Imas

This is a Kaldor fact. We should stress this. It’s incredibly surprising that it’s over 60% after the Industrial Revolution and all of the automation we’ve ever seen. Some people are worried it’s an accounting error that it’s been so constant.

There’s even a controversy right now. Some might say labor share has been falling in the last 20 to 30 years. But there have been a lot of accounting changes in the last 30 to 40 years. For example, Atkinson has a paper showing that if you keep the accounting constant over the years, labor share hasn’t even fallen ever.

Dwarkesh Patel

But it’s not that surprising, right? Phil, you made this point that if labor and capital are complements, you need both to do anything. It would make sense that you’d need to pay both of them to get something done.

Alex Imas

You have had stuff be completely automated.

Phil Trammell

There’s a sense in which nothing has yet been completely automated. Look at the network-adjusted factor shares of a good. Look down the supply chain and not just the final step and how much of that is done by capital and labor, but what went into the machines that can automate that final step. You’ll find that labor is adding a lot of value down the supply chain.

Computer and electronic products in the US have a very stable network-adjusted capital share of around 50%. It’s not 100%. I do think there’s this qualitative shift that I think we agree is coming, which is that there will be at least some goods whose network-adjusted capital share goes to one. The whole supply chain can be automated, and there’s no part in it that we care intrinsically about having a human do. That will be a qualitative shift.

Interestingly, the implications of that shift for the overall capital share are ambiguous. Let’s say we’ve got two sectors: the human-intrinsic sector with the ballerinas, and everything else. Right now, everything else has been scarce because of the lack of labor in it. But if we fully automate the supply chains for everything else, and we satiate in everything else really fast, then the quantity of everything that’s not a ballerina goes to infinity, but the marginal utility in that stuff goes to zero faster than the quantity is rising.

Alex Imas

I also want to move away from the ballerina example. The point I was trying to make in my post—working backwards from a particular scenario—was that the ballerina and the performer are the wrong reference class. Right now we have a lot of jobs where you have different tasks. This is the task-based model of jobs. Take a doctor, what is their job? They’re filling out insurance documents. They’re going and calling different pharmaceutical companies. One of their tasks is to see the patient and talk to them, but that’s not the main part of the job.

You could have a job and a service or a good be a product of different types of tasks, and you can automate a ton of those tasks. If the consumer is willing to pay more for a product or service where every single task is automated except for that one part where the doctor is delivering the diagnosis and providing support, we would call that job part of the relational sector. People are willing to pay more for the human to stay in the loop in the job.

We don’t have data to say, “Here are relational jobs, here are not.” You literally need to collect data of the following sort. Do a conjoint analysis of your willingness to pay for this service or good. Here’s the counterfactual where everything is produced by machine. Here’s the counterfactual where this one task is not produced by a machine. What is your willingness to pay? What is your elasticity for the human to not be in the loop?

If I don’t have that data, what prediction am I going to make in this story?

Dwarkesh Patel

Isn’t there another point, which is that there are a lot of fully automated goods that don’t even exist yet? And you can’t collect any data right now about, say, how much people will want to keep buying more and more of some drug that makes you healthier that is fully produced by the AIs.

Alex Imas

Absolutely. That’s kind of Phil’s point. You could have an increase in variety in capital where you don’t get the satiation. You’re increasing variety, so you’re not hitting that diminishing marginal utility point where most of your income is going to the human sector. If that increasing variety is fast enough, and there is no such increasing variety in the human sector, then you can get all of the relational goods you want, but it doesn’t matter for labor share. It goes to zero.

Dwarkesh Patel

Phil, I liked your analogy to some Mongolian economist sitting around in 1400 thinking about what will be scarce and the limits of that kind of analysis. I think you should talk about that.

Phil Trammell

Just look at the goods available to a Mongolian of the distant past. I’m no expert on this society, but I know that they didn’t have nearly the variety that we have now. Look at the jobs that were intrinsically human, like being a singer. And then you look at the things that were not intrinsically human, like the transportation services provided by their horses or the different kinds of food they had.

If they just held the varieties fixed in both categories and asked, “What will happen once we have a lot more automation?”, they might have said, “We’ll just satiate in horse-like transportation and in yogurt and in yurts. Those shares will all go to zero, and we’ll be left spending all of our money on singers.”

But of course, that’s not what happened. As we’ve accumulated more wealth and more advanced machines, we’ve expanded the range of things other than singers to spend our money on, and the share spent on singers has stayed negligible. Likewise, that’s my central prediction about how the future unfolds, though it could go either way.

Dwarkesh Patel

I was going to make a point and I realize it’s a fallacy, but the reason it’s a fallacy is interesting. It’s just hard to imagine a world where there are trillions upon trillions of robots, but there’s only some billion-odd humans, and the cumulative amount we’re spending on robots is less than what we’re spending to pay Magnus Carlsen or—

Alex Imas

Financial advisors or doctors or tutors.

Dwarkesh Patel

Right, or podcasters or whatever. But then I realized why it’s a fallacy. The number of transistors in the world has literally trillion-X’d, maybe quadrillion-X’d. Your colleague Chad Jones has a very interesting result about how the share of the economy that is going towards paying for computing, paying for the transistors, has been decreasing.

The point you made is that one way to think about Moore’s law is… What sets price? Supply and demand. So not only are we producing more transistors more cheaply, but also the value of the marginal transistor is decreasing. As you were saying, another way of saying Moore’s law is…

Phil Trammell

I like the pessimistic framing of Moore’s law: every 18 months, the value of computation halves. We’re running out of uses for computation so fast that it’s sustaining Moore’s law.

Dwarkesh Patel

This is relevant to a conversation about AI where maybe for the first time, this is no longer true. The famous fact here is that an H100 costs more to rent now than it did three years ago, even though we have much superior technology and much more compute in the world. Because as models get smarter, the opportunity cost of compute gets higher.

Alex Imas

This is Phil’s point about increasing variety. What we have done is increased the types of things that people demand from capital. Now all of a sudden you have a new variety that you could be using capital for, and you jump back up.

Dwarkesh Patel

You could imagine we just never satiate demand for compute. As long as that stays the case, the share of the economy that is going towards compute would keep increasing.

Alex Imas

That’s the big question. That is the ultimate question that we need to be looking at. What number of new uses are we finding for that compute where you have the demand for these uses? What I want to emphasize is that a lot of models in economics, especially in the space that we’re talking about, take demand as almost exogenous. They don’t unpack the psychology of what people actually want.

What got me thinking about the idea of the relational sector is work that I was doing on the fact that there does seem to be this intrinsic value. It’s not just because it’s scarce; it’s because there’s some intrinsic preference that people have for empathy, connection, and interacting with another person.

One of the experiments that we ran involved an art print. We have an incentive-compatible way of asking, “How much are you willing to pay for this art print?” People are actually paying real money for it. Then we say, “Look, there’s only one of those art prints, and it’s either made by AI or by a person.” These are between-subject conditions. With one, you get the effect that the person-produced art print is valued much higher than the AI version.

Then, in a set of other conditions, we say there’s 500 of these being produced. For the human-made one, the price goes down a lot because it’s no longer seen as making a connection with this one artist. With AI there’s no difference. AI is already viewed as a commodity.

We need to do a lot more research on this, but it seems that’s the key difference between this and something like a horse. A horse was an input into an output, where you can replace the horse with something else. You only care about the output.

The only way this relational story works—and this is what we need more data on—is if a human is not a horse in the sense that they are providing value from the output, where if you replace the human, the value of the output decreases. If that’s not strong enough, and if it doesn’t hold for enough sectors or enough jobs, then this story doesn’t work anymore.

00:19:36 – Messy Middle scenario

Dwarkesh Patel

There’s one possibility which Molly Kinder has written about, this “Messy Middle” scenario. That possibility made me think about whether it might be better to have—at least as far as wealth distribution and redistribution go—a much faster AI takeoff.

I want to ask you whether the following possibility is at all likely, or if there’s any set of assumptions that can make it so. AI makes it possible to automate jobs such that many people are losing their jobs, but it doesn’t create enough wealth, while the process of automation is happening, to basically pay off the people who are getting laid off and create a Pareto improvement, where everybody’s getting better as a result of AI automation.

Of course, there’s a trivial sense in which that must be true. Whatever money the company is saving by not paying the humans instead of just paying the AIs, those resources still exist in the economy and can just be paid out to people.

But there’s going to be some allocative inefficiency. The government doesn’t know exactly who got laid off because of AI. There’s a political problem. If the Meta worker gets laid off first and they were making $200,000 a year, is there a politically sustainable situation where you give them a $200,000 check a year when there are many working people making much less? Do you find this scenario plausible, where AI is automating a bunch of things, but there isn’t as much wealth creation as there is automation?

Phil Trammell

I think it’s possible. To me, it does seem like a pretty narrow window. My guess is that if we have the technology to automate so many jobs that it becomes a new kind of political problem, then the pie will also be growing really fast.

Well, unless in all of those professions it’s automating, it’s just a hair more productive. So the cost of all the capital to replace all the software engineers is just a hair less than the cost of what we’ve been paying the software engineers.

Dwarkesh Patel

Why is it implausible that a company can save money by laying off a bunch of software engineers? And in the long run, there’s a Jevons paradox, and we can’t anticipate in advance what we’d do with more software, and surely there will be more uses. But in the short run, the effect is just that a lot of people are laid off, and they still need to figure out how they can use a million times more JavaScript tokens.

Alex Imas

Phil and I have been writing about these things, and we have mathematical models in the back of these things. We don’t have any political economy in any of our models.

Andy Hall wrote a really nice blog post about the politics of AGI, and he made a really interesting observation. If there’s a 2% increase in unemployment, the political winds completely change. Unemployment has a huge effect on what happens politically.

Referring to Molly’s excellent essay, I think in some ways one of the worst scenarios is a drip scenario because of the political economy piece. What you might see is people not really being unemployed en masse, but moving into sectors that pay them less money. This is what happened with phone operators between 1920 and 1940. Phone operators were completely automated, but it took 20 years, even though the technology existed. There was this drip. It wasn’t like this giant sector just disappeared.

There’s a really nice QJE paper on this showing that they got reabsorbed into the economy, but at lower salaries, and they were mostly underemployed. That’s the scenario Molly was writing about, this messy middle where things aren’t a disaster. We saw with COVID that the fiscal response can move quickly if there’s an emergency. An emergency is a quick uptick in unemployment, which could even look like 2-3%. That becomes a national emergency if it happens fast.

Dwarkesh Patel

The concern is that whatever you’re saving on those white-collar workers, if that’s not growing the economy but just creating saved resources that can be allocated elsewhere, is that enough to do a broad-based redistribution scheme? You have the money you’ve saved off a couple of people. Unless you can figure out exactly how to get it to them specifically, you have the problem of, “Can I do a UBI off the money I saved by laying off…?”

Alex Imas

You’re basically saying the pie did not grow that much. You’re just displacing a bunch of people, but that didn’t grow the technological frontier of what the economy can produce.

Dwarkesh Patel

Then there’s a question of whether every time this has happened in history, the technological frontier has expanded a bunch.

Alex Imas

I think that’s the case. Simply in history, the technological frontier has expanded. I think Phil made the same point. It’s hard to imagine that sort of scenario where you are getting intelligence that’s just enough to replace the software engineer but still costs a lot of money. It’s just a hair less expensive than the software engineer, so you’re not getting this abundance effect. Where is the redistribution going to happen because the pie didn’t grow?

Dwarkesh Patel

This is very helpful. Many different things have to be true for this scenario to come to pass, each of which seem unlikely.

One, it has to be the case that it is possible to automate entire white-collar jobs, but only in a piecemeal way. That is to say that you can only automate software engineers, but that same program can’t also automate an accountant and an analyst and whatever.

My model of intelligence is such that—both the breadth of tasks it requires to do something like software engineering and what intelligence is—if you can really just lay off all the software engineers, you’ve got enough in the bucket there that you could automate all kinds of white-collar work. There are huge amounts of potential savings that have happened as a result of these layoffs, and also AI is going to be cheaper than human labor.

If both of those things are true, this messy middle scenario where we literally don’t have the wealth to go around seems unlikely. Then the question is, what is the best way to tax it and redistribute it?

00:25:57 – How to tax and redistribute AI wealth

Alex Imas

I have some thoughts. I think it’s really important to outline the costs and benefits. First, there’s differential complexity in implementing these things. Two, they differ in the timeline of being actually helpful. Something like universal basic capital is not going to generate returns for something that happens in six months. You probably are going to end up with a layer of things.

Take a negative income tax, for example. You implement it, and the day it turns into law, you already have this insurance that there’s a floor where everybody gets a certain amount of money, and if you earn more money, you get taxed more. But there are positives and negatives to a negative income tax.

With UBI, for example, I worry a lot about the political economy implications. If people are just dependent on a check, it really matters who’s in power. Right now, we’re endowed with labor that can turn into income. When that is no longer the case and we are at the mercy of the elected official for basic needs, that feels like a power-sharing arrangement that’s really dangerous.

Dwarkesh Patel

But wouldn’t that be true of any sort of government redistribution program?

Alex Imas

With something like universal basic capital, where you have an ownership share and property rights for capital, you just have a share.

Dwarkesh Patel

You’re a normal shareholder.

Alex Imas

You’re just a normal person.

Dwarkesh Patel

But this goes back to the question of indexing, because if indexing is hard, then universal basic capital is hard.

Alex Imas

That’s the problem of universal basic capital: targeting. What do you target to put into people’s portfolios?

Dwarkesh Patel

Like, what if Anthropic goes to zero, but some random robotics company takes all this over?

Alex Imas

Exactly. That’s the risk of universal basic capital. With a negative income tax, you have the same sort of issues as with UBI, where somebody comes into power and says, “We’re not going to do that anymore,” and people can’t work, and then you have the issue of the floor being gone.

Dwarkesh Patel

One concern with the wealth tax is that there’s no politically sustainable equilibrium at a 0.5% wealth tax. This happened with the income tax, of course. It starts low, it’s for war or something, and then it slowly escalates until the marginal income tax rate in the US is on the order of 40%, and in certain states, upwards of 50%.

With a capital tax, is there a reason to worry that it would distort investment? Would people just say, “Why would I invest in Anthropic or Intel? The government is going to take larger and larger shares of it and dilute my share.”

Phil Trammell

Hold on. It’s worth separating how the revenue is raised, what’s taxed, and how it’s distributed. It could be that the government hands out shares of Anthropic to everyone by a broad-based tax and then buying Anthropic. Which would probably be the right thing to do. Hopefully, some populist proposal doesn’t interfere with that and expropriate some particular company that everyone happens to know about.

Dwarkesh Patel

You’re suggesting there could be some sort of optimal tax. We’re taxing externalities or we’re taxing land. I guess we probably need to tax something other than just those two things.

Phil Trammell

Or consumption.

Dwarkesh Patel

Ok, a consumption tax, like a European value-added tax, allows the government to go buy a bunch of stocks, and then they just distribute those stocks to everybody.

Alex Imas

That’s David Autor’s...

Phil Trammell

That’s not going to be that different from just redistributing the stocks, but it will be a little different.

Alex Imas

That was the proposal for Social Security, by the way. That was privatizing Social Security. It’s worked so far, but there are questions about how long it’s going to keep working. Privatizing Social Security was basically giving everybody a basket of stocks.

00:30:02 – Why demand collapse is unlikely

Dwarkesh Patel

People talk about whether there’s a white-collar apocalypse already. Is there any evidence that suggests there is mass automation or unemployment as a result of AI already?

Alex Imas

A lot of people are looking at it. This is an area where there’s a lot of eyes and a lot of data being produced. The Budget Lab over at Yale is doing really good analysis on this. They just recently released a report, and you really have to squint to see anything happening. If you want to take an approach across the entire economy, even looking at software engineering, the most exposed sectors, there’s just not really anything going on.

There might be a little bit of a signal about junior developers getting jobs less than before. But that’s a “less than before” rather than a level shift, as in there’s actually an increased demand for senior software engineers, if anything. If you look at the trend, for junior developers, it’s a bit below trend.

Dwarkesh Patel

So you’re saying the growth is slower than before, but there is still growth even for entry-level software engineers. What do you think is going on with the anecdotal evidence of graduating college students saying that they’re finding it harder to find CS jobs?

Alex Imas

I think that’s anecdotal evidence.

Dwarkesh Patel

You think it’s always been hard to get jobs for some people, and now it’s getting turned into an AI narrative? Same with the layoffs, where it’s probably just a normal layoff, and they turned it into an AI layoff.

Alex Imas

You have to be careful with all of this. There are these public coordination devices. Let’s say we get into a narrative where if you’re a firm and you’re not laying people off, then you’re seen as not adapting AI enough. Then you’re going to just get a cascade effect of firms needing to keep up with the Joneses in terms of starting to lay people off. That’s super worrying, where the firm might actually be worse off after the layoffs than before, but it’s just doing the layoffs to have the perception that, “Look, we’re not behind the times. We’re using AI.”

You probably heard these anecdotal stories of token counters, where you have to maximize tokens and things like that. Right now, we don’t really have any evidence of a white-collar bloodbath.

Dwarkesh Patel

Is that surprising at all, given all these things AI can do? This is a story as old as time. If you automate some complementary task, the overall bucket of things—the human labor which complements the automation—will increase in value.

Alex Imas

One of the statistics that’s really important for that argument is elasticity of demand.

Take the O-ring model of jobs. A job is a series of tasks. Let’s say the AI automates nine out of ten tasks. One task is not automated. If that person can now focus in on that task, the job will become more productive. If that translates into a price effect where the product is actually cheaper, and if demand responds enough where it’s being bought more and the service is being used more, that could actually lead to more hiring.

A lot of people on the internet have been making that argument very generally, saying, “Look, if anything in the data, we’re seeing an uptick in software engineering demand.” Which suggests that at least for now, given the way that jobs work, it might be elastic enough.

Dwarkesh Patel

I think this elasticity of demand argument is incredibly important for a lot of arguments that people make, or just a lot of labels that people use without understanding what the underlying causation is. People often talk about Jevons paradox. This is the idea that as something gets cheaper, you will want so much more of it that the total amount you spend on the thing increases. Famously, this happened to coal in Britain ~200 years ago.

But really this only happens if the demand for something is highly elastic. There are many things for which there is not super elastic demand. If oil, for example, gets super cheap, it’s not like magically—

Alex Imas

Or insulin.

Dwarkesh Patel

Exactly. It’s not like magically there’s going to be so many more cars that now we’re going to be using way more oil than before.

Phil Trammell

At least not in the short run.

Dwarkesh Patel

Exactly. The long-run elasticity is higher than short-run elasticity. But even in the long run, agriculture famously is the example where we can produce way more food if we dedicated the same portion of the economy that we dedicated to agriculture in the past. We’re already producing more food regardless, but we could produce even more if the same portion of the economy that was producing food 100 years ago was currently producing food. But you eat enough, and then you’re done.

The claim with software is that it is not some inherent property of markets that as it gets cheaper, you’ll just keep wanting more of it. The thing about software is this is a particular kind of good where as it gets cheaper, we’ll want more and more of it.

It is also highly relevant, and you wrote an essay about this—a lot of this podcast is me summarizing your essays back to you. There’s this very viral scenario planning about the future by Citrini, predicting that as a result of automation and very powerful AI, there will be a recession. White-collar workers will get automated, their salaries will no longer be available, and so there will be a slump. Do you want to recapitulate why this might be implausible?

Alex Imas

Part of it is plausible, part of it’s not. The part that we started the conversation with is the idea that there could be a lot of unemployment. If the speed of automation is quick, people could get laid off, and they may not find work very quickly. We can quibble about the unemployment part of the Citrini essay, but that’s not the issue. The issue is that they talked about negative economic growth.

What I did in the piece, that Phil and I had a back and forth on, was to say, let’s start with the proposition that there’s negative economic growth. What conditions do you need in the economy to get negative economic growth? It turns out the conditions are pretty improbable.

One thing that you need is for the holders of capital, rich people basically… Basically what you have in those sorts of scenarios is a reallocation of wealth and income from lower-income people who are using their labor towards tech capital owners. So you need demand to be bounded, like a hard bound, not even a soft diminishing sensitivity. You need for them to eventually say, “I’ve had enough. I don’t want to spend any more money.” And for that money to not enter as investment. Then you can get negative growth.

Dwarkesh Patel

The crucial thing is, even if we don’t want more shit, the world in which there’s a singularity and we don’t want to invest more money is crazy. We’re not saying, “Let’s build more data centers. Let’s build more fabs.” Even though we have AGI, we’re not investing in more data centers to run the AGI and that’s driving more economic growth.

Alex Imas

I sent the essay to Phil, and Phil wrote back being like, “This is pretty dumb,” like my essay. He said, “You’re trying to say that there’s going to be negative economic growth, but these are very implausible conditions.” And I was like, “That’s the point of the essay. These are very implausible economic conditions.”

That’s where scenario planning really shines. You have the Citrini essay, which was great that it was written because it started a conversation. It’s so intuitive, this idea that if there’s demand collapse, we can get the economy to shrink. You could get that with a depression. In the Depression, the technological frontier didn’t expand.

Here, the technological frontier is expanding. You actually have abundance. For abundance to generate negative economic growth, that’s really hard to get.

00:39:26 – Human employees would be hard to integrate into the machine economy

Dwarkesh Patel

We were talking a second ago about why there isn’t more automation as a result of LLMs. One plausible mechanism could be, as you were saying with the O-ring theory… O-ring theory refers to the fact that the Challenger shuttle blew up because one component malfunctioned, and it destroyed the whole thing. Maybe that’s a more general model of how goods are produced in the economy. You have to make sure everything is reliable and works well. So you can’t automate an entire job to an AI right now. Even though it might be able to perform it at some probability, you need extreme reliability in order for it to not destroy the finished good.

This might explain why there’s a lot less automation now than there otherwise could be. But I think it works in the other direction once AIs get advanced enough. Integrating humans into the production flow of future goods will become difficult. Even beyond the arguments about how humans will be more expensive or less capable, there will be whole production flows organized for AI labor. They’re talking in neuralese. They’re thinking many thousands of times faster.

So even if there’s some comparative advantage where it makes sense to hire a human, there will be transaction costs and worries of reliability that will actually make it hard to integrate humans into future production flows.

Phil Trammell

That seems right to me. In particular, I just want to distinguish between the point that if you automate nine-tenths of a job, people might shift over to the last tenth, but there might be ten times more work demanded of them.

Compare that to the model of O-ring automation from Gans and Goldfarb recently. If you can only automate nine-tenths of the job, but you do it to a lower standard of quality than the human could, you might not want to automate even those nine-tenths.

That’s the thing that could totally port over. Symmetrically, it could be a reason why we don’t use a human for one-tenth of the job anymore, because a human just can’t perform it to the level of quality that the AI can perform the other parts of the job, or the level of speed. They end up pulling down the quality or speed of the finished product.

Dwarkesh Patel

By the way, the model you’re talking about seems extremely plausible to me for why more lawyers, accountants, or even software engineers are not automated. There are cases where there’s a pretty good probability that the thing worked as you expect, but the thing you’re paying the lawyer for is: “No, really, my company’s not going to go under because—”

Alex Imas

You’re also paying for a lot of regulation-type stuff. With lawyers particularly, you need some entity to back up the product. You need ownership of the product. You need somebody to be able to fire or hire, and there are licensing issues. There’s a lot of regulatory layers that are also going to be keeping—even if there’s no relational element—humans in the loop that have nothing to do with the ability of the human to actually perform the service.

Phil Trammell

All of these frictions on the political-type decisions that we are accustomed to only trusting humans for—legislation, being a judge, being a juror, or all the licensing that keeps certain professions human—that all strikes me as transitional.

What we expect to come from a human and how we organize our politics has changed so many times throughout history, from little hunter-gatherer bands to empires to whatnot. Once an AI-run political system is much more efficient than the alternatives, those will probably tend to out-compete the others.

00:43:08 – What if some humans (or AIs) value wealth accumulation intrinsically?

Dwarkesh Patel

Speaking of which, we’ve been talking about what preferences humans currently have and what impact that has on what kinds of goods will be scarce in the future. But of course, we’ll have different kinds of entities in the future: AIs. There was a time when there were no humans on Earth, but evolution selected for agents that have specific drives and preferences because those tend to survive the most, and those preferences now determine what a hundred-trillion-dollar world economy produces.

Why not expect the same thing from AIs in the future? This is not even a world with catastrophic misalignment, where they just kill everybody. But there will be evolution of, even if not individual AIs, firms which have AIs as part of them. What will that evolution favor? It will probably favor firms or agents that grow. There’s a selection argument that things which grow will be more prevalent.

Maybe just based on that, you can make some predictions about what their preferences will be. Is the kind of entity which prefers to have human-intrinsic goods going to be the kind of entity that accumulates resources the most? Probably not. Probably it saves more and has unsatisfiable demand for whatever the relevant resource happens to be. Compute is an obvious one. Can we use that to make some predictions about the non-human preferences that will be guiding the future?

Alex Imas

If there’s an AI that has its own welfare, is fully autonomous, and is making its own decisions that are welfare-relevant, to be honest, I have absolutely no prior that it would prefer to deal with humans. There’s no reason.

But let me take the other side of that argument. Humans’ preferences to be interacting with one another, to trust and empathize with other humans versus a simulated AI, I think it’s a really important question whether those will change.

I’ve heard a lot of arguments saying, “Look, right now we’re just not used to the technology. What you’re thinking of as relational… At some point, people are just going to see an AI therapist as a superior product, and they’re not going to need the empathy that the human is providing.”

I think this is actually a really complicated question. Here’s one argument for why it’s not going to go away, and it has to do with evolution. Let’s say there are two types of people. One person doesn’t really have this preference. They can just interact with an AI, whatever can simulate it better. The other one has almost a moral emotion—using Jonathan Haidt’s framework—against offloading those sorts of social interactions to an AI.

Which of those two people are going to reproduce, find a mate, all of these sorts of things? I think the answer is clear. It’s the second one that has the preference for other people.

Dwarkesh Patel

Depends on how the reproduction is happening.

Alex Imas

Fair. But if we’re in the world where reproduction is still happening the way that it’s happening, I think… And this is a big question, I’m not making a prediction.

You had David Reich on the show. His point on the last podcast was that we’re buzzing with natural selection. So even if you get some sort of indifference now, you might get selection to point into an even stronger preference for other humans.

Dwarkesh Patel

Here’s one way to think about it. How is the wealth of the richest people in the world instantiated? We were having a call earlier, and you made the point that their consumption is more geared towards relational goods. Like Mark Zuckerberg is hiring MMA instructors and dancers for his wife’s birthday, and so forth.

But most of his wealth is just stock in Meta. As a controlling shareholder, he could say, “Meta, turn all this wealth into dividend income, and I will just spend that on consumption.” Instead, he would rather have his wealth compound and have Meta build more data centers.

So you don’t even have to change humans for this to be the case. The humans who are wealthiest—and growing wealthier because their wealth is compounding—just have this almost Nick Landian preference for accelerating capital. That does seem to suggest that this is an important determinant of what kinds of things are produced in the future.

Phil Trammell

There are two ways you could get the two kinds of people, one of whom prefers a human therapist and one of whom is fine interacting with the AI. If they both satiate equally quickly in capital but the one who likes the human therapist also just likes having some human-intrinsic services, then the marginal value of capital in the future, compared to the marginal value of capital today, for each of them if they start out equally rich, should be basically the same. There could be interactions and whatnot, but basically, that should be the same.

If what’s driving the difference is that one person just doesn’t satiate in capital because they’re engaged by the prospect of exploring the universe and turning their head into a galaxy brain or whatever, and the other one satiates, then the person who doesn’t satiate in capital is going to, if they’re being rational, have a higher savings rate.

So in the long run, they’re going to have most of the wealth, and the overall capital share will basically be the capital share of that person’s spending, which is going to be one.

Dwarkesh Patel

It’s important that we’re not talking about a hypothetical future. Elon Musk is talking about mass drivers on the moon. He’s by far the wealthiest person in the world. Obviously, currently his investments are going towards humans as well as machines, but I don’t think he cares particularly that his future researchers and engineers are humans versus AI.

Phil Trammell

And he manages to reproduce fast as well. So I just think it’s worth drawing that distinction. There are currently some rich people that don’t seem to satiate quickly in capital, and so maybe in the long run they’ll save the most. That does seem right to me.

I would also say, even if they do reproduce more slowly biologically, that might just not matter that much in the long run if they can live forever.

Alex Imas

The living forever is key. Again, we’re scenario-building here. If you could live forever, a lot of stuff changes for my story as well. To your point about rich people not consuming a lot and investing, this will all depend on the returns to capital. Right now, the returns to data centers are super high, but if we get into a situation where people are satiated with capital, then the returns to accumulating capital are going to be lower. Then these rich people are going to be consuming more, because the incentive to invest is smaller.

Basically, you think about the general equilibrium of this sort of process… We have gotten tremendously richer since 1820. Many more people are investing, but you’re still getting a consumption response which keeps people employed and labor share high. That’s because—

Phil Trammell

Hold on.

Dwarkesh Patel

Wait, not necessarily. I think you’re probably making the same point. It could be that their investment has to be titrated through actual laborers who have to do things for their investment to work. In the future, only the consumption is human-mediated, right? Because the investment can just be done by the robots.

Alex Imas

So we’re in the scenario of how you can keep high labor share. Let’s take that scenario. In the scenario with high labor share, for whatever reason, the returns to capital are going to be lower.

Dwarkesh Patel

That’s right. To the earlier thing where we were saying why the messy middle is implausible, I feel like we can do a similar thing here. For our returns to capital to be lower, the growth rate has to be lower, right? It certainly has to be lower than what we’re expecting through the period of transformative AI. If there’s explosive growth…

Phil Trammell

Yes and no. The capital stock could grow quickly, but the price of capital goods relative to consumption goods could be falling faster than the capital stock is growing.

Alex Imas

It’s the difference between the potential frontier of technology and the realized prices of these things. Because you have relative prices.

Dwarkesh Patel

So you’re saying I could be putting my money towards earning 30% interest and investing in data centers, or whatever. There will be something in the future, if the growth rate is high, that earns high returns. Or, as a result of all these technological breakthroughs, there’s some cool product that I really want to buy right now, and both of those will be compelling options.

Phil Trammell

Yeah. It doesn’t have to be a new product. It could be a human-intrinsic product.

Dwarkesh Patel

Although, if it’s a human-intrinsic product, we would want to have it much more in the future than we want it now, because the thing it compares against is—

Phil Trammell

We might want it the same as we want it now in the sense that the marginal utility in a ballerina performance is exactly the same as now. But the marginal utility in a robot might just be a lot lower than now. So in units of robots, we want it a lot more than we want it now.

Would the interest rate be 30%? It depends what you mean by the real interest rate. It might be that every robot now can turn into 100 robots next year. So in units of robots, the interest rate’s 10,000%. But if the price of robots is falling really fast...

Alex Imas

Prices adjust. I think that’s the whole point.

Phil Trammell

Here prices are adjusting in this interesting way that too many macro models don’t allow for. What’s happening is what would be called investment-specific technical change.

The price of capital is falling relative to the price of consumption, instead of doing the standard macro thing of saying there’s just output, this chimera of a thing called output, which one for one can be allocated to capital or consumption. That’s not going to be true in this world.

Every unit of capital next year is giving up way less consumption than each unit of capital this year. One robot now turns into many robots next year, but the number of ballerinas is the same.

Alex Imas

Again, we’re going to go back to the increasing varieties thing. If all of those extra robots next year are actually different varieties of robots and I’m not getting satiated on those robots, then it’s a very different story.

Dwarkesh Patel

But now we’re talking about the consumption world. For the investment side of things, there could be just some greedy titan of industry who keeps wanting more and more robots. That alone would be enough to increase the marginal value of robots and therefore decrease labor share?

Phil Trammell

Yes.

Dwarkesh Patel

Why are we not expecting greedy titans of industry to keep existing?

Alex Imas

Greedy titans of industry historically have built libraries and—

Dwarkesh Patel

But that’s because they die, and they’re like—

Alex Imas

Oh, they all die. Everybody dies.

Dwarkesh Patel

Well, we’ll see.

Alex Imas

Conditional on people dying… You had a guest on the show who said to understand the future, you should think about the past. You could have new types of titans being born whose entire reason for accumulating wealth is just to accumulate wealth.

But a lot of the time, at least historically, the wealth accumulation process is part of a large social interaction amongst peers, amongst the community, where you want to be admired in some way. The stylized fact of titans of industry is you accumulate the capital, and then you buy a bunch of stuff.

Dwarkesh Patel

I guess this is a historical question, but it does seem to me that in a lot of cases what is happening is that as they near the end of their life, they either hand it off to their children, who are worse stewards of capital than they are. They don’t even manage to grow their wealth at the rate the economy grows, much less faster than the economy grows, which their parents were doing.

They’re like, “Well, I care less about my children having it than me playing this game of accumulating wealth. So I’m just going to give it to some trust.” If people are living longer or if they can figure out some way to align their trust to this wealth accumulation process… It just feels like the evolution here is so strong. You just need a couple of agents that think this way for this to be the dominant thing determining the preferences of the whole economy, because this part is growing much faster than the other parts of the economy.

Alex Imas

The part about satiation and diminishing marginal utility keeps coming up, I think it’s really important. If a person has an intrinsic preference for accumulation, that’s just what they want, then I think your story is totally right. But that’s just not how preferences usually work. You have enough hedonics in your life, and then the social status… Rousseau wrote about this, St. Augustine wrote about this. This is a basic part of preferences.

Now, you guys are arguing about something else, where you could have such high concentration that you could just have a couple of exceptions to the rule, and that’s going to be enough. I have nothing to say about that.

Phil Trammell

I think the claim is a little stronger, not just that you could have some exceptions, but that historically and today we see the exceptions. They just haven’t really taken over the economy historically because there have been these dissipation shocks, as they’re called. They’ve given it to their kids who squandered it, or they put it in foundations which spent it. It’s not really a shock, but…

People might have liked to fill the universe with monuments to themselves and live forever, very wealthy. It’s a weird preference, but it’s not a hypothetical preference. I think that’s the claim.

But who knows what’s going on in their heads? Even without the intrinsic preference for accumulation, there are some instrumental reasons why some people might value accumulation, which is also worth bringing up. There’s a desire for political, philosophical, or religious influence. People get into an arms race over what society looks like and what people believe.

Similarly but differently, because it’s not an arms race, there’s just total utilitarian philanthropy. When I think about why it might be good to have a lot of wealth in the future as a good classical utilitarian, to me, the value—or at least one way you could have an almost unsatiating utility function in having wealth in the future—is to create new happy beings. They just add to the total welfare of the world.

This idea goes at least as far back as Bostrom’s astronomical waste point, that we could put Dyson spheres around the stars and turn all the energy into really happy simulations and whatnot.

Dwarkesh Patel

I think the particular greediness of this optimizer doesn’t matter, what they’re greedy for. Forgetting about utilitarian philosophy or whatever, a pure von Neumann probe has… I don’t know, is this an accurate way to say it? They just have high marginal value for the random solar system they’ll occupy because that turns into more solar systems. A von Neumann probe is a thing that can exist. That’s a very greedy optimizer.

Phil Trammell

If we’re talking about whether they’ll dominate the economy, maybe this is a technicality. But we only count final consumption goods and investment goods as GDP. If there’s just this phenomenon—

Dwarkesh Patel

How does a von Neumann probe show up in GDP?

Phil Trammell

Exactly. If we recognize it as a person that owns itself, and it’s optimizing on the margin between spending a bit more on a baby von Neumann probe that colonizes another star system or a ballerina or something, and it just doesn’t value the ballerina very much…

Alex Imas

When we’re talking about AI beings, it just completely depends on how we’re doing the accounting there.

Dwarkesh Patel

What does the world look like in a world where von Neumann probes are possible? Is it possible labor share is high?

Phil Trammell

I think it’s possible the labor share is high the way we usually count it.

01:01:28 – What should developing countries do?

Dwarkesh Patel

Do economists have any advice for countries which are not in the AI production chain? If you’re not either producing the AI models, you’re not producing the hardware that goes into AI models, if you’re not Korea making HBM or Taiwan with the fabs or the Netherlands with ASML. India or Nigeria, what should they be doing right now? If you’re talking to Modi right now, what do you say?

Alex Imas

I think the biggest lack of resources that we have allocated in the economics profession is thinking about middle-income developing countries in the age of AI. This is something I fault myself with as well. There’s not enough people thinking about this question.

There are scenarios where you get AI technology being allocated and dissipating to Nigeria and developing countries, leveling the playing field, essentially giving them a level up as far as capabilities.

But there’s another world where, because they don’t have enough resources, they’re not training the models, they don’t have the hardware, and they just completely get left behind. And because of automation, we can produce commodities in developed countries now. Then we don’t even have the consumer market. That world looks pretty bad.

Phil Trammell

This seems to me like an extension of the messy middle case. One of the ways in which the messy middle might only be bad in a narrow range of scenarios isn’t just that it would be easy to redistribute because the pie would be bigger, but because the interest rate would be way higher, and/or, equivalently, the price of everything except human-intrinsic goods would be falling really rapidly. They’re sort of two sides of the same coin. A little bit of savings would turn into a lot of consumption next year.

Things have to go really wrong for us to just get over the threshold of capital being productive enough to automate lots of work, but not be productive enough that the interest rate is high and the price of capital-produced goods is falling a lot. Even without redistribution, a little bit of savings will save a lot of people.

Dwarkesh Patel

You’re saying if the developing countries have some savings in the developed world, that will be enough to produce a lot of surplus that they can then—

Alex Imas

They will now be able to consume a lot using their savings.

Phil Trammell

But the messy middle could be wider in this case. They’re starting from such a lower level in terms of how much they have and how much it’s actually indexed to the global economy. I think it’s important for them to get on it now. I don’t have strong feelings about whether it should take the form of sovereign wealth funds that invest in the right supply chains or just subsidies to their own citizens to buy a little bit of—

Dwarkesh Patel

This is actually a crucial point. We were talking earlier about why the Rockefellers of the world, why their descendants don’t control everything, if our argument about the selection of these greedy optimizers holds. One argument is just that it’s very hard to index the economy. Maybe they would’ve just decided to have their heirs index the economy and have their wealth grow at the rate of economic growth, and their heirs would be trillionaires by now.

Before index funds existed, it was just very hard. A very small fraction of the economy, going back 100 years, accounts for a majority of the value created now. If you missed those particular things, your wealth would’ve just stagnated.

Maybe there was a brief golden window from the creation of index funds up until five years ago where you could actually index the economy and have your wealth grow at the rate the economy grows. But now we’re in this world with very concentrated returns, especially to private companies. As we were making the point in our blog post, this is capital that the average person has disproportionately less access to. Most of their capital is having a random house, at least in the US.

Phil Trammell

Or a part of a house.

Dwarkesh Patel

Which, as we were saying, is capital that is uniquely ill-suited to be complementary to the production of AI or the serving of AI or to robots.

Phil Trammell

Or the kinds of goods that the rich will bid up the prices of.

Dwarkesh Patel

Exactly. What is the value of a house currently? It’s that the land is close to other humans and modulo relational stuff that is just not going to be the main factor of production in the future.

Alex Imas

This would be why a Georgist tax would not raise enough money for the sort of programs that we will be discussing.

Dwarkesh Patel

Right. But stepping back, the point I was trying to make is, if it gets harder to index the economy now, and that is the main way in which normal people are supposed to—modulo some sort of universal basic income—

Phil Trammell

In the developed world.

Dwarkesh Patel

—are supposed to have some purchase on the wealth from AI. And it’s also the way that developing countries are supposed to have some purchase on the wealth gains from AI.

But it’s very hard. Does Nigeria own a lot of SK Hynix and Anthropic? I’m guessing not. It’s not enough for them to just own the S&P 500.

Alex Imas

This brings up a really important point. Is AI going to be like electricity or social media?

Think about ConEd, or whatever the electricity provider here is. It’s a monopoly. It provides a resource that everybody uses. But do we think about electricity as creating a concentration of power? Does ConEd have this huge amount of political power, social power, or something like that? No, because with electricity, a lot of the downstream benefits actually came to the users of the electricity rather than the actual entity producing it.

On the other hand, with social media, it was the opposite case. Social media was everywhere. Everybody uses social media, but the rents went to the platform.

Dwarkesh Patel

That’s a really interesting point. I don’t endorse this take yet, I’m going to talk out loud. The more you think our economy is going to be run on AGI the way our economy currently runs on electricity—that is, there’s a broad fundamental transformation of the entire economy—the more it looks like electricity… Every company in the S&P of the future, if it’s going to make it to the S&P 500, it is because it has leveraged AI.

Alex Imas

Exactly. And then you’re indexed again.

Dwarkesh Patel

But then again, I guess if you just look at how concentrated the S&P is over time, just these big tech companies much more so… I guess this goes to a fundamental point that it’s hard to reason about how much of the gains from AI these individual private companies will be able to control.

Alex Imas

I think the open model thing is going to be a big point here. If we’re indeed in a world where the open models are six months behind the frontier—or nine months—then we’ll hit AGI, we’ll hit whatever, and in six months, everybody has access to this resource.

Dwarkesh Patel

This goes to show you that every question is connected to every other. That question about whether there’s runaway gains connects to questions about recursive self-improvement. Even if not recursive self-improvement, then continual learning, or online learning, which lets a model learn on the job. So if it’s deployed, it gets to learn more. These are just forecasting technical questions which then impact whether Uganda will have any purchase on the returns of AGI.

The reason I’m emphasizing the question is I think both for the messy middle and for developing countries, a recommendation that is often made naively is that you’ve got to do some kind of retraining. You’ve got to do some kind of jobs program, or you’ve got to have them build data centers in your country. I think you guys are suggesting something closer to just buying the index of AGI. That’s probably a much cleaner strategy and much more likely to succeed.

Alex Imas

These are the two scenarios. I think there is a world where it is concentrated, in which case it’s going to be really hard to index AGI. There is another world where it is electricity. Basically every company has access to AGI. So you just buy the index. Nigeria just needs to buy the index, and Nigeria has access to AGI because of the open models.

Phil Trammell

Just to get back to the question about whether to go with retraining or trying to index. I would prioritize trying to index, just given how fast AI could hit the world. But I definitely wouldn’t just rely on that.

In the messy middle cases or the long-timeline cases where we don’t get anything like AGI all that soon, it would be leaving a lot of value on the table if you could have retrained to be a bit better educated on how to use the latest wave of computing. I don’t think there’s that much of an either/or there.

Dwarkesh Patel

Maybe the reason to be pessimistic about this is because one of the reasons a country is poor is that it has a bad education system, so becoming the best in the world at retraining people at using AI doesn’t seem like a particularly promising strategy for that poor country.

Alex Imas

Although there are cases where, in developing countries, you had this leapfrogging effect with, for example, mobile banking. It’s much more prevalent in Nigeria than it is in Germany. Everybody is doing mobile banking. They have it on their phones, and they’re constantly doing this sort of thing.

Again, I’m not putting probabilities on this, but with a transformative technology like AI, you could get leapfrogging where you skip the step in the middle and get really astronomical growth.

Phil Trammell

Just about the ease of indexing, I think it’s definitely something to worry about a bit and keep an eye on. But as discussed in our own essay, and as other people have pointed out, it’s already not that hard to index. There’s been a bit of an increase in the privatization of returns, but still, well under 20% of the total market cap of non-tiny companies in the US is private.

Everyone thinks about OpenAI and Anthropic. If that’s where all the wealth will accrue, then all these questions about whether open models will stay only a little bit behind, those are important. But even they look like they’re going public before too long, probably. The frictions that have been keeping companies from going public might themselves be alleviated by AI a lot, just all of the disclosure requirements and whatnot. They want to get access to more potential investors, too.

If I had to guess, I would guess that the long general trend of lowering those frictions and making it easier for more and more people to index will continue, despite the recent bump in the other direction.

Dwarkesh Patel

This actually makes me hope even more so than before that the labs do get commoditized, or at the very least they go public as soon as possible. But hopefully they just get totally commoditized. I think AI will be much more popular and, more importantly, will be much more likely to lead to broad increases in prosperity if it is as hard to capture the gains of AI as it is to capture the gains of electrification.

Alex Imas

Exactly. There’s no anti-electricity people out there.

Dwarkesh Patel

I mean electricity doesn’t take your job, but—

Phil Trammell

Well, it takes some people’s jobs.

Dwarkesh Patel

It took some people’s jobs, yeah.

Alex Imas

This is maybe tangential to the conversation but I think narratives matter. There’s this really negative narrative around AI right now, but that’s because people are not putting out the positive narrative. There’s a reason. It’s more difficult to imagine a good thing that doesn’t exist than losing something that exists. It’s much easier for somebody to go on a podcast and say, “These jobs that you like, they’re going away,” than for somebody to spin up a utopia which doesn’t exist yet.

Phil Trammell

I hope this isn’t too out of left field, but I would be remiss if I didn’t point out one big cost of having commoditized frontier AI models, which is the tech race dynamic. For safety purposes, you might want fewer frontier companies so that each one has a buffer in case they want to slow things down to make things safer.

The way this relates to our point before about the widespread access of the returns, is that I think there’s a lot less of a trade-off there than some people imagine. Some people think either frontier AI gets commoditized and we all enjoy the benefits—but there might be some risk, because the market’s really competitive and cutthroat—or things are safer because there’s a big gap between the leader and the laggard. But that means the leaders get fantastically wealthy? No. You could just have a relatively big gap, but it’s a public company, and ownership in it is widely distributed.

Dwarkesh Patel

More recently, I have been thinking that the risk of commodification—which is that it diffuses the ability to use AI to harmful ends—is worth the benefit. I worry that having these concentrated labs not only makes it so that the surplus isn’t as widely distributed through society, but also creates a very tangible, clear political target for the government. We saw this with the Defense Production Act threat against Anthropic. If there wasn’t one lab, or a couple of labs, that are clearly ahead of others, this kind of threat would be much harder to make.

Thank you guys for doing this. I feel like there’s a lot of unresolved questions, but it is helpful to know what the first branch is along all these important dimensions.

Phil Trammell

Great.

Alex Imas

Thank you.

Reiner is CEO of MatX, a new chip startup (full disclosure - I’m an angel investor). He was previously at Google, where he worked on software efficiency, compilers, and TPU architecture.

Watch this one on YouTube so you can see the chalkboard.

Sponsors

• Crusoe was one of only five GPU clouds that made the gold tier in SemiAnalysis’ most recent ClusterMAX report. Gold-tier providers like Crusoe delivered 5-15% lower TCO than silver-tier clouds, even with identical GPU pricing. This is because optimizations like early fault detection and rapid node replacement don’t necessarily show up in the sticker price, but still matter a ton in the real world. Learn more at crusoe.ai/dwarkesh

• Cursor is where I do most of my work—from reading research papers to visualizing technical concepts to coding up internal tools for the podcast. Most recently, I used it to build two different review interfaces for my essay contest, one that anonymizes submissions for scoring and another that lets me see applicants’ essays next to their resumes and websites. Whatever you’re working on, you should try doing it in Cursor. Get started at cursor.com/dwarkesh

• Jane Street let me ask Ron Minsky and Dan Pontecorvo, two senior Jane Streeters, a bunch of questions about how they use AI. We discussed everything from the types of models they’re training to how they think about the future of trading to why they’re more bullish than ever on hiring technical talent. You can watch the full conversation and learn more about their open positions at janestreet.com/dwarkesh

Timestamps

00:00:00 – Building a multiply-accumulate from logic gates

00:16:31 – Muxes and the cost of data movement

00:26:10 – How systolic arrays work

00:39:11 – Clock cycles and pipeline registers

00:51:51 – FPGAs vs ASICs

01:03:25 – Cache vs scratchpad

01:07:27 – Why CPU cores are much bigger than GPU cores

01:12:00 – Brains vs chips

01:15:33 – A GPU is just a bunch of tiny TPUs

Transcript

00:00:00 – Building a multiply-accumulate from logic gates

Dwarkesh Patel

I’m back with Reiner Pope, CEO of MatX, a new AI chip company. Last time we were talking about what happens inside a data center. Now I want to understand what happens inside an AI chip. How does a chip actually work? Full disclosure, by the way: I am an angel investor in MatX. So hopefully you have designed a good chip.

Reiner Pope

Hope so. I’ll start with the smallest fundamental unit of chip design, and we’ll build up to what an actual production chip is and what its components are. At the very bottom level of a chip, the primitives we work with are logic gates, very simple things like AND, OR, and NOT. These are connected together by wires that have to be laid out physically as metal traces on a chip.

The main function that AI chips want to compute is the multiplication of matrices. Inside that, the fundamental primitive is a multiply-accumulate of pairs of numbers. We’re going to demonstrate what that calculation looks like by hand, and then infer what a circuit would look like for that.

It’ll be easiest if I do a multiply-accumulate of a four-bit number with another four-bit number. The clearest primitive is actually multiply-accumulate. So there’s a multiply of these two terms, and then we’re going to add in an eight-bit number.

Dwarkesh Patel

Can I ask a clarifying question? Why is this the natural primitive for whatever computation happens inside a computer?

Reiner Pope

There are a few reasons. It’s a little bit more efficient, but the reason it’s natural for AI chips is that if you look at what’s happening during a matrix multiply… What is a matrix multiply in short? There’s a for-loop over i, over j, and over k, of output [i, k] += input [i, j] x other input [j, k]. A multiply-accumulate happens at every single step of a matrix multiply.

The other observation is that the precision will almost always be higher in the accumulation step than in the multiplication step. This is specific to AI chips. You’re multiplying low-precision numbers, and then when you accumulate, errors accumulate quickly, so you need more precision there. This is why we’ve chosen to do a four-bit multiplication and an eight-bit addition.

Dwarkesh Patel

Let me make sure I understood that. There are two ways to understand that. One is that the value will be larger than the inputs. The other is that if it was a floating-point number it would be… Maybe that part is less intuitive to me. But maybe it’s the same principle?

Reiner Pope

It really is the same principle. The separate principle is that as you’re summing up this number, you’re summing up a whole bunch of numbers, so you’ve got a lot of rounding errors accumulating. Whereas in this case, there’s only one multiplication in the chain, so there aren’t a lot of rounding errors accumulating in the multiplication.

Dwarkesh Patel

Why are you summing up a whole bunch of numbers? There’s just two numbers there.

Reiner Pope

This summation is repeated j many times.

Dwarkesh Patel

Any errors accumulate. I see.

Reiner Pope

So how would we perform this calculation by hand? As a human, we would probably separate it into two, but we can do it all in one using long multiplication.

For the multiplication term first, we’re going to multiply this four-bit number by every single bit position in the other four-bit number. We write that out. First, 1001 multiplied by this bit position. That is the number itself. Then shifted across by one, we’re multiplying by 0. That gives us an all-0 number. Shifted across one more to multiply by this one, we get 1001. Finally, for this last bit position, we get an all-0 number again.

This gives us a bunch of terms that we have to add for the multiplication. While we’re doing that summation, we might as well add in the actual accumulator term as well. So we just copy that directly across. So this is the sum. It’s a five-way sum that we want to compute.

What logic gates did it take us to get to this intermediate step? We needed to produce all 16 of these partial products. How do I produce one of these partial products? Let’s take this number 1, for example here. We produce it by multiplying this number by this one over here. We can produce that with an AND gate. This number is 1 if both this bit is 1 and this bit is 1. If either of them is 0, then the multiplication of 0 times anything is 0. To produce all of this, we ended up consuming 16 AND gates. In the general case, if I were doing a p bit multiply times a q bit multiply, this will be p times q many ANDs.

Finally, I sum them. Most of the work is going to happen in the summing. Let me describe the other logic gate that we use here. AND is almost the simplest logic gate that exists on a chip. It’s almost the smallest. At the other extreme, the very largest logic gate you’ll typically use is something called a full adder.

Coming from software, you might think that a full adder adds 32-bit numbers together. In this case, it just adds three single-bit numbers together, so you can think of it as adding 0, 1, and 1 together. When I add these together, the result can be 0, 1, 2, or 3, so I can express that in binary using just two bits. As input, it has three bits. As output, it has two bits. The number 2 in binary is 10. This is also known as a 3→2 compressor because it takes three bits of input and produces two bits of output.

Dwarkesh Patel

Just to make sure I understood: the two inputs are an X and a Y value and then some carry that came in…

Reiner Pope

The three inputs are all bits in the same bit position, like three bits in a column here. The two outputs, I’ve drawn them vertically here and horizontally here to match this vertical versus horizontal layout. This expresses that things in the same column are in the same bit position, whereas things in adjacent columns are different. This is a carry out, whereas this was the sum.

Dwarkesh Patel

So if the inputs in the full adder were, say, 101, then the output would be 10. If it were 111, it’d be 11. If it were 000, it’d be 00. If it were 010, it’d still be 01. Got it.

Reiner Pope

Yeah. It’s essentially just counting the number of things and expressing that in binary. This circuit captures what we as humans naturally do when we’re summing along a column.

I’ll show one iteration of using the full adder to sum. The way I sum here is going to be a little unnatural for humans. We would sum along the column and then remember the carry, but instead of remembering the carry, we’ll explicitly write it out. We proceed from the rightmost column toward the left. On the rightmost column, we sum the 1 and the 1, and that produces a zero here and a carry of one. We’ve used this full adder circuit on this pair of bits and produced a pair of bits as output.

Now we can do the same thing with this column. We have a column of four numbers, so we’ll take the first three of them, run a full adder on them, and that gives us a 0 and a 0 as output. The sum of these is 00. That’s the full adder applied to all these bits. As I’ve used up bits, I’ll cross them out to indicate that I’ve handled them.

Let’s keep going a little bit more. I take these three numbers, I add them, and that gives me a 1 and a 0. I’ve dealt with these three numbers. Now I take these three numbers and add them, and that gives me a 1 and a 0, and I’ve dealt with these numbers.

The way to view this is that I have this whole grid of numbers that need to be added. I’m going to keep applying full adders to all the bits here, constantly removing three numbers from a column and writing out two numbers as output. Keep going over and over again until I eventually get just one single number coming out.

This approach is called a Dadda multiplier. This is the standard for how you do area-efficient multipliers using full adders. Let’s try to quantify the circuit size of this so we have a sense of how big things are and can compare them later.

How many full adders did I use? How many numbers did I start with? I have the 16 partial products, which is a product of all of these terms with all of these terms, plus the eight terms that I’m adding here. I started off with 24 bits. Eventually I produced eight bits on the output. In every step, I was crossing off three numbers and writing two numbers out as a result.

Every single use of a full adder eliminates one of the bits here. So how many full adders? It must be 24 minus 8, so there were 16 full adders in this circuit. This is true in the general case as well. There will be p times q many full adders in this circuit.

Dwarkesh Patel

Let me make sure I understand the logic of that. The input bits, 24, is p x q, plus p + q. The output bits are just p + q. So p x q plus p + q, minus p + q equals p x q.

Reiner Pope

That’s right. I think this explains, or at least hints at, the second reason we chose to do a multiply-accumulate. The first reason is that it’s what shows up in matrix multiplication. The second is that it gave us this very slick, simple p x q, very simple algebra.

We’ve described this whole procedure. Every single atomic step that I took here becomes a logic gate, and then the wires are connected together. When I had these three inputs that I used to produce these two outputs, if I think of mapping this to a physical device, there would be a wire connecting all three of these things together into a logic gate that produced this output.

This is the main primitive, at different bit widths, that’s inside an AI chip. We’re going to build up from here to how you would use it to run all the other operations you might want.

Dwarkesh Patel

This might be the wrong time to ask, but whenever Nvidia reports that this chip can do X many FP4 or half as many FP8, it seems to imply those circuits are fungible, that there’s not a dedicated FP4 versus FP8. But the way you’re mapping it out here, it seems like if it has to be mapped out in the logic, you would need a dedicated FP4 multiply-accumulate and then a dedicated FP8 accumulate. Can you “funge” them?

Reiner Pope

As drawn, they’re not particularly fungible. This is actually one of the main choices you have to make when designing a chip: how much of FP4 and how much of FP8 do I have? Sometimes I’ll make that consideration from the point of view of the customer requirement. Another angle is to equalize the power budget between FP4 and FP8.

Dwarkesh Patel

When they report those numbers and it just happens to be the case that it does 2x as many FP4 as FP8, they’re just choosing to give equivalent die areas to all the floating points, and as a result it ends up being…?

Reiner Pope

Why is the ratio exactly 2x? Part of it is that surely it won’t be exactly equivalent to die area. There’s a data movement reason. We’ll maybe come back to this when we look at how it goes into and out of memories. There’s something really nice from a software level about the fact that I can pack two four-bit numbers into the same storage as an eight-bit number. When I store that to memory, the sizing of the buses that I wire out within the chip makes that work out really nicely.

Dwarkesh Patel

Come to think of it, it’s not just 2x. The amount of area it takes sounds like it’s quadratic with the bit length. That’s why smaller precision is even more favorable than you might think.

Reiner Pope

This is a really big reason. In fact, Nvidia made a change. Historically, up until B100 or B200, every time you halved the bit precision, you doubled the FLOP count. For the reason you said, because of this quadratic scaling, that ratio is actually slightly wrong. You should get an even bigger speedup than you might otherwise think. Nvidia’s product specs have started acknowledging that in B300 and beyond, where the FP4 is three times faster than the FP8.

Dwarkesh Patel

Though it should be 4x.

Reiner Pope

Yeah. What I’ve shown here is the simplest case of integer multiply. When you’re dealing with floating point, as you do in FP4 and FP8, there’s this other term, the exponent, that complicates the calculation.

What can we see already from this? I think the big observation you’ve made is that there’s this quadratic scaling with bit width, which is very effective and is the single reason low-precision arithmetic has worked so well for neural nets. The other thing we’re going to do now is compare the area spent on the multiplication itself with all the circuitry around it.

00:16:31 – Muxes and the cost of data movement

Reiner Pope

We’ll walk back in time a little bit and see how GPUs prior to Tensor Cores worked, which is in fact the same way CPUs worked. Where do we stick this multiply-accumulate unit? Generically, I’ll describe a CUDA core or a CPU. You’ll have some register file which stores some number of entries, maybe eight entries of, in this case, 4-bit numbers, but typically 32-bit numbers.

Inside the CUDA core, I’ll have some register file of some depth, and then I’ll have my multiply-accumulate circuit. What it’s going to do is take three arbitrary registers from this register file, perform the multiply-accumulate, and then write back to the register file. It’s going to write to this one, but it was able to read from this one, this one, and another random one. It will take three inputs like this.

This is the core data path of many processors. Most processors look like this. You’ve got some set of registers, and then you’ve got some set of logic units, or ALUs. We want to analyze the cost of the data movement from the register file to the ALU and back.

Ultimately, there’s going to be some circuit that says, “Well, I don’t always have to select this guy. I might select any of the registers at any point in time.” The first question is: how can I build a circuit? The circuit I’m going to look for is a mux. In this case, it’s going to have eight inputs, one from each entry of the register file, and it’s going to have one output, which is actually producing this output.

What is the cost of this thing? All we have to build it out of is AND and OR. How do we build it? We do the dumbest thing possible. We form a mask. When we want to read the third entry, we’re going to AND every single entry with either 1 or 0 based on whether that’s what we want to read, and then we’re going to OR all of them together.

Dwarkesh Patel

Just to make sure I understand the basics. What the mux is doing is just selecting an input?

Reiner Pope

Just selecting, invisible to software. You say “I want input number three,” and that means there’s a mux here.

So what is the cost of this mux? An n-input mux operating on p bits. I have n rows. That’s just eight rows, and each row is p bits wide. I have to AND every single bit, so I get n x p many AND gates. For every single input I have to decide whether I’m going to mask it out or not. Then I’m going to OR them all together.

There’s going to be n – 1 times p many OR gates. I’ve got all of these different things, almost all of them are 0s, but I need to collapse them from my eight options down into one option. Every step, I need to OR one row into an existing row.

Dwarkesh Patel

It’s actually funny that you don’t think at the level of hardware. You just think, “Oh, I’ll just select element three,” and something as simple as that is in and of itself quite a complicated circuit.

Reiner Pope

This is the first step of all of the hidden data movement costs that show up. We’re just going to compare. I have to pay this cost. I’ve got one mux here, and in fact I have two more copies of that for each of the three inputs to my multiply-accumulate operation. I have this cost, which is 3 x n x p AND gates over here, compared to p x q gates in the actual circuit that is doing the thing I care about.

If we plug in actual numbers, with n being eight, I get 24 x p gates just in the data movement, compared to—if q is four—4 x p gates just in the multiply-adder.

Dwarkesh Patel

Where is the three coming from?

Reiner Pope

Three different inputs here. What I’m hinting at is that all of this work, which scales as the size of the register file—and this is a very small register file—all of this work just moving the data from the register file to the logic unit is many, many times more expensive than the logic unit.

Dwarkesh Patel

It may be helpful to just see what a mux looks like, maybe a two-bit or a four-bit mux.

Reiner Pope

We’ll do a two-way mux. We’ve got two different numbers, we’ve got these two inputs. These are the inputs that are being selected between, and the selector can either be “I want this one” or “I want the other one.” This is a one-hot encoding.

This is what we start with. Let’s focus on this case. This is the actual input we got, and we want to produce this guy as the result. Very laboriously, we AND this bit with all of these. That produces ANDing this bit with this row. Likewise, we AND this bit with this row. That produces all 0s. There are four ANDs here.

Finally, we OR these two together, and this gives a 1. We OR these two together, this gives a 1. We OR these two together, it gives a 0. We OR these two together and it gives a 1. Those are the four ORs. This actually ends up looking a little bit like addition. We did exactly the same set of ANDs. We’ve ANDed all of these things together, but then instead of collapsing it by using full adder circuits, we just get a very simple collapsing with OR gates.

Dwarkesh Patel

But that doesn’t look like n times p.

Reiner Pope

This was with n=2 inputs. In the general case, we will have n rows, and we’ll have p bits per row. That gives us the n times p many AND gates. In this circuit I’ve described, almost all of the cost, seven-eighths of the cost, is in reading and writing the register file, and only a tiny fraction of the cost is in the logic unit itself.

This is the problem to solve. This essentially was the state of play prior to the Volta generation of Nvidia GPUs. This kind of thing is what was inside the CUDA cores. This problem statement is what motivated the introduction of Tensor Cores, which are more generically called systolic arrays.

Think about how we’re going to solve this problem. We’re spending almost all of our circuit area on something that we really don’t care about and is hidden to the software programmer, and the thing that we actually care about is not much of the area. Make this one bigger somehow while keeping this at the same size. That’s the goal.

00:26:10 – How systolic arrays work

Reiner Pope

The evolution was that we had baked this much into hardware at this stage. This single line is a multiply-accumulate, and this single thing was baked into hardware. The idea of a systolic array is to go two levels of loops up and bake this entire loop out here into hardware. The idea is that if we have a much bigger granularity fixed-function piece of logic, maybe the taxes we pay on the input and output are much smaller.

Dwarkesh Patel

Interesting. It sounds like you’re suggesting that if you go up one step in the matrix multiply loop, you can tilt the balance more towards compute than communication.

Reiner Pope

That’s right. There are two effects we’re going to take advantage of here. One is that we can do more stuff per every trip through our register file. The other is that in some of this loop, we can take advantage of certain things staying fixed.

Visually, we’re going to look at this matrix multiplication. This portion of the loop corresponds to a matrix-vector multiplication. We’ll take a matrix and multiply it by a vector. How do we do this? Every column gets multiplied by the vector and then summed. We’re going to sum along columns.

This 0 and 3 gets multiplied by the 3 and 7 and gets summed, and then the 1 and 2 gets multiplied by the 3 and 7 and gets summed. There is a multiply-accumulate associated with every single one of these entries in the matrix. We’ll draw out these four multiply-accumulates.

Dwarkesh Patel

Just to make sure I understand why there are four multiply-accumulates: each entry in the column that corresponds to the output vector is a dot product, and in this case it will be two multiplications and then the addition of those two multiplications. You’re accumulating...

Reiner Pope

Really there’s only one addition per dot product, but we like to start with zero.

Dwarkesh Patel

But it includes the initialization of zero.

Reiner Pope

Yeah. We want to have quadratically more compute. We have x times y as much compute as we had before. But we want to aim for having only x times as much communication. The intention is to get this advantage term going as y.

We’ve laid down the multiplications. We want to bring in a vector of size two, and that is already in line with our columns target. That’s fine. However, we need to manage the communication of this matrix, which exceeds our budget of x.

The idea is that in an AI context, this matrix is going to stay fixed for a long period of time. We’ve got some register files sitting over here.  The amount of stuff coming out of this register file… this is the term that we want to go as x, in some sense. We don’t want to bring this full matrix in from the register file every cycle, because that would cost too much in terms of wiring from the register file.

Our key trick is that this matrix can be stored locally to the systolic array. We’ll store these numbers 0, 1, 2, and 3 in a gate called a register that physically stores these numbers, and we’re going to reuse these numbers over and over again for a large number of different vectors.

Dwarkesh Patel

The optimization here is that the nature of matrix multiplication is that you can store this square quadratic thing directly where the logic is happening, which has an extra dimension compared to the inputs that you keep swapping in and out.

Reiner Pope

That’s right.

Dwarkesh Patel

This is the nature of what a matrix multiplication is. You do a lot of multiplication to get one value out. A dot product is the result of a lot of multiplications. So that optimization means you can stuff a lot of multiplication in before you get some value out of it.

Reiner Pope

That’s right. Just to complete the picture of concretely how that looks: I swapped the 3 and the 2 here. Just like this 0 and 3 is going to multiply by the 3 and the 7, we’re going to form a dot product along columns here. We’re going to feed a 3 and a 7 in here. This feeds into this multiplication and also feeds into this multiplication. Likewise, the 3 feeds into here and also into here. Then we’re going to sum along here. Starting at the top of a column, we feed in 0s, and then coming out the bottom we get results.

Visually, there’s a dot product performed along columns in a matrix, and that maps exactly to what is done spatially in the systolic array. This is one dot product summed vertically, and this is a second dot product also summed vertically.

What is the data that needs to go into and out of the register file? We have x amount of data coming out on the output, and we also have x amount of data coming from the input. With respect to the input and output vectors at least, we’ve met our goal of having only x as much data going in and out of the register file.

This leaves open the question: I said the weight matrix is stored locally in the systolic array, so how did it get there in the first place? At some point, you need to boot your chip and populate this data, so where did that come from?

The trick is that we just do it very slowly. We very slowly trickle-feed it into the systolic array. The simplest strategy is that we run this daisy chain: feed a number in here, and on the next clock cycle it will move down to the next entry of the systolic array. We can do that in every column in parallel, this is also going to come from here, and that gives us another factor of approximately x units of bandwidth coming in.

Dwarkesh Patel

Would you mind repeating that sentence one more time?

Reiner Pope

We know that we’re going to be bringing numbers only rarely into the matrix. We just want to come up with any construction at all such that the amount of wiring that crosses the boundary of the systolic array is bounded to x and not go as xy.

A particularly simple strategy is that we bring a number into the top row of the systolic array in one clock cycle. Then, for y consecutive clock cycles, we bring in the top row every time and shift all the other rows down by one. This keeps the wiring that needs to come from this expensive register file only down to a factor of x rather than xy.

Dwarkesh Patel

I see. There are two questions in terms of communication: communication time and communication bandwidth. You’re saying that since we’re only going to be loading this in once, let’s minimize bandwidth, because bandwidth equals die area. We load it in slowly over smaller lanes because we’re just going to keep this value in there for a while.

Reiner Pope

Exactly.

Dwarkesh Patel

It’s interesting to me that when we were talking last time about inference across many chips, the big high-level thing we’re trying to optimize for is increasing the amount of compute per memory bandwidth, that is to say, per communication. Here also, we’re trying to increase the amount of actual multiplies or additions relative to transporting information from registers to the logic. In both cases, you’re trying to maximize compute relative to communication.

Reiner Pope

This shows up all the way up and down the stack. This is close to the bottom, to the gates. There’s a version that’s maybe even closer to the gates in the precision of the number format that you choose to use. We saw that same effect. There’s a squared versus linear term going on both purely in the precision of the ALU, but also in the size of the matrix.

This unit is the next bigger unit. We had the multiplication circuit, and on top of that we have a pretty large systolic array. I drew it as 2x2, but older TPUs were described as 128x128 of this circuit shown here. This ends up being the most efficient known circuit for implementing a matrix multiply.

Dwarkesh Patel

We’ve talked about how it seems obvious that you should try to maximize compute relative to communication. What are non-obvious trade-offs that keep you up at night, about whether you should do X or Y and it’s not obvious what the answer is?

Reiner Pope

Most of the decisions in chip design are sizing decisions. Already in what we’ve drawn so far… AI chips all have this circuit in them. They have a systolic array, and somewhere near it a register file providing inputs and outputs.

Even within this scope, the sizing questions you have are: how big should I make my systolic array, and how big should I make the register file? These two questions are coupled. One way to think of it is to set a budget for what percentage of chip area you want to spend on data movement. Maybe I say that I want this to be 10% and the systolic array to be 90%.

Then I can size my register file. Bigger register files are more flexible. They allow me to get more application-level performance out, but they take away from the area spent on the systolic array.

00:39:11 – Clock cycles and pipeline registers

Dwarkesh Patel

Where does the clock cycle of a chip come in? What determines what that is? And what is a clock cycle of a chip?

Reiner Pope

At baseline, it’s worth observing that chips are incredibly parallel. You’ve got 100 billion transistors in a chip. A key thing you need to do whenever you have massive parallelism is synchronize between the different parallel units.

In software, typically you have these very expensive synchronization methods like a mutex. One thread will finish what it’s doing, grab a lock stored somewhere in memory, and notify the other thread that it’s done. On chips, we take a very different approach. Every nanosecond or so, all circuitry in the chip will pause for a moment and synchronize. That is the clock cycle. The entire chip typically goes in lockstep to the next operation in one fell swoop.

What this looks like in circuitry is that the clock is mediated by registers, which are these storage devices we’ve drawn elsewhere. The way to think of it is: I have some storage holding a bit, which might be 0 or 1. Then I have some cloud of logic, which maybe is this systolic array or multiplier. I have a bunch of inputs feeding into this cloud of logic, and eventually there’s going to be some output register that it writes to.

There is a global clock signal driving all these registers. At a certain instance in time, when the clock strikes, whatever value happens to be on that wire at that instant is what gets stored.

The challenge is that I would like to have my clock speed run as fast as possible. If I run at two gigahertz, I get twice as many operations done per second as if I run at one gigahertz. But what that ends up meaning is that I’m very sensitive to the delay through this cloud of logic, because any computation happening in there needs to finish before the next clock cycle hits. A major point of optimization on any chip is to make this delay as short as possible.

Dwarkesh Patel

Interesting. The constraint here seems to be that if you add too much logic, you might risk missing the clock cycle. But if you don’t add enough, you’re leaving potential compute on the table. Is there ever a situation where you take a probabilistic chance that a computation finishes, or is it strictly that it either finishes by the clock cycle or it doesn’t?

Reiner Pope

In standard chip design, you margin it such that there is a probability, but it’s many standard deviations out. For all intents and purposes, it is a reliable part and will always meet the clock.

There are some weird exceptions, like clock domain crossings where you go from one clock to another. Then you actually do have to reason about this probability. But in the main path, you margin it such that you’ll get there 25% of the clock cycle in advance, making it very unlikely that it misses.

Dwarkesh Patel

Where the clocks synchronize, where the registers are, is this something you determine as a chip designer? Or is it an artifact where you want a certain sequence of logic, and the software you use to convert your Verilog into what you send to TSMC just determines that to make it work, you have to put a register here, here, and here, making sure no single step makes the whole chip’s clock cycle longer than it has to be?

Reiner Pope

Inserting them is actually a huge part of the work of designing a chip. It’s done by a combination of manual and automatic methods.

To show the very dumb version of what you can do here, you can take this logic and split it in half. Instead of just one cloud of logic, I can have two smaller clouds of logic that do the same thing, but split them up by a register. If you split it in the middle, you can hit twice the clock frequency. That’s great, you get twice the performance, but at the cost of an extra register, which means more storage.

Dwarkesh Patel

Stepping back, why do we need to synchronize the whole chip? If you imagine playing Factorio or something, there’s no global clock cycle. Things are just done when they’re done. There’s iron on the plate, and you can take it if you want.

Reiner Pope

Taking that analogy, the thing you need to be mindful of is if I have two different paths through some logic. Say I have to do computation f here and computation g here, and they’re going to meet for computation h.

There’s going to be manufacturing variance. In some chips f will take a little longer; in some chips g will take a little longer. If I have a signal propagating through, and the results from f and g have to meet up at h, what can go wrong is that f gets there early and it meets the previous value of g, or the next value of g.

Dwarkesh Patel

Ah. And h needs to know when to start, when the next iteration has…

Reiner Pope

Exactly.

Dwarkesh Patel

This explains why different chips made at the same process node, the same TSMC technology, can have different clock cycles. Two chips made at 3 nm might have different clock cycles based on whether they were able to optimize to ensure no single critical path is so long that it slows down the whole chip’s clock cycle.

Reiner Pope

That’s right. This optimization I showed here is called pipeline register insertion. We’ve inserted a register in the middle of the pipeline. This is a pure trade-off between clock speed and area. That is the easy case. There is a harder case too. I drew out a pipeline of logic, but in other cases you may have some calculation which actually feeds back in on itself. It runs some function f and then writes back to itself. For example, this might be an addition where you’re adding a number every clock cycle. This little circuit essentially sums all the numbers presented on different clock cycles.

The challenge is, if this plus takes too long, what can I do? If I try to put a pipeline register right in the middle of it, it changes the computation that’s done. Instead of forming a running sum of everything that comes in, I will actually have two different running sums. I’ll end up with a running sum of the even numbers and a running sum of the odd numbers. This constraint—where I have a loop in my logic, which all chips have somewhere—is the hardest thing to address and sets the clock cycle.

Dwarkesh Patel

I don’t understand why it would be a problem to have that. I’m not even sure what it would mean to have a register there. Is it a sort of atomic operation?

Reiner Pope

Well, plus is not really atomic.

Dwarkesh Patel

As you just demonstrated.

Reiner Pope

It took a whole lot of work to do a summation. You can take the early parts of that work, stick a register in the middle, and then take the late parts of that work.

Dwarkesh Patel

Okay. TSMC offers a PDK which specifies the primitives of logic they can grant you in the chip. It’s up to them to determine that no primitive is bigger than the clock cycle they’re hoping a process node targets. But other than that, can’t you just take all the primitives from TSMC and keep adding registers between them as much as needed until you get to your desired clock cycle?

Reiner Pope

As a logic designer, the chip architect sets the clock cycle. For example, the primitives you get from TSMC are on the order of AND gates or full adders. It depends a lot on voltage and which library you choose, but generally you can have about 10, 20, or 30 of these sequentially in a clock cycle. These primitives are very fast, maybe 10 picoseconds.

As a logic designer, in principle, if you just had a register and an AND gate in a loop, you could get an insanely fast clock speed, more than four, five, or six gigahertz. But if you take this really simple circuit and look at the area you’re spending here… This is called one gate equivalent in size, so unit of one in area. This thing is maybe a unit of eight in area.

Again, almost all your cost becomes synchronization or communication cost compared to the actual logic. This would be a case where you’ve gone too far. You’ve made your clock speed really fast at the cost of spending almost all of your area on pipeline registers.

Dwarkesh Patel

Interesting. So you’re hinting at a dynamic where you can have a really fast clock speed but you’re not getting that much work done. You can have low latency but low throughput.

Reiner Pope

It hurts your throughput, in fact, because the throughput of your chip is the product of how much you get done per clock cycle—which is based on area efficiency—times how many clocks you get per second.

Dwarkesh Patel

This is actually so similar to the thing we were discussing last time about batch size, where if you have a low batch size, any one user can receive their next token really fast, but the total number of tokens processed in, say, an hour will be lower than it could otherwise be.

Reiner Pope

Exactly. You get less parallelism out if you drive your clock speed up really high.

00:51:51 – FPGAs vs ASICs

Dwarkesh Patel

I remember talking to an FPGA engineer at Jane Street, Clark, who helped me prep for the previous interview we did together. He was explaining why they use FPGAs. I imagine that for high-frequency trading, throughput is less important than latency, so having very specific control over the clock cycle in a deterministic way is the most important thing. Maybe it’d be interesting to talk about why you can’t just achieve that with an ASIC, or why you might use an FPGA to have deterministic clock cycles for high-frequency trading.

Reiner Pope

Let’s consider the business case for an FPGA versus an ASIC. FPGAs and ASICs use largely the same conceptual model. You have a series of gates built from small primitives—ANDs, ORs, XORs—connected together with wires running in a fixed clock cycle. Anything you can express in an FPGA you can express in an ASIC too. It will be about an order of magnitude cheaper and have better energy efficiency on an ASIC than an FPGA.

The trade-off is that the first FPGA costs you $10,000, whereas the first ASIC you make costs $30 million because it requires an entire tape-out. The business use case for an FPGA is when you want something that has very deterministic latency, fast runtime, and high parallelism, but you are going to change the workload frequently, maybe every month. You don’t want to pay that tape-out cost every time.

How does an FPGA actually emulate the ASIC programming model in a fixed piece of hardware? At its core, it has the two components we just talked about. It has registers as storage devices, and it has lookup tables (LUTs) which provide all of the gates.

Then there’s a third component. We have a swarm of these registers and LUTs, and they are connected by a big set of muxes. In front of every single one of these, we have a mux which selects an input from everywhere else. We have a whole bunch of different options feeding into all of these things.

What this allows is essentially that when I program my FPGA, I can take all of these components and superimpose a particular wiring which goes through this LUT, feed it into another LUT, send it to this register, and then feed it into another LUT, or something like that.

What I’ve drawn in orange is how you… FPGA means Field-Programmable Gate Array. The orange is what has been programmed in the field, whereas the white is all the wires that must exist in the FPGA in order to actually make the device in the first place.

Dwarkesh Patel

What does it mean to be programmed in the field?

Reiner Pope

Programmed in the field means the device is being deployed in a data center. It’s sitting out in the world, and then you can come and program it.

Dwarkesh Patel

Ah, not field as in like electric field. Field as in like out there in the world, ok.

If I look at how the field programming comes out of the first lookup table and goes into a second one, how does that work?

Reiner Pope

Where are the wires that make that happen? I got a little bit lazy in drawing all of these. Every single device here has a mux sitting in front of it, which can select from all of the nearby circuits that are available. The actual configuration of the FPGA amounts to the mux control. In this mux, we have the data inputs, and we have the control that selects.

There’s a little storage device sitting next to every single one of these muxes saying, “This is where you’re going to source your input from.” Programming it consists of configuring every single one of these muxes.

Dwarkesh Patel

That makes sense. What is happening inside of the lookup table?

Reiner Pope

The lookup table is also going to have a little bit of control telling it what to do. Its purpose is to configurably take the role of an AND gate, OR gate, XOR, or any of those different things. There are many ways you could consider doing that. The way it’s done in traditional FPGAs… A lookup table has four bits of input and one bit of output. How many different functions are there from four bits to one bit? There are 16 different functions.

You can tabulate this as 16 different numbers. You’ve got a table of 0111001, 16 entries. This table is stored in this blue configuration bit. It views these four bits as binary, looks up the relevant row of the table, and emits that bit. This is essentially a truth-table view of lookup tables.

Dwarkesh Patel

Okay, so if you think about an AND gate, OR gate, NOR gate, XOR gate, these all take as input…

Reiner Pope

Those are two-input functions. Sometimes we have a three-input function, like a three-way XOR, or a four-way XOR.

Dwarkesh Patel

In this case, does it just depend on how big it is?

Reiner Pope

The typical size for LUTs is four inputs. It’s sort of just a sweet spot. There’s another compute vs. communication trade-off here. If it has too few inputs, you need to use more LUTs.

Dwarkesh Patel

Basically the lookup table is a truth table. With a truth table, you can program in any gate you want. So instead of a lookup table, you can just think of it as a programmable gate.

Reiner Pope

That’s right. One of the things you can do here is you can see where the rule of thumb that an FPGA is an order of magnitude more expensive than an ASIC comes from. You count how many gates would be inside this lookup table.

We can view this lookup table essentially as one of these muxes. It has to select between 16 different values, so it’s a mux with n=16 options and p=1 bits. As we saw earlier, this circuit costs n times p many gates. So it costs np, which is 16, AND gates, and also 16 ORs.

Dwarkesh Patel

This circuit being the mux?

Reiner Pope

Exactly, the mux.

Dwarkesh Patel

The mux that goes into the lookup table?

Reiner Pope

The lookup table itself you can think of as being a big mux that selects from all 16 rows down to one output. That’s the lookup table.

Dwarkesh Patel

But the way you’ve drawn it here, there’s a mux and then a lookup table.

Reiner Pope

It’s muxes all the way down. There is a second mux that is inside here. This mux is this mux.

Dwarkesh Patel

And the other mux is just saying…

Reiner Pope

where it came from in this mess of gates.

Dwarkesh Patel

Right, and the second mux is, “Okay, now you have one value, but that value is still a four-bit value.”

Reiner Pope

Yeah, I’ve selected four bits from the soup. Then I use those four bits to select which entry in the lookup table I’m going to use.

Dwarkesh Patel

Suppose in the first mux you’re pulling from eight nearby registers as input. That’s a total of 32 bits going in. Out of that, four bits come out. Those four bits go into the second mux, which is inside the lookup table.

Reiner Pope

In this case, these registers are single-bit registers. If there are eight nearby registers and lookup tables, then I have eight bits total coming in nearby. I select from eight down to four different values. There are actually four different muxes, a little mux associated with each of these input bits. Each of them is selecting one out of eight.

Dwarkesh Patel

Where are those eight coming from?

Reiner Pope

Nearby registers and other LUTs.

Dwarkesh Patel

Each register is one bit.

Reiner Pope

Yes.

Dwarkesh Patel

I guess AMD or whoever makes these FPGAs still has to be opinionated about which registers are connected to which registers. You can program in the actual gates, but they add a wire in the connect… the communication topology, right?

Reiner Pope

You get flexibility at a local grain. There’s a nearby neighborhood you can select from, but for more coarse, long-distance connections, they form an opinion on that.

Dwarkesh Patel

And the reason it’s 10x slower is why?

Reiner Pope

If you look at the cost of building this lookup table, it’s 32 gates. It can give me the equivalent of—what’s an interesting thing I can do here—a four-way AND gate. A four-way AND means AND, AND, and then an AND of an AND. This is a circuit I could implement in an ASIC directly using three AND gates. Using a LUT, I can also implement it, but it’s going to take 32 gates instead of three.

Dwarkesh Patel

So the overhead is really coming from the fact that there’s a more concise way to describe a truth table than listing out every single possible combination of inputs, which is just to write out the gate.

Reiner Pope

Yes, to place down the polysilicon and the wires and so on.

01:03:25 – Cache vs scratchpad

Dwarkesh Patel

Interesting. One important point you made to me is that the reason they prefer FPGAs to CPUs is that they get deterministic clock cycles. They know when a packet will come in and go out. Why isn’t that a guarantee in CPUs?

Reiner Pope

You can actually design a CPU that has deterministic latency as well. In fact, the processors inside a lot of AI chips also have deterministic latency. Groq has advertised this. TPUs have that in the core as well.

The challenge is getting deterministic latency and high speed at the same time. Non-deterministic latency comes from specific design choices in a CPU. It’s actually possible to remove those design choices and make a CPU with deterministic latency, but those are not very attractive in the market, so people don’t make those CPUs anymore.

In some sense, deterministic latency is a simpler starting point, and some chip designers have added things in to make it non-deterministic. To take a concrete example, probably the most important source of non-determinism on a CPU is the CPU cache itself.

In a CPU, you have the CPU die itself, and then DDR memory off on the side. You have a cache system inside that remembers recent accesses to DDR and stores them. When I’m running through my CPU instructions, every time I have an instruction that accesses memory, it first checks if the data was stored in the cache. If not, it fetches it from DDR.

This is a huge optimization. The cache is two orders of magnitude faster than the DDR. If you never used the cache, basically all programs would run a hundred times slower. The presence of a cache is absolutely necessary for a CPU to run at a reasonable speed.

But whether or not you get a cache hit depends on the ambient environment of the CPU: what other programs are running, what has run recently, and what the random number generator inside the cache system is doing. That is a big source of non-determinism in the runtime of a CPU.

That is the memory system for a CPU. The big thing you can do differently is, instead of having the hardware say, “I’m going to read memory” and then the hardware decides whether or not it comes from the cache, you can bake this decision into software, a different design philosophy.

You see this in TPUs, for example. I’ll draw the same diagram, but I’ll call it a scratchpad. The main difference is… This would be a TPU, and you have HBM in this case rather than DDR, but it’s still an off-chip memory. Instead of the software saying “first access memory” and letting the hardware decide, you have one kind of instruction that goes to the scratchpad and a totally different kind of instruction that goes to HBM.

This style is generically known as scratchpad instead of cache. The key distinction is that you have one kind of instruction that says “read or write scratchpad,” and a totally different instruction that says “read or write HBM.”

Dwarkesh Patel

So scratchpad being the cache.

Reiner Pope

Yeah, this thing here is the scratchpad.

01:07:27 – Why CPU cores are much bigger than GPU cores

Dwarkesh Patel

Stepping way back: people say computers have the “von Neumann architecture”, where there’s this serial processing of information. Maybe it’s just because we’ve been talking about parallel accelerators, but the FPGA is super parallel. The AI accelerators, the TPUs, are super parallel. Even CPUs are super parallel if you think about all the cores they have. In what sense is modern hardware actually the von Neumann architecture? Is that actually a fair way to describe modern hardware?

Reiner Pope

I think it’s a fair way to describe CPUs. The amount of parallelism you get on a CPU is about 100 cores times maybe 16-way vector units, so about 1,000-way parallelism on a CPU.

Dwarkesh Patel

One question: there is a die being used for the CPU, and if there are fewer threads, just as a matter of transistor voltages switching on and off, is it that there’s literally one control flow—a small part of the die—where voltages are switching on and off?

How do you actually occupy the die area of a CPU…

Reiner Pope

If there are so few cores, what are you spending all of the die on?

Dwarkesh Patel

Yeah, what is happening there?

Reiner Pope

The cores are just much bigger and more complicated. We should compare a CPU core, which takes up one one-hundredth of the die, to a LUT. A LUT is only 16 gates. It’s clear why there are so many more LUTs in an FPGA than cores in a CPU.

But why are there more CUDA cores, for example, than CPU cores? What’s the difference between a CPU and a GPU? Inside the CPU, one big use of the area is the cache. Very little is actually the ALUs. Mostly it’s these register files rather than the logic units. Both of those have equivalents in a GPU, so that’s not a big difference.

But the thing that does not have an equivalent in a GPU is the branch predictor. There is a whole big area in the CPU which is just a bunch of predictors saying when the next branch will be and where the branch target is. Stripping a lot of that out, as well as making these register files tighter, drives a lot of the GPU gains over the CPU.

Dwarkesh Patel

What is the purpose of the branch predictor? To execute both branches at once, or what does it do?

Reiner Pope

The issue is that when I’ve got a series of instructions, if I have a branch, the actual step of processing an instruction takes a really long time. It takes maybe five nanoseconds.

The time to notice that I’ve got a branch, evaluate whether the Boolean is true, update the program counter to the new target, and then read from the instruction memory could take five nanoseconds to finish. So in reality, this may finish somewhere down here. I want to run a clock speed that is much faster than what five nanoseconds allows. Five nanoseconds is a 200 MHz clock speed. I would like to run at one or two gigahertz.

So I need to run other instructions while the branch is being evaluated. I just want to keep running the instructions that happen after me. But that might have been wrong. If the branch ended up being taken, then I need to know that instead of evaluating these instructions, I actually need to jump to wherever the target is and run those instructions instead. The purpose of the branch predictor is to predict, five cycles earlier, that a branch is going to happen, before you even get to that instruction.

01:12:00 – Brains vs chips

Dwarkesh Patel

If I think about how the brain works versus what you’re describing here, at a high level the differences might be that while you can do structured sparsity in these accelerators and save yourself some area that you would have otherwise had to dedicate to gates, in the brain there’s unstructured sparsity. Any neuron can connect to any other neuron, and not in ways where they have the column aligned.

Then there’s the fact that memory and compute are co-located. Although I guess you could say in a way the memory and compute are co-located on these dies too.

Reiner Pope

This is exactly the co-location, in some sense, of the memory and compute.

Dwarkesh Patel

So maybe that isn’t a big difference. Another big difference is that the clock cycle on the brain is much slower than on computers. Partly that’s to preserve energy, because the faster the clock cycle, the bigger the voltage needs to be in order for the signal to settle and to identify what state a transistor is in.

Reiner Pope

That’s right.

Dwarkesh Patel

I don’t know if you have any commentary on what the brain might be doing versus how these chips work.

Reiner Pope

Let’s take the clock speed one first. The clock speed is quite high on a chip because that drives higher throughput. When we compare a GPU running some workload, it’s running batch size 1,000. Whereas the brain is not running batch size 1,000, there’s only one of me.

You could imagine saying, “Take a GPU and instead of running at a gigahertz, run it at a megahertz,” and that would start to look a little more like the equivalent things you’re talking about in the brain. But in the way silicon works, that does not give you a 1,000x advantage in energy efficiency.

What it ends up looking like is you just run this circuit once to stabilization, and then it will sit idle for a long period of time. It doesn’t consume a lot of energy while it’s sitting idle because most of the energy is consumed in toggling bits from zero to one and back.

Let’s talk about the energy consumption of a circuit like this. The way to think of a bit being stored is that you’ve deposited some charge in a capacitor sitting somewhere in the chip implicitly. It becomes charged when the bit becomes a one, and then it becomes discharged when it next goes to a zero.

That cycle of charging the capacitor and then dumping that charge out to ground is where the energy is consumed. This is called the dynamic or switching power, and it’s most of the energy consumption of a chip. There is some other energy consumption just coming from the fact that insulators aren’t perfect, but we’ll discard that. Most of the energy consumption comes from toggling from zero to one and back to zero.

If you run a chip much slower and you only clock it once every thousand clock cycles, you will have 1,000 times fewer transitions. It will be about 1,000 times less energy consumption. But it’s not a substantial advantage in energy efficiency.

01:15:33 – A GPU is just a bunch of tiny TPUs

Dwarkesh Patel

Okay, so you described how a TPU works at a high level. What is the difference at a high level between how a GPU and a TPU work?

Reiner Pope

There is a high-level organization principle that is different, and then inside the cores things are different. Looking at the high level, we’ll take a GPU and a TPU and see what the top-level block structure looks like.

If you think of this as the whole chip in each case, the organization of the GPU is mostly a bunch of almost-identical units, which are the SMs. They’ve got an L2 memory in the middle, and then a bunch more of these SMs on the bottom. So there is a fairly regular grid of cores.

If we look at a TPU in comparison, you end up with much coarser-grained units of logic. You end up with just a few matrix units, which are the big systolic arrays. In the middle you’ve got some vector unit, and then you’ve got your matrix units at the bottom. These matrix units with a vector unit in the middle make up the whole TPU chip.

You can think of scaling this thing down into a really tiny unit with a smaller matrix unit and a smaller vector unit, and that is sort of what an SM is. From a very high-level point of view, the GPU has a lot of tiny TPUs tiled across the whole chip.

Dwarkesh Patel

Oh, interesting. You’re suggesting the tensor core within a streaming SM is analogous to an MXU?

Reiner Pope

Yeah, it’s all very similar.

Dwarkesh Patel

I see. If you had more lack of structure, having a bunch of tiny TPUs makes a lot of sense. Whereas if you just have huge matrix multiplications, you might want to avoid the cost of having individual SMs with their own registers and warp schedulers. Why not just make a huge thing and amortize those costs across the whole thing?

Reiner Pope

This shows up in how large you can grow things. We’ve seen this theme, especially with the systolic array, where a larger systolic array amortizes the register file costs better.

This design allows you to have larger systolic arrays, whereas the GPU design constrains you to having small units of everything. There is a trade-off, however. Because of this coarse-grained separation of things, you need to move a lot of data from the vector unit to the matrix units, through just two lines of perimeter here.

If you look at the equivalent thing in a GPU, you’ve got vector units everywhere, and you can move data through many different lines. The amount of data you can move between a vector unit and a matrix unit is actually much higher in a GPU than in a TPU. Instead of having to move all the data through just two lines, you’re moving it through 16 lines of wiring in a GPU.

Dwarkesh Patel

Right. But also you might have to move across less area.

Reiner Pope

Which is an energy saving as well. So if you can operate entirely within an SM, the data movement is much smaller. But the moment you want to operate across SMs, it becomes more complicated and expensive.

Dwarkesh Patel

So you don’t have to comment, but one might expect that a thing MatX might try to do is get the GPU-like smaller structure of systolic arrays surrounded by SRAM, but at the same time make it so that the things you need in an SM to support the CUDA architecture—which take a bunch of space—you might discard.

Reiner Pope

We’ve talked publicly about something we call a splittable systolic array, which in some sense you can think of as big systolic arrays that can be small systolic arrays too.

Dwarkesh Patel

Cool. Okay, I think that’s a good note to close on. Reiner, thank you so much.

Reiner Pope

Thanks, Dwarkesh.

To be clear, these people are obviously very competent and clever. But when you think of ASI, you don’t think of Trump, but more so.

The person who kept pressing this question was correctly pointing out that I basically defined intelligence as power. And by this definition, Stalin was the most intelligent person who ever lived.

Now, of course, you could change the definition of intelligence to something more like, manipulate abstract concepts and rotate shapes.

But notice that the most powerful people in the world do not max out this quantity. The correlation between extreme power and this kind of intelligence might be even weaker than the correlation between extreme power and height. The physicists are not running the world.

We tend to conflate power-seeking AI and superintelligent (in science and tech) AI. I’m not denying that AI can be power-seeking. Whatever skills and drives Donald Trump has could be embodied in a digital mind. I’m simply pointing out that the way AI systems are currently becoming smarter (by getting trained to be to be really good at specific economically valuable tasks like coding) is not that strongly correlated with power.

We often talk about power in this way that misunderstands how it is actually derived in our world. Our intuitions are primed by games like Diplomacy or Go, which are designed to isolate and reward a g loaded kind of strategic reasoning.

But in the real world, power is more the product of having the authority and trust to get lots of people to collaborate with you, rather than some galaxy brain scheming capability. Trump is not powerful because his brain, considered in isolation, is the most effective optimization engine on Earth. He is powerful because the government which hundreds of millions of people consider legitimate gives him a lot of authority.

A group versus individual level analysis is useful here. As Garett Jones has written a lot about, individual IQ is only modestly correlated with individual income, but national IQ is strongly correlated with national outcomes. This is because intelligence has a lot of spillover effects - smarter societies cooperate more, save more, and can coordinate to build things like space shuttles and semiconductors.

Richard Trevithick, who invented the high-pressure steam engine, died in poverty, buried in an unmarked pauper’s grave. But the fact that 18th and 19th century Britain had lots and lots of people like Trevithick contributed to Britain being able to set up a global empire and outcompete lots of backwards principalities around the world.

It seems to me that the right mental model is that automated firms will outcompete everyone else in normal capitalist ways, rather than a single AI outthinking everyone else.

On why pretraining runs fails

Had an interesting chat with someone on why pretraining runs often fail. It was very interesting to get a sense of all the tangible ways that things can get fucked, and why training is such a precarious operation. At a high level, breaking causality, and adding bias, seem to be key culprits.

Breaking causality:

• When you do expert routing, you first go through the router, which gives you a score of how much each token wants each expert. There’s two ways to proceed from here: 1. Token routing, where you read the scores from the token’s perspective, and allocate to each token’s top k experts. Problem is that you could end up with wildly unbalanced allocation across experts, which is terrible for performance. Alternatively, you could (and only in training) do expert choice, where you just split the tokens by which are more relatively preferred by each expert. This way you can enforce that each expert gets roughly the same number of tokens. But the big problem is that this breaks causality, because which expert token n gets allocated to may depend on which expert token n + k might be router to. And breaking causality is very bad, because you’re getting information in training (and updating based on it) that you wouldn’t see in deployment.

  • Rumor is that this explains why Llama 4 was underwhelming.

  • I guess you could do expert choice during prefill inference? But maybe it doesn’t work well in practice to allocate tokens to experts which would not have received that token in actual training.

  • Tbh I don’t fully understand why breaking causality is so bad. I understand you can’t see beyond causality in real inference. But why is this minor deviation such a big issue?

• Another thing that can break causality is token dropping. Where experts just ignore the tokens in the batch that they’re supposed to process, but which rank not so strongly, and cutting whom would spare going outside padding. This breaks causality cause a later token being more strongly matched to this expert might lead to an earlier token getting ignored.

  • Apparently this was an issue with Gemini 2 Pro.

Adding bias:

• Bias much worse than variance - variance can average out, but bias compounds

• Apparently the original GPT 4 training was slow and got initially fucked because of the following bug: they were using FP16 on their collectives like all-reduce. FP16 distributes its granularity according to logarithmic density - between 1 and 2, the mantissa bits carve the interval ~0.001 apart. But 1024 and up, the mantissa might be carving the interval by multiple whole number values. Suppose some collective involves adding 1 + 1 … 10,000 times - you could get in a situation where as soon as you get to 1024, you add 1, it goes to 1025, you round down to the nearest interval at 1024, add one again. And so the calculated value is 10x off the real value. Huge issue if you’re trying to sum many small gradients into a large accumulator. And imagine how hard the bug must have been to find!

Implications for AI training:

• Some of the people who think we can cure aging argue that there’s basically 5 different ways people die of old age (heart disease, cancer, etc), and that if we cure these 5 different diseases, then we’d basically have solved aging. You could ask a similar question about these failed pretraining runs - are there 5 different ways training runs fail, in which case once a lab figures out numerics and , you’ll just have smooth sailing, or will you keep seeing new bespoke issues emerge at each new level of scale? The person I talked to seemed to think the later - he pointed out that even within numerics, there’s so many ways you can fuck things up. And new ones will keep emerging at scale.

• Bearish on AI fully automating kernel writing anytime soon. Presumably this is because he thinks it’s more of an AGI complete problem than some give it credit for. There’s another school of thought that says, “Hey, which kernel gets attention or MLP to run fastest on this scaleup is a super verifiable domain, thus we can RL to superhuman performance easily.” But he says, it took Nvidia, which has the best kernel engineers in the world, a long time to optimize for Blackwell, which suggests that actually it’s quite hard, and might not be super easy to close the loop on.

• Sometimes people say inference for RL generation and inference for end user generation is basically the same. But this person pointed out that in RL inference, numerical drift between inference and training engine can cause these subtle off policy biases, which matter a ton for highest quality training. But are not an issue if just serving to users.

• Emphasized how important it is to have a disciplined process for amalgamating compute multipliers, because of the risks of stacking up bugs with subtle biases.

Pretraining parallelisms

Notes from an excellent lecture that Horace He gave my friends and me.

What made this lecture so good is that Horace built up the whole topic as a chain of problems and solutions: here’s what we want to do, here’s why it breaks, here’s how we fix it, here’s why that fix eventually breaks too. Most explanations just list out a hodge podge of different strategies, without ever connecting them to the problems they solve or explaining why you’d pick one over another.

• Equation for pretraining flops = 6ND. 2 FLOPs per parameter per token for the forward pass (multiply + add). Backward pass is 2× forward because you compute gradients w.r.t. both input matrices. So 2 + 4 = 6.

• Okay we can’t do all this on one GPU. So how do we split up this problem? The obvious solution is to do data parallel - where you copy the model weights across each GPU, and you just do a part of the batch on each GPU.

  • The obvious problem is that each GPU only has a limited amount of HBM - B300 is 288GB - and this is not enough to store the weights as models get bigger and bigger, much less their activations.

• Okay so next thing we try is fully sharded data parallel - each GPU only stores 1/N of the parameters of each layer - before processing each layer, you all-gather the full layer’s parameters from all GPUs (each GPU only stores 1/N of each layer). After processing, each GPU discards the gathered parameters.

  • It was emphasized that this is the go to default. And you only move on from this when having too many GPUs forces you to move on, for reasons explained later. The reason this is the default is that it’s trivial to overlap compute and communication time - that’s because the only thing being communicated is the weights, which are not dependent on what happened in the layer before, so you can start all gathering the next layer while you’re still computing this layer. Compare this against tensor or expert parallelism, which do need to share activations for one layer before you can process the next one. The problem with pipeline parallelism is bubbles as explained below.

  • From a comms volume perspective, FSDP looks insanely expensive at first — you all-gather every layer’s full weights across all GPUs, use them for one matmul, then throw them away. But this ignores what regular data parallelism already costs you - in regular DP, you still need to do an all reduce after every layer of the backwards pass in order to sync the batch’s gradients across all the GPUs. That all-reduce has comms volume of params × 2. FSDP adds all-gathers — one per layer in the forward pass, one per layer in the backward pass. But an all-gather is half the comms volume of an all-reduce. So naive FSDP comms volume ends up being # params * 4 (all gather forward and back, plus all reduce on back). You can do even better: since each gradient shard only needs to end up on the one GPU that owns it, replace the all-reduce with a reduce-scatter (which skips the final broadcast step). That gets you to params × 3 total — a 50% overhead over vanilla DP.

• So why can’t you always just do FSDP?

  • Comms crossover: You want your compute time to be greater than your comms time - you don’t want to be bottlenecked on comms. But since compute time for FSDP decreases as you increase the number of GPUs, and comms time does not, as you scale the number of GPUs on FSDP, your MFU can totally crater. When this happens, you need to add pipeline parallelism too.

    • Compute time = (6 * # tokens * active params) / (compute per GPU * number of GPUs)

      • This decreases as you increase number of GPUs

    • Comms time = (# total params * 3) / (nv link domain size * infiniband BW)

      • Comms time does not increase as you add more domains. This was really confusing to me. Each domain collectively holds all the parameters, and you need to sync gradients across domains after each layer of the backward pass. You’d think that adding more domains means more hops in the ring, so the all-reduce gets slower. But the standard ring algorithm splits the message into one chunk per participant. More domains means more hops, but proportionally smaller chunks per hop. (This breaks down when chunks get so small that per-hop latency dominates, at which point you switch to tree algorithms.)

        • Technically, you can do better than a naive single all reduce for the gradients between all the domains. You do a hierarchical collective to optimize comms time across multiple NVLink domains. Key thing to remember is that each GPU in the domain gets its own bandwidth access to infiniband. So you wanna use it all up since interconnect bandwidth is the bottleneck. You do this by trying to do as much as possible within a scaleup before you move out. So you do reduce scatter within a scale up to give each GPU the domain-level reduced gradients for a shard of the layer, then all reduce these shards across corresponding GPUs across domains, then all gather within a domain. This shifts the comms time line down, thus moving the crossover point to the right.

        • Made an animation to illustrate it using Cursor and Composer 2:

  • If you look at the equations, you can see that if you increase batch size, crossover point moves to right, and if you make the model more sparse, moves to the left.

  • Also why TPUs are better at FSDP - because more accelerators within a domain.

• Batch size floor: FSDP is data-parallel, so each GPU processes at least one sequence. Attention is computed within a sequence and can’t (easily) be split across GPUs. If your critical batch size is 10M tokens and sequence length is 10K, you only have 1K sequences — so you can’t scale beyond 1K GPUs with pure FSDP, even if you have plenty of comms bandwidth left.

• Problems with pipeline parallelism (the next addition you’d make to FSDP in order to deal with these issues):

  • The problem with pipeline parallelism is different - there you have bubbles that emerge from the fact that at the beginning of the batch, the GPUs dedicated to the final layers are not being used, and conversely at the end of the batch, the GPUs dedicated to the first layers are not being used. The reason you can’t overlap batches in training to solve pipeline bubbles is that you need to consolidate gradients and update the model before you process the next batch.

  • But also you’re adding architecture constraints - things like Kimi’s attention-to-residuals (where each block attends to all previous layers’ residuals) become very difficult when those residuals live on different pipeline stages. Similarly, interleaving sliding-window and global attention layers could cause load imbalance across stages. Dealing with all this slows down research iteration, which is the greatest sin you can commit.

The organizing question from my interview with Michael Nielson was, “How do we recognize scientific progress?” It’s especially relevant to thinking about what it would take for AI to close the RL verification loop on scientific discovery. But it’s also a surprisingly mysterious and elusive question when thinking about the history of human science.

Some people have this idea that AI is going to be disproportionately good at making scientific breakthroughs. The reason they think this is that 1. Science is ‘verifiable’, 2. AI is absolutely crushing domains that have a tight verification loop - coding, math, etc - because you can RL on these loops.

But the history of human science shows that the verification loop for theories can be on the order of decades and centuries, and even then experiments do not definitely rule out alternatives: Ancient Athenians dismissed Aristarchus (2nd century BC) on heliocentrism because it would imply stellar parallax. The first successful measurement of stellar parallax was in 1838, achieved by Friedrich Wilhelm Bessel.

What we know today as the better theory can often actually make worse predictions: it’s well known that Copernicus’s model of circular orbits around the sun was less accurate than Ptolemy’s geocentric model, which had accumulated millenia of correcting epicycles. What is not well known is that Copernicus’s theory wasn’t even simpler (Ptolemy’s model interpreted the true elliptical nature of orbits using an equant trick where other planets are not moving in uniform circular motion around Earth exactly, but rather an off center point. Copernicus didn’t like this, because it violated his Platonic heuristics - so he discarded the quant trick, which led to a less parsimonious model, since Copernicus had to add more epicycles and epicyclets to make up for it.)

So in what sense was it a better theory in 1543? In some sense, it wasn’t! You couldn’t have known ex ante that heliocentrism married with Kepler’s 3 laws (1619) is a much cleaner and more accurate theory, or that there’s a very beautiful unification of heliocentric orbits and terrestrial gravity (Newton in 1686).

There was one ex ante reason that you should have preferred Copernicus in 1543: his theory required retrograde motion as a natural consequence of his theory, whereas for Ptolemy it was an ad hoc addition. Even more impressively, his theory, developed in 1543, actually predicted the phases of Venus before they were observed by Galileo in 1610. But both of these things were also implied by Brahe’s model, which had set the sun to orbit the earth and then all the planets to orbit the sun.

Under a naive falsificationist framework, you’d have to wait until Stellar parallax was observed in 1838 to know that Brahe was wrong. But obviously the scientific community was able to make progress faster than this. There is some mixture of judgment and heuristics in the progress of science that we don’t even understand well enough to actually articulate, much less codify into an RL loop.

Or consider the case of the discovery of Neptune in 1846. Uranus deviated from its predicted Newtonian path. Le Verrier predicted that an unknown perturbing planet must exist, calculated its mass and orbit, and Neptune was found almost exactly where predicted.

But the Neptune story is symmetric to a failure case. Mercury had an anomalous precession, where the ellipse that shows its orbit would rotate 43 arcseconds more per century than should be implied by the impact of other planets using Newtonian mechanics. This led astronomers to speculate that there’s an unknown planet Vulcan within Mercury’s orbit. But it was resolved in 1915 with Einstein’s General Relativity.

A proper Newtonian would still proceed with the research agenda, but modify it as follows. First, you predict some unknown planet. If it can’t be found, you say it’s so small, it must require a bigger telescope, and you build a bigger telescope. And if you still can’t find it, maybe there’s a cloud of cosmic dust occluding it. If still not found, maybe the satellite’s instruments are being screwed by some unknown magnetic field, and you send a new satellite. At each of these steps, had you discovered a new planet, or some unknown cosmic dust, or some new magnetic field, that would have been a sensational victory for Newtonians.

Ex ante, this is not unreasonable to do! It is only after decades or maybe centuries of patchwork that we can then analyze, are we simply adding epicycles, or is this theoretical framework progressive, in that it makes predictions we wouldn’t otherwise be able to.

What do these examples illustrate? That ex ante it is almost impossible to determine which research programs are progressive (will predict and explain unanticipated new phenomenon) and which are regressive (need to be contorted repeatedly to accommodate seemingly disconfirming new phenomenon).

But the verification loop is often extremely long and weirdly hostile, and even then, experiments do not definitely rule out alternatives (see the discussion in the Nielson episode about how physicist contemporaneous with the 1880s Michelson-Morley experiments thought that it simply ruled out a particular theory of ether. Only Einstein made the full conceptual leap to discard the ether altogether).

This means that big conceptual breakthroughs cannot be easily verified. They are recognized decades or centuries later, when it turns out they were much more productive than the alternatives available. What this means for AI for science is that 1. You can’t easily train an RL loop for big conceptual breakthroughs.

And 2. the society of AI scientists will still need individual AI instances that have idiosyncratic biases and heuristics, and to pursue them unrelentingly for decades on end - for example, like the one Einstein had in insisting that there shouldn’t be some arbitrary inertial reference frame. There should be dedicated people to keep a bunch of dormant research agendas alive in case they turn out to be productive upon further investigation. To understand the kind of intransigent dedication to hypotheses that is needed to preserve correct scientific idea - even in the face of disconfirming evidence - consider the following story: In 1815, Prout hypothesized that the atomic weights of all pure chemical elements are whole numbers, because experimentally, most elements seem to come out like this. But there’s many anomalies - for example Chlorine’s atomic weight is measured at 35.5. And so Prout’s school claimed that maybe the chemical substance in which these elements appeared were impure. But there seemed to be no chemical reaction that could get rid of the impurities. And then they said, maybe it’s fractions of full atomic weights - but the closer you measure, the less natural the fractions seem to get - Chlorine goes from 35.5 to 35.46. It takes until almost a century later for people to realize that these measurements are showing multiple isotopes of the same element, which can be separated physically, but have no chemical distinguishing characteristics.

What I’m trying to say is that ex ante, one couldn’t have known which research program would be more productive. We need to invest in all of them concurrently. But that investment looks like a bunch of different individual scientists being super unreasonable and obstinate about propping up their preferred research agenda.

What does the parallel discovery of a deep idea like Darwinism tell us?

The Origin of Species was published in 1859. Principia Mathematica was published in 1687, two centuries earlier. Conceptually, it seems like natural selection is much simpler than the theory of gravity. A contemporary of Darwin’s, Thomas Huxley, read the Origin of Species and said, “How extremely stupid not to have thought of that!” Nobody ever said the same for not beating Newton to the Principia. I wonder if the reason this happened is that, while Darwin’s theory is conceptually simpler, it cannot be decisively tested. The evidence is circumstantial, retrospective, and cumulative. There’s no equivalent of Newton running the numbers on the moon’s orbital period and radius, and confirming that it corresponds to his equations.

Also you need this concept of deep time. Charles Lyell published the Principles of Geology in 1830, which gave Darwin the vast stretches of time that natural selection needed. And the fact that Darwin and Wallace basically arrived at evolution at the same time (and both credited Lyell’s contribution) does suggest that these underrated intellectual footholds were quite important (geology, paleontology of ancient extinct species which showed intermediate species (in some cases between apes and humans), biogeography from voyages and age of colonization, more sophisticated artificial selection like pigeon breeding). It’s interesting that an idea whose essence must have been obvious to herders and parents for thousands of years actually required many millennia of ancillary intuition pumps to fully spell out.

The pattern of parallel discovery in science and technology is very interesting, and seems to contradict this vibe that certain innovations could have happened earlier much earlier than they really did.

Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn.

Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second.

Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside.

Watch this one on YouTube so you can see the chalkboard.

And check out the flashcards I wrote to retain the insights.

Sponsors

• Cursor‘s agent SDK let me build a pipeline to generate flashcards for this episode. For each card, I had an agent read the transcript, ingest blackboard screenshots, generate an SVG visual, and run everything through a critic. A durable agent is much better at this kind of work than a chain of LLM calls, and Cursor’s SDK made it easy. Check out the cards at flashcards.dwarkesh.com and get started with the SDK at cursor.com/dwarkesh

• Jane Street gave me a real deep-dive tour of one of their datacenters. I got to ask a bunch of questions to Ron Minsky, who co-leads Jane Street’s tech group, and Dan Pontecorvo, who runs Jane Street’s physical engineering team. They were willing to literally pull up the floorboards and take out racks to explain how everything works. Check out the full tour at janestreet.com/dwarkesh

Timestamps

(00:00:00) – Basics of Go

(00:08:17) – Monte Carlo Tree Search

(00:32:04) – What the neural network does

(01:00:33) – Self-play

(01:25:38) – Alternative RL approaches

(01:45:47) – Why doesn’t MCTS work for LLMs

(02:01:09) – Off-policy training

(02:12:02) – RL is even more information inefficient than you thought

(02:22:16) – Automated AI researchers

Transcript

00:00:00 – Basics of Go

Dwarkesh Patel

Today, I’m here with Eric Jang, who was most recently vice president of AI at 1X Technologies, and before that, senior research scientist at what is now Google DeepMind Robotics. You’ve been on sabbatical for the last few months. One of the things you’ve been doing is rebuilding, improving, and hacking on AlphaGo.

Today, you’re going to explain building AlphaGo from scratch and what it tells us about the future of AI research and development. Before we get to that, why is AlphaGo interesting? Why is this the project you decided to do on sabbatical rather than just hanging out at the beach?

Eric Jang

I like making things, and AlphaGo and Go AI is one of those things that really got me into the field. When I saw the early breakthroughs on AlphaGo in 2014, 2015, 2016 and so forth, it was profound to see how smart AI systems could become and the computational complexity class they could tackle with deep learning. This is a problem that has long been understood to be intractable for search, and yet it was solved through deep learning. That was quite mysterious to me, and I’ve always wanted to understand that phenomenon a little better.

My training is in deep neural nets for robotics, where the decisions made by the neural networks are a bit more intuitive. But AlphaGo is a problem where the decisions are the result of a very, very deep search. It’s always been very mysterious to me how a ten-layer network can amortize the simulation of something so deep in the game tree.

Eric Jang

If you plot out how much compute it took to build various iterations of strong Go bots over the years, you can see that in 2020 there was an open-source project called KataGo by David Wu from Jane Street, which achieved a 40x reduction in the compute needed to train a really strong Go bot tabula rasa. I’m not certain if it’s stronger than AlphaGo Zero, AlphaZero, or MuZero, but it’s very strong, and this is what most Go practitioners today train against when they’re playing an AI.

Thanks to LLM coding, what took a whole team of research scientists at DeepMind and millions of dollars of research and compute can now be done for a few thousand dollars of rented compute.

Dwarkesh Patel

We should first discuss how Go works. How does the game work?

Eric Jang

Go is a very simple game that can be implemented quickly and easily on a computer. The objective is to put down black and white stones and try to occupy as much territory as possible. I might start by putting down a black stone. Black always goes first. Go ahead. The way you capture an opponent’s stones is that for every intersection, if you can surround all four of its neighbors with your stones, then it’s cut off from oxygen, if you will, and it’s a dead stone. Now I control these four stones as well as this empty intersection here.

There are slight variations between Chinese, Japanese, and what are called Tromp-Taylor rules. Tromp-Taylor rules are designed to be completely unambiguous, so this is what all Go AIs train and resolve against. In typical Go, when humans play, you’re actually not allowed to put this white stone down here. It would be instant suicide. In Tromp-Taylor, it’s actually fine. You put it down, and it immediately resolves to death, so the outcome is the same.

Let’s start over and play a few stones, and then I’ll explain some more. I’ll just start there.

Dwarkesh Patel

I’m basically playing randomly here, but I’m trying to get around your stones and see if I can surround them.

Eric Jang

This move exposes one empty neighbor for your white stone. It’s akin to a check in chess. If you don’t respond immediately by putting one here, then I can immediately capture this.

Dwarkesh Patel

I see. Because it’s the diagonals that determine whether you’re surrounded or not.

Eric Jang

The cross-section, not the diagonals. This one is surrounded on three sides, so you’re at threat of losing that stone if you don’t play one immediately there.

Now you can see that I’m starting to pressure you, because by putting a stone here, you’re forced to put one here.

Dwarkesh Patel

Otherwise, you would have this two-block to yourself.

Eric Jang

Yes. And if you think through what happens if you were to respond here, you can probably search into the future and deduce what I’ll do in response once you do that.

Dwarkesh Patel

You have a lot of confidence in my abilities, but I’m guessing you’d put the black here.

Eric Jang

That’s right, and then I would capture all three of these stones.

Dwarkesh Patel

So I should just assume that this little block is gone.

Eric Jang

Yes. In Go, it’s actually okay to let an opponent capture some stones if, for example, it lets you position to capture more stones somewhere else on the board. This is what makes Go a beautiful game: you can lose the battle but win the war. As the board size increases, the complexity of these micro versus macro dynamics gets more interesting.

Dwarkesh Patel

Presumably you’d put one here.

Eric Jang

So now I would capture this entire group, and this would be mine.

There’s one more case I want to demonstrate, which I actually had a bug in my code for recently. Let’s consider a formation like this, with other pieces on the board in play. Let’s talk about how the game ends. In this territory, who controls these areas? Is it white or black?

Dwarkesh Patel

White.

Eric Jang

It’s actually black, because I’ve surrounded this whole area. Assuming I have other black stones here, it’s very hard for you to break this out of the control of these stones.

Dwarkesh Patel

And when the final score is tallied, would these ones also count as being in...

Eric Jang

Great question. This is where different rule sets have different ways of scoring. We should talk about how you resolve scores between humans and how you resolve scores in computer Go, because there’s some ambiguity in how humans evaluate this.

Most humans would look at this board configuration and conclude that black has totally surrounded white, and white has no chance of life. We could play out more here, but at the end I would capture everything. However, if you have a way of breaking this formation and connecting white to something outside of it, then it can flip. This is where it’s a little bit hard for a computer to decide these kinds of things.

How do humans do it? It’s worth thinking about how humans resolve this, because this will map later to how we think about the deep neural network. Humans basically say, “I think the game is done,” and then you have to also say, “I think the game is done.” Then we’ll say, “I think these are my stones,” and you have to agree. If you don’t agree, we keep playing.

Essentially, once two humans—their so-called value function—agree on a consensus, then the Chinese rules resolve that. In Tromp-Taylor scoring, it’s perfectly unambiguous, so it can be decided algorithmically by a computer. If you have this at the endgame, the way you score it is that you first count how many stones you control, and that’s unambiguous.

Then you count how many empty intersections are not touched by your opponent’s stones. These intersections would not count for either player, because all of these intersections are connected to both white stones and black stones.

If this were like this, then white would get three points. This is a little odd because a human would know that white is actually losing these points. But Tromp-Taylor scoring would consider white to have all of these points as well as these points. So that is a very big difference in how computer Go scores things and how humans score things.

Dwarkesh Patel

How does the game end?

Eric Jang

The game ends when either a player chooses to resign or both players pass consecutively. Those are the rules.

Dwarkesh Patel

Now help me crack this with AI.

00:08:17 – Monte Carlo Tree Search

Dwarkesh Patel

Let’s understand how AlphaGo actually works and how somebody in the audience might be able to implement it.

Eric Jang

Let’s start with an intuition about the underlying search process used to make moves, and we’ll layer on ideas from deep learning to make it much more efficient and tractable.

Go is a game with just two players. We’re going to draw a person here, and we’re going to draw an AI here. Let’s say this person is playing black, so they go first. They go here. Now the AI is going to make a move based on what it sees here. There’s a question of how you encode these inputs into the AI. Maybe you could use ones and zeros, but you want to represent black, white, and empty. You would need at least three different values. Maybe you could use zeros, ones, and twos. The AI might see something like zero, zero, zero, zero, one. This is the input to the AI on its turn.

The AI can choose. Let’s just pick three possible random moves it can make, and I just drew these at random. Which move is best here? Well, we don’t know until the game ends. Go doesn’t have any kind of local reward for which move here is good. This is what makes Go a very difficult game. You don’t actually know who won until you really get to the end.

How deep is this tree? On a 19x19 Go board, there are roughly on the order of 361 moves on any given move, and of course, as it fills up, you have fewer moves. The number of steps in the game can be somewhere from 250 to 300 moves. Experts might decide to end the game well before that, but under Tromp-Taylor scoring, you actually have to play things all the way to the end. So this could be 300 moves, a depth of 300 in the tree.

If you keep on expanding possible moves—here the AI goes, then here the human would go, and so forth—you find that you end up with an enormous explosion in the possible game outcomes originating from just this one state. This is something on the order of 361³⁰⁰, which is far more than the number of atoms in the universe. Of course, there are redundancies and symmetries, so it’s not actually that, but if you were to do a naive tree with no merging of children, you end up with a tree about this big.

Dwarkesh Patel

What do you mean by “merging of children”?

Eric Jang

Let me use this board here. If we start here, and then you play here, and then I play here, and then you play here, that’s equivalent to me starting here, you playing here, me playing here, and then you playing here. Both arrived at the same spot, but through different paths. So this child node can be thought of as a shared ancestor.

Dwarkesh Patel

Got it. And I guess it starts at 361 but it decreases by one each time.

Eric Jang

Yes, the branching factor decreases by one each time. In any case, this is a very large tree. This is also why computer scientists for many years thought that Go was not a tractable problem this century. The amount of compute you would need to exhaustively search every possibility is just too large.

If you could, Go is actually a deterministic game. In any given state, you can compute the best possible strategy you can make in order to win the game. You can search all the possible futures where you win, and then just make sure you always stay in that set of futures.

AlphaGo’s core conceptual breakthrough was using neural nets to make this search problem tractable. Before we get into how neural networks are involved, let’s talk about how we can, assuming we had a powerful enough computer, search this tree to find the best move.

In the beginning, you’re not going to build out the whole tree, because storing it would be very expensive. Instead, you might interactively figure out which leaves of this tree are worthy of exploring and expanding into the future to see what else is there. There are some early algorithms in the bandit literature like UCB1, which is not exactly appropriate for a sequential game like Go, but very much inspired the action selection algorithm used in AlphaGo. UCB1 looks like: on every move, we’re going to take the best action—the argmax over a that maximizes Q of a, and I’ll explain what Q of a is in a moment—plus some sort of exploration bonus.

On every node, we’re going to track a few quantities. Let’s consider each of these a node. This is the root node where you’re making decisions from, and these are the children of the root node. Each node is basically a data structure that stores a visit count of this child node.

Dwarkesh Patel

How often the parent visited this node.

Eric Jang

Yes. One thing that’s easy to trip on if you come from robotics or other kinds of reinforcement learning is, where are the actions? I’m only talking about nodes. Nodes here represent states, and because this is a perfectly deterministic game with no randomness, you actually can just infer the action based on the child. If I go here, that implies an action, and this is the state we resolve to.

If you ask LLMs to vibe-code an MCTS implementation, they will most likely design the right data structure here. But it’s sort of a chef’s choice. You can rewrite the tree structure however you like. This is what Claude 4.6 wrote for me when I asked it, and it was a very reasonable choice.

Q_a represents the mean action value of this action. I’ll use a subscript a to denote that this corresponds to taking a specific action to get here from the root node. Taking a gets us to this node here. We’re also going to store the probability of taking this action

Dwarkesh Patel

Again, from the parent?

Eric Jang

From the parent, yes. What are the odds that we sample this one? This will become relevant later. We’ve talked about a deterministic tree for now, so I’ll bring probabilities into this later.

Finally, we have a dictionary of children, which is just more of these nodes in a classic linked list-style reference tree. This is the basic data structure to implement a tree.

In AlphaGo, they use a slightly different action-selection criterion called PUCT, short for Predicted Upper Confidence with Trees. When you select which child to take, you do argmax a of Q(s,a) plus a constant. The equation forms are pretty similar. These are both scoring criteria. You want to argmax this quantity and you want to argmax this quantity to determine which action to take.

Let’s break down the intuition of how you select actions here. Q(s,a) is the mean action value, so how good a given child is on average. If you actually knew the whole tree, this is all you need to select the best action. You don’t really need to do more than that. But if you’re interactively building this tree as you’re figuring out what the Q values should be, then occasionally you have to try some other actions as an explore-versus-exploit trade-off.

In both UCB and PUCT, there’s this term that basically rewards taking actions you haven’t taken before. As we mentioned, each node stores the visit count of taking that specific action. Everything is initialized to zero. For a given action, let’s call it action a, initially it’s zero. As n is increasing, if we’ve already made 10 action selections from that root node but we haven’t picked a yet, then this term starts to become quite large for a. Conversely, if we’ve chosen a 10 times out of 10, then this term is quite small. It diminishes very quickly. The same thing is true here.

Dwarkesh Patel

Just to make sure I’m understanding, let me put it in my own words. Let’s focus on UCB. Conceptually, you can think of it as two different things: the Q and the exploration term.

Let’s be clear about what Q is. Q is basically saying, once we do these rollouts—you’re actually running all these simulations—you go down the tree and figure out: if I end up at the terminal value of this tree, do I win this game or not? You average whether I win or not across all the leaves of this tree starting from this node. That average, you put in Q. So Q represents the probability that I’ll win this game starting at this node. That is your sort of exploit. It’s saying: I’ve run these simulations, and I think this is a good move or not.

The other term is saying: have I explored this branch enough yet relative to the other actions I could be exploring, or have already explored? If I haven’t explored this branch yet, maybe I think it has a low score, but I just haven’t explored that many leaves down this node in this tree. So I should try this even though Q, the exploit term, is telling me it’s not that valuable.

Because ln(n) grows slower than n, over time you will move from the argmax being dominated by the exploration term, which is the second term here, to the argmax being dominated by the Q term, which is when you’ve done enough simulations and are confident that this is the branch to go down.

Eric Jang

Yes, that’s right. The motivation for UCB was to come up with an algorithm where, if you don’t know the payoff of the different actions you can select, this strategy, given some exploration term here, bounds your regret in terms of how wrong you can possibly be.

I don’t know the proof. I also don’t know if this one is proved to have logarithmically or square-root-bounded regret, but I think the algorithm was derived to look something like this. You can tell these terms grow a little differently, and this is to account for the fact that Go has many more actions for any given move compared to your standard bandit problem.

One small clarification: you talked about simulations and probabilities. We should remember that Go fundamentally is a deterministic game. Where does the notion of probability come from here? If you had a very powerful computer, there are no probabilities. You can just compute the true average of the mean action value.

So where does the probability come in? In computer Go before AlphaGo, we’ve always done some sort of Monte Carlo method where we take the expected Q value averaged over a randomly selected tree. That randomly selected tree is where probabilities come in. The interpretation of Q is: what is the expected action value under the random distribution induced by some random search process?

Where does the random search process come in? That’s where P_a, of action, comes in. If we assume a naive algorithm where you have a uniform probability of taking any valid action, then this would just be one over the number of valid moves. You would be taking this average over a very diffuse tree. This is a valid integral, but it’s very slow because you’re going to consider a lot of trees that have very low value. It’s essentially almost like an importance sampling problem. Only a few actions and paths contribute high value, and almost everything else is low value. So that’s a tricky problem here.

This is the action selection criterion for how you decide which moves to go down. As you move down in tree search, you will eventually run into a node where it’s quite clear you’ve won or lost. At the very end of the game, when there are no valid moves left to play under Tromp-Taylor scoring, you can decide whether you won or lost. This is the final return of the whole game.

We can assign a value, U, to a terminal leaf node of the tree, but how do we assign values to the nodes prior to that, the parents? You take the mean action value, which is essentially your average. Suppose these were all leaf nodes. The mean action value of this node is just the average of whether you won or lost at the leaf nodes. Correspondingly, you can walk up the chain and say the mean action value of this node—let’s call it Q_b—is just a weighted average of these ones here.

The weighted average could depend on whether you have a different sampling distribution. But the basic intuition is that you want to resolve the game where you have a deterministic win or lose, and then you can go backwards—this is called the backup step—and assign values to these nodes or actions corresponding to the average over the final terminal leaf.

If you were to do this without neural networks, it would still be intractable. You would have trouble finding which actions to sample. A lot of the actions would contribute very low value, especially if you’re trying to fight your way out of a losing position. Only a few actions give you high value, so the search in practice is still very expensive. But the idea is that because Go follows a tree structure, you can inform a very good estimate of the value of this node based on the downstream values, assuming they’re all correct and you’ve searched deep enough.

Dwarkesh Patel

Your earlier explanation—about the sorts of states where it’s obvious to a human who’s going to win, but deterministically you still have to play it out—actually drew upon the intuition for why 1) the value function is both trainable and 2) necessary in order to learn this game effectively. Maybe it’s worth defining value in the first place.

Eric Jang

We talked about U being your final resolution of whether you won or lost. This is a terminal leaf node condition. Humans don’t play all the way to the leaves of the tree. They stop dozens of moves before, maybe even 100 moves before in high-level play.

How do they know? You can think about humans as implicitly having a neural network called a value function that takes in a board state and evaluates p(win). The human glances at the board and knows, “I’m probably going to lose.” They’re essentially running a neural network that looks at a board and implicitly amortizes a huge number of possible game playouts. They take that average and decide whether the board is winnable or not, and whether they should concede or keep playing.

This is remarkable. If you think about the beauty of something like this, a neural network in a human can somehow do all of this simulation at a glance. They just know within a few seconds, without actually playing every single game logically, based on crystallized knowledge and experience. They can do this. This gives us a hint that in games like Go, there are ways to radically speed up the search process. This is one of the fundamental intuitions behind why AlphaGo works. You can train a value function to look at a board and quickly resolve the game without playing out all of these trees to a very deep search depth.

Dwarkesh Patel

Makes sense. I will say for the audience, for previous episodes when I was prepping and it seemed relevant to understand how AlphaGo works, I would find it very confusing. But it’s the kind of thing where once you understand the problem in this way and then build the next few pieces, it is actually a lot more understandable and it makes a lot of sense. It’s okay to be confused right now, but it’s probably simpler to understand, by the end of this lecture, than you anticipate. I’ll just make that note for the audience.

Eric Jang

The important intuition at a high level—to step back about where we’re going with all this—is that classically, for games like Go, you could build a tree, but we don’t have computers powerful enough for that. Estimating the value of every action you could possibly take is also hard because you don’t know until the end of the game.

You could take averages by playing them to the end, but that’s also hard because you don’t know which actions to take to sample these averages. Conceptually, there are two problems: the breadth of the tree and the depth of the tree. AlphaGo gives us a way to shrink both of those to be tractable. That’s essentially the core idea behind it.

We take the idea that humans can glance at a board and instantly predict whether we win. That maybe gives us the opportunity to truncate how deep we search. We also know that humans can intuitively, at a glance, decide what moves might be good on a Go board. These are two things we can use deep neural networks for, to accelerate the search process.

Before we talk about neural nets, let’s go back to how this playout works. We’ve only talked about making one move. The AI looks at this encoded Go board. It has a tree. It searches deeply into the tree to find out which of its actions might be the best, takes that action, and goes back to the human. Now the human sees a Go board that looks like this, and they make their move. Maybe they put their stone here.

Now we go back to the AI, which now looks at a new encoded board. I’ve used 2 to denote the AI playing as white, 1 to denote the human playing as black, and 0 as empty. On the AI’s turn, it does the MCTS tree search all over again from scratch. It throws away the old tree that it searched last round. Now there’s a new root node and it begins to search anew.

You can basically think about MCTS as a search algorithm that decides which moves to play best, aided by neural networks, and it’s done on every move. Let’s talk about the neural network part of this.

Dwarkesh Patel

While you’re erasing, another thing that was important for me to understand was about the MCTS data structure with nodes and children of nodes. This is done per move and reinstantiated once a move is made. A human makes a move, then the AI looks at this and runs a bunch of simulations to figure out what move to make next.

A simulation is exploring one more node in the MCTS tree. Once you run 1,000 simulations, that informs the probability of what move to make next, as you’ll explain. That’s what you store. You choose the best move given those probabilities. You discard all of that, the next player makes a move, and you restart this process at the beginning of every move.

Eric Jang

Correct. One small addendum: you don’t discard all of that. You keep one thing behind that we’ll use later.

00:32:04 – What the neural network does

Eric Jang

Now that we have a basic intuition of how moves are made with search, we’re going to talk about how neural networks can speed this up by providing an analog to human intuition. There are two networks. There is the value network, which takes in a state and predicts, am I going to win or lose? It’s a binary classification problem. Then we have a policy network, which induces a distribution over good actions to take.

I’m going to draw a one-dimensional flattened move distribution, but this is really a square grid. These are the probability distributions over good actions. Both of these are categorical classification problems. You can train this like any classifier with deep learning, cross-entropy loss, that kind of stuff.

The specific architecture does not matter too much. I tried a few different architectures. Transformers work, ResNets work. For small data regimes, my experience is that ResNets still outperform transformers and give you more bang for the buck at lower budgets. But this may not always be true.

Dwarkesh Patel

Wait, why is that?

Eric Jang

They provide the inductive bias of local convolutions. Generally, transformers start to outperform residual convolutional networks when you want more global context.

One interesting finding from the KataGo paper was that they found it quite useful to pool together and aggregate global features throughout the network, to give the network a global sense of how to connect value from one side of the board to the other.

Dwarkesh Patel

What does it mean to aggregate global features?

Eric Jang

If you have a very large 19x19 Go board, and you’ve got some battles going on here and some battles going on there, when you pass this through a convolutional neural network, the receptive fields are going to be good at computing local things and making that invariant. But they won’t be able to connect these two features easily. They need to be pooled together and attend to each other somehow.

The argument for why transformers are good for computer vision tasks—vision transformers and so forth—is that because they have global attention across the whole thing, they can more easily draw these connections. But you do need more data there so that you can learn the invariant local features through data.

I’ve tried very hard to make transformers work for this problem because I was curious if transformers would present some sort of breakthrough in Go and just remove a lot of those tricks. But try as I might, I haven’t figured out a way to make transformers better than ResNets for now.

Dwarkesh Patel

One more tangential question. It makes sense why transformers, with their global pooling of information, would be better if you need to consider information that is not just spatially… CNNs give you a bias that the things next to you are especially relevant.

Eric Jang

And then they’re aggregated up slowly.

Dwarkesh Patel

For games where it isn’t that relevant what is happening locally—you just have to consider the whole thing—you’re saying transformers would work better. We’re talking about the spatial dimension. How about the temporal dimension? Right now we’re only considering the previous move because it is a deterministic full-information game.

What if it were something like poker or Diplomacy, where a bluff they made a while back is relevant to understanding now? And to decide your next move, you need to consider all those previous states. Would that change the consideration of what inductive bias and which architecture is most relevant?

Eric Jang

Great question. Go is a perfect information game. In perfect information games, there does exist a Nash equilibrium strategy for which you can do no worse than any other strategy.

If you know your opponent has a particular bias, like they love to play aggressively, you can in principle counter that specific strategy better than a Nash equilibrium policy. But to counter any given strategy, there exists a single Nash equilibrium that can be decided solely using the current state. That is a design choice AlphaGo made, which in hindsight turned out to work very well because the Nash equilibrium seems to be superhuman. No human strategy seems to be able to beat it.

There are variations of this where you would need to consider temporal history. This is a very exciting research area, and I’d encourage people to fork my repo and try it out. If you were to play 2v2 Go, you actually need to model your partner’s behavior. You may not have information on how they play, so you need to aggregate information on how they play so you can respond accordingly. These are situations where it’s no longer a perfect information game. In games of imperfect information or partial observability, you do need context to build a model. That’s a place where things can get very exciting in terms of self-play or Diplomacy-style games.

Returning to the neural network, the architecture again is not super important. You can get it to work with transformers. You can get it to work with ResNets. I found that for low-budget experiments, ResNets work a little better. You can also use Karpathy-style AutoResearch hyperparameter tuning to make your architecture pretty good. You don’t have to worry too much about that. You just need to set up the problem so that you have a target optimization.

We’re going to pick a somewhat arbitrary architecture that worked for what I did. But again, this part is not super important. You have your encoded board state and, similar to an RGB, we’re going to have three channels. One channel to encode black, one to encode white, and one to encode empties or a masked region if you want to train on multiple board sizes. I’m actually not going to talk about multiple board sizes for now. That’s a bit too complicated.

We have this two- or three-channel RGB-like image, and we feed it into a ResNet. Then we have two branching heads. One head predicts the value function, which is a single logit, let’s just call it R¹. Then we have the policy, which is R³⁶¹. This is the architecture. We’re going to train this to predict the outcomes of games given the board state, and we’re also going to train it to predict what are good moves.

The original AlphaGo paper, called AlphaGo Lee, initialized this network with a supervised learning dataset of expert human play. Later, they removed this restriction by having the model teach itself how to play well. I find it super nice for implementation, for your audience, to always initialize your experiments to something easy and get the problem working before trying to bite off the whole thing and learn tabula rasa.

In deep learning, initialization is everything. You always want to initialize your research project to something as close to success as possible, especially if you’re doing something new that you haven’t done before. Always pick something that works and then get it to do something better, rather than start from something that doesn’t work at all and try to make it work.

Under that philosophy, it’s a great idea to start with something that has a good initialization. We’re going to take human expert plays and train this model to predict good actions. We’re going to take all the moves in which an expert won and predict those actions. Regardless of board state, whether you won or lost, you’re going to predict the outcome.

You might be wondering, for some of the early boards where only one stone has been put down, how could you possibly know who the winner of this game is? Well, if you have hundreds of thousands of games, on average you’ll probably see that boards starting like this have half of the games branch off and win, and half branch off and lose. So that’ll actually be fine. When you train this model to predict those outcomes, the logit will sort of converge to 0.5. For these things, it’s expected that once you train the model, a starting board state will look like 0.5, and then as you progress towards the end of the game, the win probability will either go up or down.

This is your move number. As you get hundreds of steps into the game, it becomes much clearer who’s more likely to win or lose under your expert data distribution.

Dwarkesh Patel

I didn’t understand the significance of why this way of thinking about value is especially relevant to the expert data.

Eric Jang

It is not relevant to the expert data. It’s true for any data that you trained it on. If you were to learn tabula rasa, you would also expect this to fall out.

Imagine you’re vibe coding AlphaGo. You gather some expert datasets from KataGo online, or you have a dataset of human players, and you train this model. It turns out this model is already a pretty good Go player. It will most likely beat most human players. If you just take this policy recommendation and take the argmax over these probabilities—if you take the argmax and just take this action as your Go play—it’ll be a very fast Go player that doesn’t think in terms of reasoning steps. It just shoots from the hip, and it’ll be a very strong Go player. This is already quite miraculous if you think about how ten neural network layers, maybe under 3 million parameters, can already do something that impressive.

You can start this way. It’s important when implementing this to verify that this is probably true. It’s good to verify that your Go rules are implemented correctly and that you can run these simulations relatively quickly. Just as a checkpoint, you want to make sure you can actually do this basic step before you try to layer on more complex things like search.

But we can do a lot better than taking the raw neural network and playing the moves. So let’s apply the neural network to improve Monte Carlo tree search. We start with our root node, and we now have a four-step iterative process to do MCTS.

This tripped me up when I was first reading the paper and trying to understand it. Essentially what we’re going to do is choose a number of simulations, and this number varies. It can be somewhere between 200 and 2,048. I believe in the AlphaGo vs. Lee match, they used tens of thousands of simulations per move because they really wanted to boost the strength of the model as much as possible. But in training, you don’t actually need too many. KataGo, I think, uses something in this order as well.

Dwarkesh Patel

Do you know if they used… If you watch the documentary, they had a laptop out during the game. They didn’t use the laptop itself. It was on some—

Eric Jang

It was on some TPU pod, I think.

Dwarkesh Patel

Honestly, kind of unfair. Lee is not using 1E22 FLOPS to do a move.

Eric Jang

Fair enough. Interestingly, modern Go bots don’t need that much compute at test time. What we’ll find out, as we talk about how the MCTS policy improvement works, is that over time the raw network actually takes all of the burden of that big TPU pod and just pushes it into the network. You can do all of that work with one neural network forward pass. The TPU pod will always add the extra oomph on top, and that’s what they wanted for the match.

We’re going to pick this number of simulations thing, and for every simulation, we’re going to do several things simultaneously. We’re going to see which moves are the best in the current tree. We’re going to add extra leaves to the tree if we get to a point where we need to add a leaf, and we’re going to update the action values for the tree. Every simulation involves this four-step process: selection, expansion, evaluation, and backup.

At the beginning of our Monte Carlo tree search, our tree is very basic. It only has the root node, or our current board that our AI wants to play at. We’re basically going to select the best action for this. When this root node is created, we also know that we can evaluate it under our neural network and get the quantities V_θ, as well as our probability over actions. And I’m going to say root. For all of the actions here, we can create a bunch of children. In this case, I’m drawing a 3x3 board with one missing, so there are eight possible children associated with this root node. Each of these has an associated probability of taking that action, so there’s P₈, P₁, ... P₂, and so on.

So at the beginning of our Monte Carlo tree search, we have our root node, and we can initialize it with some children. The policy network evaluated on the root node gives us—on a 3x3 board with one existing stone placed—eight possible children that this AI could take. With each of the children, the policy network also gives us the probability of selecting that child. The first step is to do the selection of the tree. Again, this is a very shallow tree. All we have so far is essentially a tree of depth one.

Our first move is to select by maximizing, or argmaxing, the PUCT criterion, which is basically Q(s,a) + C_PUCT x P_a x (√N / (1 + N_a)). For each of these, N_a is zero for all the actions initially. N is zero. So we’re going to pick according to this. Initially, the chosen action is most likely going to be biased towards the highest-likelihood action, because these are uniform for every node.

Let’s suppose P₁ was the highest-probability node, so you selected this one here. Now you get to this node and you realize it’s not a leaf node. It’s not a terminal game, so you cannot resolve the final resolution. The next step is expansion. You will then run this board state through the policy network. Note that this is the AI’s move. The AI is making this move. When we expand this tree, we’re now thinking about what the human, or any opponent, might do.

When we evaluate the node here, we’re going to evaluate it from the perspective of this player. This node has possible actions that we could take, and we expand the leaf nodes here. For each of these nodes that we could arrive at, we’re going to now check how good they are. From here, the human could play here, here, or here. We’re going to store V_θ for each of these, so V_θ of Node₁’, V_θ of Node₁’’, and so on. We’re basically using our neural network to make an intuitive guess of how good this board is from the perspective of this player.

Fortunately, because it’s a zero-sum game, it’s easy to deduce that the value for this player at this step is just one minus the value from the other perspective. It’s easy to flip the search process depending on which player you’re at. This is the expansion step. You’ve taken a non-leaf node, expanded it, and evaluated the value. This is essentially a quick guess as to whether I’m going to win or not if I were to play to the end. You can almost think about V_θ as a shortcut for searching to the end of the tree, for any given simulation.

This is essentially the evaluation step. We’re evaluating the quality of each of these boards. In the original AlphaGo Lee, they did something kind of interesting. They took this value and averaged it with the value of a real Go playout. They actually played a real game from here all the way to the end. I’m just going to draw this squiggly line to indicate some path. They play this all the way to Tromp-Taylor resolution of a full board. So this is a zero or one. They took this value and averaged it with this one here. The formula was α x V_θ of some node plus (1 - α) of a truly randomly sampled playout.

You might be wondering how they play this out. It would be very costly to do another search on this playout, almost like a tree within a tree. They don’t do this. Instead, they just take the policy network and play it against itself. They use it as both players, and they just play it all the way to the end. This helps ground the estimates here in reality because you can get a single-sample estimate of whether you win or not. In the endgame, where the board is almost resolved, this becomes quite useful because the play according to the policy will most likely decide a pretty reasonable guess of the game, so you’re not facing a problem where this becomes untethered from reality.

It turns out this is totally unnecessary. In all subsequent papers after AlphaGo Lee, they got rid of it. In my implementation, I did the same, and it speeds things up a lot because you don’t have to roll these games out on every single simulation.

Dwarkesh Patel

Okay, just to reinforce my own understanding and re-explain it. For the audience, by the way, in case it’s not obvious, the P there in the select, that is the probability coming from the network in this case.

Eric Jang

Correct. The policy network here.

Dwarkesh Patel

Fundamentally, a simulation, just think of it as rolling out one more node in the search process.

Eric Jang

Almost. A simulation is easy to think about when the whole tree already exists. You just walk down the tree using the PUCT selection criterion, and then you keep going. In AlphaGo, the data structure is such that we begin with a tree that basically only has depth one—its only children—and you want to iteratively build out the tree as you’re also selecting actions down the tree. That’s the core thing here. Because Go is such a combinatorially complex game, you cannot afford to build the tree in advance and then search it. You must search while building the tree.

Let me finish up with the last step, which is the backup. Once you’ve scored these things, the Q value assigned to the node here for taking this action is just the average across your evaluated values. You take a running mean over all the simulations you’ve taken, averaging the values of the children nodes. That’s the backup step, and once you evaluate this, you can recursively go back up. If you know the action value of this node, you can then take the average on its parent, and so on.

You have this four-step process: 1) You’re choosing the best action you know of so far. 2) You may run into a node you haven’t been to before, so you need to grow the tree a bit. 3) You run it through the network to guess whether you’re going to win or not. 4) You walk all the way back up to the root node to update your values on what the best moves are.

As you do this iteratively, because you’re always selecting according to this criterion, you’re always going to be selecting the best action you think at any given branch. The final visit counts of how often you chose these things will reflect your correct policy distribution as induced through this search process. The visit count we stored in the node earlier actually becomes the vote for which action we should finally select.

As a test of understanding, it’s worth thinking a little bit about whether we could make this even simpler. Could we actually maybe even get rid of this one and still make the thing work? Recall that when you do an expansion and then an evaluation at this node, you are checking the win probability of each of the child nodes. So if this one is one and these are zero, you do know something about which action might be better to take. Why would you still need this, right? Why not just normalize the values into a distribution and call that your policy distribution?

This is fine. You can do this, and it probably does work. But in practice, having a single forward pass that gives you a pretty good guess is how the breadth is pruned out. There is a duality here. It would be weird if the policy recommended an action that disagreed with the value. If a policy said this was very high probability, but this one said it was a low value, then there’s something fundamentally wrong between your policy head and your value head. They are linked, and you probably could get rid of this if you came up with a different way to recover it from just the value evaluations.

Dwarkesh Patel

Right. Just to make sure I understand, the reason you don’t do that is so you don’t have to do 360 independent forward passes to say, “Okay, here’s the value of everything. Let’s argmax over it.” Instead, you can just do one forward pass and get the probabilities of all of them.

Eric Jang

You can usually batch these somewhat efficiently, so it probably isn’t a huge computational burden in practice. But yes, you would have to pass up to 361 boards into a single mini-batch update to evaluate all the values, then normalize them.

There’s actually a more important reason why we still do this, which is how Monte Carlo tree search is used to feed back on itself and recursively improve its own predictions and search capabilities. That’s where having this as an explicit entity you’re modeling, rather than an implicit normalization over your value, is a good idea.

Dwarkesh Patel

Makes sense. Okay.

Eric Jang

We talked about the simulations. What you end up with as you roll out the number of simulations is a tree that looks like… I’m drawing a very low-dimensional version of this. Of course, in the real game, it’s much more high-dimensional. You’ll end up with a tree structure that has a lot of leaves that terminate and are not visited again because their value is deemed to be too low. But along one path, there will be a set of actions with very high visit counts that gravitate towards that one set of decisions as you increase N.

This is the mental picture of what the tree in Monte Carlo tree search looks like. You should contrast this with an exhaustive tree like in tic-tac-toe, where there are nine actions, then eight, then seven and six, and so it’s a nine-factorial-sized tree. The Monte Carlo tree search in Go is very sparse. It only considers the paths that you’ve expanded children nodes on.

Now that we have the search algorithm that applies the value function as well as the policy function, we can talk about how the Monte Carlo tree search algorithm can act as an improvement operator on top of these guys here.

01:00:33 – Self-play

Eric Jang

We now talk about the RL part of how this thing gets stronger by playing itself. Let’s say we play a game. You make a move. The AI will compute the search, and this is the visit count distribution. Let’s say this is your initial policy recommendation at this node.

After MCTS, it gets more confident about one of these actions. Maybe the distribution looks a bit more peaky like this based on the search. Of course, you can tune the search process so that it ends up more diffuse, but that’s probably not a good idea. MCTS should get more confident about specific actions over others. It might place a lot of weight on other actions initially, and then as you increase the number of sims, it should converge to a very peaky distribution.

Let’s call this π. Let’s wrap this in an MCTS operator of (a | s). After applying the MCTS process, your policy recommended distribution looks like this. It’s a bit more peaky than the previous one. Then you take the argmax, or maybe you just sample from this. It doesn’t have to be the argmax. You make your move, throw away the tree, and begin anew on the next move. Again, you compute a new distribution. Initially, maybe your guess looks like this, and then you refine it through MCTS.

Dwarkesh Patel

There should be one more X on the board, right?

Eric Jang

I’m sorry, that’s correct, yes. Something that looks like this. On every move, you have your initial guess from your policy network. Then the search process, which combines your policy network and your value network, arrives at a more confident action that you take, and so on and so forth.

Then the game ends, and one person wins and one person loses. The beauty of how AlphaGo trains itself is that it can actually take this final search process—the outcome of the search process—and tell the policy network, “Hey, instead of having MCTS do all this legwork to arrive here, why don’t you just predict that from the get-go? Why don’t you not use this guess and predict this to begin with?” If you have this guess to begin with in your policy network, then MCTS has to do a lot less work to get things to work.

If we draw a test-time scaling plot… Let’s say this is the number of simulations. At zero simulations, your implicit win rate is here. If you just take this raw action, this is what your win rate is. Let’s say we increase the number of sims, maybe you have a win rate curve that looks like this. When you search for 1,000 simulation steps, that gets you to a policy here that gets you to here, which is great.

But if you were to distill this MCTS policy network back into your shoot-from-the-hip policy network, then you could actually start here. Let’s say this was zero, by distillation, if you then spend another 1,000 sim steps, you actually get to here. It’s almost as if you could amortize the first 1,000 steps into the policy network instead of the search process, then you could begin at a much better starting point and get a much better result for the number of sims that you play.

Dwarkesh Patel

The sigmoid type nature of test-time scaling as the number of simulations increases, the increase in win rate gets smaller. Is that true even for the distilled network? That is to say, is there some gain where we start from the distilled network and get these early gains again, or is that just inherent to the nature of MCTS?

Eric Jang

To be honest, I actually don’t know the test-time scaling behavior of MCTS simulations. I believe it might be quite sensitive to how strong this one is in practice. I’m just drawing a monotonically increasing function that gets to one.

Don’t pay too much attention to the shape of the curve. Just know that it’s monotonic with respect to sims. The idea of MCTS is very brilliant. We got something better by applying search. Now, on our next iteration of updating this network, we’re going to train it to approximate the outcome of 1,000 steps of search.

Instead of starting here, we get to have our neural network start here, and then the play gets stronger once we then apply another 1,000 steps on top of it. You can keep going. The training algorithm for AlphaGo is to basically take the games where you’ve applied the search on every move that the policy encountered—whether you won or lost, and that’s quite important—and just train the model to imitate the search process.

There’s an analogy to robotics, which is the DAgger algorithm. First I’m going to draw a schematic of the states: S₀, S₁, S₂, S₃. Let’s say we took a series of actions in an MDP to get a trajectory. These actions may be suboptimal. Maybe we lost at the end of this game. There is a family of algorithms that basically take trajectories and relabel the actions to better trajectories. Maybe a better action here would have been to take A₀’. A better action here would have been A₁’, and yet another one, A₂’, A₃’.

What MCTS is doing is saying: you played this game where you eventually lost, but on every single action, I’m going to give you a strictly better action that you should have taken instead. It does not guarantee that you are going to win, but it does guarantee that if you take these tuples as training data and retrain your policy network to predict these actions instead of the original ones, you’re going to do better.

This is very related to DAgger in robotics and imitation learning, where you want to collect an intervention here. Even if you’re in a not-great state—for example, a self-driving car that veers off the side of the road—there is still a valid action that corrects you and brings you back.

Dwarkesh Patel

Pedantic question, but is there a guarantee that MCTS must be better than the policy? For example, you could imagine early on in training, because MCTS is informed by the value network, that when the value network hasn’t been well-trained on finished games, MCTS is worse than a randomly initialized policy. Is it just a heuristic that MCTS is better than the policy, or is there some guarantee?

Eric Jang

In practice, it is a heuristic. It does work in practice, but let me illustrate an example where MCTS can give you a worse distribution than your policy network. This can often happen if your self-play algorithm has trained to a good point, but then somehow it collapses because it’s not trained on diverse data or something.

Let’s say we have a board state where the policy recommendations are very good. So Π_θ (a | s) is great. But somehow, maybe because we’re playing a lot of games where the bots just resign instead of playing all the way to the Tromp-Taylor resolution, they forget how to evaluate those late-stage plans. Like in the case we showed with the corner play, maybe 100% of our training data in our replay buffer has lost examples of how to evaluate the value function at those states.

You might end up in a scenario where your terminal value is very bad. If the terminal values of the leaves are not good, then this will propagate all the way up and cause your PUCT selection criteria and your backups to be off. And then you end up visiting a very different distribution than what your policy initially recommended.

Also, if your number of sims is low, you might have a variance issue where you just don’t explore enough. It’s only guaranteed to converge when you take N to infinity. Variance in your search process as well as inaccuracies in your evaluation can definitely screw with the quality of your policy recommendation. That’s why it’s not a guaranteed improvement.

That is why I suspect AlphaGo Lee had the playouts to the end in their training algorithm, so that they could ground this thing in real playouts. In practice, what you could also do is, for 10% of the games, prevent the bots from resigning and just say, “Resolve it to the end.” That way you get some training data in your replay buffer to really resolve those late-stage playouts that normal human players would not play to.

If you assume that the value functions are correct, this is why MCTS gives you a better policy. It’s a very critical chain of assumptions. Assuming this is accurate, your search process should give you a better recommendation than your initial guess.

Dwarkesh Patel

If you have a cold-started policy—if you have an AlphaZero-type thing—really what’s happening for the first few epochs is that the policy is kind of useless. What you’re really just doing is, “Hey, let’s play full games, and once we have played full games, for the preceding moves, we’ll have labeled who won and who didn’t win.” The loss for AlphaZero has two components: how good is the policy relative to MCTS, and how good is the value prediction relative to who actually won the game from this move. You can think of this being applied to every single action or every single move.

Really what’s happening at the beginning of AlphaZero training is just that we’re trying to get the value function to actually predict who will win the game if you find yourself in this state and you’re this player. Functionally, that’s all that’s happening. Later on, once that’s well trained, now the policy is also improving.

Eric Jang

Correct. There’s one trick I found to be pretty useful. This is not a peer-reviewed claim, so take it with a grain of salt. I found it useful in my own implementation to do the following. You want to first make sure that this is good before you invest a lot of cycles doing MCTS. It doesn’t really make a lot of sense to do search on garbage value predictions. You want to start at a good place where this works.

AlphaGo Lee does a very good thing where it just takes human games, you train on them, and it just works. It totally works. You can also take an open-source Go bot, play it against itself, generate data, and that also works. If you have some offline dataset that has realistic, good play, you can easily learn the late-stage value functions pretty well. That’s what you need to start the search process.

Dwarkesh Patel

Sorry, can you just repeat that sentence one more time?

Eric Jang

Sure. It’s quite easy to evaluate a late-stage Go game. When almost all the pieces are on the board, it’s almost like a decidable problem because there’s lower and lower uncertainty as to the depth of the tree. So most games played to the end by reasonable people will be good training data to train a good value function at terminal parts of the tree.

Then, as you play more games, the search will back up good values into the intermediate nodes of the tree. As you increase the amount of data, your value head gets a good intuition of what is a healthy board state versus a not-healthy board state. Those are much more subtle to judge in the mid-game than at the beginning or the end.

The most difficult part to score is not the beginning, because the beginning is obviously 0.5, and at the end it’s pretty obvious who’s winning. The hard part that you want to learn in the value function is who is winning in the middle.

Dwarkesh Patel

So this is actually very analogous to TD learning.

Eric Jang

Yes. There’s a beautiful connection to TD learning that we can talk about in a bit, contrasting it with Monte Carlo tree search.

You first want to get good value functions, and expert data can give you a quick shortcut. I recommend for practitioners, just do that first to initialize to a good starting point. If you want to do the AlphaZero thing or KataGo tabula rasa learning, what you can try to do is play random games on a small board. Just take a random agent. If you play 50,000 games, you’ll actually learn a pretty good value function as well. On a 9x9 board, you can see enough of the common patterns with random play.

If you train on both 9x9 and 19x19 data—and KataGo proposed one of these architectures—there’s pretty good transfer learning from the value head evaluated at 9x9 to 19x19.

Dwarkesh Patel

Right, because this, unlike other games, has very much a sense… There’s not like a new kind of piece introduced when you increase the size or something.

Eric Jang

If we take it to its limit and consider a very tiny 4x4 Go board, if you play 50,000 games, you’re going to have a lot of end states that look like human play. It’s just Tic-Tac-Toe at that point.

If you broaden this a little bit to 5x5 or 9x9, it’s not unrealistic to imagine that purely random play will generate pretty reasonable-looking boards. You can score those pretty easily. That is what gives you the bootstrapping to then improve your policy with search.

But it’s very critical that MCTS has accurate value estimates. You need to ground the value. Ultimately, MCTS will fall apart if you don’t have a grounding function for the value.

Dwarkesh Patel

I’d be curious how much compute you save by training the value and policy on the same network. Because they share the same representations, how much more efficient is learning? We’ve just talked about how they’re making similar predictions, or they should be in line with each other. So I’d be curious if you’re halving the amount of compute you have to do by keeping them in the same network.

Eric Jang

AlphaGo Lee, the original AlphaGo paper, had two separate networks. In all subsequent papers, they merged them into two heads, and presumably this saves compute. But answering that question in a rigorous scientific way is a simple question that in practice, if you really want to chase the question down to its limit, takes quite a bit of work to really resolve.

Intuitively, yes, they share a lot of representations. As we mentioned, your policy network and your value network, when doing evaluations, should agree. There really should be this consistency between them.

Dwarkesh Patel

Tell me if this is the wrong way to think about it. When I learn how an LLM works and how simple RLVR is as an algorithm, I’m stunned by the kinds of things it can do. It can learn how to build very complicated code repositories simply from getting a yes or no.

Here, if you understand it more deeply, just predicting MCTS, AlphaGo seems less impressive in retrospect the more you understand it. You’re putting in a lot of bias by telling it how it should titrate exploration as things go on. You’re building this very explicit tree search for it. I don’t know if you share that intuition where the more you understand it, the less impressive the accomplishment in 2017 seems.

Eric Jang

I personally disagree. I think they’re profound for different reasons. I don’t understand the LLM RL enough to comment on it on your podcast. But why is AlphaGo a profound accomplishment? It’s worth stepping back a little bit. It is different from modern RL, and we can talk a little bit about some of the algorithmic choices there.

I think the most profound thing here is that a 10-layer neural network pass, basically 10 steps of reasoning… Of course, the reasoning is not just one trail of thought. It could be distributed representations and a lot of thoughts going on at the same time. But by construction, let’s say a 10-layer neural network can only do 10 sequential steps of thinking. 10 steps of neural network parallelized distributed-representation thinking is able to amortize and approximate to very high fidelity a nearly intractable search problem.

This was a breakthrough that I think most people don’t even fully comprehend today, how profound that accomplishment is. This is what also girds AlphaFold, for example, where you have a very, very difficult physical simulation process where you would need to roll out as so many microscale simulations, and yet 10 steps of a somewhat small neural network can somehow capture what feels like an NP-class problem into a single problem.

It actually makes me wonder if our understanding of problems like P=NP, or these fundamental computational hardness problems, is incomplete. Obviously, it’s not a proof of P=NP, but there’s something to it that is very disturbing, where what felt like a very hard problem can fall to a very simple macroscopic solution.

Dwarkesh Patel

That is a very interesting insight, that a lot of problems which are proven to be NP-hard—I don’t know if Go is proven to be NP-hard, but protein folding, et cetera—neural networks can solve. They’re NP-hard in the worst case, but we’re usually not concerned about the worst case. These problems usually have a lot of structure to them.

Eric Jang

I think the question we should be asking ourselves is about how we’ve been formulating solutions to NP-hard problems in worst-case complexity. I wouldn’t say this solves Go. It doesn’t give us an exact solution of the optimum, but in practice, it is extremely useful. The same thing has been shown in AlphaTensor and AlphaFold. Yes, there is a very hard problem that, in the worst case, seems intractable, and yet we’re able to make almost arbitrary amounts of progress.

In the limit, what might this look like? If you want to simulate something very complex like weather, or predict the future—do we live in a simulation or not—the computing resources you need to build a very complex simulation might be much smaller than you think, based on our ability to amortize a lot of that computation into the forward pass of a single network. To me, AlphaGo was the first paper that really showed this profound level of simulation being compressed into a small amount of compute.

Dwarkesh Patel

I feel totally not qualified on the computational complexity or the math to comment on this, but I wonder if there’s an important role of chaos here. What is the problem with weather, and why does it take 10x the amount of resources to predict weather a day out, and continually so for every additional day out? It’s because it’s a chaotic system, so small perturbations can totally change the final estimate as time goes on. I guess you would expect that for Go and protein folding as well.

Eric Jang

Here’s an analogy to weather that might be relevant in Go. Here’s our current board state. Given what we know about both players, what is the exact board state in the future? This is extremely sensitive to initial conditions. A single stone placed here can disrupt the entire prediction. This is hard. Intuitively, this is the chaotic problem.

Yet somehow, we can predict who’s going to win. This captures a lot of possibilities. There’s a more macroscopic quantity that we really care about, which is the average or expectation or some sort of global macrostructure over a lot of possible futures.

In weather, it could be the same thing. We don’t exactly care what the velocity of wind 6,000 feet above a specific latitude and longitude is. We care where the hurricane is, and things like that. In chaos, there’s the classic Lorenz attractor. If you start anywhere on the Lorenz attractor, you don’t know where you’re going to end up, but you do know that the thing looks like this. There’s a kind of beauty here. Sometimes we don’t necessarily care about the microscale things. We actually care about the macroscopic structure, and these things can be predictable.

Dwarkesh Patel

Contrast that to something like a hash function, which is also incredibly dependent on initial conditions but doesn’t have a macrostructure, or at least hopefully doesn’t, if the algorithms work.

Eric Jang

One would hope.

Dwarkesh Patel

There’s no equivalent of a value function or broadly how the weather is going to be that’s interesting there. It’s really just about what the board is going to look like exactly 100 moves from now exactly.

Eric Jang

Intuitively, that seems correct. This is also out of my area of expertise, but I find it interesting that the tools of cryptography and hashing have also not been able to prove that you cannot come up with fast approximations. You cannot come up with fast approximations. If they were able to do that, then you could prove P is not equal to NP.

Dwarkesh Patel

In fact, we know that there’s structure in many cryptographic protocols, obviously RSA cryptography. There is structure, and that structure is what quantum computers exploit to break them.

Reiner has a very interesting blog post, which we’ve talked about in the episode, where he points out that if you look at a high level at what cryptographic protocols look like and what neural networks look like, they’re extremely similar. You have sequential layers of jumbling information together. There’s a convergent evolution in the algorithms. In cryptography, you want the final state to be incredibly sensitive to initial conditions, so that it comes out looking jumbled if you change anything. In neural networks, you similarly want everything to be dependent on all the information, because you want to process all the information and consider how it relates to itself.

Eric Jang

You have the maximum power of a neural network at the edge of chaos. I think there are some research papers from Jascha Sohl-Dickstein on this. There’s something quite fundamental about chaos that’s not just hopeless noise. There’s something useful in chaotic systems, at least at that boundary. But this is just thinking about it as a philosophy. I don’t know the math well enough to comment on it.

We’ll talk about LLM RL in a little bit, because there’s some connection there, but let’s go back to MCTS. What is it doing? Crucially, it is not saying that we’re going to increase the probability of winning directly. It’s not going to upweight all actions that won and downweight all actions that didn’t win.

Importantly, what it is doing is saying: for every action we took, we did a pretty exhaustive search on MCTS to see if we could do better, and we’re going to make every action that we took better by having the policy network predict that outcome instead. This is a very nice idea because you have one supervision target for every single action. The variance of your learning signal is very low compared to the alternative naive RL thing.

Let’s consider a very naive algorithm that looks a lot more like modern LLM RL today, where we do something like take the winner of a self-play game and encourage it to do more of that.

01:25:38 – Alternative RL approaches

Eric Jang

It’s worth thinking a little bit about what alternatives we could use to train self-play agents instead of MCTS. We use a lot of LLM-style RL these days. Is that relevant? Could we do that instead?

Suppose we have a very naive algorithm where we take a league of agents of different checkpoints and play them against each other. For the games where a single player wins, we reinforce those actions up and retrain the policy network to imitate those winners, instead of the MCTS objective. Let’s say you have a chain of actions that led to a win, and you have a matchup between two agents that are basically the same. Assume policy a (Π_a) and policy b (Π_b) are evenly matched, so their true win rate is 50%.

Let’s say you play 100 games, and each game lasts 300 moves. You’re doing some sort of evolution strategy or some way to perturb these things to get them to do different things. Or maybe you don’t, and you just play them against each other to see that occasionally one might have a better strategy than the other. Let’s say policy a wins 51 games and policy b wins 49. This is just due to random luck, or maybe you perturbed policy a in some way that let it do this.

Just to have a very simple model, let’s pretend that for 50 of the games, they played exactly equally. In the one game where a won, it played slightly differently. It made one critical move that normally it would have done differently, but due to some exploration or random noise, it happened to make a smarter move than it did previously.

So you have one true supervision signal for your policy network, and then you have 99 games times 300 moves for which imitating those actions gives you exactly the same policy you had before. The scale of your variance is actually very bad, because you only have one label out of this enormous dataset of supervision actions where you want… Actually, let me clarify a little bit.

Dwarkesh Patel

We’re talking about how the good move, the out-of-distribution move, is a small fraction of all the moves that are played across all the games on which you’d want to train. This, of course, reminds me of how LLMs are trained with policy gradient methods. When Karpathy was on the podcast, he called it like “sucking supervision through a straw”. It’s interesting that this thing you’re saying—which would be intractable and prevents you from actually getting beyond a certain level in Go—is just by default how LLMs are trained?

Eric Jang

Right. This is not to say it doesn’t work. If you imagine increasing the number of games to millions of samples, you actually can get some meaningful supervision samples, so long as you find a way to mask out the supervision from these guys. This is where things start to get pretty related to RL in terms of advantage and baselines.

Let’s look at the gradient variance of a very naive approach—I’m going to call it gradient RL—that’s basically the sum of rewards. The sum of rewards is the return. In our naive setup here, we only have an indicator variable for the return, where either you won or lost. In the case where you lost, your gradient is zero, so you don’t train on those examples, and when you won, you try to predict those things. You can think about this setup as a special case of this general formula here.

The trouble here is that this is very high variance. When you multiply these terms out to compute the variance of the gradient, it’s equal to the expectation of… Just for simplicity, we can pretend this is on average zero or something if you’re centering it at no signal. The variance here basically means that you’re taking the square of this product term. You end up with a term that grows quadratically with T. When you have a setup like this, this thing acts as a coupling effect on top of these terms here.

Let’s actually map this to an LLM case, and we can answer why LLMs only do one-step RL instead of a multi-step RL scenario. In LLMs, you have a decoder that might predict some words like “hello world”. In current LLM RL, they treat this entire sequence as a single action, a_t, and big T is just one. It’s true that because of how transformers are formulated through the product of conditional probabilities, the log probability of the whole sequence is equal to the sum of the probabilities of individual tokens.

In this case, I would say something like log(hel) + log(lo) + log(world). This is true, and if this term were one, then they would be the same thing. However, when you’re sampling, if you have a reward term assigned to every specific token, now you have these interaction effects between the cross-multiplication of these terms and these terms. The problem becomes, how do you ascribe the credit associated with every episode to all these different terms?

Dwarkesh Patel

The thing I’m confused on is what it would even look like to do it that way in LLMs, because you only get a reward at the end of the episode.

Eric Jang

You could imagine a reward that says, “I’m going to give you some process supervision where you get a reward for each of these actions on every step.”

Dwarkesh Patel

Okay, so you’re saying if instead of doing it that way… Well, the way you’ve written it, it would be a sum at the end anyway, so they wouldn’t have to be multiplied. But you’re saying instead of doing it that way, you would just add up these process rewards at the end and then treat that as one single reward signal?

Eric Jang

Correct, for one single log prob action.

Dwarkesh Patel

But isn’t that how it’s written to begin with anyways? The sum of the rewards?

Eric Jang

The thing that’s a little bit hidden here in the math is that we’re assuming that when you decompose the problem into a multi-step problem, you’re now introducing correlations between your actions through the computation of this guy. If you separate these things out, this will magnify the variance of this one.

In the case where you don’t separate it out, if you just have T=1, you have a single estimate of log prob and a single estimate of reward. This term still shows up. In LLMs, the naive REINFORCE estimator looks a bit like the return of the single action… It looks kind of like this. This is the basic form here, but it’s still a contributor to variance. You want to make sure that, similar to how in this case we were training on a lot of neutral labels, you’re sort of penalizing the labels that don’t help and only rewarding the ones that actually make you better.

Intuitively, the analogy is: can we find a term in our training objective such that it’s actually discouraged from doing this, or these don’t have any effect on the gradient, and this has an effect on the gradient?

Dwarkesh Patel

If you applied that there, the only thing you could do is eliminate 49 of the games. At least the way you have it written there, it would be 51 times…

Eric Jang

Actually, the optimal case is to discard all of these moves, and only get a gradient on that single move that you got better on.

Dwarkesh Patel

How would you do that?

Eric Jang

This is a pretty tricky problem in practice. This is where advantage estimation happens in reinforcement learning. You want to subtract a term from your multiplier. Instead of an indicator function of one and zero, you want something that behaves like a zero for all of these guys, and then a one for these ones.

Dwarkesh Patel

You could do that if you can say, “Hey, I won this game, so this is slightly above baseline performance.”

Eric Jang

Well, you won a lot of games. But you don’t know which ones let you win because they were truly better versus winning by accident.

Dwarkesh Patel

How would you design a baseline where it’s truly better?

Eric Jang

This is where in RL people use things like TD learning to better approximate the quality function, the Q that we mentioned earlier. You can try to subtract that from your return.

Ideally, what you really want to do in RL is push up the actions that make you better than average and push down the actions that make you worse than average. They call this advantage. There are multiple ways to compute it. I highly recommend John Schulman’s “Generalized Advantage Estimation” paper as a good treatment on how to think about various ways to compute it. At the end of the day, you want to reduce variance by trying to make this smaller, so it doesn’t magnify the variance of this one.

Dwarkesh Patel

That makes sense. This requires you to have a very good estimate of what average performance from a state would look like. This gets us back to the value function thing we were talking about earlier.

Eric Jang

Keep in mind that this model-free RL setting is trying to solve a credit assignment problem where you don’t know which actions were actually good and which ones were bad. Monte Carlo tree search is doing something very fundamentally different. It’s not trying to do credit assignment on wins. It’s trying to improve the label for any given action you took. We can actually think about a completely different algorithm called neural fictitious self-play, which was used to great effect in systems like AlphaStar and OpenAI’s Dota.

Let me talk a little bit about how you can unify some of these RL ideas in the model-free setting as well as the self-play setting. What happens if you don’t have the ability to easily search a tree? In Go, it’s a perfectly observable game. You can easily construct a pretty deep tree that completely captures the game state. In a game like StarCraft where you don’t have complete control over the binary, it’s a little bit hard to do this, and I’m not even sure if it’s a deterministic game. That makes this difficult from a data structures perspective.

What is done instead is that the basic idea of supervising your actions with a better teacher is still there. We’re going to talk a little bit about how neural fictitious self-play works. It’s the same idea. We’re going to come up with better labels for each of the actions we took, just like in MCTS. But how do we derive the better labels? In MCTS, we perform search. Assuming we have a good value function, the search will give us a better result than our initial guess.

In a game where you can’t easily simulate a search process, what they do instead is train what’s known as a best response policy. You fix your opponent. Let’s say you’re currently training Π_a against a strong opponent, Π_b. In StarCraft, maybe these are the Zergs and you’re playing Protoss or something. You fix your opponent, and you treat this as a classic model-free RL algorithm, where your goal is just to beat this guy. Here you use your standard TD-learning-style tricks, or use PPO or any model-free RL algorithm to try to hill climb to winning against this player. You have a reward function where the return is one if it wins against Π_b, and zero otherwise. This is no longer a self-play problem. It’s just a fixed opponent, and you’re trying to maximize a score against them. You have a fixed environment where all you care about is beating this guy.

Once you have a good policy that you train with your favorite model-free RL algorithm—PPO or SAC or V-MPO—you now have a good policy that gives you a good label for what this one should do when playing against that player. When you train multiple best response policies, you can basically distill the RL algorithms into the labels for a given opponent. You might have a best response policy against π_b, and then maybe you have a league of opponents like π_b, π_c, π_d. You’re going to take the best response policy that you train against each of these fixed opponents, and supervise them with the label that this one would provide.

This is almost like a proxy for your MCTS teacher. Instead of an MCTS teacher, you use a model-free RL algorithm to find the best search action that you could do to beat your opponent. Then you’re distilling the policy here into what’s known as a mixed strategy, where it’s trying to average across all possible opponents you could play against. This is what gives you something that can do no worse than an averagely selected opponent from the league. This gets around the problem of having to derive a teaching signal from MCTS, but fundamentally it’s still about relabeling your states with better actions so that they improve your policy.

Dwarkesh Patel

Just to make sure I understand, if you win a game against this other policy, you reinforce all the actions on that trajectory.

Eric Jang

Yes. Here you can use a number of algorithms like PPO, V-MPO, or even Q-learning if you want. The specific algorithm here is usually a model-free thing because you don’t have search, but there’s an interesting connection between MCTS and Q-learning that I want to bring up.

In MCTS, you have a tree, and through the resolution of your value function at the approximate leaves of the tree, you can back up through many sequences and obtain some sort of mean value estimate. Your Q is derived from the average of a bunch of simulations.

In model-free algorithms, there’s often a component of estimating a Q value. Q values are often learned through TD learning, although in PPO, the way they do advantage estimation is not necessarily through a Bellman backup. In Q-learning, there’s this very cool trick where Q(s,a) is backed up as r plus some discount factor times the max over a, Q of your next step. Intuitively, if you have an MDP and this is terminal, it’s saying that the best action you can take at this state is equal to the reward you get for taking this action, plus the best that you can do at the next state.

There’s a recursive, dynamic-programming property of MDPs, and you can train neural networks to basically enforce this consistency. You can say, “Well, once I know the Q value of this action, I can then use that to compute something about the Q value for the next.”

Dwarkesh Patel

Earlier I was like, “Hey, why are we training policy? Why don’t we just train the value alone?” That is what this is.

Eric Jang

This is an algorithm for recovering value estimates of intermediate steps when you don’t have the ability to do forward search. You must collect a trajectory first of n steps before you’re able to do this trick. But the intuition is kind of the same. Knowing something about the Q value here can tell you something about the Q value here. And indeed, you can recover a policy from a Q value. You don’t need to explicitly model the policy distribution. You can recover the policy distribution by doing argmax over your Q values.

Q-learning, or approximate dynamic programming, propagates what you know about the future Qs backward like this. You can see that there’s a similar structure going on here, where in one case you’re planning over trajectories your agent hasn’t been to yet, whereas in the other you’re planning over trajectories your agent has visited.

Importantly, why was Q-learning a big deal? It’s because historically we just haven’t had the ability to do search on fairly high-dimensional problems like robotics or whatever. For a long time, we made the assumption that if we can’t model the dynamics with a world model, we’re going to instead just collect trajectories and then plan with respect to the only number that really matters, which is reward.

01:45:47 – Why doesn’t MCTS work for LLMs

Dwarkesh Patel

This is very interesting. To unify this with our discussion of LLMs, you don’t have Q values, but you’re doing this backwards learning where you find the trajectories that pass some unit test in some coding environment and then reinforce those trajectories.

There’s a huge difference between that and this forward approach with MCTS. The reason it’s much more preferable to do MCTS is because you can do it per move and make each move better, rather than having to learn per trajectory and hope, as Karpathy said, to learn this…

Eric Jang

Through a straw.

Dwarkesh Patel

Right, you get supervision through a straw. You basically just upgrade all the tokens in a trajectory that might or might not have been relevant to getting the answer right.

The reason you can do this much more sample-efficient, favorable thing with Go is that because MCTS works in Go, you basically know that if you just do the search locally here—and this search is truncated at the end by a value function that works even if you haven’t unfolded your whole trajectory—you can just say, “This is my new policy,” and improve in a more iterative, local way, rather than having to unfold all these trajectories.

Eric Jang

There was some research from Google in 2023 or 2024 where they tried to apply tree structures to reasoning. The jury is still out as to whether this can ever work. We probably will see a revisiting of this idea of forward search in the future.

But there are two things that make MCTS very simple for Go. Value estimation is concrete. You can determine it for real, and then you can use it to truncate depth, as you said. The breadth is also determined. What’s critical is that the action selection algorithm, where you iteratively visit and grow the tree, is well suited for the size and depth of problem that Go is.

But for something like LLM reasoning, PUCT might not be a good enough heuristic. It might be too greedy with local tokens, and it might only give you obvious thoughts that are correct but don’t really solve your final problem. I would say the jury is probably still out on what the final instantiation of reasoning for LLMs will look like. I wouldn’t rule out that this stuff could come back, but it’s a bit hard.

Dwarkesh Patel

Don’t LLMs natively learn to do MCTS, where they’ll try an approach and be like, “Oh, that doesn’t work. Let’s back up. Let’s try this other thing,” and then go in the direction that proves to be more fruitful?

Eric Jang

Certainly, LLMs manage to do something that looks like real human reasoning without having to do an explicit tree structure. That being said, I think the idea of doing forward search and simulation to get a better sense of what is valuable might make a comeback, even if not in exactly the same instantiation as AlphaGo.

Dwarkesh Patel

Just to make sure I understand the crux of it: the breadth comes from the number of legal actions being wider, and the depth comes from not being able to train a value function as easily, because…

Eric Jang

Here’s an example where LLMs break down. The C_PUCT rule involves the √N over 1+N_a. In an LLM, you’re most likely never going to sample the same child more than once. If you have multiple steps of thinking, because language is so broad and open-ended, a discrete set of actions is not really an appropriate choice for an LLM. Even though they’re discrete tokens, it’s just such a large number that this type of exploration heuristic is probably not the right thing to guide how to search down a tree.

Dwarkesh Patel

I guess the crux comes down to the fact that in Go, you know that MCTS is almost certainly better than your current policy, even though you haven’t explored the end of any trajectory. In normal reasoning for LLMs or robotics, there’s no way to just locally evaluate and improve your next move in a way that’s independent of actually solving the problem.

Eric Jang

“No way” is a strong word. Lots of people have thought about how to apply MCTS or its successors like MuZero to continuous control spaces, and I’m sure very cool research work is still ongoing to try to crack that problem. But yes, the apparent challenge right now is that most problems in much higher-dimensional action spaces, or something combinatorially much bigger like language, don’t seem as amenable to the discrete action-selection heuristics and game-evaluation type stuff that Go has.

That’s not to say the idea of thinking into the future along multiple parallel tracks might not give you some information about which way to search. If you think about mathematics, it often occupies more of a logical search procedure where you can back up and see which paths seem good or not. There’s more of a rigid structure there, whereas in a business negotiation it’s less of a tree and maybe something a bit different.

Dwarkesh Patel

We’re now seated, so I can ask you some more questions about AlphaGo and AI research more generally. In 2021, Andy Jones had a paper called “Scaling Scaling Laws with Board Games”, and he basically anticipated inference compute, or inference scaling, by showing that you can trade off test-time compute and training compute. That is to say, you can spend more compute searching through MCTS, and if you do that, you can get the equivalent performance to having spent more time training the model.

If you see this pattern, you might think that with LLMs you might do something similar in the future, and in fact that’s what ended up happening.

What is a fun exploration one could do now to explore other axes of scaling in toy settings, which will be important to understanding what AI development might be like in a few years?

Eric Jang

Test-time scaling and reasoning, and how they interact with model size, are quite profound when it comes to how much needs to actually be done as explicit search versus how much can be packed into the forward pass of a neural network. How does a forward pass of a neural network learn to do something that should be sequential and recursive? That’s quite interesting.

The Andy Jones “Scaling Scaling Laws with Board Games” paper is quite cool. There was another really nice result from that paper. Not only can you predict scaling laws of the LLM variety—where as you increase parameters you can decrease the amount of compute for search or vice versa—but he also showed that you can predict how much compute is needed to solve a larger version of the board game.

For example with Go, which can scale from 3x3 to an infinitely sized Go board, you might be able to revisit this question and try to reproduce whether this shows up. I actually started this project with the motivation of asking: does the “Bitter Lesson”, or our knowledge of scaling laws, allow us to execute a lot better on a compute-optimal Go bot? Can we build a strong Go bot without all of the KataGo tricks, just by really focusing on the Bitter Lesson and scaling laws?

I have not been successful so far, but usually when you want scaling laws to work, you want to be in the regime where the recipe already works and the data sets are good, rather than trying to figure out how to do scaling while also trying to figure out what the right data sets are.

The scientific understanding component in research often follows a step where you get something to work first. Then you use that system to collect data that then helps you build a mental model of how things work, such as scaling laws. Usually, if you want to build a strong Go bot using scaling laws, you actually have to make a strong Go bot first and then use the scaling laws to extrapolate a bit further into the future.

Dwarkesh Patel

Say more. Just so I understand, are you saying scaling laws did not work? There was no scaling-laws pattern that you could see in your Go bot?

Eric Jang

A mistake I made initially, when I had some bugs around how MCTS labeling was working, was that I would collect a bunch of data with an expert policy and then treat it as a supervised learning problem and try to identify scaling laws with expert data sets. You can indeed plot things that look like this, but if you’re in a regime where your policy is not working well, you might just be studying scaling laws on bad data.

One important implementation detail is that if you want to study a scaling-laws problem, you have to have a problem for which the data is good, the architecture is good, and there are no bugs, and then you solve it from there.

Ex ante, I wasn’t able to apply scaling laws to direct what to look at until I had the rest of the system working. This sounds obvious to researchers. Of course you want a working, bug-free system before you study scaling. But as advice for practitioners, on where I tripped up when I started this project, you don’t necessarily want to jump into the science of studying your man-made artifact before your man-made artifact is interesting enough to be studied.

Dwarkesh Patel

Speaking of compute, you can look at these charts of compute used to train the best AI model in the world over time going back 10 years. It’s a very smooth line in log space that grows exponentially year over year. Except there’s this huge aberration, and that aberration is AlphaGo Zero, which was trained on way more compute than any other AI model at the time. It was 3E23 FLOPS. It’s comparable to a frontier LLM, orders of magnitude off, but still.

The question is, especially with you being able to get something off the ground, did you train it on your own?

Eric Jang

I got a donation from Prime Intellect for about $10K. I spent maybe the first $4K doing exploratory research, and then about $3K on the final run. Some of it remained for serving the model.

Dwarkesh Patel

Is your sense that they did a bad job training it, if you can do it in $10K now?

Eric Jang

The compute required to be the first to do something is always much larger than the compute it takes to catch up. It’s the same story playing out in LLMs. Once someone else has done it, you can use tricks like distillation. You can use all sorts of crutches to bootstrap your way to success.

With my own bot that I’ve hosted online, I actually used best-response training against the KataGo models to get a strong level of performance. As of the time of recording, I’m validating whether I can do that first step, which is to do the tabula rasa play.

Importantly for research, you often want to start from a good init. The simple thing I did first was train best-response agents against KataGo. The AlphaZero team did not have any policy they could train against, because they were trying to do everything tabula rasa. Being the first to do it means you’re prioritizing getting the thing working rather than the most compute-efficient possible implementation.

This plays out in robotics as well. If you look at the frontier of large models trained for robotics, the scatter plot is all over the place, and there isn’t a very clean line the way there is for frontier LLMs. That’s because the folks training these models often are not at the scale where every FLOP counts and they need to squeeze out the performance of every single FLOP as the deciding factor in pre-training. Instead, their focus is more, “We want a certain capability to show up, so we optimize the training setup to make it easy to derive that capability.”

Once you have that capability… Invariably if you scale up the compute, you are forced to make it compute-efficient, because this is hundreds of millions of dollars we’re talking about. But in the past, when compute for experiments was more plentiful—or not accounted for in a way the researcher was really responsible for—you end up with people optimizing for things besides being on the compute-optimal Pareto frontier.

Dwarkesh Patel

I see, like speed or something?

Eric Jang

Yeah, time to result, or just getting it to work. The first AlphaGo probably had lots of compute, and they didn’t need to worry too much about making it the most compute-optimal thing.

Dwarkesh Patel

How much of the improvements to compute efficiency are methods that did not exist as of 2017 versus things which they could have done in 2017 but didn’t?

Eric Jang

Great question. Going into this project, I knew in the back of my mind that things always get easier to do over time. I wanted to see where Go was at, given that there hadn’t seemed to be any major open-source strong bot after KataGo in 2020.

Reading the KataGo paper, there were a lot of clever ideas. I was thinking, “Let’s see if the Bitter Lesson had happened, where a lot of these tricks just go away because Nvidia made faster GPUs. Roughly, where are we on that?”

Again, this is not a peer-reviewed claim. It’s just my preliminary vibe guess based on what I’ve seen with my own experiments. It seems like architecture choices don’t matter that much. Transformer versus ResNet… We’re at the speed of GPU where the size of the model is not so big that this really matters.

You can simplify the setup quite a lot. Instead of doing a distributed asynchronous RL setup with replay buffers and pushers and collectors, you can do a dumb synchronous thing where you collect, train a supervised learning model, and then collect again. There are opportunities to simplify infrastructure.

Nvidia GPUs have indeed gotten faster. Whereas KataGo was trained on V100s, you can train on half the number of desktop Blackwell GPUs and it still works. Some of the auxiliary supervision objectives that KataGo developed aren’t really necessary if you have a strong initialization. If you’re initializing best-response training against KataGo itself, your own model needs none of the tricks that KataGo needs. The core thing is getting as quickly as possible to some strong opponents. That matters a lot more than the specific architectural innovations.

But there are still some nice compute multipliers. I found that training on 9x9 boards was very nice for resolving endgame value functions. If you can co-train that on an architecture that can transfer between 9x9 and 19x19, you can really cut down the warm-start time to learn from scratch. AlphaGo Zero’s plot showed the first 30 hours or so spent basically catching up to the supervised learning baseline. You can cut down that time a lot by pre-training on a small board and then warm-starting that into your 19x19 board play.

There was some other stuff, like varying the number of sims between episodes. This turns out to be not that sensitive. You can fix it or increase it. It doesn’t matter too much. Anyway, it’s nice from a scientific perspective, revisiting an old paper and seeing what really matters.

02:01:09 – Off-policy training

Dwarkesh Patel

This is a tangential question, but why is it okay to have a buffer in AlphaGo? Every time I talk to an AI researcher, they’re telling me about how bad it is to be off-policy. But the way a naive implementation of AlphaGo Zero would work is that most of the moves in a given backward step, or in a batch of backward steps, would not be among the ones made by the most recently trained model. So why is that okay?

Eric Jang

Great question. This gets into the fundamental off-policy versus on-policy reinforcement learning questions. As you’ll recall, in MCTS, you take actions that you took, and you relabel them to take different actions on the same states.

The off-policy part comes in where, what if you’re relabeling states that your new policy would never visit? What’s the point? You’re wasting capacity. In the extreme limit, imagine the distribution of states in your training buffer are all states you would never visit. Then you’re supervising them to take good actions on states you would never achieve. Therefore, your policy can get really bad. This is where off-policy can really hurt AlphaGo.

However, if you interpret this from the DAgger perspective—which is basically a way to correct yourself back to the optimal trajectory given some data—what you want in an algorithm like this is to have mostly states that you would visit, but then a small or reasonable percentage of states in this high-dimensional tube around your optimal trajectories. Any of those states are given a supervision target to funnel you back into your optimal trajectory.

Maybe I can just draw real quickly here. In a DAgger-style setup, your optimal training data distribution is your optimal states and actions. You want to be in this state, you want to be in this state, you want to be in this state, and then you win here. These are your optimal policy actions. These are the things you definitely want to train on. But to make it robust to disturbances, you want to make sure that if you happen to drift off into some other states, you can funnel yourself back in.

Dwarkesh Patel

But why isn’t this a fully general argument for off-policy training?

Eric Jang

This is actually why you want to do off-policy training sometimes. You don’t want a compounding error where, if you make a mistake, you don’t have the data for how to return to your optimal distribution.

Optimal control doesn’t really say too much about how to not accidentally get here, because it’s making the assumption that once you learn the policy, you’re going to get to here. But in applications like robotics, a gust of wind blows you slightly off and now you need to correct. Or the friction on one of your tires is slightly lower than the other wheel, and now your car is drifting and you have to correct it. These kinds of things often happen in more real environments.

There was a funny quote about chess and Go. The problem with Go and chess is that the other player is always trying to do some shit. Things can drift off, and you always want to be able to correct back to your winning condition. Your replay buffer really should have the states your policy would visit, plus some distribution of states you might drift to, and then how to return back to your optimal states.

Now, if you take this to the extreme and you say, “We don’t have any of this data, and we’re just going to be labeling with MCTS states that are so far away from our optimal behavior, like this bag of states over here…” Well, now each of them gets an MCTS label, and your policy learns how to take the best possible action here, but you never get here. You’re training your model on states you would never reach. This is where off-policy can really hurt.

As part of this project, I did try an experiment where I took a bunch of trajectories, and to try to saturate the GPU as much as possible, I took random states from the dataset and reran MCTS on just those states. Instead of playing a whole game where I’m doing MCTS on every move, I ignore the causality of moves and pick random board states, and I label those with my current network. I might revisit old states I’ve labeled before and relabel them again with my current network.

In practice, this actually does work. You can take some states that are reasonable and constantly be relabeling them while training. This starts to converge on a very robotics-like setup, which is very common. You have your dataset of trajectories, and then you have something like a replay buffer pusher. These are off-policy offline trajectories. Your replay buffer pusher pushes transition tuples to the replay buffer, and then you have a job that’s continuously replanning what the best action you should have done instead of the action you took.

In robotics, it’s very common to minimize TD error. Your Bellman updater is constantly pulling things from here and trying to satisfy Q(s,a). From here you have your trainer, which is trying to fit the Q to the ^Qtarget. You can think about this as a sort of planner. You visit old states you’ve been to, and you take your current model and rethink. What could I have done better if I visited this? This is how off-policy robotic learning systems are usually trained. These days there’s a simpler recipe, but in the Google QT-Opt days, we did things like this.

Dwarkesh Patel

So what is the trainer?

Eric Jang

The trainer is where you try to minimize Q(s,a) and Q^target.

Dwarkesh Patel

Can you explain the whole setup again at a high level?

Eric Jang

You have your off-policy data that came from various policies. You’re constantly pushing transitions you saw before to a replay buffer.

Then you’ve got this thing called a Bellman updater, which basically replans: instead of this action, what action should I have taken at s to have a better value? The way you enforce that is by trying to minimize the TD error. Given this, you have s’. You compute Q(s’,a) and you find the action that should go with s’ that makes this Q value as high as possible.

Then you add that to the reward here, and that gives you your actual target. For this current s and a, your Q^target is this. Now you send back the Q^target to this transition. With this tuple, you pair a Q^target. Then on the trainer side, you simply use supervised learning and minimize your current network’s Q(s, a) with its target.

Dwarkesh Patel

Got it. So in the background you’re basically thinking through how valuable all these actions actually were.

Eric Jang

Yeah. In a more optimal policy where you’re trying to maximize this, what is the Q^target of this transition?

Dwarkesh Patel

It’s sort of like daydreaming.

Eric Jang

Exactly. You can think about it like you’re going back in hindsight. Given what I’ve seen in the historical buffer, was there a better action I could have taken?

The connection to Go here—I tried and it was moderately successful, but too complex to open source—was replacing this with an MCTS relabeler. Instead of doing this target network computation, you run MCTS on your transition. In this case, you have your state, your action, and whether you won or not at the game. Actually, you can just toss those two. You don’t care about these ones. You just take your state and plan MCTS to get your best policy, Π, on your current network. Not the network that took this action, but your current best policy network. You rerun your search offline on these transitions.

If these are transitions your policy can get to, this actually acts as a very nice stabilizing effect. Another benefit is that you can fully saturate your GPU better because you’re not blocking on the Go game to give you board states. You simply search across all board states at any depth in parallel. Here the trainer would just predict the MCTS label as best as possible.

This kind of works, and it is quite relevant in robotics where you have a lot of offline data and you can’t simulate things like MCTS. But in practice, it does run into the problem that if the current model is looking at states it would never reach, it’s wasting capacity. You have to be a little bit careful here.

Much of RL has converged to a much more on-policy setup where they don’t really try to directly train on off-policy data. At best, they use off-policy data as a way to reduce variance, but not to directly influence the objective.

Dwarkesh Patel

Why have they converged to that?

Eric Jang

It’s just more stable. You might use the off-policy Q as a way to do advantage computation, like Q minus the sum of Q. If there are N actions, this is your value, and these are your current Q values. Your advantage for that action is the average value minus your current one. People can try to estimate Q in an off-policy way and then just use advantage here.

If there’s a problem in these dynamics, it doesn’t blow up your loss as much. In robotics, there’s a convergence toward using off-policy data to shape your rewards but not actually be directly here.

02:12:02 – RL is even more information inefficient than you thought

Dwarkesh Patel

I’m reminded of our earlier conversation about why MCTS is so favorable compared to the REINFORCE or policy gradient thing LLMs do. This might be totally wrong, but I wrote a blog post a few months ago about how policy gradient RL is even more inefficient than you might think.

The inefficiency one thinks about naively is the fact that you have to roll out a whole trajectory to get any learning signal at all. As these trajectories become longer and longer—as an agent has to do two days’ worth of work to figure out if it did a project correctly, rather than just completing the next word in a sentence—the amount of information per FLOP decreases. You have to unroll two days’ worth of thinking to see if you implemented a feature correctly, so the amount of samples per FLOP has been decreasing.

You’re trying to maximize as you’re learning, bits per FLOP. You can think of bits per FLOP as samples per FLOP times bits per sample. What I mentioned a second ago is that the samples per FLOP go down as RL becomes more long-horizon.

But this kind of naive RL is also terrible from a bits-per-sample perspective, at least compared to supervised learning. Early on in training, let’s say you have a vocabulary size for an LLM that is 100K long, so there are 100K possible tokens that one could answer. You have a totally untrained model and a prompt like, “The sky is…” With supervised learning, the model would have some probability distribution over all the things it could say. There’s a label that says the term here is “blue,” and it would learn through cross-entropy loss exactly how far its distribution is from correctly saying “blue.”

If you were doing this through RL, the model would try, “The sky is halycon.” Nope, that’s wrong. “The sky is told.” Nope, that’s wrong. This is a totally untrained model. You would have to do this on the order of 100K times just to stumble on “blue” and get some learning signal. In the supervised learning regime, you have your distribution of probabilities, you get told that it’s “blue,” and you figure out how far off you were. The amount you learn is a function of your pass rate. The further away you are from “blue,” the more you learn to go toward “blue” using cross-entropy loss.

You can think of it as like your pass rate, your prior probability of having said “blue.” As a function of that, in supervised learning, through cross-entropy loss, you would learn negative log(p), p being pass rate, bits, once you get this label. Whereas in RL, if you’re just randomly guessing and seeing if it works, that’s basically just the entropy of a binary random variable.

Eric Jang

What’s also tough here is that the distribution you’re sampling under is your policy’s distribution. If your policy has no chance of sampling “blue,” then you will never get a signal.

Dwarkesh Patel

Exactly. That’s modeled by the fact that your probability of sampling “blue” is extremely low. If you sample it, you do learn as much as you would have in supervised learning. In all other cases—99.99% of the time in an untrained model—you’re learning incredibly little from seeing that “halycon” or “told” is not the correct word.

That’s what happens most of the time. You learn very little. If you put your pass rate on the X-axis and the bits you’re learning from a sample on the Y-axis, with 0%, 50%, and 100%, so at the end of training you’re here. If you have supervised learning, the negative log pass rate would look something like this. The entropy of a binary random variable would look like this, depending on whether you’re doing nats or bits. If you do bits it’s one here at the peak. This is like a coin flip. You learn the most from a coin flip. This is supervised learning. This is RL.

However, the problem is you spend most of training in this regime, in the low pass rate regime. How fast you’re learning is a function of how many bits per sample you’re getting, and you’re getting very little signal here. If you chart the pass rate on a log scale—where at the beginning of training with a vocab size of 100K the pass rate is 1/100,000, then 1/10,000, 1/1,000, 1/100—what this graph looks like here, supervised learning would look like this, and then RL would look like that.

Eric Jang

And arguably you spend all your time here, potentially never even getting a single success. It’s a depressing plot in the sense that once you’re here, it’s not at all obvious how you get to there. Once you’re here you have something, but you actually spend all the time here in many RL problems. There’s a question of how you initialize so you’re at least at a non-zero pass rate.

One more thing I’d like to add about bits per sample that’s very relevant to any machine learning problem is that there’s a connection to soft targets and distillation. If you have access to the logits, not just the one-hot token answer, if you have access to the soft targets the entropy of this distribution is far higher than the one-hot. There’s way more information in bits per sample in a soft label. That’s why distillation is so effective per sample. It’s giving you way more information per sample.

Dwarkesh Patel

I wonder what the equation would be, but obviously it’s…

Eric Jang

It would just be the entropy of this distribution. The entropy of this is zero. The entropy of this is the entropy equation.

This is also why AlphaGo is quite beautiful. In AlphaGo, you don’t train the policy network to imitate the MCTS action. You train it to imitate the MCTS distribution. Both of these are valid, and if you wanted to do a scientific experiment of how important this soft label is, dark knowledge distillation, you can run an experiment where you retrain the policy network on the action MCTS selected rather than the soft target.

Dwarkesh Patel

Earlier I was stumbling around. Intuitively, why is this ability to do iterative search—where you don’t necessarily need to be able to win the game in the beginning, you just need to be able to improve your current policy—so powerful a capability in learning compared to how LLMs currently learn RL? It’s exactly this thing of considering your pass rate of the entire trajectory. I just don’t know a formal way to think about this. Maybe you should help me out here.

Eric Jang

Why is AlphaGo an elegant RL algorithm? The major reason is that you never have to initialize at a zero percent success rate and solve the exploration problem of how to get to a non-zero success rate. This is what allows you to hill-climb this beautiful supervised learning signal.

If you look at the actual implementation of AlphaGo, every step of the way, there’s actually no TD error learning or dynamic programming, at least explicitly. It’s just supervised learning on a value classification as well as a policy KL minimization. It’s just a supervised learning problem on improved labels.

The training is very stable. You can train as big of a network as you want. You can retrain this on the data set. Everything will just go stably. The infrastructure is very simple to implement as well. You don’t need a complex distributed system to keep everything on policy.

At the end of the day, you’re just saying, “I have some improved labels. Let’s retrain my supervised model on these targets.” You’re always in this beautiful regime where you’re just trying to improve the policy, rather than escape local minima where every signal is flat all around you.

One way to draw the curve is if you draw the win rate of an MCTS policy versus the raw network… Let’s say this dotted line is the raw network. The MCTS policy looks like this. Every step of the way, this supervision signal is very clean. You’re never in a situation where the MCTS is giving you no signal, unless your MCTS distribution converges to exactly what your policy network predicts.

Dwarkesh Patel

That’s a great way to explain it. Maybe we sit down and I ask some questions about automated research.

02:22:16 – Automated AI researchers

Dwarkesh Patel

One thing I really wanted to talk to you about is that you did a bunch of the research for this project through this automated LLM coding assistant loop. There’s an idea that if you fully automated AI research, you could have some sort of singularity.

Obviously we’re not there yet, but to the extent that we have early indications of what this process might look like, I’m curious about your observations about what the AI is good at, what it’s not good at, what you think about this scenario’s likelihood eventually, and what thoughts you have about this in general.

Eric Jang

I think automated scientific research is one of the most exciting skills that the frontier labs are developing right now. It’s important for everyone who’s doing any kind of research to get a good intuition of what it can do now and what it can’t, and how the science process might work in the future once we have AIs automating a lot of this investigation.

In brief, I mostly used Opus 4.6 and 4.7 while working on this. What works is that the models can do a very good job of hyperparameter optimization. In the past, people would come up with a search space of hyperparameters like learning rate, weight decay, and maybe how many layers are in your network. They would do a grid search or a Bayesian hyperparameter optimization approach, and it would find some tuned parameters.

The really cool thing that automated coding can do now is search a much more open-ended set of problems. It can say, “I’ve identified that the gradients are small in this layer, so let me change it up here. Let me rewrite the code so the data loader has a new augmentation I came up with. Let’s try to find the best way to fit the constraints of the optimization problem.” You end up with this much more flexible, high-level, almost grad-student-like ability to just grind a performance metric.

This can squeeze out quite a lot of performance. On a fixed data set with a fixed time budget, you can improve perplexity by quite a lot on a classification problem like LLMs or Go. It is also fantastic now at basically executing any experiment. I have a Claude Skill that I wrote called Experiment where I give it a description of what I want it to plot. I just describe, “Here’s the x-axis I want, here’s the y-axis. Answer this question for me.” It’ll run off and do all the experiments, compile the plot, make a report, and suggest what might have caused it and so forth. That’s what works quite well today, and I think we can expect these abilities to get better in the future.

But it’s also useful to know what it’s not doing so well today. In the blog version of this tutorial, I have a plot of all the experiments I did grouped in a tree, where every node represents a failed, successful, or mixed experimental result. From there, it branches off into a child representing the follow-on experiment. Occasionally, I’ll rabbit-hole down a track like off-policy MCTS relabeling, do a few experiments, and then realize it’s probably not worth it. So then I’ll jump to a completely different track. I call these things rows.

What I find is that the current closed models the public can access today don’t seem to be that great at selecting what the next experiment should be in a given track. They don’t seem to be able to step back and do the lateral thinking of, “Wait a minute, this track doesn’t really make sense. Let’s go back to first principles and think about what the bottleneck might be, or what we are trying to achieve.” Often I had to catch infra bugs myself by prompting the right question to Claude to investigate what’s causing the discrepancy, and then it’ll answer the question.

With Mythos-class models or Mythos++ models coming online, maybe this just completely changes and these problems fall to improved scaling. But at the same time, I think there’s a rich opportunity to develop RL environments that might incentivize this kind of lateral thinking. One of the motivations for setting up this Go environment was that Go captures a lot of very interesting research problems, often overlapping with LLMs or robotics. Yet it’s very quick to verify. The outer loop is ultimately: does the agent do what I think it does? You can check the outcome of a Go game quite easily.

The inner loop involves all this research engineering around distributed systems, predicting whether an idea is going to work or not, and predicting the difference a particular modification to your training algorithm might make. I think there’s a rich library of subtasks and sub-environments that you can train an automated scientist to work on, with Go as a sort of outer verification loop. Once you acquire these skills, maybe you can apply them to other domains like biosciences or robotics.

Dwarkesh Patel

Or automating AI research.

Eric Jang

Or automating AI research.

Dwarkesh Patel

Which is the real crux, the scary/incredible thing of making AIs make future versions of AIs. You’re suggesting the outer loop here could just be your win rate against KataGo?

Eric Jang

That’s one of them. I think there are a lot of deeper questions that one could tackle. For example, let’s say you have an idea on how to improve a scaling-law compute multiplier. The outcome isn’t necessarily “achieve the best Go bot ever”. The outcome might just be, “Can I predict what the win rate of my Go bot will be?” Or, “Can I predict the scaling-law plots that emerge from my idea?” But then you can verify that you haven’t reward-hacked anything by using a very verifiable game like Go on the outer loop.

Dwarkesh Patel

I think there are a couple of interesting follow-on questions. There are questions on the inner loop and the outer loop. On the inner loop, there’s a question of how locally verifiable any modification you might make is. That is to say, would you know whether some idea you try out is actually an improvement or a degradation? Would you know if something isn’t working as a result of a bug, or is it the result of the idea itself being wrong?

Ilya was talking about how one of the things he thinks makes him a good researcher is that he has a strong belief in what the correct idea is. He’s able to persevere through bugs and know which things are bugs versus mistakes in the fundamental idea, based on his high-level belief that “this idea should work, so therefore there has to be a bug”, versus the other way around. Why don’t we start with that question? How locally verifiable are things which are good ideas?

Eric Jang

As in the case of the success story for deep learning, you can think about this as a decades-long idea that took a lot of faith to get it to work. This presents a very challenging long-horizon RL problem where every step of the way you have a committee telling you that this is a bad idea, and then ultimately you break through. How do you design RL environments that maybe give you some feedback earlier? I think this is a very tough open question that I don’t have an answer to.

Ultimately, to play a very strong Go bot, you probably did need to discover deep learning. Having a challenging game that cannot be cheated easily on the outer loop could be used as an outer-loop signal for something like discovering the principles of deep learning. Now, of course, to make it tractable—and this is where research taste really matters—you have to come up with ways to initialize your problem so that you don’t try to solve a very intractable problem. Maybe you can leverage LLMs as a universal grammar in the middle to give you some sort of local feedback.

The fact that LLMs are a universal grammar means that they can move at almost any level of the stack. They can think very locally, as well as step back and think in very broad steps. I think that’s where a lot of the lateral thinking ability of humans comes from: knowing when the track you’re pursuing or the objective you’re pursuing is not right, and you should be asking a different question.

Dwarkesh Patel

The other question is how stackable local improvements are in the attempt to get to a better result on the outer loop. I’ve heard rumors that at some AI labs, the thing that has gone wrong is that people will individually pursue good ideas, but those don’t end up stacking well, and so the training run fails because of some weird interaction between two seemingly good ideas. Having a single top-down vision of how things should work is very important. Having worked at different AI labs and also played around with parallel agents trying different ideas, what’s your sense of how parallelizable AI innovation is?

Eric Jang

Great question. I think the research taste for executing well on the Bitter Lesson is that you need to know how much the Bitter Lesson can buy you and how much is too much to ask for, at any given moment. Of course, in the fullness of time compute is the single most important determinant of how things work. It’s almost inevitable that as you scale up energy and compute and parameters, intelligence will just fall out of that. That’s super beautiful, super profound. No algorithmic detail really matters beyond that.

But in the present day, we don’t have infinite compute and parameters and arbitrarily good initialization, so we have to come up with heuristics that give us that. These heuristics are probably somewhat redundant. That’s probably why you see this effect where a lot of these compute multipliers don’t necessarily stack. They might have some correlated benefit. And then three years down the line, when the Nvidia GPUs have gotten even stronger, maybe they stack even less well.

Maybe at any given point in time, the benefit of any given compute multiplier is transitory, which is what I suspected with the KataGo paper. There were many algorithmic ideas applied, and then you can see that with modern Blackwell GPUs and Ada-class GPUs—which are much better than the V100-grade GPUs that that paper used—some of these algorithmic tricks to speed up convergence just don’t matter so much compared to something else. I think that’s a matter of taste in the present time.

Dwarkesh Patel

How about the outer loop? How verifiable for making AI smarter? With Go, you do have this outer loop of win rate against the best open-source model out there. Even there, as you were saying, there are other outer loops of whether you discovered a new phenomenon, which is actually very hard to…. If you didn’t know scaling laws were important… When were Chinchilla or Kaplan scaling laws released?

Eric Jang

2018.

Dwarkesh Patel

So if you’re back in 2015, there’s not an automated procedure one can easily imagine for knowing which paper is the scaling laws paper versus which is just another random plot. Even in the Go case, it’s a hard-to-verify outer loop, and the whole idea of an outer loop is to have some backstop on improvement, let alone for general AGI, where of course we have a bunch of these benchmarks.

But there’s a problem. We know the things we can measure, and we improve on the things we can measure. We care about this broader ability to do economically useful work, which is not super easy to measure, at least until you automate everything. There’s a question of how good the outer verification loop is for AI self-improvement, and does that matter?

Eric Jang

I’m going to give a non-rigorous argument, but one that I intuitively believe. DeepMind started with a focus on games. They used games as their outer loop, and then their researchers learned from the experience of solving games, and now they’re working on LLMs.

Presumably, there was some positive transfer from their time working on games and Atari and Go and StarCraft that now helps them make good LLMs. I assume that there’s positive transfer in some regard, whether it’s coding or general research ability or project management. All these things probably help them do well.

If that’s the case, why wouldn’t it also be true for automated AI researchers? They should be able to positively transfer experience tackling quick-to-verify, quick-to-iterate environments to something more ambitious and economically useful, like automating drug discovery.

Dwarkesh Patel

I don’t know. Isn’t the issue… Historically, until Gemini 3 or whatever a couple years ago, people were saying, “Look, Google isn’t catching up in LLMs because they’re too tied to the old approach.” There are gains, but there are also ways in which it actively hinders you. It’s not obvious to me.

Eric Jang

The jury’s still out. Let’s say currently Google’s doing quite well. Who knows if the initialization of training on games is ultimately going to hobble their ability to be the winner in the long term. It’s hard to say for sure.

Likewise, who knows if the seeming late start was really just them pre-training for longer on how to scale up TPUs. They invested all their tech tree in getting TPUs to be good, which seemed not that useful in the short term but in the long term it becomes… So it’s even hard for humans to reason about what the optimal research strategy should be, even with the data we have today.

Dwarkesh Patel

Okay, we should let people know how they can find out more about this project, whether to fork it themselves, or check out your blog post where you do an excellent job explaining many of these ideas. Where do people go next?

Eric Jang

My website is evjang.com. There’s a blog post that links to an interactive version of this tutorial. On my GitHub, which is the username ericjang, there’s an AutoGo repo that people can fork and reproduce the training results.

Dwarkesh Patel

I also highly recommend people check out this blog post, “As Rocks May Think.” We touched on some of the ideas in this conversation, but it’s this grander thesis of what happens when you have thinking as a primitive in—

Eric Jang

Computer science.

Dwarkesh Patel

Exactly. I highly recommend people check out that blog post as well.

Eric Jang

I encourage the audience to think about the relationship between thinking and Go via MCTS and search, and how it relates to LLMs. I think there’s something quite profound there, and probably underexplored just because Go has been relatively underexplored compared to the boom in LLMs.

It’s not to say that I think we should have trees in our LLMs, but there is some very interesting duality between them. You can actually do a lot of research on Go, MCTS, and reasoning with very small budgets. So that’s very exciting.

Dwarkesh Patel

Awesome, Eric. Thanks for doing this.

Eric Jang

Thank you. It’s an honor to be on the podcast.

He and collaborator Ali Akbari just published a paper that overturns a long-standing consensus about human evolution — that natural selection has been dormant in our species since the agricultural revolution.

By scaling ancient DNA sequencing and developing a new statistical method, they found that selection has actually sped up.

Selection went especially bonkers during the Bronze Age (around 3,000 years ago).

That’s when gene frequencies for everything from immune function to body fat to intelligence were most in flux.

Over the last 10,000 years, selection pushed the genetic predictor of cognitive performance up by roughly a full standard deviation — most of it between 4,000 and 2,000 years ago.

After we finished recording, David sketched out on a whiteboard his new heretical model about who the Neanderthals really were. Luckily, I took out my iPhone and managed to record it.

He thinks the standard story (that Neanderthals are some separate archaic lineage we interbred with a little) just doesn’t fit the evidence. Instead, he proposes that Neanderthals are essentially genetically-swamped modern humans.

A small population somewhere around the Caucasus invented Middle Stone Age technology roughly 300,000 years ago and expanded outward. The ones that moved into Europe interbred with local archaic humans, got genetically swamped, and became Neanderthals. The same expansion went into Africa, met much more diverged archaic Africans, and that mixture became us.

This means Neanderthals and modern humans share the same cultural ancestry — the only difference is which archaic humans they mixed with afterward.

David is a brilliant and rigorous scholar. It was a real delight to learn from him again.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Cursor was super useful as I prepped for this episode. Whenever I had a question, I’d have Cursor kick off a few different models simultaneously and then compare their responses. I found that this led to better results than I could get out of any individual LLM. If you’ve only used Cursor for coding, you should try using it for research. Check it out at cursor.com/dwarkesh

• Jane Street uses an internal currency called “hive bucks” to allocate compute through a real-time auction – and anyone can change anyone else’s bids or even kill their jobs! Everyone just trusts each other to act in the firm’s best interest, which is what lets the system work in the first place. If this weird and high-trust culture sounds like your kind of thing, Jane Street’s hiring at janestreet.com/dwarkesh

• Crusoe’s ML infra team built fastokens, an open-source tokenizer that delivers a ~9x speedup over Hugging Face and up to 40% faster time-to-first token – on real production workloads! Crusoe achieved these results by parallelizing things and using some clever engineering to handle duplicates without cross-thread coordination. Learn more at crusoe.ai/dwarkesh

Timestamps

(00:00:00) – Ancient DNA suggests strong selection over last 10,000 years

(00:15:45) – Natural selection intensified during the Bronze Age

(00:35:02) – Why didn’t evolution max out intelligence?

(00:57:21) – Evolution is limited by time, not population size

(01:09:02) – Why no farming before the Ice Age?

(01:17:13) – The Neanderthal puzzle David can’t stop thinking about

(01:54:10) – The methodology behind this breakthrough

Transcript

00:00:00 – Ancient DNA suggests strong selection over last 10,000 years

Dwarkesh Patel

I am back with David Reich, who is a professor of ancient DNA at Harvard. How do you describe what it is that you study?

David Reich

I’m a geneticist, and I work on human history and how ancient people relate to each other and people living today.

Dwarkesh Patel

We did an interview two years ago, which ended up being one of the most popular interviews I’ve ever done. I think people found it really compelling that there’s so much about human history we don’t know and are just learning about now as a result of the kinds of techniques your lab is using.

You have a new preprint that’s very exciting, and I wanted to talk to you about it. Can you give me a little bit of context on what we’re talking about today?

David Reich

The dream was that when this ancient DNA field started, more than 16 or 17 years ago, we were going to learn a lot about biology —about how people’s biology changed over time— by getting DNA out of ancient human remains and tracking changes over time. And that dream has really not been realized since the beginning of this field.

The field has been a big success with regard to learning about human history. It’s resulted in surprising findings about human migrations —people not being descended from the people who lived in the same place hundreds or thousands or tens of thousands of years before— and mixture being common in human history, and sex-biased processes being common. And there have been things that were not expected from archaeology. The field’s been a big success from that perspective, but what’s not been successful is learning about biology and biological change.

One big reason has been that the sample sizes have been too small. When you have a single person’s DNA, it provides a tremendous amount of information about history. That’s because when you look at one person’s DNA, it’s not a single person. It’s many people. It’s your two parents, your four grandparents, your eight great-grandparents, 16 great-great-grandparents, and so on. Going back in time, thousands, tens of thousands, even hundreds of thousands of ancestors are contributing to people today.

When you look at the DNA of a single person’s genome or a Neanderthal genome, you have effectively tens of thousands of ancestors all represented in your data. And you can position that individual exquisitely with respect to other people from whom you have data. But when you are interested in how a particular genetic variant—that affects something like your skin pigmentation, or your ability to digest cow’s milk into adulthood, or a behavioral trait—changes over time, a single person gives you only one sample, or maybe two samples: the one in their mother and the one in their father.

To get a high-resolution picture of how the frequency changes over time, you need very big sample sizes, truly very large numbers of people. We just didn’t have that until the last few years. What motivates the study we’re talking about today, and the work that hopefully a number of groups will be doing in the coming years, is the fact that we now finally have those numbers. We can do something with the data to see how frequency changes over time.

Dwarkesh Patel

Can I ask a question? I’ll be asking a lot of naive questions through the next few hours, but why are frequency changes especially interesting?

David Reich

What we’re interested in is using the experiment of nature that’s occurred in our history, over the last tens of thousands of years, to understand what’s biologically significant in our DNA. If there has been a change in environment that a population has experienced—for example, people shifted to agriculture, began living close to domesticated animals, or moved from a cold place to a warm place, or a low place to a high place—then there’s pressure on the population to adapt to these new stresses and new needs.

The way you’re going to detect that is by seeing that the frequency of a genetic variant—that for example might allow you to live at higher altitude, or that might nudge you to have a different behavioral pattern advantageous in the new situation—pushes systematically in some direction in a way that is enough for you to detect. It’s very hard to detect slight shifts in frequency by a few percent or ten percent unless you have a very big sample size. What we’re looking for are those changes in frequency that are too extreme to be due to chance. That will tell us there have been pushes against the biology as a result of the changes in environment that people have experienced.

Dwarkesh Patel

Interesting. What did you guys find?

David Reich

Seven years ago, Ali Akbari, who at the time was a postdoctoral scientist in my laboratory and a few years later became a permanent staff scientist, set out to use the data we were producing to learn about biological change over time. I think the reason he was interested in our laboratory rather than other places was that a focus of our lab has been generating truly large amounts of data from ancient humans. We’ve been trying to industrialize the process, make it very inexpensive, make it high quality, and generate large numbers of samples with lots of good data for this purpose.

There’s been this large amount of data we’ve generated. And it made it possible to conceive again of asking whether there have been frequency changes over time. The mainstream view in human evolution in the last several decades has been that natural selection has been pretty quiescent over the last several hundred thousand years of human history.

There are several lines of evidence that have been deployed to document this. One is that if you compare diverse populations from different continents around the world, for example Europeans and East Asians, and you look at mutations that differ in frequency between these groups—all mutations differ a little bit in frequency, sometimes a lot—you can say, “What are the most different mutations in terms of frequency between Europeans and East Asians?” And there are almost no genetic changes that are 100% different in frequency between Europeans and East Asians.

Europeans and East Asians descend from a common ancestral population 40,000 or 50,000 years ago that came out of Africa and the Middle East. This population had a set of gene frequencies, and these variants bopped around randomly—a process known as genetic drift—or perhaps under selection in one direction or another. The time that’s passed since 40,000 or 50,000 years ago is sufficiently small on an evolutionary timescale that there’s just not much genetic differentiation on average between these two groups.

However, if there’s been natural selection, for example to help people in one place digest alcohol better, or digest milk better, what you might expect is that there would be some mutation that would have rocketed up to very high frequency. Forty or fifty thousand years is a lot of time, it’s maybe 1,500-2,000 generations. That might easily be enough time to see a 100% difference in frequency. Yet you don’t see any more than what you would expect by chance.

This combination of things made it seem that selection has just been quiescent. Maybe a few hundred thousand years ago, the ancestral human population got to some kind of optimum, and after that there hasn’t been much genetic change in one way or the other.

There have been small amounts of natural selection, or selection to remove bad mutations that are constantly raining down on the genome, but not what we call directional selection. That would be newly arising mutations, or mutations being pushed in a systematic direction, to help the population get to a different adaptive set point more favorable for the conditions that population is living in.

We were able to partition how much of the changes in frequencies of all the mutations that we’re seeing in the DNA—we’re looking at about 10 million positions that vary—is due to directional selection (adaptation) versus other factors, especially genetic drift. And 98% of it is other factors, especially genetic drift. It’s overwhelmingly migrations and population structure causing fluctuations in frequency.

As a result, it’s super hard to detect the signals of adaptive natural selection because they’re a tiny fraction of the total frequency change. The vast majority of it are these migrations and mixtures. Nevertheless, there’s so much natural selection, as our study has shown, that it’s actually been rampant in the genome.

Dwarkesh Patel

Can I ask a clarifying question here? Why are we discounting population admixture or replacement as selection? If you think about it at a group level, if one population replaces another population, isn’t that selection?

I remember from the last episode you were explaining how there have been huge changes in what kinds of people are in a specific area. One population came in and replaced the previous one, and then a new population came in and replaced that one. To the extent that the genetics are relevant to why that population replaced the other one, why should that not count towards what we understand to be selection over the last 10,000 years?

David Reich

It could count, and may count, and probably should count in some respects. But it could also be that this population replacement is due to some cultural phenomenon —technology held by one of these groups and not others. And maybe there are some genetic mutations that are contributing to this. Who knows? It’s possible.

But what you’re seeing is a whole-genome shift. What we’re looking to see is whether there’s one place in the DNA that is driving the change in a way that’s different from the rest of the genome. From a statistical point of view, what happens at these times of migration is there are just huge fluctuations in frequencies. These are extremely uninformative times for detecting natural selection. The best moments to detect natural selection are when migrations and population admixtures are not happening for a few hundred years. During these times, you can actually see the mutation slowly blowing in one direction as a result.

The way we think about the history of Europe and the Middle East for the purpose of this study is as an archipelago of little populations in space and time, each pretty isolated from each other. You have a little population in Britain isolated for a few hundred years, or a little population in Hungary isolated for a few hundred years, between big events of migration and mixture. In each of those little experiments of nature, we can ask: does this mutation slightly increase in frequency? Does that same mutation slightly increase in frequency? If all the arrows point in the same direction, we win. They’re telling us that natural selection is occurring.

For example, 4,500 years ago in Europe, almost all mutations went through huge frequency changes. That’s not because of natural selection. It’s because of the steppe migration from north of the Black and Caspian Sea. 40-80% of the DNA becomes Yamnaya from steppe pastoralists. Their frequencies of mutations were different not because of selection necessarily, but just because they had evolved in different places for thousands and tens of thousands of years. When you look at the descendant populations, there are huge changes in frequency. What you need to do is see if natural selection is explaining a shift more than you would expect by chance.

Dwarkesh Patel

So you found these locations that seem to be under selection. I have another clarifying question. You say you found 3,800 locations which you’re 50% confident have been under selection in the last 10,000 years.

David Reich

It’s 7,200 where we’re 50% confident. We’re getting about 7,200 positions in the DNA that have 50% confidence of being real. Only half of those are real—we don’t know which ones—so 3,600 of them are real.

Dwarkesh Patel

Does that also mean that outside of those 7,200, you’re confident the other locations in the genome are not under selection?

David Reich

No. If you look at the 25% probability cutoff, there will be tens of thousands, and there will be many real ones there too. In fact, multiple analyses we do suggest that the genome is vibrating with natural selection.

There are all sorts of weaker effects that would be picked up in even larger studies than we’ve done. In fact, almost every position in the DNA is correlated to another position that is being dragged in one way or the other by natural selection. Instead of being quiescent, natural selection is everywhere. Even though it’s only 2% of the frequency change, it’s tugging the positions in one direction or the other everywhere.

So we analyzed these positions that we had identified, the hundreds of positions we were super confident about. We looked to see whether they were randomly distributed in the DNA or whether they had patterns. We looked at maybe 100 or so traits where there had been genome-wide association studies for all sorts of different traits, associated with immunity or autoimmunity or behavior or metabolism, and other things.

For each of these we could ask: do the genetic variations that are known to affect these traits from genome-wide association studies have an unusual number of genetic selection signals? What we found is that there was a vast enrichment, by about four or five-fold, for immune traits. There was a super concentration of selected signals in immune traits. We also saw a strong enrichment for metabolic traits—things that might impact obesity or fat traits or Type 2 diabetes—and almost no detectable enrichment, as far as we could tell, for behavioral or psychiatric traits.

Dwarkesh Patel

Just to make sure I understand. This is not to say that behavioral or psychiatric or cognitive traits are not under selection. It’s just that the individual sites where such traits are controlled are not especially likely to be among the locations you’ve identified as under selection.

David Reich

That’s exactly right. It might seem from the results of that analysis that immune traits are highly selected and that there’s been no selection for behavior in the last 18,000 years in this part of the world. But that’s a wrong conclusion, and we have evidence that it’s a wrong conclusion. There’s clear evidence of selection also on behavioral traits.

The reason we think we see much weaker signals for behavioral traits is that behavioral traits, we know from medical studies, are underpinned by much larger numbers of genes than immune traits, which are underpinned by relatively small numbers of genes of strong effect. Behavioral traits are shaped genetically by a very large number of genes of weak effect, and we just don’t have the statistical power to detect these very weak signals.

When we do an analysis looking at our very strong signals of selection, that collection of very strong results is very effectively querying the immune traits, but is not very effectively querying the behavioral traits. It may still be the case, and I guess it is, that immune traits are the most selected category. But it is not at all the case—and we can prove it’s not the case—that behavioral traits are not selected.

Dwarkesh Patel

Interesting.

00:15:45 – Natural selection intensified during the Bronze Age

David Reich

We’ve been able to prove that there are two ways to reconcile the previous observations with our new observations. Remember, the previous observation is that natural selection seems to have been quiescent over a timescale of hundreds of thousands or many tens of thousands of years. Reason? That you don’t see 100% difference in frequency variance across Europeans and East Asians.

Now we’re seeing hundreds of positions that are rocketing up in frequency with selection rates of 1% or more in a lot of cases. A 1% or more selection rate will mean a rapid doubling over periods of dozens of generations. Over the 1,500 or 2,000 generations separating Europeans and East Asians, shouldn’t you see many genetic variants that are 100% different in frequency across populations?

We were able to show that this is explained by at least two factors. One is that in this part of the world—Europe and the Middle East—we are actually in a period of accelerated natural selection. One way to see this is to look at the enrichment pattern we’re observing, where immune traits are unusually associated with these selection signals. We could compare the last 5,000 years of our time period, what’s called the Bronze Age and further onward, to the previous 5,000 years. What we see is that this intensification of selection around immune traits, and similarly the intensification around metabolic traits, has accelerated over this time period.

It’s not like natural selection has been at the same rate over all places and times. It’s increasing over the time period we’re analyzing. Plausibly the whole time period has increased compared to previous periods. We’re in a period of intensified selection. That’s not implausible, because this is a population that went through a huge shock in terms of the way people live and the culture. Almost everyone we’re analyzing are farmers or food producers in one way or another. Farming was invented for the first time anywhere in the world in the Middle East 11,000 or 12,000 years ago. The people who invented farming exploded into Europe after 8,500 years ago, spread across the continent, and expanded rapidly.

In the Bronze Age, there was an intensification of how people lived, with much higher population densities. People were living more and more next to their animals and getting their diseases, and exchanging their diseases with the animals and with each other. This is a period of rapid change in how people are living, resulting in different biological needs of this population. It’s not surprising, perhaps, that in the context of these dramatic changes, the biology of the population might not be ideally adapted.

There might be what some people call an evolutionary mismatch, where you take a genetic variation that evolved in hunter-gatherers and put it into farmers or pastoralists, and it’s not exactly right. What you’re seeing is the DNA of this population, which descended from hunter-gatherers only 10,000 years ago, reacting to the shock of having been moved into an agricultural, Bronze Age, high-population-density, urban environment. A hypothesis is that what we’re seeing is the adaptation that occurs as a result.

Dwarkesh Patel

In the paper you have many examples of this intensification of selection around the Bronze Age. It might be helpful to go through some of these.

David Reich

One of the things we do in this work is look carefully at many of these positions in the DNA. We actually have an internet browser called the AGES browser, which Ali and a colleague of his—who’s a co-author of our paper—built. It allows you to query each of these 10 million positions and see the trajectories at each position and the evidence for selection.

One of the things we see is that, while for the most part the signals of natural selection we detect are consistent with constant natural selection over time, in a handful of them we’re able to see that there’s been a reversal or a radical change in natural selection. Very often that occurs in the period between 5,000 to 2,000 years ago, which is the Bronze Age and the Iron Age, a period of rapid population growth and rapid movement to intensive use of many technologies that were not used that way before.

An example of this is the TYK2 genetic variant that is a major risk factor for severe tuberculosis, which is the most important infectious disease killer in the world today. If you look at this major risk factor for tuberculosis, this variant rockets up in frequency from 8,000 or 6,000 years ago to maybe 9% or 10% in this part of the world. Then it rockets down in frequency in the last 3,000 years. In both cases, there’s very clear evidence of natural selection, in the first case to increase in frequency, and in the next case to decrease in frequency.

A possible reason is the spread of tuberculosis. It maybe becomes endemic in the population 2,000 or 3,000 years ago. That’s potentially consistent with pathogen sequence data and other lines of evidence. And maybe this variant was protecting against something before then, but then tuberculosis became significant after that point, and it was so bad that it pushed in the opposite direction. That’s speculative.

Dwarkesh Patel

The thing it was protecting against was probably another disease?

David Reich

Maybe.

Dwarkesh Patel

One of the big takeaways for me from the paper was just that something weird happened in the Bronze Age. As you said, across trait after trait, the selection intensifies during the Bronze Age.

This makes sense for some things. For example, why do we see lactase persistence, where adults can process milk, intensified during this period? This is the time when we start using cattle not just for the meat, but also for milk and wool and other secondary products. So it makes sense why lactase persistence would matter more.

But then there are other things that seem like they should have been relevant since the dawn of agriculture. I forget the exact name of the allele, but was it FADS1, which helps convert plant fatty acids into long-chain fatty acids that your body needs? That’s obviously relevant when you move from a diet of meat as a hunter-gatherer to a diet of cereals.

That is also one I think you found was under especially high selection 5,000 to 3,000 years ago. So what’s going on? Why is the Bronze Age so special across all these different traits that you’re observing?

David Reich

So this FADS1/2 variant is a vegetarian/meat-eating adaptation. Already in work prior to this, Ian Mathieson, who worked with me in 2015, identified this as a very strongly selected variant. It’s actually ancient. You see copies in archaic humans too.

One of the findings of our paper is the ABO blood system. You get your blood typed as A, B, and O. The B variant has increased up to 10% at the expense of A, but previous work has shown that A and B were both already present in the ancestor of humans and gibbons and other apes. Some of these mutations have been going back and forth and fluctuating over different time periods.

But we’re talking about changes in the Bronze Age. The TYK2 variant for tuberculosis risk, a multiple sclerosis risk variant, inflected and increased in frequency before the Bronze Age, and then 2,000 or 3,000 years ago reversed in that period. There are differences in Northern Europe where this process is super strong, very strong positive selection, very strong negative selection. And then in Southern Europe, only a little bit, and not even very strong negative selection.

For haemochromatosis, which is pathogenic iron buildup that causes problems in Europe, that too has reversed around this period. In some of the complex traits that maybe we’ll talk about later, these traits too have periods of intensification of natural selection. For example, depigmentation: Europeans have gotten lighter skin over the last 10,000 years. You can see it in our data. The period of strongest depigmentation is between about 4,000 to 2,000 years ago, and then after that it’s much less.

This seems to be a very impactful, eventful, important period where a lot of the processes we are seeing become very powerful. It’s surprising on first principles. You might think, before you walked into this genetic data, that the big change is going to be starting to grow plants and maybe farm animals. That happens in the Neolithic, beginning 10,000-12,000 years ago, and spreads into Europe after 8,500 years ago. But actually, the intensification happens 5,000 years ago, 4,000 years ago.

It’s really interesting. This observation of that being an inflection point tells us something about when humans, at least in this part of the world, were wrenched into a way of living that was so different from how their hunter-gatherer ancestors lived that the organism had to adapt very strongly. It may be that the degree of that wrenching process moving into the Bronze Age was qualitatively greater than the degree of the wrenching process that happened from the initial transition to growing plants. That’s surprising, because our cartoon picture is that the big transition is farming. But the biological readout is saying our genome is reacting much more strongly to these events that happened 5,000 years ago.

Dwarkesh Patel

You did some work with Bhatia and many other colleagues in 2014 where you were looking at 20,000 or 30,000 African American genomes today. You were saying, “Look, there’s 80% West African DNA and then 20% European DNA. Can we look at their genomes today and see that their allele frequencies are much different than we’d expect from this admixture?” Correct me if I’m wrong, but you found that they weren’t.

That is to say, over 200 or 300 years of extremely intense environmental change—going from chattel slavery to a completely new environment—there’s no effect of natural selection. So we see episodes like this where we don’t see natural selection, but then the Bronze Age apparently must have had an even stronger effect, where the change in environment is even stronger than what we see from Africans in Africa being migrated to the New World and living under slavery.

David Reich

That may be the case. It also may be the case that that period is just too short to see much effect. In the Bhatia et al. paper, where we looked at about 30,000 African Americans, we looked to see whether—instead of the average percentage of around 80% West African ancestry—there were some places in the DNA with significantly more than 80%, or significantly less than 80%. That’s what you would expect if there were natural selection for some genetic variant from Europeans or from Africans.

We didn’t see any place in the DNA that was significantly different from what you would expect by chance. One possible explanation is just that there’s only a handful of generations, maybe five, over which natural selection would operate. So if the selection was 2% a generation, you would still only see a 10% compounded effect, and there’s just not enough time to detect it.

But the Bronze Age is not 300 years, it’s 3,000 years. It’s the power of compound interest, and you have enough time to begin to see a strong effect. This really, really does seem to be a very impactful time in terms of human history, and you can see it in our complex traits. Look at pigmentation, for example, which is the strongest signal of selection for a complex trait in our data set. You look at genetic mutations that are known to affect pigmentation. You add up their effect across all of the DNA, there’s dozens or hundreds of them. You look to see when natural selection is strongest, and the time period is really 2,000 to 4,000 years ago.

For some of these other traits as well, you see again that the time period over which selection is strongest is 2,000 to 4,000 years ago. For example, if you look at genetic variants that affect measures of cognitive performance, such as performance on intelligence tests in white British people today. This is of course a very strange trait to measure in the past because there were no intelligence tests and there was no school. But it is a predictor today, and you can look at how it’s changed in the past.

We see very strong natural selection for this combination of genetic variants that predicts people’s performance on IQ tests and is also highly correlated to the predictor of the number of years of school or the household wealth of people. All crazy traits in the past because there was no wealth in the past, there was no school in the past. But if you look at the predictors today, there is a strong movement in a systematic direction, a large effect, about a standard deviation on the scale of modern variation.

We can do this trick of looking to see whether there are periods of time when this natural selection has occurred more intensely or less intensely. We drag a 2,000-year window through our data, and we repeat our whole analysis, not on 18,000 years, but just on a short 2,000-year window. We can measure the strength of selection in each of these 2,000-year windows. What you see when you look at intelligence is that this maxes out in the Bronze Age, between 5,000 and 2,000 years ago.

The impact in the last 2,000 years is almost nothing. There’s no evidence of natural selection at all. Your bias coming into this, my bias perhaps, might be that if there’s any signal of natural selection on this trait at all, that it would be unusually strong in the last 2,000 years. Maybe this is a time of industrialization. Maybe this is a time of greater need for this particular trait. But in fact, there’s no evidence of natural selection at all in the last 2,000 years. There’s very strong evidence between 2,000 and 4,000 years ago, where instead of a one standard deviation strength of selection, it’s a two standard deviation strength, averaged over this time period.

Dwarkesh Patel

The standard deviation here is how much the polygenic score for the trait itself moves?

David Reich

How much the polygenic score for the trait moves over a 10,000-year period within a population that is held constant in terms of its ancestry. What we’re actually doing is looking in our data set at a heterogeneous group of people. There’s Southern Europeans and Northern Europeans and hunter-gatherers and farmers. At different times in the past, those groups are more or less represented.

The whole strength of the methodology Ali Akbari developed is that it corrects for that changing ancestry over time. Really what’s being asked here is that we’ve divided up our whole data set into an archipelago of little populations in different places in space and time. We’re asking in each place in space and time: a little pocket of people in Britain from 4,000 years ago to 3,500 years ago, a little pocket of people in Hungary, a little pocket of people in Italy from 2,000 years ago to 1,500 years ago. In each of these places, where the ancestry is relatively similar without being too disrupted in that short period by migrations, we watch to see if the genetic changes blow in the same direction. We’re measuring the strength of selection at each point in time after correcting for the big population changes that have occurred.

Dwarkesh Patel

The effect here is huge then. One standard deviation above the median would be somebody in the 85th percentile. You’re saying the effect of selection has been so strong that comparing 10,000 years ago to now, the median has gone to the 85th percentile. That’s just a huge effect over the last 10,000 years on something like intelligence or the thing that predicts household income.

Especially given that this is only 2% of the change in allele frequencies, and the 98% is coming from migration… It’s stupendous to think about what the impact of migration is, if this alone is driving a standard deviation change in these kinds of qualities, at least among the kind of variation we see in the world today.

David Reich

One thing you can see in the data is that the migration impact is huge. For example, if you look at the trajectory for measures of cognitive performance—scores on intelligence tests in white British people today—but you look at the predictor of that in people in ancient times, the estimate for the hunter-gatherers of Europe is three standard deviations below the modern mean. So that’s hugely different.

Then you see a huge jump from them to the farmers, who are at the mean, at zero. That’s migration. What you’re seeing is that those two groups had different set points for those traits. And then the steppe pastoralists have a lower set value.

You see huge fluctuations in the predictor of this trait over time. That doesn’t prove selection. That’s just migration. But what our test is telling you is: in addition to those fluctuations due to migration, is there a consistent effect of natural selection blowing the trait in the same direction over all places and times? That’s what we’re detecting.

00:35:02 – Why didn’t evolution max out intelligence?

Dwarkesh Patel

There’s this theory called the collective intelligence hypothesis, which is the idea that selection for intelligence has actually been in the opposite direction. As society has developed, there’s been more specialization, and if there’s more specialization, each person only needs to understand a smaller and smaller part of the world. Therefore, the ancients were actually much smarter than us, and we’ve evolved down in intelligence.

Your results seem to point in the opposite direction. Although there hasn’t been selection in the last 2,000 years as society has gotten more complicated, at least when society began, there was more need for the kind of thing that predicts intelligence today.

The reason that’s surprising is, if you think about hunter-gatherers—reading your colleague Joseph Henrich’s book—the amount of information they needed to hold onto and assess, everything from how to process food, to how to build shelters, fire, et cetera, compared to my world, where I just need to know how to set up mics and ask questions… It seems like the demands on intelligence should have been way higher in the ancestral environment. So it’s very surprising that the beginnings of civilization increased the selection on intelligence.

David Reich

This is the power of data. I think if you asked Joe prior to this work what the hunter-gatherer selection would be and where their set point for this particular trait would have been… I think he probably wouldn’t have made a very strong prediction, but he would have said, “Maybe you would have expected it to have a high predicted value of this trait because these people were really having to do a lot of things and figure a lot of stuff out. Maybe once you have more complex societies, there would be more of a collective brain, and maybe there’d be selection against this trait.” In fact, it’s the opposite in some ways.

It’s the power of data. It’s not what you expect.  It’s actually the value of data to try to make sense of all these things. It’s very interesting. The genetic predictor of intelligence, there are lots of things that are confusing about it, so it’s worth talking about. Or the genetic predictor of years of schooling, which is highly correlated to it and is measured even better.

If you look at the genetic predictor of years of schooling, there’s another amazing study from 2017 from a group in Iceland that looked at this measure over the last hundred years in Iceland. It looked at older people and younger people born more recently. There’s an estimated 0.1 standard deviation decrease in the genetic predictor of intelligence in Iceland just within one century. It’s an absolutely huge effect over a short period. This is selection against years of schooling. If I said intelligence, I didn’t mean to. It’s selection against the genetic predictors of the number of years of school.

One possible interpretation of this—hand-wavy—is that what’s being measured here is not selection for years of schooling or for real intelligence, but for another trait altogether that’s correlated to both of them. For example, the predictor of the number of years of schooling is very strongly correlated to the age at which women have their first kid. If you control for that, all of the signal of years of schooling goes away. So maybe what you’re measuring is women’s decision about when to have children.

If you have children earlier, you don’t go to school as much. If you have children later, you go to school more. Maybe it’s some kind of measurement of delaying gratification or putting things off or planning. The same trait is correlated to body mass index, to obesity, and to walking pace. So is this really intelligence as we think about it, or is it something else that manifests itself differently at different times in the past?

Dwarkesh Patel

Obviously, a trait like years of schooling was not itself a meaningful thing in the past. The underlying things for it seem to have been under strong selection. Whatever in the genome predicts years of schooling seems to have been under strong selection. How should we think about this? What’s the actual thing that’s changing in the genome?

David Reich

There are two things going on that you need to think about. Years of schooling is connected to so many other things genetically. If you look at the genetic predictor of years of schooling—this trait has been measured in millions of people now—it’s correlated to really surprising things. It’s correlated to the age at which women have their first kid. It’s correlated to people’s obesity. It’s correlated to people’s walking pace. It’s correlated to people’s household wealth. It’s correlated to a variety of other traits that seem quite different from it.

If you think you’re actually measuring the genetic prediction of intelligence, or actual studiousness, you should think again because there are many things that it’s correlated to. There seems to be some kind of general trait that you could maybe think of as executive function or a propensity to defer gratification—I’m just waving my hands—that is under selection. It pushes all these traits in the same direction one way or the other, and at different times in the past, it’s advantageous or disadvantageous.

When we found this signal of the genetic propensity to go to school for more years as it manifests itself in white British people today, we were incredulous. How could this be? Maybe this is a problem. So we did a few tests to try to figure out whether this was real. One of the tests we did was that we looked for a study where this measurement of the number of years of school was done not in Europeans, but in Chinese people in China. We looked at the effect size of many variants as they affected the number of years of school in China, and we saw whether they had a correlation to the trajectory of those same genetic variants in Europeans over the last 10,000 years.

These are two parts of the world where the populations have been essentially completely disconnected. There’s no way by chance that the trajectory in Europeans over the last 10,000 years would have anything to do with the effect on years of schooling in China today. But there’s actually a huge statistical correlation, a five or six standard deviation correlation between the effect size of variants on the number of years of school in China today and the trajectory in Europe. It’s just as strong, actually, as the effect size of variants in Europeans on years of school to the trajectory in Europeans. We just could not see a way this could happen by chance. Once we saw that, we felt quite convinced that this was a real signal and that somehow there has been natural selection to increase the genetic changes that today manifest themselves as predicting more years of schooling.

Dwarkesh Patel

Just to make sure I understood, you’re looking at this ancient DNA in Europe. You’re saying it seems to predict years of schooling for modern people in Europe, or at least selection on that ancient DNA seems to predict more years of schooling in modern Europe. You also find that the same variants predict more years of schooling for Chinese people in China. So this is not just some weird artifact from the way these GWAS were done in Europe. These parts of the genome seem to robustly predict the kind of thing that actually leads to more years of schooling, at least in people today.

David Reich

Correct.

Dwarkesh Patel

Stepping back, I want to understand what this tells us about what actually changed in our environments over the last 18,000 years. We talked a little about what happened after the Bronze Age. We were talking about this during the collective intelligence part of the conversation. It’s surprising to me that things like intelligence, or lack of schizophrenia—things that just seem robustly good—were not maxed out before the Bronze Age.

The diversity among different populations was so big that you have the European hunter-gatherers having three standard deviations less predicted value for what they would score on an intelligence test if it existed. But they were existing in the real world in a place where intelligence matters.

How can it be that this was not a trait… You just look at the human body or any animal, and evolution has been acting on it so strongly to make it functional for the things it needs to do. And this one thing, which seems so relevant—especially to what human hunter-gatherers needed to do—doesn’t seem to have been under that strong selection in the Mesolithic or Paleolithic eras?

David Reich

I think that’s a great question. As we talked about before, selection is very effective. It can move the mean value of traits within hundreds or thousands of years in one direction or the other if that’s adaptive in a particular environment. So you might wonder, isn’t intelligence good in all contexts and places in time? There are a number of ways to think about that.

First of all, we are speaking from the point of view of a society which intensely values this particular trait, the ability to score well on IQ tests or things like them, or to go to school for a long time. I think it’s unprecedented in human history that we live in a time like this. If you look at the Hebrew and Christian Bible, and you look at how much intelligence is valued, it’s basically not at all.

Dwarkesh Patel

But when the Bible was being written, especially the Old Testament, that’s exactly when selection for intelligence is at the highest point it’s apparently ever been.

David Reich

Exactly. But there it’s about strength or courage or religiosity. Those are the values. If you read Homer or the texts of other religions, it’s not intelligence. It’s beauty and other things. This value system which has a hyper-focus on smarts is not obviously a trait value that’s been common in the past. You might think that in certain communities there might be valuation of things that are more proximate to years of schooling. But really broadly, it’s not been a high value in the population.

Dwarkesh Patel

Obviously, the thing we care about is not direct performance on an IQ test, especially in the past. The thing I’m trying to understand better is intelligence more broadly. Maybe IQ-test intelligence is just not that correlated with, “Here is a new-world environment, go figure out how to process food there and make shelter and everything else.”

Your colleagues like Joseph Henrich have talked about how modern people underestimate the difficulty of doing this kind of thing with a small band of people. Maybe that’s not IQ-test intelligence, and that’s why we don’t see that strong a selection effect on this thing. But intuitively, regardless of the value system, it just seems very valuable to have this trait maxed out.

David Reich

I’m being very speculative. Let me give you two examples of how I’m thinking about this, not that I’m a particularly good authority on these things. As I mentioned, a lot of these traits, which are quite disparate, are highly correlated to each other. Obesity, years of schooling, walking pace, performance on IQ tests, household wealth, all these crazy traits seem to be governed to a substantial extent by a shared combination of genetic variants.

Let’s think about what this might mean. In Iceland in the last hundred years, there’s been selection against this combination of variants. One possible interpretation is that it’s basically selection for two ways of investing in your children: having many kids and not investing a lot in them, or having few kids and investing more in them. If you invest in deferring having kids, having more wealth, having more resources, and putting more into each kid, you’re going to have lower fertility and fewer kids. That’s going to result in lower fertility, but those kids might survive more and do better in society. Alternatively, you can just have as many kids as you can and invest less in them. They might individually have less good outcomes, but in a time of plenty—which is potentially Iceland in the 20th century—it might make sense to have more kids and invest less in them.

There’s a toggle between having more kids and investing less in them, and having fewer kids and investing more in excelling in various ways. You can imagine that at different times and in different places… In ecology, there are different ways. Mammals often invest a lot with a pregnancy and a small number of children, whereas fish will spawn huge numbers of offspring into the river, the great majority of whom will be eaten. But that is an effective way to produce offspring in certain conditions. So there will be a toggle depending on the environmental conditions back and forth between investing in large numbers of offspring with less investment, or smaller numbers of offspring with more investment. Maybe we’re just seeing that move back and forth over different places and times.

Similarly, for schizophrenia and bipolar disease, how could this ever be advantageous? Maybe what we’re seeing with these diseases is a readout of some spectrum of traits that in some contexts might be advantageous. Maybe being anxious, imaginative, or neurotic might be helpful in a shamanistic tradition or a religious tradition which values people who can have visions or be creative. Maybe these are subclinical versions of schizophrenia or bipolar disease that in certain times may be advantageous and in other times may be disadvantageous. You might just be seeing selection for different types of creativity or other thinking that can be valuable in different contexts.

I’m waving my hands here, but my sense is that these complex traits have not pushed in one direction because there are advantages to both ends of the spectrum, and there are multidimensional impacts of these different traits.

Dwarkesh Patel

Julian Jaynes has this famous theory in The Origin of Consciousness in the Breakdown of the Bicameral Mind. I’m butchering this, but fundamentally, the way I understand it is that up until Homer, basically everybody was schizophrenic. People genuinely thought that gods were real people that you were communicating with. His claim is that ancient texts seem to show people behaving in this way.

David Reich

You’re being asked to believe in visions. Even today, there’s valuation in some religious communities in communicating with God, having visions, and having supernatural communions. So I just don’t know.

But I think it’s super interesting to ask the question of why certain traits are not always advantageous. For schizophrenia and bipolar disease, there is a sense in which most of the mutations are disadvantageous. We can see that from the patterns of variation, where the variants that are risk factors tend to be low frequency and they tend to be small effects.

Dwarkesh Patel

So another trait you find under selection is the trend away from body fat since the agricultural revolution. Why is that?

David Reich

What you see is a reduction in the combination of genetic mutations that make you at risk for obesity, body mass index, and similarly very correlated to it, higher fat mass, higher waist-to-hip ratio, and higher type 2 diabetes risk. There is clear selection, by about a standard deviation on the scale of modern variation for these traits, reducing over the last 10,000 years in this part of the world.

What can be going on there? Why wasn’t there selection for this combination of traits before? There’s a longstanding idea known as the thrifty gene hypothesis. The idea is that once you have hunter-gatherer populations that move into a farming environment where there’s plentiful food, there is no longer a need to the same extent to be able to build up body fat to survive in times of stress, because there are more constant stores of food.

As a result, there will be natural selection against body fat once you move into an agricultural environment and into periods of food plenty. Maybe what you’re seeing is that this group of people in Europe and the Middle East over the last 10,000 years has moved into a period of relatively more stable food, where building up stores of fat is not as advantageous, and there’s been selection against this combination of traits. Europeans are actually relatively better protected genetically against type 2 diabetes than some other populations around the world, like African Americans and Native Americans, that have perhaps not been exposed to agriculture for as much time. So you may be seeing the effect of more exposure to more stable food accessibility.

Dwarkesh Patel

This is also another way in which the data go against a common story. The common story is that hunter-gatherers actually had much more stable diets because they were more varied, and they weren’t reliant on a single cereal or crop for their calories. If one game went away, they had other things they could scout for. They could move locations more easily because they weren’t tied down to the land. So they were more food-stable. But if there’s been selection against storage of body fat, that suggests that as unstable and as common as famines might have been in agricultural societies, it’s at least more stable than what the hunter-gatherers had.

David Reich

There’s a timescale issue. You’re absolutely right. As I understand it, I’m no anthropologist, when there’s a hunt in traditional societies or communities that hunt, people will often gorge themselves, eat a huge amount, build up a temporary store of fat, and then go multiple days without eating meat until the next hunt. There is this boom-and-bust access to high-value nutrition that is not true to the same extent in farming communities.

On the flip side, famines are something that occurs more commonly in agricultural societies, but the timescale and the tempo of them is very different from the hunting tempo. Maybe there’s a famine every three years. Indeed, if you look at the bones of farmers, at least in some communities, there’s more stress in them, maybe due to a famine every three years or every five years. But selection might not be acting on that three-year time period. Your fat store from the latest hunt is not going to carry you through to the famine three years later. Survival of famines is a different thing than building up body fat to be able to survive two weeks later.

Dwarkesh Patel

A random question I have. You were mentioning that compared to these other things which matter much more for fitness in the ancestral environment—the immune system, especially after the Bronze Age—all these other things have mattered more than intelligence. They’ve been under much more selective pressure than intelligence.

That makes you wonder whether there’s much more room at the top for intelligence. If humans had been selected especially for intelligence, they could have been much smarter. The reason that’s relevant is that we’re currently building AI systems, which we’re trying to make as smart as possible. In fact, the only goal of the training process is intelligence. We don’t have to worry about at the same time making their immune systems powerful—

David Reich

We have lots of energy to spend on it.

Dwarkesh Patel

And at the same time making sure they’re not schizophrenic. I guess we kind of do worry about that. But if intelligence has not been the dominant trait under selection for humans over the last 10, 20, or 100,000 years, does that mean there’s more room at the top for this trait?

David Reich

I think there’s more room at the top for a lot of these traits. You can move height extremely in one direction, much more than it is today. You can move any of these traits much more extreme in the other direction. There are probably very strong negatives to doing that. You’re probably sacrificing other things, and there are trade-offs. But it’s highly likely that if natural selection pushed any of these traits more in one direction than it is, the mean would move.

00:57:21 – Evolution is limited by time, not population size

Dwarkesh Patel

So all of this evolution since “Out of Africa” is acting on alleles that already existed in the pool of human variation from that first group we were talking about last time, on the order of 10,000 people, that exploded out of Africa. Is it surprising that across all these different traits, from cognitive profiles to disease resistance to height, that one pool of people contained so much latent variation that they could supply enough stretchiness to accommodate all of these different traits you’re studying now?

David Reich

That’s a rich question, and I think the human population has within it a tremendous amount of variation for complex traits. There’s a huge amount of variation that affects height. There’s a huge amount of variation that affects body mass index. If you take all these mutations and set them to the high-height variant, a person will be extremely tall, like as tall as a tall building. Of course, that will never happen.

But if you take all these variants that affect schizophrenia risk and you point them all in the same direction, there will be extreme risk or extreme protection for schizophrenia. For complex traits, ones underpinned by many mutations, all the variation already exists to move the population to a different adaptive set point that’s optimal in the environment it’s in.

If you push the population into a new environment, within hundreds or thousands of years, the population can rapidly move to a new adaptive set point. There are some unusual traits, like the ability to digest cow’s milk or protection against sickle cell anemia, that require a single very important mutation that may not yet exist in the population. You have to wait for the mutation to occur in some people. When the populations are relatively small, only 10,000 people, you might have to wait dozens or hundreds of generations for that mutation to arise. But when the populations are large, there’s no mutation limit anymore.

Every mutation that can occur does occur. There are eight billion people in the world. There are maybe 30 new mutations every generation, so that’s 240 billion new point mutations every generation. There are only three billion DNA bases in the genome, so every mutation that can occur does occur about 100 times every generation. We’re not mutation-limited anymore. The mutations can arise again. They do arise again. But when the population is only 10,000, you sometimes have to wait dozens or hundreds of generations for the new mutation to occur.

Dwarkesh Patel

How likely is it that the thing that changed with the Bronze Age is just that the human population was big enough? By 3000 BC, you go to a population of 50 million-ish people. The population is big enough, and the gene flow between different areas is high enough, such that things which don’t have an overwhelming selection coefficient, which aren’t overwhelmingly favored by evolution, are finally visible to selection.

David Reich

I think that’s not likely to be true, but it’s an extremely interesting thing to think about. Already when population sizes are on the order of a million or so, every mutation that can occur does occur within a few generations. That’s well before the Bronze Age if you take the population even of a place like Europe, but also of other places. Or maybe it’s at the dawn of the Bronze Age or the farming period. The question you’re asking is whether, when the population is small, maybe natural selection doesn’t work effectively.

A common thing people think about with natural selection, which is true, is that in small populations selection doesn’t work effectively. That’s because mutations bop around in frequency from generation to generation a lot in a small population, just randomly. If you have a population size of 1,000, mutations will bop around by a frequency of one over 1,000 every generation. If the selection coefficient is less than that, it will be drowned in the random bopping around of frequencies due to genetic drift. But that is already for a population of 1,000. A 0.1% selection coefficient is very weak. We’re talking about 1% effects, and that’s very strong. It will work very well even in a population of size 1,000 or 10,000.

If you are talking about mutations of the type that will start rising only in large populations but not small populations, those are selection coefficients on the scale of one over 10,000 or one over 100,000. Those will take 10,000 or 100,000 generations to rise in frequency, which is hundreds of thousands or millions of years. That’s not going to do anything over the timescale we’re talking about. There’s just a timescale issue. We’re talking about strong, measurable selection coefficients on the order of half a percent or more in this study. All of those are going to work in small populations or large populations. It’s not going to be affected by the population size.

Dwarkesh Patel

Interesting. You’re saying that more generally, once you hit a given threshold of population, the dominant factor is time span, not population size.

David Reich

Correct. It’s very interesting, and it’s actually not widely understood.

Dwarkesh Patel

Speaking of data contradicting what you might have otherwise assumed, one of the papers you sent me beforehand, Mallick 2016, found that there are no fixed differences between modern and archaic humans 50,000 years ago. We know this is the period in which the so-called cognitive revolution happened, and modernity started, and people are making art. Does this suggest that nothing biological changed to make modern humans modern? The thing that happened was some cultural change? How do we understand what this data tells us?

David Reich

Right. 100,000 to 50,000 years ago, there’s a quickening of the pace of change in culture. You see the first extensive representational art, bead necklaces, drawings on the wall, and a rapidly increasing pace of innovation in the types of tools that people use. The thought might be that there would have been some important genetic switch, a kind of important genetic change that occurred in the population and swept to high frequency that everybody soon had. That made it possible to do these things. Maybe some genes allowed people to have complex, representational language, for example.

One thing we did in 2016 in this paper by Swapan Mallick and colleagues was look across the DNA for places that might be expected to look like this, where nearly all people living today share a common ancestor maybe 100,000 or 200,000 years ago. We looked really hard, and right across all the DNA we could look at, we couldn’t find anything more recent than four or five hundred thousand years ago.

This is a crazy result because it looks like there are no key selective sweeps that have occurred in this period that are ancestral to everyone living today. We talked before about no selective sweeps between Europeans and East Asians, but there don’t even seem to be any selective sweeps shared between all humans in this really important period when a lot of evidence in the material culture record appears. It could be that there’s biological adaptation in this period, but it’s polygenic. There are lots of mutations that all shift in the same direction to help the population move to a new set point, but there’s no key biological change that rises to high frequency in this time.

Dwarkesh Patel

This group 50,000 years ago, are they the ancestors of everybody out of Africa or also some Africans?

David Reich

This is 100,000 to 50,000 years ago. This is the population that’s ancestral to West Africans, to most East Africans, to all non-Africans.

There are a couple of populations in Africa that have substantial ancestry coming from more divergent groups. For example, Khoisan from Southern Africa or Central African rainforest hunter-gatherers have substantial fractions of their ancestry from groups that diverged maybe 200,000 years ago from the other lineages. But all of these groups today are able to go to college and do everything everybody else does. There is no evidence that there is any key mutation lacking in some groups that is not present in the others.

Dwarkesh Patel

The differences we see between different groups of people, especially if this group 50,000 to 100,000 years ago had a very small population size… I think last time we were discussing on the order of 10,000 people. So almost everybody in the world, or the variance we see between different humans today, was latent in this group.

I get your point that if you just stack up different things across the genome, stacking them up really has a big effect. But it’s interesting that we have so many different groups in the world today, and all that diversity comes from a very small population size.

David Reich

A lot of us in human genetics think that our population contains within it the clay that’s needed to make almost any trait. And that depending on environmental conditions or selection conditions, the mean value of these traits will move in different directions. There’s an empirical question about how much selection there’s been in different human populations over time.

One of the things this new work we’re involved in is showing is that at least in the last 18,000 years in this part of the world, there has been significant movement, at least for a handful of important traits. We looked at more than 500 traits. About 100 complex traits showed significant movement in a systematic direction over this time period. It really does seem that there is a response to the environments people are living in that has occurred over this period, and that is potentially stronger than in previous periods.

01:09:02 – Why no farming before the Ice Age?

Dwarkesh Patel

We were talking earlier about how there are no fixed differences between humans 30,000 years ago and humans today. So if there’s no genetic basis for the kind of thing that allowed humans to have more symbolic representation, have farming, et cetera—I think I asked you this question last time we talked, but especially with this context—why no farming before the Ice Age? Genetically we were there.

David Reich

That is such an interesting question. Genetically we’re there. The common ancestral population has all of the ingredients for farming 50,000 years ago. These people are distributed into different parts of the world: the Americas 15,000 years ago or whatever it is, New Guinea 40,000 years ago, East Asia, Europe, West Africa. No farming developed before 11,000 or 12,000 years ago. It only developed in the last 12,000 years, the period known as the Holocene, which is the end of the Ice Age.

If you talk to climate scientists and archaeologists—I keep asking people this question every time I meet someone who’s an expert in this—how can it be that farming develops in all these places? Are we really living in such an unusual time? People tell me, indeed, we’re living in a very unusual time on a scale of two million years. That is, 12,000 years ago we switched into this period of not just warmth, but climate stability.

It’s hard to believe that we’re living in such a special time. But if you look at data from the bottoms of ponds where you can measure the fluctuations of temperatures using isotopic signatures, apparently we’re in a period where it’s fluctuating a lot less year to year, 10 years to 10 years, and 100 years to 100 years. It’s a period of relative stability that we are miraculously living in.

When this period of relative stability happens, it follows that multiple groups independently turn to agriculture, even though they all have the same genetic complement that arose 50,000, 100,000, 200,000, 300,000 years ago. It’s a crazy observation that people just accept, but it’s unbelievable.

Dwarkesh Patel

Oh, so you increased the range there. You said 100,000, 200,000, 300,000 years ago. Based on the genetic differences between modern people and people from 300,000 years ago. Do you basically think they’re modern 300,000 years ago?

David Reich

I don’t know. This is actively what I’m thinking about all the time right now. There’s a big transformation in terms of the culture of humans 300,000 or 400,000 years ago: this invention of Levallois technology, the ability to make stone tools out of cores.

The Middle Stone Age Revolution, or the Middle Paleolithic Revolution depending on what you call it in Africa or Eurasia, is a new way of making stone tools that’s shared by Neanderthals and by modern humans, but is not shared in East or South Asia. It’s a big change, and it presumably involves a cognitive change in order to make this sort of technology.

Then there’s a further change to the Upper Paleolithic Later Stone Age, maybe 100,000 to 50,000 years ago, when there’s a second transition with a new type of tool making, but it’s not as revolutionary as the earlier one. So when the cognitive leap happens is unclear.

The diversification of the lineages leading to people living today, like Khoisan Southern Africans and rainforest hunter-gatherers, all occurs more on the timescale of 300,000 or 200,000 years. All of these people are capable of going to college and doing everything. So it’s not obvious that the cognitive toolkit, the behavioral toolkit, and the genetic abilities were not all in place 200,000 or 300,000 years ago, and that even Neanderthals had them. It’s not obvious that this was not the case.

I just don’t know. You distribute these people descended from this diversification that happened 200,000 or 300,000 years ago to different parts of the world, and then after 12,000 years ago, you start having agriculture popping up in different places. It’s an outstanding mystery of human history. I find it unbelievable that we live in a time period that climatologically is so unique on a scale of two million years, but my colleagues tell me it’s true.

Dwarkesh Patel

The climate thing seems surprising given there are so many different environments in which agriculture was independently developed. I understand that across environments the variance could have gone down. If it had only happened in one place at one time, I could have bought that explanation. But the fact that they’re making maize in the New World and they’ve got cereals in the Old World in very different environments makes it surprising.

David Reich

It’s very, very surprising. We accept it, but it’s a crazy observation that most normal people don’t realize.

The thing that basically everybody accepts is that the common ancestral population of almost everybody in the world, except for rainforest hunter-gatherers and Khoisan, is around 70,000 years ago. Everybody accepts that these people all have in place the cognitive, behavioral, and intellectual ingredients that are necessary for the farming revolution and building state societies. Because when these descendants get distributed to West Africa, East Africa, the Americas, Europe, South Asia, East Asia, New Guinea, and so on, their descendants all do this. They do it independently, semi-independently, or demonstrably completely independently in all these different parts of the world.

The cognitive resources for doing this must have all been in place, but it’s a very long fuse. It delays for 40,000 or 60,000 years in all these different places after the common ancestral population splits up, and then ignites into agriculture and all these other things after that point.

It’s a crazy claim. Then you could argue about whether the actual fuse is 300,000 years, from when Neanderthals separated and from when different lineages of extant modern humans separate, and that’s also plausible. It’s a crazy set of things that we’re being asked to believe.

Dwarkesh Patel

Is it possible that agriculture existed, but you didn’t have modern metallurgy or whatever it was that allowed populations to explode starting in 5000 BC with the Bronze Age? Population-wise, it doesn’t seem like much is happening from 10,000 BC to 5000 BC in the early Neolithic. Is it possible that they had farming but they didn’t have copper or tin, which you needed to go to the Middle East for, to develop a civilization that could make use of bronze at a large scale, and so they just disappeared from the historical record?

David Reich

I think we would see their archaeology. There are extraordinary developments in the Americas which are entirely Stone Age.

Dwarkesh Patel

You would see them today if they had completely vanished?

David Reich

Oh, yeah. We should go for a trip to Teotihuacán in Mexico. It’s so impressive. When I went there when I was 20, it was totally as impressive as ancient Egypt. It’s huge. It’s massive. It’s without metal.

Dwarkesh Patel

It’s even more impressive because it’s not only without metal, but without animals and without wheels, which is crazy. The marble is just hauled without wheels.

David Reich

Right. Take any person who has an old world superiority and take them to these places, and they will not have it anymore. It’s just extraordinary what’s in these places. These are people who separated 20,000 years ago at least from the ancestors of East Asians and 40,000 years ago from the ancestors of West Eurasians.

They just had the same biological and cultural shared toolkit from then, but there’s a long fuse delay until all this stuff happens. It’s an amazing thing, and we don’t question it.

01:17:13 – The Neanderthal puzzle David can’t stop thinking about

Dwarkesh Patel

What are other questions you are either investigating right now or want to investigate, these kinds of big picture questions of human history?

David Reich

I’m perplexed. I don’t know if we talked about it before, but I remain very confused about the relationships between archaic and modern humans.

We have genome sequences now from archaic humans who lived in Europe, West Eurasia, and Central Eurasia, and the Neanderthals. We have archaic sequences from these enigmatic Denisovans, who we now have a skeleton for since we last talked. There’s now a skull that’s been shown to be a Denisovan. We have data from lots of modern humans, and there are really big mysteries about the relationships amongst these groups.

Genetically, the Denisovans and the Neanderthals are sisters. They descend from a common ancestral population 500,000 or 600,000 years ago. That group descends 700,000 or 800,000 years ago from the common ancestors of modern humans. Genetically, the whole genome data says that Neanderthals and Denisovans are archaic humans from a common ancestral archaic population.

But there are so many things shared between Neanderthals and modern humans that don’t seem to be shared with East Asians. They both share Middle Stone Age stone tools, Levallois technology, this cognitively unique way of making stone tools that wasn’t used in East Asia. They both have the same mitochondrial DNA and Y chromosome sequence.

The Y chromosome sequence of Neanderthals and the mitochondrial DNA of Neanderthals, is actually modern human that came through interbreeding 200,000 or 300,000 years ago and then shot up to 100% frequency. Neanderthals and modern humans are both the product of mixture events that happened between archaic and modern humans 300,000 or 200,000 years ago, demonstrably through patterns of variation in ancient and modern DNA.

It feels that there’s something shared between Neanderthals and modern humans that’s not shared with Denisovans, even though the vote of the whole genome says that Denisovans and Neanderthals are related. One wonders whether there’s something connecting Neanderthals and modern humans that’s different from Denisovans, even though genome-wide, Denisovans and Neanderthals cluster. I’m thinking about that all the time now.

Dwarkesh Patel

Connecting them would be interbreeding events or being in the same place at the same time that we missed?

David Reich

There’s a known interbreeding event from the lineage leading to modern humans into Neanderthals, but it’s supposed to be only 5%.

I’m interested in the possibility that that 5% is actually a sign of something much more impactful, that somehow Neanderthals are in some sense deeply modern in some ways, and even though they get swamped by archaic genes, they actually have more of a modern impact than one would think. The Middle Stone Age and Middle Paleolithic Revolution that they share with modern humans is more fundamentally a part of who they are, in some sense, than we think.

Dwarkesh Patel

Interesting. Sorry, when was this interbreeding event?

David Reich

300,000 to 200,000 years ago.

Dwarkesh Patel

So the common ancestor between Neanderthals and most humans alive today is potentially more recent than the common ancestor between all humans alive today.

David Reich

Oh, for sure.

Dwarkesh Patel

Which is crazy.

David Reich

Well, the divergence to all the archaic humans, including Denisovans, is within human variation.

Dwarkesh Patel

Wait, what?

David Reich

Yes. The average time to the common ancestor of any two human genes is one or two million years ago. If you look at the copy of chromosome 3 you get from your mother and the copy of chromosome 3 you get from your father, the typical time they share a common ancestor is one or two million years ago. That’s before the split from Neanderthals and Denisovans. So there are many places in your DNA where you’re more closely related to a Neanderthal on your mother’s side than you are to your father.

Dwarkesh Patel

I’m sure there’s a simple explanation, but how?

David Reich

It’s the same reason that if you have a sister, in some places in your DNA you’re more closely related to her than you are to me because you share a parent. But in other places you’re more closely related to me than you are to your sister because you happen not to share the same DNA from your parents.

It’s just that the DNA we get from our common ancestral population was already quite variable 500,000 years ago, 700,000 years ago, a million years ago, and some of us descend from some of those ancestors and others descend from other of those ancestors. Neanderthals split from our lineage really close in time on human evolutionary timescales, such that in some places in our DNA we’re more closely related to Neanderthals than to each other.

Dwarkesh Patel

Interesting. What are the other big questions?

David Reich

That’s the main thing I’m thinking about a lot these days. I continue to be very obsessed with questions about the spread of human populations around the world and trying to reconstruct that with ancient DNA.

The thing I’ve been thinking about a lot recently is the possibility that maybe we’re not thinking in the right way about the relationship between archaic and modern humans. The standard model is one where Denisovans—these archaic humans that were found from ancient DNA—and Neanderthals descend from a common ancestral population 500,000 or 600,000 years ago, and these two separate earlier, maybe 700,000 to 800,000 years ago, from the ancestors of modern humans, people like us. That’s the big result of a lot of studies since 2010.

But there’s also evidence of an interbreeding event that happened maybe 200,000 to 300,000 years ago that resulted in modern humans contributing DNA to the ancestors of Neanderthals. So maybe 5% of the DNA of Neanderthals comes from this interbreeding event, and a lot of studies have shown this. I’m very interested in this because, from the archaeological record, Neanderthals and modern humans look quite similar to each other, much more similar to each other than a lot of them do to Denisovans, these archaic humans in East Asia.

For a lot of history, people have thought that Neanderthals are our sisters. But in 2010, the sequencing of the Denisovan genome made it very clear that on average, Denisovans are closer to Neanderthals than to modern humans. This was a very confusing result. Most people now think that Neanderthals and Denisovans descend from a common ancestral population that separated earlier from the ancestors of modern humans.

I’m interested in the possibility that the right way to think about Neanderthals is actually as somehow culturally modern humans, even though genetically they’re mostly Denisovans. The model I’m thinking about is motivated by this archaeological phenomenon known as the Middle Stone Age Revolution. If this is Africa and this is Europe, we know that the new way of making stone tools—with cores that were very carefully mined far away from the locations they were used, made out of high-quality stone like flint—starts being used 300,000 or 400,000 years ago, first in the Caucasus, places like Georgia today, or East Africa.

This way of making stone tools is quite revolutionary. It is known in Europe as the Middle Paleolithic and in Africa as the Middle Stone Age, and is associated with much more widespread use of fire and moving stone around at much further distances than before.

I’m interested in the idea that this is something shared between modern humans and Neanderthals. There’s somehow some shared cultural feature that’s absent in East Asia, and that might have a relationship in the genetic data and is somehow related to this 5% DNA.

The idea I’m interested in is the possibility that there is a population here that invents the Middle Stone Age and the Middle Paleolithic, sometimes called Levallois technology, and that people from this population expand into Europe and mix with the local archaic humans who are there. That is what this 5% interbreeding event is. It happens 200,000 to 300,000 years ago. It produces a group that, as it expands across this landscape in Europe, mostly picks up the local DNA and becomes mostly archaic genetically, but retains its modern human culture, the way of making stone tools and some of its traditions.

One of the things that’s super interesting about this is that if you actually look at the genetics, across the whole genome, Neanderthals and Denisovans cluster. But if you look at the mitochondrial DNA—which humans and Neanderthals get from their moms—Neanderthals and modern humans cluster. If you look at the mitochondrial DNA, Denisovans and modern humans share an ancestor well more than 700,000 or 800,000 years ago, as you’d expect from the history. If you look at the Y chromosome that you get from your dad, Denisovans and modern humans share an ancestor more than 700,000 or 800,000 years ago, which is consistent with this history.

But if you look at the Neanderthal mitochondrial DNA, it’s only 300,000 to 450,000 years. If you look at the Y chromosome, it’s only 300,000 to 450,000 years. What the current genetic work is asking us to believe is that even though this is only 5% of the whole genome, it introduces mitochondrial DNA and Y chromosomes, and they jump up to 100% frequency. It’s kind of a crazy claim because the probability of this occurring by chance is low, maybe 5% times 5%, a very small number. It’s what we actually all believe, but it’s a very surprising event. Somehow it’s accreted to all the findings in the literature so that we make ourselves believe this, but it seems unlikely on first principles that somehow only 5% would introduce both the Y chromosome and mitochondrial DNA.

And it really does look like this. There’s amazing data from a site in Spain that’s 300,000 to 400,000 years old, called Sima de los Huesos. They have a nuclear genome that looks Neanderthal-like for most of the genome, but their mitochondrial DNA and Y chromosome are Denisovan-like. So it really looks like there was a population related to modern humans that pushed into this Sima de los Huesos-like population, displaced its mitochondrial DNA and Y chromosome, but kept the rest of its genome. It really looks like something like this happened.

The idea I’m playing with—and probably it’s wrong, who knows—is that there’s a landscape… This is Europe and you can break up into a hundred or so demes, little areas. Modern humans get introduced at the bottom right corner, in the Middle East or somewhere, and they spread into Europe. As this population spreads, there’s a wave front of expansion, and they’re interacting with the local archaic humans.

The theory from simulations and studies of different species like mammals and birds shows that even if there’s a small amount of interbreeding—when there’s an invasion or a movement of expansion of one group into the territory occupied by another—there’s massive introgression of local genes. The pioneers at the wave front will sometimes interbreed with the local population. There are so many of them around that their DNA will get swamped by the local group, so by the time they make it to the other side, they’re largely local.

Maybe this is what we’re seeing. You have a modern human population that’s matrilineal, for example, where the transmission of making stone tools this way is happening from mother to child. That’s why they’re retaining their mitochondrial DNA, but by the time they get to the other end of Europe, they’re mostly local archaic. You end up with a 95% population replacement. This would explain why the mitochondrial DNA is shared between Neanderthals and modern humans, and it would also explain why the mixture proportion is only 5%.

The really interesting thing is that there’s other evidence from studies of modern humans showing that modern humans are also admixed. The right way to think about this is that modern humans are a mixture of two groups that diverged maybe 1.5 million years ago, and they come together 200,000 to 300,000 years ago with maybe 20% ancestry from this archaic African group and 80% ancestry from this early modern lineage. And that same group then mixes with Neanderthals, and it’s 5% modern and 95% local.

So you actually have this key population that makes the Middle Stone Age or Levallois technology. It appears and expands in all directions—into Europe and into Africa 200,000 to 300,000 years ago—bringing this technology, new ideas, and perhaps some genetic adaptations. It expands into archaic humans in Europe, mixes with the local population, and gets 95% replaced but still retains its cultural features and maybe some genetic features.

It expands in Africa too, but here it’s not 95% replaced, it’s only 20% replaced. Probably the reason is that this group is much more diverged. It’s 1.5 million years diverged rather than 700,000 to 800,000 years. As a result, there are many more genetic incompatibilities and barriers to gene flow. But there’s still a lot of mixing, maybe 20%, and we have evidence that this is a big mixture event.

So what you’re actually seeing is a modern human expansion both into Europe and into Africa. In one place, it forms Neanderthals. In another, it forms the ancestors of everybody living today. But all of these groups are descended from this key revolutionary event that happens here.

We often talk about the revolutionary events 50 to 100,000 years ago, the more symbolic behavior and so on, that first appears in Africa and the Middle East and spreads beyond. But there’s also this earlier event, and it is contemporaneous with the breakup of all the different groups in Africa today, the Khoisan Southern Africans and the Central African rainforest hunter-gatherers. One wonders whether this is an equally important formative event.

If that’s true, it makes you think of Neanderthals as somehow our cousins. Their shared Y chromosome, their shared mitochondrial DNA, they share the formation of this 200,000 or 300,000-year-old event, and their shared toolkit. Even though the genome is telling us they’re cousins of Denisovans, the correct way to think about them may, in an important sense, be as close cousins of modern humans.

Dwarkesh Patel

I have so many questions. Do you have 15 more minutes? First of all, what is going on with this group of archaic Africans 1.5 million years ago? Where in Africa are they, and what happens to the portion of them that don’t form modern humans? Do they survive?

David Reich

This is not from ancient DNA, but from analysis of modern DNA from different people, mostly in Africa, but also non-Africans. In multiple studies—there’s at least three, maybe four or five studies that I know about—they have looked at the patterns of variation in people today and say the data in modern people today, including in Africans, is not consistent with a homogeneous population.

It looks like a population that split well more than a million years ago into multiple groups—at least two, but maybe many—and then came together a few hundred thousand years ago. The papers have different models that they fit, but they all have this feature of a split-up more than a million years ago, and then on the order of a few hundred thousand years ago, a coming together and a remixture event forming the ancestors of anatomically modern humans.

Dwarkesh Patel

This includes the Khoisan and whatever other groups?

David Reich

Yes. All of these groups have this, maybe in slightly different proportions. So you ask, where are these people living? Who knows. In this scenario, the 80% is coming from the Caucasus or Northeast Africa, where the Middle Stone Age forms. It’s from this population that the Middle Stone Age comes. They mix with local groups, and who knows where they are: Southern Africa, Western Africa, Central Africa, Eastern Africa.

We don’t have any ancient DNA, but this is a very rich environment. People have been living there for seven million years at least, and there would have been different groups of people everywhere. Probably it’s not just two groups, it’s probably more.

The important theme here is there’s evidence of substructure that’s well more than a million years old. This place would have been a landscape full of archaic humans that would have been differently related to these expanding people and would have admixed with them when they came through.

Dwarkesh Patel

So with the Neanderthals, the first time around 300,000 years ago, our ancestors share culture with them. They share the Middle Stone Age technology, but they don’t replace the population. The technology spreads through culture, basically.

David Reich

It spreads through genes too. If you look at Yamnaya in India, there’s almost no Yamnaya ancestry in India. It’s just diluted down. As Yamnaya expanded into Central Asia and into Europe, it makes the Corded Ware. There’s a 25% dilution. It expands back across Central Asia. It goes through the Hindu Kush and gets into northern South Asia. It admixes more with local people. Today, the most Yamnaya ancestry you see in India is 20% or 10%. Most people have less than 10% or 5%.

There’s been a lot of mixture on the way, but it is the tracer dye. It tracks Indo-European languages, and important aspects of Indo-European culture are coming through Yamnaya. So if you know where to look, that tracer dye is only 10%, 5%, or 2% in some groups. But it’s the languages people speak, and important shared cultural elements, that connect them to people on the other side of the Indo-European-speaking world. So this 5%, you shouldn’t sneeze at it. That’s tracing something important in this model.

Dwarkesh Patel

I understand that if things are transmitted more through women… Sorry, let me back up. I don’t understand why the maternal mitochondrial DNA and the Y chromosome would be especially privileged as the spreading is happening. Can you explain that?

David Reich

The reason I’m talking about these matrilineal or patrilineal expansions is that I’m really troubled, and have been troubled for many years—especially in the last three or four years—by the fact that the mitochondrial DNA and Y chromosome cluster Neanderthals and modern humans, but the rest of the genome clusters Neanderthals and Denisovans.

This is a crazy result that is not seen in any other species. I’m very interested in patterns that would explain it. If you assume that there was a matrilineal or a patrilineal expansion—it could be either—then modern humans, when they were expanding across the landscape of Europe, retained their identity along one of the lines.

If it’s matrilineal, when they incorporate a male from the local community, he’s brought into the community, and the kids are raised based on the culture of the mothers. If it’s a patrilineal expansion and they incorporate a female from the community, she’s raised with the culture of the fathers. If that happens, it guarantees that one of these two parts of the genome looks like it does, because it’s a modern human expansion. If it’s patrilineal, it will retain the Y chromosome. If it’s matrilineal, it will retain the mitochondrial DNA. So it will solve one of your two problems.

Dwarkesh Patel

But not both.

David Reich

It won’t solve the other one, so you need to solve the other one. You can solve it either by natural selection or by social selection. By the way, patrilineality and matrilineality are the rule, not the exception, in human communities. Usually communities have continuity along the male or the female line. Usually it’s patrilineality, sometimes it’s matrilineality.

You can also have phenomena like social selection. It could be that once you have kids of someone whose father, for example, is from the outside community… Usually in most communities, females all reproduce. That’s typical today. Usually women have kids if they can. But men in traditional societies are actually very variable in their reproductive success. A large fraction of men never have kids. Then there’s a subset of men who have many kids with many women.

There’s competition among men for kids. In this context, where males are competing for access to females, female mate choice begins to be an important process. You have a phenomenon where if your dad is an archaic male, it could be the case that you’re not going to be as successful in the competition for local females as if your dad is a non-archaic male.

Some simple social phenomenon like that could explain the data, and we actually see this in human society. For example, if I remember right, in Central African rainforest hunter-gatherers, there’s different treatment of boys and girls depending on whether their dad or mom is one group or the other.

Dwarkesh Patel

I guess I don’t understand how the maternal… The group spreads, and it gets to the next front. They have kids. From the humans that have just entered, the kids will have the mitochondrial DNA from the humans. But from the existing people, they will have the mitochondrial DNA of the archaic humans. Why are the people with the archaic mitochondrial DNA not surviving?

David Reich

It’s a question. There are multiple possible explanations, but it’s much easier to explain that than both the mitochondrial DNA and the Y chromosome. One possibility is that the mitochondrial DNA was less biologically fit. Another possibility is that there’s social discrimination against people based on whether their parents are archaic or not, which is not at all surprising in a human context. It’s the weakest link in this argument. This argument’s probably wrong, but I’m just telling you what I’m thinking about.

Dwarkesh Patel

Okay, the Neanderthals. So 300,000 years ago our lineage interacts with them, but mostly their lineage survives, and there’s cultural and genetic diffusion. And then is it 70,000 years ago that we interact again?

David Reich

Yes.

Dwarkesh Patel

And they don’t survive.

David Reich

The genetic ancestry doesn’t survive.

Dwarkesh Patel

The genetic ancestry doesn’t survive. Presumably there was also other contact between 300,000 years ago and 70,000 years ago.

David Reich

Probably. But these are the ones we are detecting currently.

Dwarkesh Patel

Is it just contingent that one time there’s this kind of diffusion where most of the archaic genome survives, and the other time it’s total replacement?

David Reich

This is not at all surprising given the context. If you think about this model, this is 700,000 or 800,000 years ago. This is 300,000 years ago. So this is 400,000 years separated. You talked about the Bhatia paper with me earlier. That’s two populations 70,000 years separated. There are no biological incompatibilities between West Africans and Europeans. There’s no natural selection against biological incompatibilities.

We know when Neanderthals and modern humans met and mixed, there were biological incompatibilities. That was 700,000 years ago. As populations become further apart, biological incompatibilities rapidly develop, probably as the square of the separation distance, because you need pairs of interacting genes. Here, it would have been maybe only 400,000 years separated between this lineage and that lineage. But here, it’s 1.2 million years. That’s a lot. These are at the edge of not being able to produce children. These are quite different humans. These are actually three times closer than these.

If you look at mixtures of humans today, there are mixtures in Southern Africa of people who are half this distance. If you look at Khoisan and Bantu people mixing in Southern Africa, like the Xhosa, which is the population of Nelson Mandela, these are groups that are separated by almost 200,000 years, which is half of this. Totally compatible.

What you’re seeing is a group that’s actually completely permeable genetically, or nearly completely permeable. This other one almost certainly has substantial biological incompatibilities. Because 200,000 or 300,000 years later, we see interbreeding between Neanderthals and modern humans, or between Denisovans and modern humans, and there’s clear evidence of incompatibility at that point. But this would be even bigger.

What you would expect to see is that as this group spread, they would be moving into a territory full of archaic humans. There would be some interbreeding, but the kids would not be very fit. They would die off. There would be a lot of infertility. The barriers to gene flow and to interbreeding would be greater.

To me, it’s not at all surprising that as this group moves into Eurasia, you have Eurasian archaics—the ancestors of Denisovans—who are only 400,000 years diverged from these people over here. And then you have African archaics, and these are 1.2 million years diverged. They just don’t interbreed as much, and you don’t get as much gene flow.

But the key thing is the timing. It’s the same time. It really feels like the signature of an explosion of people from one place, interacting with people here and interacting with people there. It’s the same cultural or technological revolution impacting this place and that place, and creating populations that are impacted by this cultural revolution, which we know is the case because they share the same toolkit.

Some people argue that Levallois technology is independently invented. But it’s very similar, and this model would be a way that it could have the same origin. So there’s a culturally shared thread, this shared toolkit. There’s a mitochondrial DNA and Y chromosome thread. And then there is a shared timing thread, which is they both form by mixture.

Dwarkesh Patel

Because otherwise you’d have to believe that Neanderthals independently developed Stone Age tools.

David Reich

Yes, which is not inconceivable. But it’s a little bit like believing that farming independently developed in multiple parts of the world.

Dwarkesh Patel

Right. But it did.

David Reich

It did. So as I said, this is probably wrong. I’m trying to tell you that we don’t really know the world we live in. This is not obviously wrong. In fact, to me, this is much more plausible than the model we currently write down. It’s probably wrong, but it’s much more plausible. It explains many more things, and it’s no more complicated.

Dwarkesh Patel

Do you want to recapitulate the thing you were saying about the analogy to Ptolemy and the epicycles? I thought that was quite interesting.

David Reich

I think the model that we’ve put together collectively about the relationships between archaic and modern humans has accreted over time. There was this idea that modern humans are distinct and that Neanderthals and Denisovans are sisters of each other. Over time, we detected additional mixture events, like this modern human into Neanderthal, and then these other ones I didn’t even talk about, like a super-divergent lineage going into Denisovans and all this other stuff. We still say, “Oh, the whole genome says Neanderthals and Denisovans are sisters, so that’s the truth.”

We’ve patched it all together and gotten it all to work. You look at the mitochondrial DNA and the Y chromosome, and they have this odd pattern, and it’s improbable, but we can get that to work if we invoke natural selection, things like this. You patch it all together.

It reminds one of what happened in the ancient world, where there was this idea that the sun revolves around the Earth, but it doesn’t quite explain the movements of the planets properly. In order to get the movements of the planets to work right, Ptolemy and the astronomers made up these epicycles, these special extra rotations and movements to make everything work about right. It was such a convoluted model. When Copernicus and colleagues suggested instead that everything is revolving around the sun, it simplified things ever so much.

What was happening is that as astronomical information accumulated, it kept being contradictory to the standard model, but it could be made to work by proposing another complication and another complication and another complication. This is not as fantastic as proposing that everything revolves around the sun rather than the Earth, but it is much simpler. And it actually explains many things.

Dwarkesh Patel

What is counterintuitive or unexpected or hard to accept about this alternative model? What is the hesitation that people have for adopting this?

David Reich

I don’t know. Nobody’s thinking about this model right now. It just seems obviously a very natural model to me.

Dwarkesh Patel

The reason I ask is that Aristarchus, the ancient Greek, had the heliocentric theory because he had deduced how far the Earth is from the sun and noticed other things. But it was not adopted, because his fellow Athenians were like, “Look, if we believe that the Earth revolves around the sun, for it to be the case that we don’t see relative movement of the stars to the Earth, the only possible explanation is that the stars are so far away that it is just incomprehensible and implausible.” So the heliocentric theory was dismissed.

What I’m trying to ask is, what is the equivalent here of “for this to work, the stars have to be so far away that it’s inconceivable,” where actually the stars are so far away? Maybe we should adopt the implausible implication that this theory gives us.

David Reich

That’s a great question. I think we have to assume that there’s a linkage between the cultural transformations in Africa and Eurasia at this time, and that’s not something the community has really put together with the genetic data.

There’s this thread in the genetics about substructure in Africans, and then there’s this whole world based on ancient DNA, and they’ve never been put together. Nobody’s put together the now extensive work on modern human substructure with the extensive work based on ancient DNA of archaic human relationships to modern humans.

If you put them together, you realize they line up in terms of their time of substructuring. I don’t know if that’s improbable. It seems parsimonious to me.

Dwarkesh Patel

It also seems significant that different groups of humans at this time were capable of adopting Stone Age technology. Once one group had figured it out, the genetic difference between different human lineages was not so big that you could not show people how to use stone tools.

David Reich

Who knows? It could be that this was genetically driven. We talked before about the time to the common ancestor of human genes. There’s nothing at 100,000 years or 150,000 years, but there’s a lot at 400,000 or 500,000 years.

If that’s what happens, and you have a mutation that occurs in the Caucasus, somewhere in the Middle East, or Northeast Africa, there could be key genetic mutations that make people able to do this. Then this population expands. When it moves into Europe, it’s swamped by local genes, but there could be retention of those genes through selection as it expands.

Maybe what you’re seeing is that there are genetic developments. Most of the discussion has been focused on the 50,000 to 100,000-year event, which is anatomically modern human behavior. But a lot of archaeologists think this is an equally—if not more—profoundly significant event in many ways. Why is that not the event we should be talking about?

Dwarkesh Patel

You’re talking about how there are no fixed differences between modern humans and the humans 50,000 years ago. Do we know if there are any fixed differences between the people 50,000 years ago and the people 300,000 years ago?

David Reich

I think there are.

Dwarkesh Patel

Other than obviously these interbreedings.

David Reich

If you look at the genetic variation going back 300,000 or 400,000 years, there do begin to be places where all modern humans share common ancestry. That’s another way of saying there begin to be fixed differences at that time depth. That is where you start seeing evidence for possible fixed differences.

What’s happening, if everybody shares a common ancestor 400,000 or 500,000 years ago, is that there’s a single ancestor at that time. If you compared it to another population, they would descend from a different lineage, so any mutation that occurred ancestral to that single ancestor would be a fixed difference. This is the time at which you can begin to see fixed differences.

Dwarkesh Patel

But anatomically modern, cognitively modern humans exist by the beginning of the Middle Stone Age, before we’re breeding with this ancient group of Africans or breeding with Neanderthals.

David Reich

Anatomically modern humans occur exactly here. It’s the same moment. This is when they occur. The people who have skeletal features like ours, and Neanderthals, appear exactly then.

This is when it all happens. There is this disconnect between anatomically modern humans in the skeletal record and behaviorally modern humans, which is 50,000 to 100,000 years ago. Anatomically modern humans appear at this time, and recognizable Neanderthals appear roughly around this time, too.

Dwarkesh Patel

Interesting. But we don’t know what exactly happens, if anything, between 200,000 years ago and 50,000 years ago that goes from just anatomical modernity to behavioral modernity.

David Reich

My understanding is no. They’re busy making Levallois stone tools like Neanderthals for 200,000 years, and they are not more impressive than Neanderthals in any obvious way, as I understand it.

Then there begins to be in the archaeological record a quickening of behavioral traits, which could be not genetic at all, or could be genetic. There are lots of arguments about this. We were obsessed with intelligence earlier in our conversation. People are obsessed with art and these things that seem important to us but who knows what’s important?

Dwarkesh Patel

Interesting. Cool, thanks for the digression.

01:54:10 – The methodology behind this breakthrough

David Reich

The work that I’ve been involved in has consistently shown that I was wrong in my biases coming into the work, and I’ve really been almost traumatized by this. Again and again, I’ve come into a project with some kind of guess about what the data was showing, and then the data doesn’t show that.

For example, when I got involved in the Neanderthal genome project helping to analyze data looking at how archaic Neanderthals were related to modern humans, I was part of a group of scientists who had established that non-Africans were a simple subset of African variation and that there was no evidence at all of Neanderthal interbreeding into the ancestors of modern humans or other archaic interbreeding. Different analyses that I and many other people had done made it look like non-African variation was just a subset, a small sample of that in Africa, and that could have fully explained the data.

So when I was involved in analyzing the Neanderthal DNA sequences, what happened was I found this very strong evidence of Neanderthals being more closely related to non-Africans than to Africans. It was very surprising, and I thought it must be a mistake. I was quite incredulous. I thought it was unlikely to be true because other evidence that had been found before seemed to point in the other direction. So I spent several years trying to make these results go away, as did my colleagues, and we just couldn’t make the results go away. They just kept getting stronger.

And this experience working on natural selection was the same. What we were convinced of was that natural selection had been pretty quiescent in our species over the last several hundred thousand years. Therefore, if we look at patterns of variation in non-African people today, or in any people today, we should see not a lot of selection going on. Indeed, the first ancient DNA studies, beginning in 2015 with this paper that we were involved in with Ian Mathieson and colleagues, seemed to show relatively small numbers of genetic positions associated with natural selection.

In 2015, we analyzed data from about 200 Europeans and Middle Easterners to try to understand frequency changes over time. We compared those ancient people who were the sources of modern Europeans to people in Europe today, and we looked at frequency differences that were too extreme to be due to chance. We were very excited to find 12 positions that we were convinced were highly different in frequency between Europeans today and what we would expect, based on the history that we and others had identified as the history relating modern to ancient Europeans. Some of these were known and some of these were not known, and this was very exciting.

We hoped that as the numbers of samples would increase and we got higher resolution to be able to appreciate differences in frequencies over time, it would make it possible to detect far more. What was quite disappointing over the subsequent decade is that that didn’t happen. For example, the largest study of that type in 2024 by a group in Copenhagen analyzed much better data than we had in 2015 and found only 21 positions that were highly different in frequency across time. While that was exciting—it was almost twice as many as we had found in 2015—in a lot of ways it was disappointing because the sample size and data quality had gone up so much, and yet this is all that was found.

That suggested we might be hitting an asymptote and might not be able to get beyond where we currently were. This approach to learning about biology, which was very promising in theory, might not produce a high yield. Maybe natural selection was quiescent, and the reason we’re seeing so few changes is that there has not been a lot of adaptive directional selection. That was the situation we found ourselves in until just a few years ago when we carried out this study in our research group led by Ali Akbari.

What we did is we deployed a few innovations to try to improve our power to detect natural selection. One of them was that we just pumped a lot of data into the system, increasing the amount of data by about 14-fold. The main thing that we do in this study is report new data from about 10,000 individuals. This is a very big increase in the amount of data in the literature. The total dataset size of ancient individuals distributed over the last 18,000 years is about 16,000 people. This is a large dataset. It’s much larger than was previously possible, and when you have more data, you can estimate frequency changes with much more subtlety.

The data comes from only one part of the world, which is Europe and the Middle East. It’s not a more important part of the world than other places, but it’s the place where maybe 70-80% of the data in the ancient DNA literature so far comes from due to historical reasons. It provides us with a natural laboratory where we can see what happens to the genome in one place over time as environments change. It’s really interesting to imagine doing this type of analysis in other parts of the world, and the comparative analyses are super important and interesting, but this study right now is about this one place in the world where we have particularly fantastic data.

The other thing we did is that we developed an entirely new methodology that hadn’t been used in this area before. The methodology is based on a technique that had been developed for finding risk factors for a disease in medical studies. A simple way to explain it is that we ask how to predict the genetic type a person has based on their pattern of relatedness to other people. We have a dataset of about 16,000 ancient people, and 22,000 people if we include the modern people. Then we look at how closely related each of these 22,000 people are to each other, and we predict the genetic type at each position in the DNA—at 10 million positions—based on the pattern of relatedness to all of the other 22,000 people.

Then we ask if natural selection blowing the frequency of the mutation in the same direction in all geographic places and at all times predicts the data a little bit better than just knowing the relatedness to all the other samples in the database. We’re simply asking if the alternative hypothesis—that selection has been blowing in the same direction at all times—explains the data better. That’s a dumb assumption, because of course, the truth is that natural selection will have changed in frequency over time. But we’re just asking the simplest of questions: whether assuming a constant rate of selection explains the data more than not doing so.

Dwarkesh Patel

To summarize to make sure I’ve understood, you’re trying to make a model that predicts allele frequency changes over time. You have two different parts. One part is this genetic relatedness matrix, which captures how similar different genomes are to each other. That should capture the impact of different bottlenecks, of drift, of population admixtures, and all those things which affect the entire genome.

Then you have the separate thing, which is, if we look at specific locations, can we just say that, “Oh, this location has been selected at whatever coefficient over time”? And if we add some coefficient, does it become easier to predict the allele frequency changes than you would have just seen from this other artifact, which is just looking at, “Oh, if you look at the whole genome, are these guys in the same, have they gone through the same bottlenecks? Have they gone through the same drift,” etc.?

David Reich

That’s precisely right.

Dwarkesh Patel

Okay, what did we learn?

David Reich

When we analyzed the data this way, we looked at 10 million positions in the DNA across these 22,000 people, 16,000 of whom were ancient. We looked to see if there was more change in this consistent direction over time than you would expect by chance. When we analyzed the data, we found many hundreds of places in the DNA that were changing too much over time and in too consistent a way to be explained by chance.

There’s a bit of a statistical problem in figuring out how many there are because they’re so densely packed that they’re close to each other and interfering with each other. But when you try to piece them out and say, “Let’s count only one in each place in the DNA and blank out the others,” we find at least 479 positions that are all independently pushing in the same way. We are 99% confident that those positions are real. By another criteria of being more than 50% confident that they’re real, we think that about 3,800 positions are all pushing in the same direction. This is a crazy number of results given that in our previous work and other people’s work, there were at most a couple of dozen discoveries coming from a single scan.

So when we got this result, we were very surprised. We thought it must be wrong, and we spent the next couple of years trying to make the results go away, but they just kept getting stronger. We were trying to look for some independent type of evidence to tell us whether these positions were real. We stumbled on something really powerful for this purpose that had not been used in this way before. It relied on the fact that we had very large numbers of discoveries, many hundreds of discoveries or even thousands.

We took a completely independent dataset, which was the corpus of genome-wide association studies. These are studies that people have carried out in hundreds of thousands of people, looking for whether particular genetic mutations are more common in people with high blood pressure than with low blood pressure, or something like this. We took the UK Biobank, which is about 500,000 people from Great Britain who have been measured for hundreds of traits. The whole genomes of all these people have been sequenced. For each of these traits, we could look at whether each of these 10 million positions are connected to this trait in a convincing way. Out of 10 million positions, about 15%—about 1.5 million positions in the DNA—are predictive of at least one of these several hundred traits.

Then we could ask a question: is our natural selection signal, our statistic, related to whether a mutation causes high blood pressure or some other trait? We slid our statistic for natural selection upward, to a value of one, two, three, four, or five. As we did that, the enrichment for genetic mutations that affect traits got higher and higher. Whereas it was only 15% when we didn’t use our selection statistic, when we required the selection statistic to be above about five, there was about a five-fold enrichment for mutations that cause traits.

Dwarkesh Patel

Sorry, what is a selection statistic?

David Reich

This is the statistic we use to measure whether a mutation is changing over time significantly in a non-zero way. It can be approximately thought of as a normally distributed statistic, a Gaussian statistic, which is the number of standard deviations the statistical value is away from zero, where zero is no natural selection.

It’s not exactly that, but it’s close to that. If the statistic is above five, we see about a five-fold enrichment in mutations that affect a trait. Instead of 15% of the mutations that are at random affecting the trait, it’s 60% or 70% that are affecting the trait when we slide our statistic upward. This provides completely independent evidence that these sites are real, and as you slide above five, there’s no more enrichment. Our interpretation of these results—which we were able to validate and show made sense using computer simulations of our process—is that once you slide the statistic above five, essentially all the signals of natural selection are real.

Dwarkesh Patel

Okay, just to make sure I understand. You’re saying, in order to figure out what alleles have been under selection, your model assigns a statistic saying, “In order to explain why this allele has a specific frequency, we’re going to give it a selection statistic.” Independently, we run these studies on modern populations where we say, “If you look at height, eye color, intelligence, or whatever trait, what are the parts of the genome that are correlated with that trait?” The higher the statistic you give it in your study to explain allele frequency changes over time as a result of selection, the more probable it is that that region in the genome is associated with traits that have some functional thing we can measure.

David Reich

That’s exactly right. This is a brilliant idea that Ali had. It abandons the traditional approach of assigning statistical significance to mutations that cause a trait because we’re just using an external piece of information—the correlation to traits, measured in a completely different way—to read off the probability mutations are real.

We can ask how much enrichment for real signal is there given a particular selection statistic. If it’s halfway enriched to the plateau, we’re able to show the correct interpretation is that 50% of the mutations are really selected. If it’s three-quarters of the way toward the plateau, there’s a three-quarters probability that the mutation is real. If it’s 99% of the way to the plateau, there’s a 99% probability that it’s real. That gives us a calibrated estimate of the probability that a particular position is really under natural selection.

A major concern here is that what we’re actually seeing is not that these mutations are really under selection, but rather that both association to a disease and our selection signal are due to some third thing that’s causing both of them, which is a type of selection which is not what we’re after, not selection to adapt to new environments, but what’s called background selection: selection against newly arising bad mutations that are removed from the population that tend to be concentrated in genes. Genes are also the parts of the genome that tend to be associated to traits. This common process is causing both the enrichment for trait signals and is also causing the enrichment for selection signals that we’re observing. That’s the concern. We were super concerned about this.

So what we did is we repeated this enrichment analysis in slices of the DNA that all were affected to the same extent by background selection, by this rain of slightly bad mutations, and we get exactly the same pattern. We also repeated this experiment just using mutations of the same frequencies because there’s different statistical power to detect these signals at different frequencies. We see the same pattern where above a value of the selection statistic of around five, we get this plateau.

Dwarkesh Patel

The thing that changed that allowed you to increase the amount of sequences you’re generating by two orders of magnitude is just the statistical method you’re using to identify which part is human? Or what exactly changed in 2014 and since then?

David Reich

There’s been a whole series of improvements. The big ones have been the huge drop in sequencing cost, which made it possible to generate ancient DNA in the first place. The drop in cost has been a millionfold since the late 2000s, and another maybe one to two orders of magnitude from 2010 to today. That’s one big change.

Another change has been in-solution enrichment. It’s been this way of taking a sample that has very small percentages of human DNA, but then suddenly creating a process that will mean that the great majority of the sequences that one’s analyzing will be useful for analyses. The approach that we used was we took the DNA samples that we had, most of which were very low percentages of human DNA—less than 10%, often less than 1%—which is such a low proportion that it’s prohibitively expensive to sequence them and to just brute-force sequencing them given the technology that we had available at the time.

We took these samples and washed them over an artificially synthesized set of short DNA fragments that targeted positions of the DNA that we were interested in analyzing. This is more than a million positions that are highly variable in people, and we picked many of these to be biologically interesting. We had a whole set of known biological targets that affected traits in genome-wide association studies, which is the way that people look to see if there are particular genetic variants in modern people that have particular impacts in phenotypes and traits.

And so, what we did is we had this artificially synthesized set of DNA fragments that we washed our ancient sample over, and it bound the parts of the DNA that we targeted. The resulting sequence that we generated was very enriched for the parts of the genome that were informative about history. Even though only 10% or 1% of the DNA was human, it ended up that a very large fraction was from the parts of the genome that we were interested in, and it became economically efficient to do it.

Dwarkesh Patel

What was the other 99% of the DNA?

David Reich

It’s mostly microbial. It’s from bacteria and fungi that colonize a person’s body after they die. Depending on how they die, there’ll be more or less of these bacteria and fungi. When you typically sequence DNA from a person, it’ll just be full of microbial sequence. Sometimes the microbial sequence is very interesting; it might be pathogens that a person died of. There’s amazing work, for example, about different plagues of malaria and Black Death and hepatitis B and so on that have been obtained from the sequences of these pathogens in people’s teeth and other parts of their body when they died.

But we’re focusing here on the human DNA. This changed the amount of data that was possible to produce from tens per year to hundreds per year, and then we further roboticized and industrialized the process so that there were many hundreds or even thousands per year. Just in our laboratory, we’ve been generating genome-scale data from more than 5,000 individuals per year. I know this is true also of several other laboratories in the world now.

This huge jump in data, this semi-exponential or even super-exponential jump in some cases, has made it possible to ask and answer questions. While we were only on the order of 10 genome sequences from humans in 2010, this year it’s passed more than 20,000 reported sequences. There are several orders of magnitude increase, and the questions we were able to ask in 2014 are just not the same as the ones we can ask today.

Dwarkesh Patel

Awesome. Excellent. David, thanks for your time.

David Reich

Thank you, Dwarkesh.

It’s shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk.

It’s a bit technical, but I encourage you to hang in there – it’s really worth it.

There are less than a handful of people in the world who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him.

Wrote up some flashcards and practice problems to help myself retain what Reiner taught. Hope it’s helpful to you too!

Reiner is CEO of MatX, a new chip startup (full disclosure - I’m an angel investor). He was previously at Google, where he worked on software efficiency, compilers, and TPU architecture.

Highly recommend the scaling book he coauthored for further study.

Watch this one on YouTube so you can see the chalkboard.

Sponsors

• Jane Street needs constant access to incredibly low-latency compute. I recently asked one of their engineers, Clark, to talk me through how they meet these demands. Our conversation—which touched on everything from FPGAs to liquid cooling—was extremely helpful as I prepped to interview Reiner. You can watch the full discussion and explore Jane Street’s open roles at janestreet.com/dwarkesh

• Google’s Gemma 4 is the first open model that’s let me shut off the internet and create a fully disconnected “focus machine”. This is because Gemma is small enough to run on my laptop, but powerful enough to actually be useful. So, to prep for this interview, I downloaded Reiner’s scaling book, disconnected from wifi, and used Gemma to help me break down the material. Check it out at goo.gle/Gemma4

• Cursor helped me turn some notes I took on how gradients flow during large-scale pretraining into a great animation. At first, I wasn’t sure the best way to visualize the concept, but Cursor’s Composer 2 Fast model let me iterate on different ideas almost instantaneously. You can check out the animation in my recent blog post. And if you have something to visualize yourself, go to cursor.com/dwarkesh

Timestamps

(00:00:00) – How batch size affects token cost and speed

(00:32:09) – How MoE models are laid out across GPU racks

(00:47:12) – How pipeline parallelism spreads model layers across racks

(01:03:37) – Why Ilya said, “As we now know, pipelining is not wise.”

(01:18:59) – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal

(01:33:02) – Deducing long context memory costs from API pricing

(02:04:02) – Convergent evolution between neural nets and cryptography

Transcript

00:00:00 – How batch size affects token cost and speed

Dwarkesh Patel

Today, I’m interviewing Reiner Pope, who is the CEO of MatX, which is a new chip startup. Previously, he was doing TPU architecture and many other things at Google. This is a very different format from my usual interviews. This is going to be a blackboard lecture. We’re going to get up in a second. We in fact built this whole new studio with specifically this format in mind, so it’s a pleasure to get to inaugurate it with you.

We’re going to be talking about model architecture, ML infra, and many other things. The reason I think it’s an important topic is because once you understand how training and inference work in a cluster, a lot of things—about why AI is the way it is, why AI architectures are the way they are, why API prices are the way they are, and fundamentally why AI progress is the way it is—start making sense. You need to understand the details to get there, and you need a blackboard to understand the details. Reiner, thank you so much for doing this.

Reiner Pope

Very happy to be here.

Dwarkesh Patel

Full disclosure, I am an angel investor in MatX, but that’s unrelated to this podcast. Reiner, to kick us off I’ll ask this question. We have a couple of companies like Claude and Codex and Cursor offering something like Fast Mode, where for 6x the price, they’ll stream you tokens at 2.5x the speed. Mechanically, I’m curious what’s going on here. Why is it the case that you can pay more to get faster latency?

Two, could you keep going? Could you pay 100x more and somehow get much faster speeds? Three, could you go the other way? Could you have something like Claude Code “Slow Mode”, where if you are willing to wait for minutes on end, you could get even cheaper prices? Maybe this will help motivate the analysis that you’ll be doing through the lecture.

Reiner Pope

Great. To jump to the conclusion a little bit, the big effect is batch size. What we’re going to do now is quantify exactly what that looks like and what its implications are on latency and cost. There’s another effect, which you can call speculative decoding or multi-token prediction. We can maybe come back to that later, but the first thing that we’ll talk through is batch size.

What I’d like to introduce is the two principles of analysis. First, we’re going to look at a roofline analysis of how we run a transformer model on a cluster of chips. We’ll take a Blackwell NVL72 cluster, so a rack of 72 GPUs. The roofline analysis means we look at memory bandwidth and compute performance. The other side of that is that we’re going to look at just two simple factors of the model: the time to operate on the weights, and the time to operate on the context, the KV cache.

Let’s jump in. We’re going to try and estimate the time that it takes to run an inference of a certain shape. We’re not perfect here. We can’t exactly predict the time, so instead we’re going to approximate. We’re going to say that the time must be greater than or equal to a certain quantity. We’re going to consider two different aspects: the time it takes to do the memory fetches, and the time it takes to do the compute. It will turn out that this gives us very strong predictive power, even with a simple model.

One by one, what is the time that it takes to do the compute? There are really two things I need to do in the compute. I need to multiply by all of the active parameters, and then I need to do some work on the attention. Multiplying by all the active parameters, I have a certain batch size that I’m running, and I’ve got a number of active parameters in my model. Then I’m just going to divide this by the compute throughput, which is the FLOPs of the chip. This is a hardware concern.

This accounts for all of the compute time for all of the weight matrix multiplies. There’s a little caveat here. We’ve ignored the time to do any of the attention computation, but that in general will be quite small in comparison to this. So we’ll ignore this.

Dwarkesh Patel

I’ll just interrupt from time to time to ask some very naive questions or to clarify some basic points. For the audience, you’re not serving one user at a time. The batch refers to the fact that you’re serving many different users at the same time, and that’s a whole batch.

Reiner Pope

I can motivate the batch at least a little bit. We will see exactly why batch is such a favorable optimization. What will turn out to be the case is that if you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together. We’ll be able to see that quite explicitly.

Then, number of active parameters. If I look at, for example, a DeepSeek model, the DeepSeek V3 model has about 37 billion active parameters, and 700 billion total parameters. We’re focusing on just the ones that are active for a single AI token.

We’re modeling compute performance. I’m going to keep writing equals, but in all of these cases, you can think of this time as being at least this much, and maybe there will be some terms we ignored.

On the memory side, what do we need to do with memory? We need to fetch all of the weights, so there is some time to fetch the total number of parameters, not just the active parameters. There’s weight fetch time, and then in addition, there’s a KV cache fetch time. This actually depends on batch size. For every element of the batch, we have to fetch an entire context length worth of tokens, and there’s a size per token, bytes for one token. This is a model parameter.

Dwarkesh Patel

Maybe just backing up, let’s explain what the KV cache is real quick.

Reiner Pope

When I do a forward pass… Let me draw how the autoregressive inference works. This is during decode. If I have a bunch of text tokens… I’m drawing a tensor because ultimately the tokens are represented as a tensor in some embedding dimension. In this direction, I have the sequence length.

The work of running a decode is that I have to run each token through a whole bunch of matrix multiplies over a bunch of different layers. In general, I’m going to have to do that work over all of these tokens. But one step of decode is to produce just this one additional token up here.

What I’m going to do there is run a full forward pass of multiplying by all of the weight matrices in the entire model. But then I’ve got this attention mechanism where this token is looking at all of the past tokens, and what is it looking at specifically? It is looking at some internal representation that the model has produced of the tokens, and we call that the KV cache. This process of this single token attending to all of the history of tokens is attention. It is mostly dominated by memory fetches rather than matrix multiplies.

So we’ve got the amount of memory that we’re fetching shown over here, and then this is of course just divided by the memory bandwidth, so the memory bytes per second. In fact, these equations here are enough for us to now draw some fit lines. The things that we’d like to look at are sensitivity to batch, and then also, which we’ll draw separately, to context length. We said that the big effect you can get is some trade-off in latency versus cost in batch size.

Let’s draw them out. I think there are just really two graphs that we want to draw. We’ll first draw batch size versus time here. When we look at the shape of this, we’ve got a maximum of the sum and then another term. Let’s look at these terms one by one and how they scale: the time for compute and memory, and how they show up.

Let’s first look at this compute time. This is just purely linear in batch size with no offset, so it is some curve like this. This is t compute. On the memory side, we’ve got some portion here that is just this constant in some base offset here, which is the weight fetch. Finally, we have this term here, which is the KV fetch, which is pretty linear in batch size, and so it looks like that. The sum of this plus this maxed with this… Let’s at least first draw the sum. The two memory times in conjunction end up looking on this curved slope like this. Then the overall maximum is—I’ll draw a little thicker here—the maximum of these two curves.

What does this mean? This is a latency plot. If I grow my batch size, initially I get some not very strong dependence on batch size, so there is some lower bound on latency here. This already partially answers the question. For a given hardware configuration—and we can talk about varying the hardware configuration—there is a lower bound on latency. It is simply that I need to read all of my total parameters from memory into the chips, and that takes a certain amount of time. If I use all of my memory bandwidth, I can’t do any better than that.

Dwarkesh Patel

It seems like the way you’ve drawn the slopes for compute time and how the KV grows—and what implication the KV has on memory time—

Reiner Pope

What if this were above or below?

Dwarkesh Patel

Yeah, is that necessarily the case? If this is always true, then as batch size grows compute always dominates KV, which suggests that if you have a big enough batch size, maybe memory is never an issue.

Reiner Pope

This is really sensitive to the context length, so I think we should come back and explore this. As you vary the context length, the KV fetch time will go up and up, and that will cause a transition from compute-limited to memory-limited.

Dwarkesh Patel

Is there something especially significant about the slope being exactly the slope of the compute time?

Reiner Pope

Whenever we have balance points, it says that you’re getting it exactly right. For the particular context length where the slopes match, that says I am equally memory-bound and compute-bound, which is a really desirable place to be.

Dwarkesh Patel

This is a very simple algebra problem, but suppose the optimal is 100K context length, and you go to 200K context length. Does your MFU go down to 50%? Does it have a humongous impact on MFU to be slightly outside of the optimal context length range, the Goldilocks zone?

Reiner Pope

That’s right. That is true as modeled here. There is a key point here that I’m modeling the memory fetch as linear in context length. That depends on model architecture. It is true for all of the model architectures with dense attention. Sparse attention actually scales much better than that.

Dwarkesh Patel

Got it. Is sparse attention what everybody uses in practice?

Reiner Pope

I’m pretty excited about sparse attention. It’s hard to know what the labs are using. DeepSeek has published a sparse attention mechanism. I’ll just put a plug in that some of the DeepSeek papers that have published sparse attention end up putting a square root in this term.

So far, we’ve looked at the latency. It’s hard to read off cost from this. If I think about what cost means… To run this inference, I’m going to use the GPU for a certain number of seconds, like one millisecond or 20 milliseconds. I have to pay the rental time for that time. So it’s $2/hour per GPU or something like that.

That’s the cost of this inference, but how many tokens have I processed during that inference? That is the batch size. What we actually want to plot is the cost versus batch size, which is t over B versus batch size. This is the cost per token. We have to imagine dividing each of these three curves by B, so multiplying by this reciprocal. What we end up with there is… The compute curve was linear. We divide by B, and that makes it a constant here. This is t compute. The KV fetch was linear, and now it becomes a constant as well. Then the weight fetch was constant, and now we’ve divided by B, so it becomes this hyperbola.

Again, we’re going to compute the max of the sum. The sum of these two terms shifts the hyperbola up. The sum of the KV fetch and the weight fetch gives us a higher hyperbola that’s like this. Then we’re going to take the max with the compute here. We end up with this being the overall shape that we care about.

Again, we see some limiting behavior. The cost initially starts very high at a batch size of one. It almost goes to infinity because we’ve got so many weight fetches that are not amortized over a large batch size. But as we increase the batch size, the weight fetches become amortized over so many different batch elements that their cost grows very small, and eventually the compute time ends up driving the cost. So there is a limiting lower bound on cost, which is this line here.

Dwarkesh Patel

So Claude Code Slow or Codex Slow or whatever would just live on this line. It wouldn’t help much because you’re not able to amortize the KV values over a much bigger batch.

Reiner Pope

They’re unique per batch. The compute is also unique per batch. So what is the minimum work you can do per batch after amortizing everything else away?

Dwarkesh Patel

This point where you are no longer memory bandwidth bound, practically how big a batch do you need? How big are the batches practically for frontier models?

Reiner Pope

You can just solve for that. It’s not even particularly sensitive to model architecture. Let’s go ahead and do that.

What we’re talking about is when the memory time is equal to the compute time. That’s what that question is. Because we’re focused on what the batch size is—and really there’s a question of when the weights are amortized over the multiplies—I’m going to focus on comparing the weight fetch time to the weight multiply time. I’m going to disregard the KV fetch term just to simplify the analysis so we can get a clean answer out. We’re going to equate this portion with these two times.

Writing that out, we get N, number of total parameters, over memory bandwidth, is equal to batch size times number of active parameters divided by the compute performance. Looking over here, everything on the top are model parameters. Everything on the bottom are hardware parameters. It turns out to be nice to rearrange them such that we have the hardware parameters on one side.

This is equivalent to FLOPs over memory bandwidth being equal to batch size times number of active parameters, divided by the number of total parameters. This hardware parameter ends up being a dimensionless constant. If you look in terms of FLOPs… What are the dimensions of this? This is multiplies per second. This is bytes per second. So that’s not quite dimensionless. But what you do is you say, how many FP4 multiplies per second times the fact that each FP4 is half a byte. I can actually make this end up being dimensionless. On most GPUs, this ends up being somewhere around 300.

Dwarkesh Patel

Has that ratio changed over time as we’ve gone from model generation to model generation, where the FLOPs keep increasing?

Reiner Pope

This is a hardware parameter. To what extent has the hardware changed? From A100 to H100 to B100, the FLOPs have increased substantially, the memory bandwidth has also increased substantially, and it has remained reasonably stable.

We can express this one as well. This is a sparsity parameter. I might even phrase this slightly differently. Let’s solve for batch size in total. Moving this back over to the other side, we end up with batch size needs to be bigger than approximately 300 times sparsity. For example, in DeepSeek I activate 32 out of 256 experts, so this would be 8 for DeepSeek.

This actually gives you a ballpark which is remarkably accurate to practice. Generally, people will go a little bit larger than this. They don’t really want to be exactly at the balance point because real-world efficiencies aren’t as good as a roofline analysis would say. But take this and maybe double or triple it.

Dwarkesh Patel

Okay, so it’s two to three thousand tokens per batch. But then if you included the KV cache, the implication would be that the optimal batch size...

Reiner Pope

Should grow larger. We solved for the equivalence between when compute time is equal to memory time. If I add in something that consumes more memory bandwidth, then I have less available for the weight loads. I need to grow the memory bandwidth more, and therefore the batch size more.

Dwarkesh Patel

This seems incredibly small. This would be less than one sequence, right?

Reiner Pope

Keep in mind that I’m talking about the number of tokens that I’m generating one more token for. It’s actually 2,000 unique sequences.

Dwarkesh Patel

Got it. We’re just talking about a single forward pass on these sequences. You think of the batch as the number of sequences.

Reiner Pope

That’s right.

Dwarkesh Patel

If you’ve got a frontier model and you are actually doing inference, surely they must have more than 2,000 concurrent users. Is there any added latency from the fact that you need to have the whole batch fill up? Or if you have a reasonable amount of users, is it so unlikely that it would take you 100 milliseconds to fill up the next 2,000 slots?

Reiner Pope

The way to think about this is: when does the train depart, as a model? Let’s say I’ve picked a batch size that I’m going to run at. By the way, this intersection point is the same intersection point here. I pick this batch size, and I know that it’s going to take, for example, 20 milliseconds, which is a common place this ends up landing.

This is a timeline of what is running on the GPU. It’s going to start a new batch every 20 milliseconds regardless. You can think of this as a schedule for the train. A new train departs every 20 milliseconds. Any passengers who are ready board the train. If the train is full, they wait until the next train. If the train is not full, the train is going to go anyway.

In terms of what that means for queuing latency, the worst case is that a request arrives just after the train departed. It has to wait for the next train, so that’s up to 20 milliseconds, and then it has to wait for that train to complete. So the worst-case latency is 40 milliseconds.

Dwarkesh Patel

How is the 20 milliseconds derived?

Reiner Pope

It’s a rule of thumb, but where it comes from is not fully explained yet. So far we’ve focused on memory bandwidth and compute time. When we look at memory, the other consideration is that we want to use all of the memory capacity we have. Generally, we’re going to use all of that memory capacity to store the weights or the KVs. In the time of doing a forward pass, we want to read all of the memory capacity into the chip. That is capacity divided by bandwidth. That tends to be 20 milliseconds on many different generations of HBM.

Dwarkesh Patel

The units make sense. You would have a byte divided by bytes per second.

Reiner Pope

For example, on the Rubin generation, it is something like 288 gigabytes divided by 20 terabytes per second. This comes out to about 15 milliseconds.

Dwarkesh Patel

Let me make sure I understand what this is saying. I understand the unit analysis. What it’s saying is we can evacuate and replace the HBM in this amount of time. So we don’t want to be in a situation where the HBM is not big enough that we’re not actually able to write everything we want to it or take everything out of it. Or we don’t want to be in a situation where our ability to write back and forth is so small compared...

Reiner Pope

There are sort of two scenarios. Why don’t we pick a latency that is bigger than 15 milliseconds? If I think about what that means, it means I actually have time to read the HBM twice. By the way, most HBM accesses are reads, not writes. It’s almost all reads because the weight matrices are read-only, and almost all of the KV cache accesses are reads. In around 30 milliseconds, I can read all of HBM twice, but what’s the point of that? I don’t want to read the weight matrices twice. I don’t want to read the KVs twice.

Dwarkesh Patel

Makes a ton of sense. A couple of quick questions. If it is the case that the optimal batch size is something like 2,000, it’s totally dependent on the sparsity, not dependent on the model size or anything.

Reiner Pope

Sparsity shows up in model size, but beyond that, it only depends on sparsity, not on scale.

Dwarkesh Patel

That’s a very interesting result. One question is how much of a push towards centralization is it that you would have these economies of scale from inference for batching? But it seems like it’s not that big a deal. Is 2,000 users at the same time a lot? It doesn’t seem like a lot.

Reiner Pope

We can do a bit of analysis on this. You can think of it in terms of number of users, but a more productive way to think of it is in terms of tokens per second. What does this batch size mean in terms of tokens per second of the system?

Tokens per second is going to be equal to the batch size. We run a batch of tokens, and we do that every time interval, which is equal to the 15-millisecond or 20-millisecond number. This ends up being batch size times about 60, so 64 x B. This ends up being around 2,000 x 64, so 128,000 tokens per second. This is in more digestible units.

It’s hard to reason about concurrent users, but what is the global traffic for a system? When you look at some of the announcements, sometimes the API providers will brag about how much traffic they have. The numbers I remember from some announcements of Gemini last year were in the hundreds of millions of tokens per second worldwide. This is one-thousandth of that.

Dwarkesh Patel

Gemini is big. One-thousandth of Gemini is a lot. To actually be competitive at scale, you need to be able to serve at least one-thousandth of Gemini. That’s interesting.

The more sparsity you have, the less compute you need. It does seem that as batch sizes get bigger, compute ends up being the bottleneck, according to this analysis. Then the question is, how far can you take sparsity? As the sparsity ratio increases, as you have fewer active parameters relative to total parameters, how much is the performance of the model degrading? Is it degrading faster than you’re saving compute by increasing the sparsity factor?

Reiner Pope

You mean the quality of the model, rather than the speed of the model. Unfortunately, we’re not able to answer that analytically. That is an empirical question of model quality. The best I can do is pull up a paper and answer that empirically.

Dwarkesh Patel

Should we pull up the paper now?

Reiner Pope

This paper is “Unified Scaling Laws for Routed Language Models.” It’s a somewhat old paper by this stage, but one of the things they looked at is if I keep increasing sparsity, what is the model quality impact? This answer is very sensitive to the actual choice of mixture of experts. Mixture of experts has been around for a really long time, maybe even back in 2017, but the techniques have changed a lot. DeepSeek’s mixture of experts was a big change in how it worked. There have been older papers, like “GShard” and “Switch Transformer”. The actual empirical results are going to depend on all of that.

On one of the older techniques shown here, you can see if I hold constant the number of active parameters at a certain size, and then I increase the sparsity, which they call expert count, the quality keeps increasing. If you imagine drawing a horizontal line from 1.3B dense across, you end up seeing that, in this case, the 64-expert 370-million activated parameter model is as good as a dense 1.3-billion model.

Dwarkesh Patel

So in some sense, it’s actually not amazing returns where you need to increase total parameters a hundredfold to get the equivalent of 10x as many active parameters.

Reiner Pope

Actually even more so. It’s a huge increase in parameter count for a modest increase in efficiency.

Dwarkesh Patel

So in this case, actually it’s 4x?

Reiner Pope

64x for 4x.

Dwarkesh Patel

So while it is true that you get this benefit of being able to economize on your compute time if you increase sparsity, naively it would seem like a trade-off worth making. But if you’re decreasing this by 2x and then having this go up by 8x every time you double sparsity...

Reiner Pope

Is that good or bad, actually? Even from a memory point of view… Keep in mind you are doubling this portion of the memory fetches, which is amortized by batch. So just keep running a larger batch size. From the point of view of the analysis we’ve done here, this is a pure win. Keep doing it until you run out of available users, basically.

There’s this equivalence where if I have a lot of users, I can go to a much sparser model. From that point of view, it’s a reasonable trade-off. The other trade-off that shows up here is that it also consumes memory capacity. We’ve only reasoned about memory bandwidth here, but it also consumes memory capacity.

Dwarkesh Patel

I see. Let me make sure I understood. You’re saying we want to spend less time computing, therefore we do more sparsity. To make that work, we need bigger batch sizes. Which means we need more memory capacity to have more sparsity.

Reiner Pope

Maybe this would be a good point to talk about how a mixture of experts layer is typically laid out on a rack of GPUs.

00:32:09 – How MoE models are laid out across GPU racks

Dwarkesh Patel

Cool. Makes sense. Where were we?

Reiner Pope

Sparse mixture of experts. Maybe how we lay that out on a GPU.

Let’s zoom in on the mixture of experts layer first and draw what that looks like. Typically, we’ll have some kind of a router layer, which is making the decision of where we route the tokens to. We get tokens coming in here, they go through a router layer, and then we have a bunch of different experts. I’ll draw a few more to line some up.

The router will make a decision of which experts it’s going to route to, and it will be a small fraction of them, maybe 1 in 32. Maybe it will make a decision to route to this one, maybe this one, and maybe this one. Each expert itself is a normal MLP. It has an up projection and then a down projection with a nonlinearity in between.

Then finally, we do the inverse operation. Where we were broadcasting things out here, we’re going to bring them back in and sum them up. Bringing them in like this. Then finally, we have our residual connections. The token is also passed through here, and it gets added to the result of the MoE layer. This is a normal MoE layer.

What I want to talk through is how this is mapped to a GPU rack and what this means for communication, because I think this will start to show some of the limits of how sparse we can go. The standard practice here, and it is the best solution, is to use expert parallelism. That means different experts go on different GPUs. If we take something like a DeepSeek model, they have 256 experts. Let’s say we want to run that on a Blackwell rack. There are 72 GPUs.

We have a divisibility problem. This is not a power of two. We’ll just simplify and say we’re only going to use 64 of them. Just ignore the other eight. It’s not a big deal. So we have four experts per GPU. Very simple. For the sake of the diagram, actually let’s just say we have two experts per GPU. We end up just putting these GPU boundaries. Every pair of experts is on its own GPU.

Then we can look at the communication cost. We had some tokens stored centrally here. They get routed to all of these experts, and there is some communication cost paid here. There’s the same communication cost paid on the output. The hope is that this does not become communication limited.

Now what is the traffic pattern here? The traffic pattern here is that any GPU will be talking to any other GPU, depending on the decisions made by the model. This is an all-to-all traffic pattern.

Dwarkesh Patel

When you say any GPU in the pre-tense, the router is more than one GPU?

Reiner Pope

I drew this as one router. In reality, you would actually have many copies of the router, and you would have as many routers as GPUs, in fact.

Dwarkesh Patel

As the incoming traffic.

Reiner Pope

Yeah. These are 64 GPUs and these are 64 GPUs. It’s actually the same GPUs, we just draw them as separate because they’re serving different purposes. So at this point, any GPU can be sending to any other GPU.

This all-to-all pattern of communication that shows up and how the Blackwell racks are configured is a perfect fit for the communication pattern that the MoE actually wants to do. However, if you think maybe one rack is too slow and I want to do two racks, then I have this challenge that maybe I’ve got some sort of rack boundary drawn outside here like this, and I no longer have all-to-all communication between all the GPUs in two racks. The rack-to-rack communication ends up being a substantial bottleneck.

The fundamental thing here is that one rack bounds the size of an expert layer you can do. This has been part of what’s been driving towards larger and larger interconnect domains.

Dwarkesh Patel

Before we continue, it may be worth you explaining what exactly a rack is. The differences in bandwidth between a rack and within a rack, and the all-to-all versus not all-to-all nature of communication within versus outside.

Reiner Pope

This is a place where it starts to be very different between Nvidia, for example, and Google, and then others, including us. Generally, a rack is a physical structure. It’s a few meters tall, a meter or two wide, depending on configuration, and it stores some number of GPUs or XPUs, which is typically about 64.

What constrains it being a certain size is power delivery, weight, and cooling ability. It ends up being about this size in many cases because of these physical constraints. When I deploy a data center, a data center may have thousands of these racks. I’ve got one of these tall racks, it’s got a bunch of GPUs in it, and so on. And then I put another rack next to it.

Dwarkesh Patel

You make it sound so easy.

Reiner Pope

Right. I just drop them in. In Nvidia’s case, the communication topology… They actually put the GPUs on the outside of the rack, and then they put these switches on the inside of the rack. What this ends up being is that there’s a set of switches in here. These are the NV switches. Then they run a bunch of cables. Every single GPU has cables going to the switches in the middle. The switches have connections to all the GPUs. All of the GPUs can talk to all the other GPUs in just two hops: going to the switch, going to the other GPU.

Now, when I want to leave the rack, I end up going via a different path. The GPUs also have a much slower connectivity, which is typically about eight times slower. The green that I drew here in the GPU cases is the NVLink. More generally, it’s called the scale-up network. You will typically also have a scale-out network, which allows you to connect to some data center switch. All of the GPUs will have some connectivity up to some data center switch somewhere. This is the scale-out, and it tends to be about 8x slower in bandwidth.

The challenge, if you want to lay out a mixture of experts layer across two racks, is that half of the GPUs here are going to be wanting to talk to the GPUs here. On average, when I look at where the tokens on these GPUs want to go, half of the tokens want to go inside the rack. That’s great. They can use the fast scale-up network. But half the tokens are going to want to leave the rack and go to the other rack, and that’s not as good. They need to use a much slower network, and so that becomes the bottleneck on the all-to-all pattern.

A different choice would be, why don’t I have a big switch here and connect everything to a much bigger switch that actually combines the two racks together? There are many ideas in this direction, but in general, the reason you have this hierarchy of switches rather than one big switch is to manage the cabling congestion. You just need to run a large number of cables.

Dwarkesh Patel

Sorry, is that question you just asked basically, why isn’t it a bigger scale-up?

Reiner Pope

Exactly. Why not just have a million chips in scale-up or a thousand chips?

Dwarkesh Patel

What has changed that has allowed Nvidia to go from Hopper, which was 8, then Blackwell is 72, and now Rubin will be... is it 500 something?

Reiner Pope

Yeah, 500 and something.

Dwarkesh Patel

What has allowed that to happen?

Reiner Pope

From Hopper to Blackwell is mostly just the decision to switch from trays as the form factor to switching to racks as the form factor. That’s a product decision. There wasn’t a substantial technical barrier there.

Switching from 64 to 500 or so, there’s a bit of Jensen math there, but there is at least a genuine 4x increase, which is coming from a much more complicated and difficult rack design. That is actually a new physical design to run more cables.

Dwarkesh Patel

The cable complication is just the cost of figuring out which cable hops to which, or which signal goes from what to what?

Reiner Pope

Let’s zoom in on this and look at the wire density. I’ll draw this diagram just once more so we have a bit of a cleaner and larger version to work with.

Let’s say I have some switches in the middle. Initially, I’m going to start with just two GPUs on each side or two trays of GPUs on each side. Let’s say maybe each tray wants to have two cables coming out of it. I physically run vertical cables that look like this running out to the switches. Now if I want to double the number of GPUs in a rack, I need to run literally twice the density of cables. I need to run these as well.

Dwarkesh Patel

Extremely naive question. But if you look at a physical data center, it seems like there’s a lot of space within a rack. I don’t know. The cables are really big and...

Reiner Pope

There is space outside the rack. Inside the rack… As they become more optimized, these racks are very tight. There’s connector density going from the tray into the rack and the rack’s backplane, and the backplane itself has a really high density. There are other physical constraints including the bend radius of cables. You don’t want to snap them and so on.

Dwarkesh Patel

Okay, so it’s literally the physical space to put a cable that’s constraining it. I had no idea. Interesting. That seems surprising. The rack is so big and we can’t just stuff more cables in there.

Reiner Pope

Rack design is not my expertise, but when I talk to folks on what constraints they’re up against, it’s a combination of things. What are the big physical things you’re optimizing for? Space, weight of the rack. It’s actually really heavy, so you need enough metal to not sag and fall. But then you add more metal, and it’s heavier. Then power and cooling. All of those are competing. Modern racks are pushing all of those to very extreme physical limits.

Dwarkesh Patel

When was GPT-4 released again? Was it 2022 or 2023?

Reiner Pope

2023.

Dwarkesh Patel

Okay. And it was rumored to be over one trillion parameters. It seems like only now, within the last six months, have models been getting released that have significantly more parameters than the model released three years ago, when supposedly there should have been this scaling in the meantime.

Is the reason that we were just waiting for racks with enough memory to hold a five-trillion parameter model, along with its KV cache for enough users for a lot of sequences? Or if you’re doing RL, a similar consideration of actually holding the KV cache for the batch of problems you’re trying to solve.

If you look at Hopper, you had eight Hoppers, and I think that’s 640 gigabytes as of 2022. With Blackwell finally, which was deployed in…?

Reiner Pope

Very recently. Maybe last year.

Dwarkesh Patel

Last year. You finally have a scale-up on the order of 10-20 terabytes, which is enough for a 5T model plus KV cache.

Reiner Pope

Deploying in larger scale-up domains is a huge unlock. I’ve drawn here the Nvidia Blackwell deployment. The Google deployment has actually had very large scale-up domains for a long time.

Dwarkesh Patel

That also explains why Gemini seemed to be ahead. It just seems like Gemini has had successful pre-training for longer than some of the other labs.

Reiner Pope

Not having been there at the time, I’m not sure how much is coming from successfully deploying higher sparsity ratios, which it could be. It could also be a whole bunch of actual modeling things, specifically how you do the mixture of experts. We’ve seen the DeepSeek mixture of experts activate more experts, but finer-grained experts. That was a big innovation. I’m sure there are many other innovations on the model architecture as well as on the training data.

It’s hard to disentangle all of them, but what shows up in terms of the limits of what you can do is that the active parameters, as we saw, are limited by the compute cost, and the total parameters are limited by the scale-up size.

00:47:12 – How pipeline parallelism spreads model layers across racks

Dwarkesh Patel

When you’re operating within a single scale-up domain, is that a consideration specifically for either forward or backward, or specifically for prefill versus decode? Or is it preferred to always be within a scale-up whatever kind of workload you have, whether you’re doing a pre-training run, RL generation, or inference for users?

Reiner Pope

Really interesting. To answer that question, we’re going to need to talk about the communication patterns. We’ve talked about the mixture of experts communication pattern. That is this all-to-all. All-to-all very strongly favors full connectivity, which is what we’ve just shown here, and it favors being within one rack.

There are other kinds of parallelism besides expert parallelism, which we just showed here. In the literature is tensor parallelism. With the trend towards smaller experts, this has become much less relevant, so we can ignore that. But the other two things we have available are data parallelism and pipeline parallelism. They can be a much better fit for using multiple racks.

Let’s focus on pipeline parallelism specifically. This is one layer of MoE. I’m going to have a hundred more layers up above. I could decide at this point, for example, to move to a different rack, change rack. Now, is that going to become a communication bottleneck? We can actually solve for when this becomes a communication bottleneck. Before we do that algebraically, let’s visualize it out and sketch the path. We’re going to have another MoE layer, and another MoE layer here, and so on.

Let’s say I change rack here, and then some number of layers later, I change rack here as well. The methodology we’re going to use to determine whether we have a communication bottleneck at the point where we change rack is we’re going to compare the scale-out bandwidth requirements to the scale-up bandwidth requirements. Let’s write this. The hint is going to be that there’s a lot more sends here. We’re sending many things here, whereas we’re only sending one thing here, and we’re also maybe doing it many times. That’s what makes the difference.

Dwarkesh Patel

Can I try to guess? Just out of curiosity, to see if I’m actually understanding, it seems like you’re sending batch size into the rack.

Reiner Pope

In here? Yes.

Dwarkesh Patel

But the communication within the rack is batch size times number of GPUs.

Reiner Pope

Number of activated GPUs. I don’t send to this GPU at all. There’s an explosion from 1-3x larger here in this diagram. The key thing is that I didn’t even need to send to this GPU at all, and so that’s a big saving.

We’re going to talk through to what extent scale-up is a bottleneck over scale-out. We will directly jump to the ratio of the time spent on scale-up over the time spent on scale-out. This is the quantity we’re talking about.

The first consideration is that scale-up is 8x faster than scale-out generally. At a baseline, if the bandwidths were the same, we would have this 1/8, which is coming from bandwidth. But then we have some amount of expansion in how much data we’re sending. If one token comes in here, then this one token gets routed to, in the DeepSeek case maybe 32 experts or 16 experts. It gets routed to some number of experts. So this is the number of activated experts. This same thing applies on multiple different layers, so maybe I’m going to run two layers. There’s also multiple times the number of layers per stage.

Dwarkesh Patel

Don’t you need to multiply the whole thing by two for the all-to-all?

Reiner Pope

For the up and down. Yes, there’s a factor of two. Thank you.

What we would like is for the scale-up time to be greater than the scale-out time, because the scale-up time is the more important and precious resource. We would like this number to be greater than or equal to one. This really doesn’t seem hard. There’s just a factor of 8 that we need to overcome. So we need the product of these three things to be bigger than 8. Typically we have a fairly large number of activated experts. It could be 8 by itself. Then we can increase the number of layers per stage a lot until we satisfy this.

What this ends up looking like is that I can have an entire pipeline of racks where one rack does one layer, and then I move on to the next rack and do another layer, and then I move on to the next rack and do another layer.

Dwarkesh Patel

It’s interesting to me that the best parallelism strategy in practice ends up being one which physically resembles the actual architecture. It’s not some galaxy brain thing. It’s like, “Oh, we have experts, we’re going to put them on different GPUs, or we have different layers, we’re just going to put them on different racks.” I feel that’s interesting.

Reiner Pope

The cutting matches the model architecture.

Dwarkesh Patel

Exactly. It could have been something wackier with tensor parallelism and whatever.

Reiner Pope

The galaxy brain way to think of it is, what are all the different dimensions in which a model is scaled up? It is scaled up by layers, it is scaled up by the model dimension, it is scaled up by the DFF dimension, it is scaled up by the number of experts. Every single one of those numbers you can choose to cut along. If those numbers are big enough, it eventually becomes profitable to cut along there. We have selected two of them. The other two, in the way models are typically sized, are not profitable.

Dwarkesh Patel

So there’s a talk by Ilya where he says, “Today we know not to do pipeline parallelism.” And Horace He gave my friends and me… I hate that it sounds like a Dr. Seuss quote. But he gave us a lecture on these different kinds of parallelisms. He said the problem with pipeline parallelism is that, other than the bubbles, it creates these architectural constraints. Kimi, for example, has these residuals where attention attends to layers a few back, so it becomes hard to implement in this way.

Reiner Pope

I guess we didn’t fully articulate even what is the benefit that we’re getting from pipelining. These complexities are real. Pipelining is a massive hassle, but it does give you some benefits. You can then decide whether those benefits are worth the costs. It has some benefits in inference, maybe bigger benefits in training. In inference, what are we saving on? Are we saving on memory time or compute time? Not really. We’re just moving the memory time from one chip to another chip, or one rack to a different rack. There’s no actual benefit in runtime.

However, what we are saving on is memory capacity. If we think that the memory in a rack is a bottleneck, then there’s a constraint on how fast we can go. Pipelining allows us to massively reduce that bottleneck.

Dwarkesh Patel

The opposite connotation to this… Before this interview, I was chatting with Axel, who’s a GPU performance engineer at Jane Street. He was explaining that to do pipelining, you have to do micro-batches rather than full batches. If you do micro-batches, then you’re by definition not able to amortize loading the weights across all the users or all the sequences. The positive connotation of that is you don’t have to use as much memory. The negative connotation is that we can’t amortize loading the weights across all those users. Maybe it’s worth explaining why you have to do micro-batches.

Reiner Pope

Shall we draw the pipeline bubble? What is this micro-batching that shows up in pipeline parallelism?

I’ll focus on inference first. It’s a slightly simpler problem. I’m going to draw time, and then which rack we’re on. The idea is that maybe I’ll have four racks. I’ve got an inference that is going to step through these four racks in some time like this. This is inference number zero. It runs at a certain batch size and steps through all the pipeline stages like this.

Now, if we were to say, “Well, we’re going to run inference number one here,” this is clearly a massive waste. Like three-quarters of the time each of the racks is doing nothing. We don’t actually run inference one here, we run it as soon as we can, which is immediately after inference zero finishes. And then we keep going. If we hadn’t filled this in, we would call this the pipeline bubble. When I’ve drawn it in this inference context where we’re only going in a forwards pass, it’s obvious. Why would you do this stupid thing? In a training context, it’s maybe less obvious. But in the inference context, it’s really natural to make this change.

Dwarkesh Patel

Oh, interesting. This is sort of obvious, but the difference between micro-batch and batch doesn’t matter at all in inference because you can just call it whatever you want. It only matters in training because there is an optimal batch size.

Reiner Pope

Yes.

Dwarkesh Patel

Before you do a full backward step, you want to have accumulated all the sequences in that batch. If you want to do pipelining in training, in order to avoid that bubble, you need to—

Reiner Pope

Should we draw the training diagram with that? Let’s do that. This is the inference diagram, and I’ll call this forward so we don’t have the wrong thing showing up there. Let’s do the same thing for training now. We’ve got a forwards pass, but at some stage we’re going to have to transition to a backwards pass.

We’ll do some number of batches in the forwards pass, and then we’re going to transition to the backwards pass for everyone all in one go. The inference part is the same here, but then we do a hard stop at this point and transition everyone to the backwards pass, with similar numbering like this.

Dwarkesh Patel

It may be worth clarifying the reason there is that hard stop is because you want to do a whole batch at once for the backward step. And then there is an optimal size for how big that batch should be.

Reiner Pope

Smaller is always better, actually, is a way to put it. From an ML convergence rate perspective, smaller is always better because you’re getting the freshest information from the gradient descent.

Dwarkesh Patel

But from a total training time perspective?

Reiner Pope

From a total training time perspective, smaller is worse from a systems perspective. The optimum is the trade-off between those two.

So you pick a batch size, and for that batch size, you do some amount forwards and then some amount backwards. You asked why there is even a hard stop there. With pipeline parallelism, because you’ve got this idle time here which is the bubble, there are so many techniques in the literature for how to lay this out differently and avoid that. There are more complicated schemes called zero bubble or one-forward-one-backward, which interweave the forwards and the backwards in complicated ways.

Dwarkesh Patel

You can mine Bitcoin in that bubble.

Reiner Pope

Right. More usefully, you can do the weight gradient step, but you can also mine Bitcoin.

In inference, the effect of pipelining on anything you care about, like batch size or latency, is neutral. It doesn’t improve it, it doesn’t make it worse. If you look at the latency of this inference, running it if it were pipelined versus if it were all on one rack… If it were all on one rack, we would just slide all the boxes down and still put them in a row, and the latency would be the same.

Pipelining is neither better nor worse for latency. It does mean that you just use less memory capacity per rack. Because now instead of needing the whole model, you only need a quarter of the model, and you can expand.

Dwarkesh Patel

Makes a ton of sense. So it’s a no-brainer to use pipelining during inference, but there’s this harder trade-off during training.

Reiner Pope

Even in inference, in fact, it is not used a ton. It reduces your memory capacity requirements, but there’s actually a huge surplus. I think you were saying that a rack of Blackwell has many tens of terabytes. That’s much bigger than a trillion parameter model. A trillion parameter model only needs one terabyte, so it already fits. There’s not a huge benefit from pipelining because you’re reducing a number that’s already pretty small.

But it does say that theoretically, maybe you had too much memory there. You could have built different hardware that has less memory. If you were designing your hardware, you could say, “I didn’t need that much memory because I don’t need the weights to fit in one rack. I can fit the weights in eight racks, then I could have built hardware that didn’t have so much HBM per GPU.”

01:03:37 – Why Ilya said, “As we now know, pipelining is not wise.”

Dwarkesh Patel

Macro question: everybody’s talking about the memory wall right now. Memory is getting super expensive. There’s not enough memory. Smartphone volume will go down 30% because there’s not enough memory. This is shocking, Dylan said hyperscalers are spending 50% of their CapEx this year on memory.

Reiner Pope

That’s believable.

Dwarkesh Patel

What is hyperscaler CapEx? That’s high hundreds of billions, maybe a trillion, and they’re spending half of that on memory? That is a huge constraint. That’s why we’re not going to get new laptops and phones this year.

But at the same time, we have too much memory? People are willing to put too much memory into these systems. Why is Jensen shoving all this memory into these racks if you don’t need it?

Reiner Pope

In the equations we had here before we erased them, we were doing memory time, memory bandwidth and compute bandwidth. Let’s now start looking at memory capacity.

We’ll start off with memory capacity without even thinking about a parallelism scheme. The demand on memory is the number of total parameters. This is what we need to fit the weights in some system that we are using. Then we need to fit the KVs as well. KVs go as batch size times the length of the context times the bytes per token.

What I was arguing about in this context, and the case I was making for pipelining, is that there are some techniques that allow us to solve this. Let’s consider running this on some number of GPUs. We’re going to have one extent, which is E, the expert parallelism. When we had this sharding of an expert layer across many GPUs, to what extent do we do that? How many GPUs? We’re going to say that this is, for example 64. Then P is going to be the extent of pipelining. This is the number of racks, maybe we’ll pick 4 or something like that.

This is the total memory requirement across the system, but now I’m going to calculate a memory requirement per GPU. I’ll use a lowercase c_mem. Obviously, we just take all of these numbers and divide it by E and P. Really easy. It’s this N_total, plus the batch times length of context times bytes per token, all divided by E times P.

Why is this correct as divided this way? We knew that the parameters were perfectly divided amongst all the GPUs in a rack. The layers are perfectly divided amongst the different racks. So that works here. Somehow we’re going to arrange—I’ll hand-wave exactly how—the same perfect sharding of the contexts across GPUs in a rack, and then based on layer across racks.

Dwarkesh Patel

Sorry, 4 is the number of racks?

Reiner Pope

Yeah, for example.

This is the place where we actually need to go back and analyze this batch size B. You were making this comment that there’s micro-batching versus global batching. Let’s come back to this pipelining diagram here.

We’ve got one batch going forward here, and then as I drew it, it kind of just disappeared. That’s not really correct. If you think about how decode is working, I have a bunch of tokens that I have generated already. I do one forwards pass where I generate a new token, and then I write that to my KV cache. Then I do another forwards pass that generates the next token. I’m actually going to be running this batch zero in a loop. In fact, I go forwards. Once I finish, I can start the next iteration of the loop up here. We’ll just fill this in. We’ve got the two, three, two and three, and two and three.

Let’s split this batch. This batch will be the global batch size. B is going to be the number of micro-batches times the batch size per micro-batch. How many micro-batches do we need? The number of micro-batches in this diagram is 4: zero, one, two, three. The micro-batch size is still this 2000-ish number. Sorry, no, this is the 300 times sparsity.

Dwarkesh Patel

This is how big the train that takes off every 20 milliseconds is.

Reiner Pope

Right. This is going to be the 20-millisecond train. The global batch size is the number of micro-batches times the local batch size. Local batch size is set by this hardware parameter.

The number of micro-batches is as small as possible, such that we can wrap around and not leave any idle time. If we had fewer, we would have this idle time when we wrap around. You can visually see that it is equal to the number of pipeline stages. It’s a proof by visual here. It is 4, and it’s 4 this way as well. You can look and see that it goes along here, and then it wraps around to the number of pipeline stages.

Dwarkesh Patel

Sorry, very basic question. Is this what is actually done? A frontier model today will have pipelining during inference?

Reiner Pope

For sure during massive scale training this is done. It can be done for inference. I’m actually going to make the case for why it is less attractive. It is useful for weights, but not so useful for KVs.

The big challenge is... Let’s fill this in. The micro-batch size here ends up being equal to the number of pipeline stages. When we go back and substitute all of that into here, we get a number of pipeline stages times this little b showing up in here. When we factor this out, I’m going to split this plus into two terms.

We get the full division by E times P over here. We still have division by E times P over here, but the Ps cancel. What we find is that if you increase the number of pipeline stages, the memory footprint for the number of weights keeps going down and down and down, but the memory footprint for the number of activations stays constant. So it doesn’t actually work.

Most of your memory… Once you do enough pipelining—and it’s really not much, even two is often enough—this term becomes very small. The KV cache becomes the dominant term.

Dwarkesh Patel

I know this is wrong. I’m just trying to think about why my train of logic here is wrong. If you’re pipelining through many different stages, the KV values are not shared between layers. Why would it not help to be pipelining across multiple layers? Because then you don’t have to store...

Reiner Pope

You only need to store one layer rather than two layers of KVs. It helps from that perspective, you’re right. What’s competing with that, though, is that you need to be keeping all of the racks usefully busy at a time, so the number of sequences that are in flight simultaneously has gone up.

Dwarkesh Patel

Ah, that makes sense.

Reiner Pope

Those exactly cancel, and you end up not getting a saving per GPU.

Dwarkesh Patel

Right. This is going back fundamentally to the point of how you’re not able to amortize across KV caches.

Reiner Pope

First, we established you can’t amortize KV caches across batch size. Now we’re saying you also can’t shard it across pipeline stages. It sucks from both of those points of view.

Dwarkesh Patel

Interesting. So then what is done during inference?

Reiner Pope

The DeepSeek paper reports what they do, which is that they just do a lot of expert parallelism. In effect, you should increase your expert parallelism up to your scale-up domain size, and then do very little pipelining. Maybe none at all, maybe two, just enough to make the weight storage not too big of an issue.

Those are the only two parallelisms that really make sense. In the past, there was tensor parallelism, which was cutting up within an expert, but the experts are so small now that that is not a profitable optimization.

Dwarkesh Patel

Does that mean that frontier labs, when they’re doing inference, are just within a single scale-up?

Reiner Pope

Yes. You can look at how it depends on model size. You could have a very large model, one that exceeds the memory of a rack. There you should be doing a bit of pipelining. Maybe it’s extremely sparse, for example, and that would be a reason to do it.

Dwarkesh Patel

This goes back to the promise at the beginning of the lecture, which was this will actually tell you about AI progress as well. To the extent it is the case that model size scaling has been slow until recently…

Let me make sure I understand the claim. The claim would not be you could have trained across more racks. It was just that it would not have made sense before, we didn’t have the ability to do inference for a bigger model easily.

Reiner Pope

Actually, pipelining doesn’t help with context length. It totally helps with model size. Because of the ability to do pipelining, a rack at least should not be a constraint on your ability to fit the model parameters.

The other consideration you’re asking is, why hasn’t it scaled up more, and why did bigger scale-up domains help? We talked through one aspect of that, which is that it’s not because of memory capacity. We have a solution to the memory capacity at least with respect to model size, not with respect to KV cache size but at least with respect to model size. The other issue that shows up is latency.

Dwarkesh Patel