The main way that AIs have been getting better is from adding more and better data, and scaling the compute to develop that data in the first place. Obviously RL is the main way that has happened. You can think of RL as a kind of synthetic data generation - you dump a lot of compute against a verifier in order to find the “good” data. Then you train your model to predict these correct rollouts, much in the same way that you might train it to predict the next word in internet text.

For this process to work, the model must have at least prior some probability to anticipate the correct solution, which is why you also need mind-stretching amounts of human expert trajectories in every single field and skill you want the model to be competent at.

It’s hard to overstate how task specific and bespoke this human expert data is. If you want to get some intuition, go read some job descriptions at Mercor or Surge’s websites. There are listings for a word specialists who will convert legacy documents into polished Word files, and legal experts who will write realistic M&A diligences or securities filings, and management consultants who will write up template market research, and dozens more other particular categories.

And it is not only that the data have to be so domain specific, but there has to be so much of it! Each skill corresponds to at least hundreds of human experts who are generating example completions, writing rubrics, and explaining their chain of thought. There’s a reason that the data industry producing these expert labels (and the RL environments in which their meticulously catalogued skills can congeal) is earning billions a year in revenue, soon deca-billions.

Imagine if it took a couple decades worth of courses with hundreds of concurrent professors and millions of practice tasks for you to learn how to polish a word file. Even the task count difference understates the gap - the models have to grind their far more numerous tasks each far harder. Whereas a human student might practice a textbook problem once or twice, GRPO has the model generate hundreds to thousands of rollouts per task. We are building some Frankenstein’s monster, with a billion grafts of carefully constructed examples sewn together.

Epoch recently reported that open models only lag state-of-the-art closed models by 4 months. I think the reason it is relatively easy for open source and previous laggards to catch up to within months of the frontier is that data is the real driver of progress. And data can be easily distilled from public APIs, whereas hyper-parameters and training tricks and architectural micro-optimizations cannot - if the latter were driving most of progress, then catching up would be harder than we are observing it to be.

It is easy to forget how much data these models are trained on, and how much more it is than what we humans see in our lifetimes. We see these AIs as a galaxy glittering with capabilities, but at their center, invisible to the naked eye, holding all the constellations together, is an unimaginably massive black hole of data.

Intermission: Comparing human and AI sample efficiency

• If a person hears and sees on average ~2,000 words an hour, then from birth to adulthood, they’ll see ~200 millions tokens. By contrast, frontier models are trained on somewhere between 10s to 100s of trillions of tokens. That is close to a million fold difference.

• A person can learn to teleoperate any random humanoid or robot arm within hours. The reason robotics isn’t already a deca-trillion dollar industry, with a endless army of Unitree G1s doing all kinds of useful work in world, is that our AIs learn so much less efficiently than humans, and even the millions of hours of demonstrations we’ve collected is not enough to allow them to perform complex, open ended tasks.

• A teenager can learn to drive a car with about 20 hours of practice. Even if you include their ~16 years of accumulated physical intuition as relevant training data, that is at least 3-4 orders of magnitude less than the amount of data Waymo and Tesla have needed to train their self-driving car models.

I wanna deal with some common objections to this kind of comparison:

• Many billions of years of evolution is our pre-training, so it’s unfair to compare how little data we see simply within our lifetime to what these cold-started LLMs have to learn from.

  • Our genome is 3GB, about 1-2% protein coding. That is just not enough space to store the model parameters that are supposedly pretrained (frontier models are terabytes sized). The closer analogy is probably that evolution has found the right hyperparameters and loss functions (Sidenote: I had an interesting podcast with Adam Marblestone where he argued that the loss functions were the more significant find from evolution), but that the equivalent of parameter training is still happening within lifetime, and is encoded in the map of neural connections in the brain built up over a lifetime.

  • Even if it were the case that we can explain away the trillions of tokens required to pretrain a base model as catching up to evolution, it doesn’t explain why the marginal capabilities take so much data - once you have been educated, you don’t need 100 different professors to learn a new programming language, but the AIs (even once pretrained) do.

• These comparisons are not including the multimodal data we see in our lifetimes. If you include all this sensory information, we’re probably in the 10s to 100s of billions of tokens range from birth to adulthood

  • Blind/deaf people who are cut off from this kind of sensory information might lack faculty with the relevant sense, but still have the same general intelligence as everyone else. Which suggests that all these billions of sensory tokens are not really the thing making humans smart.

  • In fact, deaf people who can only communicate via sign language and reading (and not from hearing) are ingesting far less than the 200 million language tokens we calculated earlier, and even this is sufficient for them to be fully general intelligences.

• Scaling laws tell us that bigger models are more sample efficient. The human brain is 100T synapses - if each synapse is ~1 parameter, and frontier models are currently roughly ~5T parameters, then maybe we could achieve human-level sample efficiency with another order of magnitude or two of parameter scaling.

  • The way the scaling law equations work is that parameter and data terms are added to the loss independently. If you have a model that is trained compute optimally, and suppose you ask, well what if I just wanna maximize sample efficiency and use less data - and I’ll throw in as many parameters as it takes to make that happen. With the constants from the Chinchilla scaling laws paper (and the nature of the result wouldn’t change even with different constants), even if you increased the number of parameters by infinity, that would only decrease by a factor of ~10 the amount of data you need in order to keep the same loss. Humans are somewhere between thousands to millions of times more sample efficient than these models. Scaling of current models simply can’t make up for that discrepancy. This really does suggest that humans are on a different scaling curve altogether.

Does sample efficiency matter?

But you might ask, why does sample efficiency matter? The labs have two overarching objectives: automate white collar work, and automate AI research itself. Is human level sample efficiency necessary for either?

The bet with white collar work is the common tasks that a software engineer or analyst or accountant does are, well common. And we can bring common tasks into distribution quite easily through RL and SFT. The revenue curves of these AI labs suggest that there is enormous value from bringing tasks into distribution, even if we don’t replicate human sample efficiency.

Yes it is far more inefficient to train AIs to do these tasks than it is to train humans. But so what? Human lifespan does not allow for the quantity and breath of training these models experience. If you as a human had some weird learning disability where you needed to read through every public repository on Github before you could be a competent developer, it would not make sense to train you up. You’d be on Social Security by the early stages of your education, and even once you were trained, you could work on only one project at a time. But AIs can learn these skills by firehosing gigawatts of training at a time. And what they learn can be amortized across billions of sessions, so we can be ludicrously inefficient in training them and still be wildly in the green.

How much “out-of-distribution” thinking do white collar employees need to do that you simply can’t train for in advance? Well this is more a question about the nature of different jobs rather than a question about AI research. And also depends on the job - some jobs are mechanical and predictable enough that they were automated long before the modern era of AI, for example bank tellers or travel agents. And there are other jobs which require dealing on a daily basis with problems that are quite distant from the data distribution. Even software engineering (the jobs AIs are supposed to take first) is one such. I would be willing to bet that there’s overall more demand for human software engineers in 2028 than there is now, largely due to the complementary input of AI.

The labs’ plan for these later kinds of jobs is to first automate AI research, and then have the automated AI researchers solve this sample efficiency problem. So then the question is, can AIs, which do not have human-level sample efficiency, nonetheless solve the remaining research problems on the way to human-like intelligence and learning.

That question I’ll address in a future blog post - I think the way that people currently think about an intelligence explosion is pretty clumsy. Either people dismiss the possibility of AIs speeding up AI progress altogether, or they just assume that God pops out the other end. People are not reasoning about what extremely rapid progress, but starting with LLMs, looks like.

There’s a bunch of important questions about how we deal with AI that only economics can answer.

What is the optimal way to tax and redistribute the wealth that will be generated? How should countries not in the AI supply chain index into the gains? Is there any world where inequality doesn’t explode?

It might seem like these questions have obvious answers, but the first thing economics teaches you is that your intuitions can often be entirely wrong.

It was very helpful to chat through these things with Alex and Phil.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Jane Street invests heavily in turning smart people into exceptional researchers and engineers. In addition to their apprenticeship model, Jane Street runs lectures and bootcamps in their in-office classrooms -- managers clear their teams’ schedules to encourage attendance. If you’d like to work at a place that takes learning this seriously, Jane Street is hiring. Check out their open roles at janestreet.com/dwarkesh

• Google’s Gemini Omni has incredible video editing capabilities -- you can upload a video and have Omni change the background, adjust lighting, or add specific elements. But Omni is also a preview of how future frontier models will be trained -- fully multimodal on both input and output. You can try it yourself in the Gemini app at gemini.google or in Flow at flow.google

• Cursor used targeted RL with textual feedback to help train their Composer 2.5 model. One of their researchers, Sasha Rush, gave me an impromptu blackboard lecture to explain how this form of on-policy self-distillation works -- I posted the full thing on X. If you want to try Composer 2.5, go to cursor.com/dwarkesh

Timestamps

(00:00:00) – Will capital share increase?

(00:19:36) – Messy Middle scenario

(00:25:57) – How to tax and redistribute AI wealth

(00:30:02) – Why demand collapse is unlikely

(00:39:26) – Human employees would be hard to integrate into the machine economy

(00:43:08) – What if some humans (or AIs) value wealth accumulation intrinsically?

(01:01:28) – What should developing countries do?

Transcript

00:00:00 – Will capital share increase?

Dwarkesh Patel

Today I’m chatting with Alex Imas, who is Director of AGI Economics at Google DeepMind and Professor of Economics at the University of Chicago, and Phil Trammell, who is Head of Economics at Epoch and research scholar at Stanford.

In general what I want to understand in this interview is what economics tells us about what we can expect in a world with more and more automation and more advanced AI. I want to understand what that tells us about what will happen to wages and the labor share, what the best way to tax and redistribute the wealth generated by AGI will be, and what kinds of things will be scarce. What is scarce tells you where the value will accrue.

I want to start there. What are some plausible candidates of what will be scarce?

Alex Imas

Something like the relational sector, which is defined as services and goods where the fact that a human was in the loop is part of the value of that product. Because humans are naturally scarce, if we have automation where a lot of other things stop being scarce, we will still have scarcity in the things that humans are involved in and in the loop for.

Dwarkesh Patel

I’m curious to understand whether humans doing services for other humans can ever be a big part of the economy. Here’s maybe one intuition pump. In a world where AI can physically do anything humans can do, there’s this whole machine economy where they’re building factories and doing research and coming up with new ideas. Humans may or may not be involved in the physical production of those things, but probably not in the ultimate limit, if robotics is solved. If you don’t care about humans being involved in that process, why would they be?

But then there are these other things you point out where we actually do want the ballerina or the barista to be a human. That’s part of the value of going to a cafe or a performance. But only humans have that preference. So there’s this human economy where humans are doing services for each other, and part of their wealth is flowing to other humans.

But part of their wealth is also flowing out, because they will want some of the automated goods this machine-only economy is creating. This is not a closed loop. A lot of things in the machine-only economy are a closed loop because the machines don’t care about getting the human barista to make them a coffee.

Within that model, isn’t it intrinsic that the human-only economy will become a smaller and smaller share?

Alex Imas

I would like to pitch a rephrasing of that question. My view is that the individual forecasts economists like us would make, as individual forecasts, are not necessarily very useful.

There was a blog post by Andrey Fradkin, Brian Jabarian, and Andrew Koh that came out yesterday looking at economists’ forecasts about the labor market. What they found is that there’s a ton of disagreement in every single direction.

What they advocate for, and I’m in agreement here, is that rather than thinking about individual forecasts, we should be generating prediction markets where you get aggregate forecasts and wisdom-of-the-crowd effects. The reason I think this is because we have been famously terrible at forecasting.

Let’s go all the way back to 1820. This debate we’ve been having is actually 200 years old. David Ricardo is one of the classical economists, not neoclassical. When the Industrial Revolution started happening, he wrote a bunch of stuff saying, “This is going to be great for everybody. Prices are going to come down.” But then he turned around and said, “Wait, I can see all these jobs that are creating value are going to be automated by these machines. This is going to be really bad. Everybody’s going to become unemployed, and there’s going to be political unrest.”

And if you look at Ricardo’s predictions, they’re actually right. All those jobs that made money in Ricardo’s time got automated. If David Ricardo woke up and somebody told him all those jobs did get automated, and then asked him, “What do you think the prime-age employment rate is in 2026?”, I think he’d be surprised to be told it was the highest it’s ever been other than 2000. We have the highest number of employed people that could potentially be employed since 2000. That was the peak and now it’s the second peak basically.

What David Ricardo ended up missing is that you have these economics of structural change, where everything that got automated became cheap. People had more money to spend, and then they started spending it on services. This is the lump-of-labor fallacy. David Ricardo didn’t consider that new jobs would be created.

But it’s not obvious that money would go to services. Why wouldn’t it go to more automated goods and something like that? I’m not using this anecdote to say this is what’s going to happen now and that we’re going to have full employment. I’m using it to say it’s really hard to make predictions.

What may be a really useful tool that economists have is to instead start with a premise. Maybe we start today: labor share is zero. Labor share has gone down. What could possibly explain this? Let’s write down an economic model of what happened. Phil will talk about this later today. Or you can write down a model that asks, “What if labor share just stays the same? What can make that happen?”

If you don’t take anything else out of this conversation from me: We don’t have any data. I’ve been saying we need a Manhattan Project for data. We don’t have data on consumer demand elasticities. We don’t know what they are. We’re not really tracking what jobs are getting created or destroyed. The O*NET database, with all of the tasks and different jobs, has been rarely updated and is super low quality.

What is really useful is to think about the potential scenarios, map them out, and say what dimension of scarcity will generate each scenario. If there’s full employment, we can talk about the relational sector. If the labor share collapses, we can talk about other sorts of scenarios. That will tell us what data we should be collecting.

Dwarkesh Patel

It’s probably worth defining labor share and capital share real quick. The whole economy, the total sum of goods and services sold, is either paid out to people in wages or it’s paid out to capital, which is to say, there’s rent on buildings and shareholders of companies that get paid out.

For many hundreds of years, ~60% of the economy basically gets paid out to humans in wages, and the other 30-40% gets paid out to people who own machines and land and claims on companies. The question is, if 60% is going to wages right now, does that shrink as AIs get smarter and better?

Alex Imas

This is a Kaldor fact. We should stress this. It’s incredibly surprising that it’s over 60% after the Industrial Revolution and all of the automation we’ve ever seen. Some people are worried it’s an accounting error that it’s been so constant.

There’s even a controversy right now. Some might say labor share has been falling in the last 20 to 30 years. But there have been a lot of accounting changes in the last 30 to 40 years. For example, Atkinson has a paper showing that if you keep the accounting constant over the years, labor share hasn’t even fallen ever.

Dwarkesh Patel

But it’s not that surprising, right? Phil, you made this point that if labor and capital are complements, you need both to do anything. It would make sense that you’d need to pay both of them to get something done.

Alex Imas

You have had stuff be completely automated.

Phil Trammell

There’s a sense in which nothing has yet been completely automated. Look at the network-adjusted factor shares of a good. Look down the supply chain and not just the final step and how much of that is done by capital and labor, but what went into the machines that can automate that final step. You’ll find that labor is adding a lot of value down the supply chain.

Computer and electronic products in the US have a very stable network-adjusted capital share of around 50%. It’s not 100%. I do think there’s this qualitative shift that I think we agree is coming, which is that there will be at least some goods whose network-adjusted capital share goes to one. The whole supply chain can be automated, and there’s no part in it that we care intrinsically about having a human do. That will be a qualitative shift.

Interestingly, the implications of that shift for the overall capital share are ambiguous. Let’s say we’ve got two sectors: the human-intrinsic sector with the ballerinas, and everything else. Right now, everything else has been scarce because of the lack of labor in it. But if we fully automate the supply chains for everything else, and we satiate in everything else really fast, then the quantity of everything that’s not a ballerina goes to infinity, but the marginal utility in that stuff goes to zero faster than the quantity is rising.

Alex Imas

I also want to move away from the ballerina example. The point I was trying to make in my post—working backwards from a particular scenario—was that the ballerina and the performer are the wrong reference class. Right now we have a lot of jobs where you have different tasks. This is the task-based model of jobs. Take a doctor, what is their job? They’re filling out insurance documents. They’re going and calling different pharmaceutical companies. One of their tasks is to see the patient and talk to them, but that’s not the main part of the job.

You could have a job and a service or a good be a product of different types of tasks, and you can automate a ton of those tasks. If the consumer is willing to pay more for a product or service where every single task is automated except for that one part where the doctor is delivering the diagnosis and providing support, we would call that job part of the relational sector. People are willing to pay more for the human to stay in the loop in the job.

We don’t have data to say, “Here are relational jobs, here are not.” You literally need to collect data of the following sort. Do a conjoint analysis of your willingness to pay for this service or good. Here’s the counterfactual where everything is produced by machine. Here’s the counterfactual where this one task is not produced by a machine. What is your willingness to pay? What is your elasticity for the human to not be in the loop?

If I don’t have that data, what prediction am I going to make in this story?

Dwarkesh Patel

Isn’t there another point, which is that there are a lot of fully automated goods that don’t even exist yet? And you can’t collect any data right now about, say, how much people will want to keep buying more and more of some drug that makes you healthier that is fully produced by the AIs.

Alex Imas

Absolutely. That’s kind of Phil’s point. You could have an increase in variety in capital where you don’t get the satiation. You’re increasing variety, so you’re not hitting that diminishing marginal utility point where most of your income is going to the human sector. If that increasing variety is fast enough, and there is no such increasing variety in the human sector, then you can get all of the relational goods you want, but it doesn’t matter for labor share. It goes to zero.

Dwarkesh Patel

Phil, I liked your analogy to some Mongolian economist sitting around in 1400 thinking about what will be scarce and the limits of that kind of analysis. I think you should talk about that.

Phil Trammell

Just look at the goods available to a Mongolian of the distant past. I’m no expert on this society, but I know that they didn’t have nearly the variety that we have now. Look at the jobs that were intrinsically human, like being a singer. And then you look at the things that were not intrinsically human, like the transportation services provided by their horses or the different kinds of food they had.

If they just held the varieties fixed in both categories and asked, “What will happen once we have a lot more automation?”, they might have said, “We’ll just satiate in horse-like transportation and in yogurt and in yurts. Those shares will all go to zero, and we’ll be left spending all of our money on singers.”

But of course, that’s not what happened. As we’ve accumulated more wealth and more advanced machines, we’ve expanded the range of things other than singers to spend our money on, and the share spent on singers has stayed negligible. Likewise, that’s my central prediction about how the future unfolds, though it could go either way.

Dwarkesh Patel

I was going to make a point and I realize it’s a fallacy, but the reason it’s a fallacy is interesting. It’s just hard to imagine a world where there are trillions upon trillions of robots, but there’s only some billion-odd humans, and the cumulative amount we’re spending on robots is less than what we’re spending to pay Magnus Carlsen or—

Alex Imas

Financial advisors or doctors or tutors.

Dwarkesh Patel

Right, or podcasters or whatever. But then I realized why it’s a fallacy. The number of transistors in the world has literally trillion-X’d, maybe quadrillion-X’d. Your colleague Chad Jones has a very interesting result about how the share of the economy that is going towards paying for computing, paying for the transistors, has been decreasing.

The point you made is that one way to think about Moore’s law is… What sets price? Supply and demand. So not only are we producing more transistors more cheaply, but also the value of the marginal transistor is decreasing. As you were saying, another way of saying Moore’s law is…

Phil Trammell

I like the pessimistic framing of Moore’s law: every 18 months, the value of computation halves. We’re running out of uses for computation so fast that it’s sustaining Moore’s law.

Dwarkesh Patel

This is relevant to a conversation about AI where maybe for the first time, this is no longer true. The famous fact here is that an H100 costs more to rent now than it did three years ago, even though we have much superior technology and much more compute in the world. Because as models get smarter, the opportunity cost of compute gets higher.

Alex Imas

This is Phil’s point about increasing variety. What we have done is increased the types of things that people demand from capital. Now all of a sudden you have a new variety that you could be using capital for, and you jump back up.

Dwarkesh Patel

You could imagine we just never satiate demand for compute. As long as that stays the case, the share of the economy that is going towards compute would keep increasing.

Alex Imas

That’s the big question. That is the ultimate question that we need to be looking at. What number of new uses are we finding for that compute where you have the demand for these uses? What I want to emphasize is that a lot of models in economics, especially in the space that we’re talking about, take demand as almost exogenous. They don’t unpack the psychology of what people actually want.

What got me thinking about the idea of the relational sector is work that I was doing on the fact that there does seem to be this intrinsic value. It’s not just because it’s scarce; it’s because there’s some intrinsic preference that people have for empathy, connection, and interacting with another person.

One of the experiments that we ran involved an art print. We have an incentive-compatible way of asking, “How much are you willing to pay for this art print?” People are actually paying real money for it. Then we say, “Look, there’s only one of those art prints, and it’s either made by AI or by a person.” These are between-subject conditions. With one, you get the effect that the person-produced art print is valued much higher than the AI version.

Then, in a set of other conditions, we say there’s 500 of these being produced. For the human-made one, the price goes down a lot because it’s no longer seen as making a connection with this one artist. With AI there’s no difference. AI is already viewed as a commodity.

We need to do a lot more research on this, but it seems that’s the key difference between this and something like a horse. A horse was an input into an output, where you can replace the horse with something else. You only care about the output.

The only way this relational story works—and this is what we need more data on—is if a human is not a horse in the sense that they are providing value from the output, where if you replace the human, the value of the output decreases. If that’s not strong enough, and if it doesn’t hold for enough sectors or enough jobs, then this story doesn’t work anymore.

00:19:36 – Messy Middle scenario

Dwarkesh Patel

There’s one possibility which Molly Kinder has written about, this “Messy Middle” scenario. That possibility made me think about whether it might be better to have—at least as far as wealth distribution and redistribution go—a much faster AI takeoff.

I want to ask you whether the following possibility is at all likely, or if there’s any set of assumptions that can make it so. AI makes it possible to automate jobs such that many people are losing their jobs, but it doesn’t create enough wealth, while the process of automation is happening, to basically pay off the people who are getting laid off and create a Pareto improvement, where everybody’s getting better as a result of AI automation.

Of course, there’s a trivial sense in which that must be true. Whatever money the company is saving by not paying the humans instead of just paying the AIs, those resources still exist in the economy and can just be paid out to people.

But there’s going to be some allocative inefficiency. The government doesn’t know exactly who got laid off because of AI. There’s a political problem. If the Meta worker gets laid off first and they were making $200,000 a year, is there a politically sustainable situation where you give them a $200,000 check a year when there are many working people making much less? Do you find this scenario plausible, where AI is automating a bunch of things, but there isn’t as much wealth creation as there is automation?

Phil Trammell

I think it’s possible. To me, it does seem like a pretty narrow window. My guess is that if we have the technology to automate so many jobs that it becomes a new kind of political problem, then the pie will also be growing really fast.

Well, unless in all of those professions it’s automating, it’s just a hair more productive. So the cost of all the capital to replace all the software engineers is just a hair less than the cost of what we’ve been paying the software engineers.

Dwarkesh Patel

Why is it implausible that a company can save money by laying off a bunch of software engineers? And in the long run, there’s a Jevons paradox, and we can’t anticipate in advance what we’d do with more software, and surely there will be more uses. But in the short run, the effect is just that a lot of people are laid off, and they still need to figure out how they can use a million times more JavaScript tokens.

Alex Imas

Phil and I have been writing about these things, and we have mathematical models in the back of these things. We don’t have any political economy in any of our models.

Andy Hall wrote a really nice blog post about the politics of AGI, and he made a really interesting observation. If there’s a 2% increase in unemployment, the political winds completely change. Unemployment has a huge effect on what happens politically.

Referring to Molly’s excellent essay, I think in some ways one of the worst scenarios is a drip scenario because of the political economy piece. What you might see is people not really being unemployed en masse, but moving into sectors that pay them less money. This is what happened with phone operators between 1920 and 1940. Phone operators were completely automated, but it took 20 years, even though the technology existed. There was this drip. It wasn’t like this giant sector just disappeared.

There’s a really nice QJE paper on this showing that they got reabsorbed into the economy, but at lower salaries, and they were mostly underemployed. That’s the scenario Molly was writing about, this messy middle where things aren’t a disaster. We saw with COVID that the fiscal response can move quickly if there’s an emergency. An emergency is a quick uptick in unemployment, which could even look like 2-3%. That becomes a national emergency if it happens fast.

Dwarkesh Patel

The concern is that whatever you’re saving on those white-collar workers, if that’s not growing the economy but just creating saved resources that can be allocated elsewhere, is that enough to do a broad-based redistribution scheme? You have the money you’ve saved off a couple of people. Unless you can figure out exactly how to get it to them specifically, you have the problem of, “Can I do a UBI off the money I saved by laying off…?”

Alex Imas

You’re basically saying the pie did not grow that much. You’re just displacing a bunch of people, but that didn’t grow the technological frontier of what the economy can produce.

Dwarkesh Patel

Then there’s a question of whether every time this has happened in history, the technological frontier has expanded a bunch.

Alex Imas

I think that’s the case. Simply in history, the technological frontier has expanded. I think Phil made the same point. It’s hard to imagine that sort of scenario where you are getting intelligence that’s just enough to replace the software engineer but still costs a lot of money. It’s just a hair less expensive than the software engineer, so you’re not getting this abundance effect. Where is the redistribution going to happen because the pie didn’t grow?

Dwarkesh Patel

This is very helpful. Many different things have to be true for this scenario to come to pass, each of which seem unlikely.

One, it has to be the case that it is possible to automate entire white-collar jobs, but only in a piecemeal way. That is to say that you can only automate software engineers, but that same program can’t also automate an accountant and an analyst and whatever.

My model of intelligence is such that—both the breadth of tasks it requires to do something like software engineering and what intelligence is—if you can really just lay off all the software engineers, you’ve got enough in the bucket there that you could automate all kinds of white-collar work. There are huge amounts of potential savings that have happened as a result of these layoffs, and also AI is going to be cheaper than human labor.

If both of those things are true, this messy middle scenario where we literally don’t have the wealth to go around seems unlikely. Then the question is, what is the best way to tax it and redistribute it?

00:25:57 – How to tax and redistribute AI wealth

Alex Imas

I have some thoughts. I think it’s really important to outline the costs and benefits. First, there’s differential complexity in implementing these things. Two, they differ in the timeline of being actually helpful. Something like universal basic capital is not going to generate returns for something that happens in six months. You probably are going to end up with a layer of things.

Take a negative income tax, for example. You implement it, and the day it turns into law, you already have this insurance that there’s a floor where everybody gets a certain amount of money, and if you earn more money, you get taxed more. But there are positives and negatives to a negative income tax.

With UBI, for example, I worry a lot about the political economy implications. If people are just dependent on a check, it really matters who’s in power. Right now, we’re endowed with labor that can turn into income. When that is no longer the case and we are at the mercy of the elected official for basic needs, that feels like a power-sharing arrangement that’s really dangerous.

Dwarkesh Patel

But wouldn’t that be true of any sort of government redistribution program?

Alex Imas

With something like universal basic capital, where you have an ownership share and property rights for capital, you just have a share.

Dwarkesh Patel

You’re a normal shareholder.

Alex Imas

You’re just a normal person.

Dwarkesh Patel

But this goes back to the question of indexing, because if indexing is hard, then universal basic capital is hard.

Alex Imas

That’s the problem of universal basic capital: targeting. What do you target to put into people’s portfolios?

Dwarkesh Patel

Like, what if Anthropic goes to zero, but some random robotics company takes all this over?

Alex Imas

Exactly. That’s the risk of universal basic capital. With a negative income tax, you have the same sort of issues as with UBI, where somebody comes into power and says, “We’re not going to do that anymore,” and people can’t work, and then you have the issue of the floor being gone.

Dwarkesh Patel

One concern with the wealth tax is that there’s no politically sustainable equilibrium at a 0.5% wealth tax. This happened with the income tax, of course. It starts low, it’s for war or something, and then it slowly escalates until the marginal income tax rate in the US is on the order of 40%, and in certain states, upwards of 50%.

With a capital tax, is there a reason to worry that it would distort investment? Would people just say, “Why would I invest in Anthropic or Intel? The government is going to take larger and larger shares of it and dilute my share.”

Phil Trammell

Hold on. It’s worth separating how the revenue is raised, what’s taxed, and how it’s distributed. It could be that the government hands out shares of Anthropic to everyone by a broad-based tax and then buying Anthropic. Which would probably be the right thing to do. Hopefully, some populist proposal doesn’t interfere with that and expropriate some particular company that everyone happens to know about.

Dwarkesh Patel

You’re suggesting there could be some sort of optimal tax. We’re taxing externalities or we’re taxing land. I guess we probably need to tax something other than just those two things.

Phil Trammell

Or consumption.

Dwarkesh Patel

Ok, a consumption tax, like a European value-added tax, allows the government to go buy a bunch of stocks, and then they just distribute those stocks to everybody.

Alex Imas

That’s David Autor’s...

Phil Trammell

That’s not going to be that different from just redistributing the stocks, but it will be a little different.

Alex Imas

That was the proposal for Social Security, by the way. That was privatizing Social Security. It’s worked so far, but there are questions about how long it’s going to keep working. Privatizing Social Security was basically giving everybody a basket of stocks.

00:30:02 – Why demand collapse is unlikely

Dwarkesh Patel

People talk about whether there’s a white-collar apocalypse already. Is there any evidence that suggests there is mass automation or unemployment as a result of AI already?

Alex Imas

A lot of people are looking at it. This is an area where there’s a lot of eyes and a lot of data being produced. The Budget Lab over at Yale is doing really good analysis on this. They just recently released a report, and you really have to squint to see anything happening. If you want to take an approach across the entire economy, even looking at software engineering, the most exposed sectors, there’s just not really anything going on.

There might be a little bit of a signal about junior developers getting jobs less than before. But that’s a “less than before” rather than a level shift, as in there’s actually an increased demand for senior software engineers, if anything. If you look at the trend, for junior developers, it’s a bit below trend.

Dwarkesh Patel

So you’re saying the growth is slower than before, but there is still growth even for entry-level software engineers. What do you think is going on with the anecdotal evidence of graduating college students saying that they’re finding it harder to find CS jobs?

Alex Imas

I think that’s anecdotal evidence.

Dwarkesh Patel

You think it’s always been hard to get jobs for some people, and now it’s getting turned into an AI narrative? Same with the layoffs, where it’s probably just a normal layoff, and they turned it into an AI layoff.

Alex Imas

You have to be careful with all of this. There are these public coordination devices. Let’s say we get into a narrative where if you’re a firm and you’re not laying people off, then you’re seen as not adapting AI enough. Then you’re going to just get a cascade effect of firms needing to keep up with the Joneses in terms of starting to lay people off. That’s super worrying, where the firm might actually be worse off after the layoffs than before, but it’s just doing the layoffs to have the perception that, “Look, we’re not behind the times. We’re using AI.”

You probably heard these anecdotal stories of token counters, where you have to maximize tokens and things like that. Right now, we don’t really have any evidence of a white-collar bloodbath.

Dwarkesh Patel

Is that surprising at all, given all these things AI can do? This is a story as old as time. If you automate some complementary task, the overall bucket of things—the human labor which complements the automation—will increase in value.

Alex Imas

One of the statistics that’s really important for that argument is elasticity of demand.

Take the O-ring model of jobs. A job is a series of tasks. Let’s say the AI automates nine out of ten tasks. One task is not automated. If that person can now focus in on that task, the job will become more productive. If that translates into a price effect where the product is actually cheaper, and if demand responds enough where it’s being bought more and the service is being used more, that could actually lead to more hiring.

A lot of people on the internet have been making that argument very generally, saying, “Look, if anything in the data, we’re seeing an uptick in software engineering demand.” Which suggests that at least for now, given the way that jobs work, it might be elastic enough.

Dwarkesh Patel

I think this elasticity of demand argument is incredibly important for a lot of arguments that people make, or just a lot of labels that people use without understanding what the underlying causation is. People often talk about Jevons paradox. This is the idea that as something gets cheaper, you will want so much more of it that the total amount you spend on the thing increases. Famously, this happened to coal in Britain ~200 years ago.

But really this only happens if the demand for something is highly elastic. There are many things for which there is not super elastic demand. If oil, for example, gets super cheap, it’s not like magically—

Alex Imas

Or insulin.

Dwarkesh Patel

Exactly. It’s not like magically there’s going to be so many more cars that now we’re going to be using way more oil than before.

Phil Trammell

At least not in the short run.

Dwarkesh Patel

Exactly. The long-run elasticity is higher than short-run elasticity. But even in the long run, agriculture famously is the example where we can produce way more food if we dedicated the same portion of the economy that we dedicated to agriculture in the past. We’re already producing more food regardless, but we could produce even more if the same portion of the economy that was producing food 100 years ago was currently producing food. But you eat enough, and then you’re done.

The claim with software is that it is not some inherent property of markets that as it gets cheaper, you’ll just keep wanting more of it. The thing about software is this is a particular kind of good where as it gets cheaper, we’ll want more and more of it.

It is also highly relevant, and you wrote an essay about this—a lot of this podcast is me summarizing your essays back to you. There’s this very viral scenario planning about the future by Citrini, predicting that as a result of automation and very powerful AI, there will be a recession. White-collar workers will get automated, their salaries will no longer be available, and so there will be a slump. Do you want to recapitulate why this might be implausible?

Alex Imas

Part of it is plausible, part of it’s not. The part that we started the conversation with is the idea that there could be a lot of unemployment. If the speed of automation is quick, people could get laid off, and they may not find work very quickly. We can quibble about the unemployment part of the Citrini essay, but that’s not the issue. The issue is that they talked about negative economic growth.

What I did in the piece, that Phil and I had a back and forth on, was to say, let’s start with the proposition that there’s negative economic growth. What conditions do you need in the economy to get negative economic growth? It turns out the conditions are pretty improbable.

One thing that you need is for the holders of capital, rich people basically… Basically what you have in those sorts of scenarios is a reallocation of wealth and income from lower-income people who are using their labor towards tech capital owners. So you need demand to be bounded, like a hard bound, not even a soft diminishing sensitivity. You need for them to eventually say, “I’ve had enough. I don’t want to spend any more money.” And for that money to not enter as investment. Then you can get negative growth.

Dwarkesh Patel

The crucial thing is, even if we don’t want more shit, the world in which there’s a singularity and we don’t want to invest more money is crazy. We’re not saying, “Let’s build more data centers. Let’s build more fabs.” Even though we have AGI, we’re not investing in more data centers to run the AGI and that’s driving more economic growth.

Alex Imas

I sent the essay to Phil, and Phil wrote back being like, “This is pretty dumb,” like my essay. He said, “You’re trying to say that there’s going to be negative economic growth, but these are very implausible conditions.” And I was like, “That’s the point of the essay. These are very implausible economic conditions.”

That’s where scenario planning really shines. You have the Citrini essay, which was great that it was written because it started a conversation. It’s so intuitive, this idea that if there’s demand collapse, we can get the economy to shrink. You could get that with a depression. In the Depression, the technological frontier didn’t expand.

Here, the technological frontier is expanding. You actually have abundance. For abundance to generate negative economic growth, that’s really hard to get.

00:39:26 – Human employees would be hard to integrate into the machine economy

Dwarkesh Patel

We were talking a second ago about why there isn’t more automation as a result of LLMs. One plausible mechanism could be, as you were saying with the O-ring theory… O-ring theory refers to the fact that the Challenger shuttle blew up because one component malfunctioned, and it destroyed the whole thing. Maybe that’s a more general model of how goods are produced in the economy. You have to make sure everything is reliable and works well. So you can’t automate an entire job to an AI right now. Even though it might be able to perform it at some probability, you need extreme reliability in order for it to not destroy the finished good.

This might explain why there’s a lot less automation now than there otherwise could be. But I think it works in the other direction once AIs get advanced enough. Integrating humans into the production flow of future goods will become difficult. Even beyond the arguments about how humans will be more expensive or less capable, there will be whole production flows organized for AI labor. They’re talking in neuralese. They’re thinking many thousands of times faster.

So even if there’s some comparative advantage where it makes sense to hire a human, there will be transaction costs and worries of reliability that will actually make it hard to integrate humans into future production flows.

Phil Trammell

That seems right to me. In particular, I just want to distinguish between the point that if you automate nine-tenths of a job, people might shift over to the last tenth, but there might be ten times more work demanded of them.

Compare that to the model of O-ring automation from Gans and Goldfarb recently. If you can only automate nine-tenths of the job, but you do it to a lower standard of quality than the human could, you might not want to automate even those nine-tenths.

That’s the thing that could totally port over. Symmetrically, it could be a reason why we don’t use a human for one-tenth of the job anymore, because a human just can’t perform it to the level of quality that the AI can perform the other parts of the job, or the level of speed. They end up pulling down the quality or speed of the finished product.

Dwarkesh Patel

By the way, the model you’re talking about seems extremely plausible to me for why more lawyers, accountants, or even software engineers are not automated. There are cases where there’s a pretty good probability that the thing worked as you expect, but the thing you’re paying the lawyer for is: “No, really, my company’s not going to go under because—”

Alex Imas

You’re also paying for a lot of regulation-type stuff. With lawyers particularly, you need some entity to back up the product. You need ownership of the product. You need somebody to be able to fire or hire, and there are licensing issues. There’s a lot of regulatory layers that are also going to be keeping—even if there’s no relational element—humans in the loop that have nothing to do with the ability of the human to actually perform the service.

Phil Trammell

All of these frictions on the political-type decisions that we are accustomed to only trusting humans for—legislation, being a judge, being a juror, or all the licensing that keeps certain professions human—that all strikes me as transitional.

What we expect to come from a human and how we organize our politics has changed so many times throughout history, from little hunter-gatherer bands to empires to whatnot. Once an AI-run political system is much more efficient than the alternatives, those will probably tend to out-compete the others.

00:43:08 – What if some humans (or AIs) value wealth accumulation intrinsically?

Dwarkesh Patel

Speaking of which, we’ve been talking about what preferences humans currently have and what impact that has on what kinds of goods will be scarce in the future. But of course, we’ll have different kinds of entities in the future: AIs. There was a time when there were no humans on Earth, but evolution selected for agents that have specific drives and preferences because those tend to survive the most, and those preferences now determine what a hundred-trillion-dollar world economy produces.

Why not expect the same thing from AIs in the future? This is not even a world with catastrophic misalignment, where they just kill everybody. But there will be evolution of, even if not individual AIs, firms which have AIs as part of them. What will that evolution favor? It will probably favor firms or agents that grow. There’s a selection argument that things which grow will be more prevalent.

Maybe just based on that, you can make some predictions about what their preferences will be. Is the kind of entity which prefers to have human-intrinsic goods going to be the kind of entity that accumulates resources the most? Probably not. Probably it saves more and has unsatisfiable demand for whatever the relevant resource happens to be. Compute is an obvious one. Can we use that to make some predictions about the non-human preferences that will be guiding the future?

Alex Imas

If there’s an AI that has its own welfare, is fully autonomous, and is making its own decisions that are welfare-relevant, to be honest, I have absolutely no prior that it would prefer to deal with humans. There’s no reason.

But let me take the other side of that argument. Humans’ preferences to be interacting with one another, to trust and empathize with other humans versus a simulated AI, I think it’s a really important question whether those will change.

I’ve heard a lot of arguments saying, “Look, right now we’re just not used to the technology. What you’re thinking of as relational… At some point, people are just going to see an AI therapist as a superior product, and they’re not going to need the empathy that the human is providing.”

I think this is actually a really complicated question. Here’s one argument for why it’s not going to go away, and it has to do with evolution. Let’s say there are two types of people. One person doesn’t really have this preference. They can just interact with an AI, whatever can simulate it better. The other one has almost a moral emotion—using Jonathan Haidt’s framework—against offloading those sorts of social interactions to an AI.

Which of those two people are going to reproduce, find a mate, all of these sorts of things? I think the answer is clear. It’s the second one that has the preference for other people.

Dwarkesh Patel

Depends on how the reproduction is happening.

Alex Imas

Fair. But if we’re in the world where reproduction is still happening the way that it’s happening, I think… And this is a big question, I’m not making a prediction.

You had David Reich on the show. His point on the last podcast was that we’re buzzing with natural selection. So even if you get some sort of indifference now, you might get selection to point into an even stronger preference for other humans.

Dwarkesh Patel

Here’s one way to think about it. How is the wealth of the richest people in the world instantiated? We were having a call earlier, and you made the point that their consumption is more geared towards relational goods. Like Mark Zuckerberg is hiring MMA instructors and dancers for his wife’s birthday, and so forth.

But most of his wealth is just stock in Meta. As a controlling shareholder, he could say, “Meta, turn all this wealth into dividend income, and I will just spend that on consumption.” Instead, he would rather have his wealth compound and have Meta build more data centers.

So you don’t even have to change humans for this to be the case. The humans who are wealthiest—and growing wealthier because their wealth is compounding—just have this almost Nick Landian preference for accelerating capital. That does seem to suggest that this is an important determinant of what kinds of things are produced in the future.

Phil Trammell

There are two ways you could get the two kinds of people, one of whom prefers a human therapist and one of whom is fine interacting with the AI. If they both satiate equally quickly in capital but the one who likes the human therapist also just likes having some human-intrinsic services, then the marginal value of capital in the future, compared to the marginal value of capital today, for each of them if they start out equally rich, should be basically the same. There could be interactions and whatnot, but basically, that should be the same.

If what’s driving the difference is that one person just doesn’t satiate in capital because they’re engaged by the prospect of exploring the universe and turning their head into a galaxy brain or whatever, and the other one satiates, then the person who doesn’t satiate in capital is going to, if they’re being rational, have a higher savings rate.

So in the long run, they’re going to have most of the wealth, and the overall capital share will basically be the capital share of that person’s spending, which is going to be one.

Dwarkesh Patel

It’s important that we’re not talking about a hypothetical future. Elon Musk is talking about mass drivers on the moon. He’s by far the wealthiest person in the world. Obviously, currently his investments are going towards humans as well as machines, but I don’t think he cares particularly that his future researchers and engineers are humans versus AI.

Phil Trammell

And he manages to reproduce fast as well. So I just think it’s worth drawing that distinction. There are currently some rich people that don’t seem to satiate quickly in capital, and so maybe in the long run they’ll save the most. That does seem right to me.

I would also say, even if they do reproduce more slowly biologically, that might just not matter that much in the long run if they can live forever.

Alex Imas

The living forever is key. Again, we’re scenario-building here. If you could live forever, a lot of stuff changes for my story as well. To your point about rich people not consuming a lot and investing, this will all depend on the returns to capital. Right now, the returns to data centers are super high, but if we get into a situation where people are satiated with capital, then the returns to accumulating capital are going to be lower. Then these rich people are going to be consuming more, because the incentive to invest is smaller.

Basically, you think about the general equilibrium of this sort of process… We have gotten tremendously richer since 1820. Many more people are investing, but you’re still getting a consumption response which keeps people employed and labor share high. That’s because—

Phil Trammell

Hold on.

Dwarkesh Patel

Wait, not necessarily. I think you’re probably making the same point. It could be that their investment has to be titrated through actual laborers who have to do things for their investment to work. In the future, only the consumption is human-mediated, right? Because the investment can just be done by the robots.

Alex Imas

So we’re in the scenario of how you can keep high labor share. Let’s take that scenario. In the scenario with high labor share, for whatever reason, the returns to capital are going to be lower.

Dwarkesh Patel

That’s right. To the earlier thing where we were saying why the messy middle is implausible, I feel like we can do a similar thing here. For our returns to capital to be lower, the growth rate has to be lower, right? It certainly has to be lower than what we’re expecting through the period of transformative AI. If there’s explosive growth…

Phil Trammell

Yes and no. The capital stock could grow quickly, but the price of capital goods relative to consumption goods could be falling faster than the capital stock is growing.

Alex Imas

It’s the difference between the potential frontier of technology and the realized prices of these things. Because you have relative prices.

Dwarkesh Patel

So you’re saying I could be putting my money towards earning 30% interest and investing in data centers, or whatever. There will be something in the future, if the growth rate is high, that earns high returns. Or, as a result of all these technological breakthroughs, there’s some cool product that I really want to buy right now, and both of those will be compelling options.

Phil Trammell

Yeah. It doesn’t have to be a new product. It could be a human-intrinsic product.

Dwarkesh Patel

Although, if it’s a human-intrinsic product, we would want to have it much more in the future than we want it now, because the thing it compares against is—

Phil Trammell

We might want it the same as we want it now in the sense that the marginal utility in a ballerina performance is exactly the same as now. But the marginal utility in a robot might just be a lot lower than now. So in units of robots, we want it a lot more than we want it now.

Would the interest rate be 30%? It depends what you mean by the real interest rate. It might be that every robot now can turn into 100 robots next year. So in units of robots, the interest rate’s 10,000%. But if the price of robots is falling really fast...

Alex Imas

Prices adjust. I think that’s the whole point.

Phil Trammell

Here prices are adjusting in this interesting way that too many macro models don’t allow for. What’s happening is what would be called investment-specific technical change.

The price of capital is falling relative to the price of consumption, instead of doing the standard macro thing of saying there’s just output, this chimera of a thing called output, which one for one can be allocated to capital or consumption. That’s not going to be true in this world.

Every unit of capital next year is giving up way less consumption than each unit of capital this year. One robot now turns into many robots next year, but the number of ballerinas is the same.

Alex Imas

Again, we’re going to go back to the increasing varieties thing. If all of those extra robots next year are actually different varieties of robots and I’m not getting satiated on those robots, then it’s a very different story.

Dwarkesh Patel

But now we’re talking about the consumption world. For the investment side of things, there could be just some greedy titan of industry who keeps wanting more and more robots. That alone would be enough to increase the marginal value of robots and therefore decrease labor share?

Phil Trammell

Yes.

Dwarkesh Patel

Why are we not expecting greedy titans of industry to keep existing?

Alex Imas

Greedy titans of industry historically have built libraries and—

Dwarkesh Patel

But that’s because they die, and they’re like—

Alex Imas

Oh, they all die. Everybody dies.

Dwarkesh Patel

Well, we’ll see.

Alex Imas

Conditional on people dying… You had a guest on the show who said to understand the future, you should think about the past. You could have new types of titans being born whose entire reason for accumulating wealth is just to accumulate wealth.

But a lot of the time, at least historically, the wealth accumulation process is part of a large social interaction amongst peers, amongst the community, where you want to be admired in some way. The stylized fact of titans of industry is you accumulate the capital, and then you buy a bunch of stuff.

Dwarkesh Patel

I guess this is a historical question, but it does seem to me that in a lot of cases what is happening is that as they near the end of their life, they either hand it off to their children, who are worse stewards of capital than they are. They don’t even manage to grow their wealth at the rate the economy grows, much less faster than the economy grows, which their parents were doing.

They’re like, “Well, I care less about my children having it than me playing this game of accumulating wealth. So I’m just going to give it to some trust.” If people are living longer or if they can figure out some way to align their trust to this wealth accumulation process… It just feels like the evolution here is so strong. You just need a couple of agents that think this way for this to be the dominant thing determining the preferences of the whole economy, because this part is growing much faster than the other parts of the economy.

Alex Imas

The part about satiation and diminishing marginal utility keeps coming up, I think it’s really important. If a person has an intrinsic preference for accumulation, that’s just what they want, then I think your story is totally right. But that’s just not how preferences usually work. You have enough hedonics in your life, and then the social status… Rousseau wrote about this, St. Augustine wrote about this. This is a basic part of preferences.

Now, you guys are arguing about something else, where you could have such high concentration that you could just have a couple of exceptions to the rule, and that’s going to be enough. I have nothing to say about that.

Phil Trammell

I think the claim is a little stronger, not just that you could have some exceptions, but that historically and today we see the exceptions. They just haven’t really taken over the economy historically because there have been these dissipation shocks, as they’re called. They’ve given it to their kids who squandered it, or they put it in foundations which spent it. It’s not really a shock, but…

People might have liked to fill the universe with monuments to themselves and live forever, very wealthy. It’s a weird preference, but it’s not a hypothetical preference. I think that’s the claim.

But who knows what’s going on in their heads? Even without the intrinsic preference for accumulation, there are some instrumental reasons why some people might value accumulation, which is also worth bringing up. There’s a desire for political, philosophical, or religious influence. People get into an arms race over what society looks like and what people believe.

Similarly but differently, because it’s not an arms race, there’s just total utilitarian philanthropy. When I think about why it might be good to have a lot of wealth in the future as a good classical utilitarian, to me, the value—or at least one way you could have an almost unsatiating utility function in having wealth in the future—is to create new happy beings. They just add to the total welfare of the world.

This idea goes at least as far back as Bostrom’s astronomical waste point, that we could put Dyson spheres around the stars and turn all the energy into really happy simulations and whatnot.

Dwarkesh Patel

I think the particular greediness of this optimizer doesn’t matter, what they’re greedy for. Forgetting about utilitarian philosophy or whatever, a pure von Neumann probe has… I don’t know, is this an accurate way to say it? They just have high marginal value for the random solar system they’ll occupy because that turns into more solar systems. A von Neumann probe is a thing that can exist. That’s a very greedy optimizer.

Phil Trammell

If we’re talking about whether they’ll dominate the economy, maybe this is a technicality. But we only count final consumption goods and investment goods as GDP. If there’s just this phenomenon—

Dwarkesh Patel

How does a von Neumann probe show up in GDP?

Phil Trammell

Exactly. If we recognize it as a person that owns itself, and it’s optimizing on the margin between spending a bit more on a baby von Neumann probe that colonizes another star system or a ballerina or something, and it just doesn’t value the ballerina very much…

Alex Imas

When we’re talking about AI beings, it just completely depends on how we’re doing the accounting there.

Dwarkesh Patel

What does the world look like in a world where von Neumann probes are possible? Is it possible labor share is high?

Phil Trammell

I think it’s possible the labor share is high the way we usually count it.

01:01:28 – What should developing countries do?

Dwarkesh Patel

Do economists have any advice for countries which are not in the AI production chain? If you’re not either producing the AI models, you’re not producing the hardware that goes into AI models, if you’re not Korea making HBM or Taiwan with the fabs or the Netherlands with ASML. India or Nigeria, what should they be doing right now? If you’re talking to Modi right now, what do you say?

Alex Imas

I think the biggest lack of resources that we have allocated in the economics profession is thinking about middle-income developing countries in the age of AI. This is something I fault myself with as well. There’s not enough people thinking about this question.

There are scenarios where you get AI technology being allocated and dissipating to Nigeria and developing countries, leveling the playing field, essentially giving them a level up as far as capabilities.

But there’s another world where, because they don’t have enough resources, they’re not training the models, they don’t have the hardware, and they just completely get left behind. And because of automation, we can produce commodities in developed countries now. Then we don’t even have the consumer market. That world looks pretty bad.

Phil Trammell

This seems to me like an extension of the messy middle case. One of the ways in which the messy middle might only be bad in a narrow range of scenarios isn’t just that it would be easy to redistribute because the pie would be bigger, but because the interest rate would be way higher, and/or, equivalently, the price of everything except human-intrinsic goods would be falling really rapidly. They’re sort of two sides of the same coin. A little bit of savings would turn into a lot of consumption next year.

Things have to go really wrong for us to just get over the threshold of capital being productive enough to automate lots of work, but not be productive enough that the interest rate is high and the price of capital-produced goods is falling a lot. Even without redistribution, a little bit of savings will save a lot of people.

Dwarkesh Patel

You’re saying if the developing countries have some savings in the developed world, that will be enough to produce a lot of surplus that they can then—

Alex Imas

They will now be able to consume a lot using their savings.

Phil Trammell

But the messy middle could be wider in this case. They’re starting from such a lower level in terms of how much they have and how much it’s actually indexed to the global economy. I think it’s important for them to get on it now. I don’t have strong feelings about whether it should take the form of sovereign wealth funds that invest in the right supply chains or just subsidies to their own citizens to buy a little bit of—

Dwarkesh Patel

This is actually a crucial point. We were talking earlier about why the Rockefellers of the world, why their descendants don’t control everything, if our argument about the selection of these greedy optimizers holds. One argument is just that it’s very hard to index the economy. Maybe they would’ve just decided to have their heirs index the economy and have their wealth grow at the rate of economic growth, and their heirs would be trillionaires by now.

Before index funds existed, it was just very hard. A very small fraction of the economy, going back 100 years, accounts for a majority of the value created now. If you missed those particular things, your wealth would’ve just stagnated.

Maybe there was a brief golden window from the creation of index funds up until five years ago where you could actually index the economy and have your wealth grow at the rate the economy grows. But now we’re in this world with very concentrated returns, especially to private companies. As we were making the point in our blog post, this is capital that the average person has disproportionately less access to. Most of their capital is having a random house, at least in the US.

Phil Trammell

Or a part of a house.

Dwarkesh Patel

Which, as we were saying, is capital that is uniquely ill-suited to be complementary to the production of AI or the serving of AI or to robots.

Phil Trammell

Or the kinds of goods that the rich will bid up the prices of.

Dwarkesh Patel

Exactly. What is the value of a house currently? It’s that the land is close to other humans and modulo relational stuff that is just not going to be the main factor of production in the future.

Alex Imas

This would be why a Georgist tax would not raise enough money for the sort of programs that we will be discussing.

Dwarkesh Patel

Right. But stepping back, the point I was trying to make is, if it gets harder to index the economy now, and that is the main way in which normal people are supposed to—modulo some sort of universal basic income—

Phil Trammell

In the developed world.

Dwarkesh Patel

—are supposed to have some purchase on the wealth from AI. And it’s also the way that developing countries are supposed to have some purchase on the wealth gains from AI.

But it’s very hard. Does Nigeria own a lot of SK Hynix and Anthropic? I’m guessing not. It’s not enough for them to just own the S&P 500.

Alex Imas

This brings up a really important point. Is AI going to be like electricity or social media?

Think about ConEd, or whatever the electricity provider here is. It’s a monopoly. It provides a resource that everybody uses. But do we think about electricity as creating a concentration of power? Does ConEd have this huge amount of political power, social power, or something like that? No, because with electricity, a lot of the downstream benefits actually came to the users of the electricity rather than the actual entity producing it.

On the other hand, with social media, it was the opposite case. Social media was everywhere. Everybody uses social media, but the rents went to the platform.

Dwarkesh Patel

That’s a really interesting point. I don’t endorse this take yet, I’m going to talk out loud. The more you think our economy is going to be run on AGI the way our economy currently runs on electricity—that is, there’s a broad fundamental transformation of the entire economy—the more it looks like electricity… Every company in the S&P of the future, if it’s going to make it to the S&P 500, it is because it has leveraged AI.

Alex Imas

Exactly. And then you’re indexed again.

Dwarkesh Patel

But then again, I guess if you just look at how concentrated the S&P is over time, just these big tech companies much more so… I guess this goes to a fundamental point that it’s hard to reason about how much of the gains from AI these individual private companies will be able to control.

Alex Imas

I think the open model thing is going to be a big point here. If we’re indeed in a world where the open models are six months behind the frontier—or nine months—then we’ll hit AGI, we’ll hit whatever, and in six months, everybody has access to this resource.

Dwarkesh Patel

This goes to show you that every question is connected to every other. That question about whether there’s runaway gains connects to questions about recursive self-improvement. Even if not recursive self-improvement, then continual learning, or online learning, which lets a model learn on the job. So if it’s deployed, it gets to learn more. These are just forecasting technical questions which then impact whether Uganda will have any purchase on the returns of AGI.

The reason I’m emphasizing the question is I think both for the messy middle and for developing countries, a recommendation that is often made naively is that you’ve got to do some kind of retraining. You’ve got to do some kind of jobs program, or you’ve got to have them build data centers in your country. I think you guys are suggesting something closer to just buying the index of AGI. That’s probably a much cleaner strategy and much more likely to succeed.

Alex Imas

These are the two scenarios. I think there is a world where it is concentrated, in which case it’s going to be really hard to index AGI. There is another world where it is electricity. Basically every company has access to AGI. So you just buy the index. Nigeria just needs to buy the index, and Nigeria has access to AGI because of the open models.

Phil Trammell

Just to get back to the question about whether to go with retraining or trying to index. I would prioritize trying to index, just given how fast AI could hit the world. But I definitely wouldn’t just rely on that.

In the messy middle cases or the long-timeline cases where we don’t get anything like AGI all that soon, it would be leaving a lot of value on the table if you could have retrained to be a bit better educated on how to use the latest wave of computing. I don’t think there’s that much of an either/or there.

Dwarkesh Patel

Maybe the reason to be pessimistic about this is because one of the reasons a country is poor is that it has a bad education system, so becoming the best in the world at retraining people at using AI doesn’t seem like a particularly promising strategy for that poor country.

Alex Imas

Although there are cases where, in developing countries, you had this leapfrogging effect with, for example, mobile banking. It’s much more prevalent in Nigeria than it is in Germany. Everybody is doing mobile banking. They have it on their phones, and they’re constantly doing this sort of thing.

Again, I’m not putting probabilities on this, but with a transformative technology like AI, you could get leapfrogging where you skip the step in the middle and get really astronomical growth.

Phil Trammell

Just about the ease of indexing, I think it’s definitely something to worry about a bit and keep an eye on. But as discussed in our own essay, and as other people have pointed out, it’s already not that hard to index. There’s been a bit of an increase in the privatization of returns, but still, well under 20% of the total market cap of non-tiny companies in the US is private.

Everyone thinks about OpenAI and Anthropic. If that’s where all the wealth will accrue, then all these questions about whether open models will stay only a little bit behind, those are important. But even they look like they’re going public before too long, probably. The frictions that have been keeping companies from going public might themselves be alleviated by AI a lot, just all of the disclosure requirements and whatnot. They want to get access to more potential investors, too.

If I had to guess, I would guess that the long general trend of lowering those frictions and making it easier for more and more people to index will continue, despite the recent bump in the other direction.

Dwarkesh Patel

This actually makes me hope even more so than before that the labs do get commoditized, or at the very least they go public as soon as possible. But hopefully they just get totally commoditized. I think AI will be much more popular and, more importantly, will be much more likely to lead to broad increases in prosperity if it is as hard to capture the gains of AI as it is to capture the gains of electrification.

Alex Imas

Exactly. There’s no anti-electricity people out there.

Dwarkesh Patel

I mean electricity doesn’t take your job, but—

Phil Trammell

Well, it takes some people’s jobs.

Dwarkesh Patel

It took some people’s jobs, yeah.

Alex Imas

This is maybe tangential to the conversation but I think narratives matter. There’s this really negative narrative around AI right now, but that’s because people are not putting out the positive narrative. There’s a reason. It’s more difficult to imagine a good thing that doesn’t exist than losing something that exists. It’s much easier for somebody to go on a podcast and say, “These jobs that you like, they’re going away,” than for somebody to spin up a utopia which doesn’t exist yet.

Phil Trammell

I hope this isn’t too out of left field, but I would be remiss if I didn’t point out one big cost of having commoditized frontier AI models, which is the tech race dynamic. For safety purposes, you might want fewer frontier companies so that each one has a buffer in case they want to slow things down to make things safer.

The way this relates to our point before about the widespread access of the returns, is that I think there’s a lot less of a trade-off there than some people imagine. Some people think either frontier AI gets commoditized and we all enjoy the benefits—but there might be some risk, because the market’s really competitive and cutthroat—or things are safer because there’s a big gap between the leader and the laggard. But that means the leaders get fantastically wealthy? No. You could just have a relatively big gap, but it’s a public company, and ownership in it is widely distributed.

Dwarkesh Patel

More recently, I have been thinking that the risk of commodification—which is that it diffuses the ability to use AI to harmful ends—is worth the benefit. I worry that having these concentrated labs not only makes it so that the surplus isn’t as widely distributed through society, but also creates a very tangible, clear political target for the government. We saw this with the Defense Production Act threat against Anthropic. If there wasn’t one lab, or a couple of labs, that are clearly ahead of others, this kind of threat would be much harder to make.

Thank you guys for doing this. I feel like there’s a lot of unresolved questions, but it is helpful to know what the first branch is along all these important dimensions.

Phil Trammell

Great.

Alex Imas

Thank you.

Reiner is CEO of MatX, a new chip startup (full disclosure - I’m an angel investor). He was previously at Google, where he worked on software efficiency, compilers, and TPU architecture.

Watch this one on YouTube so you can see the chalkboard.

Sponsors

• Crusoe was one of only five GPU clouds that made the gold tier in SemiAnalysis’ most recent ClusterMAX report. Gold-tier providers like Crusoe delivered 5-15% lower TCO than silver-tier clouds, even with identical GPU pricing. This is because optimizations like early fault detection and rapid node replacement don’t necessarily show up in the sticker price, but still matter a ton in the real world. Learn more at crusoe.ai/dwarkesh

• Cursor is where I do most of my work—from reading research papers to visualizing technical concepts to coding up internal tools for the podcast. Most recently, I used it to build two different review interfaces for my essay contest, one that anonymizes submissions for scoring and another that lets me see applicants’ essays next to their resumes and websites. Whatever you’re working on, you should try doing it in Cursor. Get started at cursor.com/dwarkesh

• Jane Street let me ask Ron Minsky and Dan Pontecorvo, two senior Jane Streeters, a bunch of questions about how they use AI. We discussed everything from the types of models they’re training to how they think about the future of trading to why they’re more bullish than ever on hiring technical talent. You can watch the full conversation and learn more about their open positions at janestreet.com/dwarkesh

Timestamps

00:00:00 – Building a multiply-accumulate from logic gates

00:16:31 – Muxes and the cost of data movement

00:26:10 – How systolic arrays work

00:39:11 – Clock cycles and pipeline registers

00:51:51 – FPGAs vs ASICs

01:03:25 – Cache vs scratchpad

01:07:27 – Why CPU cores are much bigger than GPU cores

01:12:00 – Brains vs chips

01:15:33 – A GPU is just a bunch of tiny TPUs

Transcript

00:00:00 – Building a multiply-accumulate from logic gates

Dwarkesh Patel

I’m back with Reiner Pope, CEO of MatX, a new AI chip company. Last time we were talking about what happens inside a data center. Now I want to understand what happens inside an AI chip. How does a chip actually work? Full disclosure, by the way: I am an angel investor in MatX. So hopefully you have designed a good chip.

Reiner Pope

Hope so. I’ll start with the smallest fundamental unit of chip design, and we’ll build up to what an actual production chip is and what its components are. At the very bottom level of a chip, the primitives we work with are logic gates, very simple things like AND, OR, and NOT. These are connected together by wires that have to be laid out physically as metal traces on a chip.

The main function that AI chips want to compute is the multiplication of matrices. Inside that, the fundamental primitive is a multiply-accumulate of pairs of numbers. We’re going to demonstrate what that calculation looks like by hand, and then infer what a circuit would look like for that.

It’ll be easiest if I do a multiply-accumulate of a four-bit number with another four-bit number. The clearest primitive is actually multiply-accumulate. So there’s a multiply of these two terms, and then we’re going to add in an eight-bit number.

Dwarkesh Patel

Can I ask a clarifying question? Why is this the natural primitive for whatever computation happens inside a computer?

Reiner Pope

There are a few reasons. It’s a little bit more efficient, but the reason it’s natural for AI chips is that if you look at what’s happening during a matrix multiply… What is a matrix multiply in short? There’s a for-loop over i, over j, and over k, of output [i, k] += input [i, j] x other input [j, k]. A multiply-accumulate happens at every single step of a matrix multiply.

The other observation is that the precision will almost always be higher in the accumulation step than in the multiplication step. This is specific to AI chips. You’re multiplying low-precision numbers, and then when you accumulate, errors accumulate quickly, so you need more precision there. This is why we’ve chosen to do a four-bit multiplication and an eight-bit addition.

Dwarkesh Patel

Let me make sure I understood that. There are two ways to understand that. One is that the value will be larger than the inputs. The other is that if it was a floating-point number it would be… Maybe that part is less intuitive to me. But maybe it’s the same principle?

Reiner Pope

It really is the same principle. The separate principle is that as you’re summing up this number, you’re summing up a whole bunch of numbers, so you’ve got a lot of rounding errors accumulating. Whereas in this case, there’s only one multiplication in the chain, so there aren’t a lot of rounding errors accumulating in the multiplication.

Dwarkesh Patel

Why are you summing up a whole bunch of numbers? There’s just two numbers there.

Reiner Pope

This summation is repeated j many times.

Dwarkesh Patel

Any errors accumulate. I see.

Reiner Pope

So how would we perform this calculation by hand? As a human, we would probably separate it into two, but we can do it all in one using long multiplication.

For the multiplication term first, we’re going to multiply this four-bit number by every single bit position in the other four-bit number. We write that out. First, 1001 multiplied by this bit position. That is the number itself. Then shifted across by one, we’re multiplying by 0. That gives us an all-0 number. Shifted across one more to multiply by this one, we get 1001. Finally, for this last bit position, we get an all-0 number again.

This gives us a bunch of terms that we have to add for the multiplication. While we’re doing that summation, we might as well add in the actual accumulator term as well. So we just copy that directly across. So this is the sum. It’s a five-way sum that we want to compute.

What logic gates did it take us to get to this intermediate step? We needed to produce all 16 of these partial products. How do I produce one of these partial products? Let’s take this number 1, for example here. We produce it by multiplying this number by this one over here. We can produce that with an AND gate. This number is 1 if both this bit is 1 and this bit is 1. If either of them is 0, then the multiplication of 0 times anything is 0. To produce all of this, we ended up consuming 16 AND gates. In the general case, if I were doing a p bit multiply times a q bit multiply, this will be p times q many ANDs.

Finally, I sum them. Most of the work is going to happen in the summing. Let me describe the other logic gate that we use here. AND is almost the simplest logic gate that exists on a chip. It’s almost the smallest. At the other extreme, the very largest logic gate you’ll typically use is something called a full adder.

Coming from software, you might think that a full adder adds 32-bit numbers together. In this case, it just adds three single-bit numbers together, so you can think of it as adding 0, 1, and 1 together. When I add these together, the result can be 0, 1, 2, or 3, so I can express that in binary using just two bits. As input, it has three bits. As output, it has two bits. The number 2 in binary is 10. This is also known as a 3→2 compressor because it takes three bits of input and produces two bits of output.

Dwarkesh Patel

Just to make sure I understood: the two inputs are an X and a Y value and then some carry that came in…

Reiner Pope

The three inputs are all bits in the same bit position, like three bits in a column here. The two outputs, I’ve drawn them vertically here and horizontally here to match this vertical versus horizontal layout. This expresses that things in the same column are in the same bit position, whereas things in adjacent columns are different. This is a carry out, whereas this was the sum.

Dwarkesh Patel

So if the inputs in the full adder were, say, 101, then the output would be 10. If it were 111, it’d be 11. If it were 000, it’d be 00. If it were 010, it’d still be 01. Got it.

Reiner Pope

Yeah. It’s essentially just counting the number of things and expressing that in binary. This circuit captures what we as humans naturally do when we’re summing along a column.

I’ll show one iteration of using the full adder to sum. The way I sum here is going to be a little unnatural for humans. We would sum along the column and then remember the carry, but instead of remembering the carry, we’ll explicitly write it out. We proceed from the rightmost column toward the left. On the rightmost column, we sum the 1 and the 1, and that produces a zero here and a carry of one. We’ve used this full adder circuit on this pair of bits and produced a pair of bits as output.

Now we can do the same thing with this column. We have a column of four numbers, so we’ll take the first three of them, run a full adder on them, and that gives us a 0 and a 0 as output. The sum of these is 00. That’s the full adder applied to all these bits. As I’ve used up bits, I’ll cross them out to indicate that I’ve handled them.

Let’s keep going a little bit more. I take these three numbers, I add them, and that gives me a 1 and a 0. I’ve dealt with these three numbers. Now I take these three numbers and add them, and that gives me a 1 and a 0, and I’ve dealt with these numbers.

The way to view this is that I have this whole grid of numbers that need to be added. I’m going to keep applying full adders to all the bits here, constantly removing three numbers from a column and writing out two numbers as output. Keep going over and over again until I eventually get just one single number coming out.

This approach is called a Dadda multiplier. This is the standard for how you do area-efficient multipliers using full adders. Let’s try to quantify the circuit size of this so we have a sense of how big things are and can compare them later.

How many full adders did I use? How many numbers did I start with? I have the 16 partial products, which is a product of all of these terms with all of these terms, plus the eight terms that I’m adding here. I started off with 24 bits. Eventually I produced eight bits on the output. In every step, I was crossing off three numbers and writing two numbers out as a result.

Every single use of a full adder eliminates one of the bits here. So how many full adders? It must be 24 minus 8, so there were 16 full adders in this circuit. This is true in the general case as well. There will be p times q many full adders in this circuit.

Dwarkesh Patel

Let me make sure I understand the logic of that. The input bits, 24, is p x q, plus p + q. The output bits are just p + q. So p x q plus p + q, minus p + q equals p x q.

Reiner Pope

That’s right. I think this explains, or at least hints at, the second reason we chose to do a multiply-accumulate. The first reason is that it’s what shows up in matrix multiplication. The second is that it gave us this very slick, simple p x q, very simple algebra.

We’ve described this whole procedure. Every single atomic step that I took here becomes a logic gate, and then the wires are connected together. When I had these three inputs that I used to produce these two outputs, if I think of mapping this to a physical device, there would be a wire connecting all three of these things together into a logic gate that produced this output.

This is the main primitive, at different bit widths, that’s inside an AI chip. We’re going to build up from here to how you would use it to run all the other operations you might want.

Dwarkesh Patel

This might be the wrong time to ask, but whenever Nvidia reports that this chip can do X many FP4 or half as many FP8, it seems to imply those circuits are fungible, that there’s not a dedicated FP4 versus FP8. But the way you’re mapping it out here, it seems like if it has to be mapped out in the logic, you would need a dedicated FP4 multiply-accumulate and then a dedicated FP8 accumulate. Can you “funge” them?

Reiner Pope

As drawn, they’re not particularly fungible. This is actually one of the main choices you have to make when designing a chip: how much of FP4 and how much of FP8 do I have? Sometimes I’ll make that consideration from the point of view of the customer requirement. Another angle is to equalize the power budget between FP4 and FP8.

Dwarkesh Patel

When they report those numbers and it just happens to be the case that it does 2x as many FP4 as FP8, they’re just choosing to give equivalent die areas to all the floating points, and as a result it ends up being…?

Reiner Pope

Why is the ratio exactly 2x? Part of it is that surely it won’t be exactly equivalent to die area. There’s a data movement reason. We’ll maybe come back to this when we look at how it goes into and out of memories. There’s something really nice from a software level about the fact that I can pack two four-bit numbers into the same storage as an eight-bit number. When I store that to memory, the sizing of the buses that I wire out within the chip makes that work out really nicely.

Dwarkesh Patel

Come to think of it, it’s not just 2x. The amount of area it takes sounds like it’s quadratic with the bit length. That’s why smaller precision is even more favorable than you might think.

Reiner Pope

This is a really big reason. In fact, Nvidia made a change. Historically, up until B100 or B200, every time you halved the bit precision, you doubled the FLOP count. For the reason you said, because of this quadratic scaling, that ratio is actually slightly wrong. You should get an even bigger speedup than you might otherwise think. Nvidia’s product specs have started acknowledging that in B300 and beyond, where the FP4 is three times faster than the FP8.

Dwarkesh Patel

Though it should be 4x.

Reiner Pope

Yeah. What I’ve shown here is the simplest case of integer multiply. When you’re dealing with floating point, as you do in FP4 and FP8, there’s this other term, the exponent, that complicates the calculation.

What can we see already from this? I think the big observation you’ve made is that there’s this quadratic scaling with bit width, which is very effective and is the single reason low-precision arithmetic has worked so well for neural nets. The other thing we’re going to do now is compare the area spent on the multiplication itself with all the circuitry around it.

00:16:31 – Muxes and the cost of data movement

Reiner Pope

We’ll walk back in time a little bit and see how GPUs prior to Tensor Cores worked, which is in fact the same way CPUs worked. Where do we stick this multiply-accumulate unit? Generically, I’ll describe a CUDA core or a CPU. You’ll have some register file which stores some number of entries, maybe eight entries of, in this case, 4-bit numbers, but typically 32-bit numbers.

Inside the CUDA core, I’ll have some register file of some depth, and then I’ll have my multiply-accumulate circuit. What it’s going to do is take three arbitrary registers from this register file, perform the multiply-accumulate, and then write back to the register file. It’s going to write to this one, but it was able to read from this one, this one, and another random one. It will take three inputs like this.

This is the core data path of many processors. Most processors look like this. You’ve got some set of registers, and then you’ve got some set of logic units, or ALUs. We want to analyze the cost of the data movement from the register file to the ALU and back.

Ultimately, there’s going to be some circuit that says, “Well, I don’t always have to select this guy. I might select any of the registers at any point in time.” The first question is: how can I build a circuit? The circuit I’m going to look for is a mux. In this case, it’s going to have eight inputs, one from each entry of the register file, and it’s going to have one output, which is actually producing this output.

What is the cost of this thing? All we have to build it out of is AND and OR. How do we build it? We do the dumbest thing possible. We form a mask. When we want to read the third entry, we’re going to AND every single entry with either 1 or 0 based on whether that’s what we want to read, and then we’re going to OR all of them together.

Dwarkesh Patel

Just to make sure I understand the basics. What the mux is doing is just selecting an input?

Reiner Pope

Just selecting, invisible to software. You say “I want input number three,” and that means there’s a mux here.

So what is the cost of this mux? An n-input mux operating on p bits. I have n rows. That’s just eight rows, and each row is p bits wide. I have to AND every single bit, so I get n x p many AND gates. For every single input I have to decide whether I’m going to mask it out or not. Then I’m going to OR them all together.

There’s going to be n – 1 times p many OR gates. I’ve got all of these different things, almost all of them are 0s, but I need to collapse them from my eight options down into one option. Every step, I need to OR one row into an existing row.

Dwarkesh Patel

It’s actually funny that you don’t think at the level of hardware. You just think, “Oh, I’ll just select element three,” and something as simple as that is in and of itself quite a complicated circuit.

Reiner Pope

This is the first step of all of the hidden data movement costs that show up. We’re just going to compare. I have to pay this cost. I’ve got one mux here, and in fact I have two more copies of that for each of the three inputs to my multiply-accumulate operation. I have this cost, which is 3 x n x p AND gates over here, compared to p x q gates in the actual circuit that is doing the thing I care about.

If we plug in actual numbers, with n being eight, I get 24 x p gates just in the data movement, compared to—if q is four—4 x p gates just in the multiply-adder.

Dwarkesh Patel

Where is the three coming from?

Reiner Pope

Three different inputs here. What I’m hinting at is that all of this work, which scales as the size of the register file—and this is a very small register file—all of this work just moving the data from the register file to the logic unit is many, many times more expensive than the logic unit.

Dwarkesh Patel

It may be helpful to just see what a mux looks like, maybe a two-bit or a four-bit mux.

Reiner Pope

We’ll do a two-way mux. We’ve got two different numbers, we’ve got these two inputs. These are the inputs that are being selected between, and the selector can either be “I want this one” or “I want the other one.” This is a one-hot encoding.

This is what we start with. Let’s focus on this case. This is the actual input we got, and we want to produce this guy as the result. Very laboriously, we AND this bit with all of these. That produces ANDing this bit with this row. Likewise, we AND this bit with this row. That produces all 0s. There are four ANDs here.

Finally, we OR these two together, and this gives a 1. We OR these two together, this gives a 1. We OR these two together, it gives a 0. We OR these two together and it gives a 1. Those are the four ORs. This actually ends up looking a little bit like addition. We did exactly the same set of ANDs. We’ve ANDed all of these things together, but then instead of collapsing it by using full adder circuits, we just get a very simple collapsing with OR gates.

Dwarkesh Patel

But that doesn’t look like n times p.

Reiner Pope

This was with n=2 inputs. In the general case, we will have n rows, and we’ll have p bits per row. That gives us the n times p many AND gates. In this circuit I’ve described, almost all of the cost, seven-eighths of the cost, is in reading and writing the register file, and only a tiny fraction of the cost is in the logic unit itself.

This is the problem to solve. This essentially was the state of play prior to the Volta generation of Nvidia GPUs. This kind of thing is what was inside the CUDA cores. This problem statement is what motivated the introduction of Tensor Cores, which are more generically called systolic arrays.

Think about how we’re going to solve this problem. We’re spending almost all of our circuit area on something that we really don’t care about and is hidden to the software programmer, and the thing that we actually care about is not much of the area. Make this one bigger somehow while keeping this at the same size. That’s the goal.

00:26:10 – How systolic arrays work

Reiner Pope

The evolution was that we had baked this much into hardware at this stage. This single line is a multiply-accumulate, and this single thing was baked into hardware. The idea of a systolic array is to go two levels of loops up and bake this entire loop out here into hardware. The idea is that if we have a much bigger granularity fixed-function piece of logic, maybe the taxes we pay on the input and output are much smaller.

Dwarkesh Patel

Interesting. It sounds like you’re suggesting that if you go up one step in the matrix multiply loop, you can tilt the balance more towards compute than communication.

Reiner Pope

That’s right. There are two effects we’re going to take advantage of here. One is that we can do more stuff per every trip through our register file. The other is that in some of this loop, we can take advantage of certain things staying fixed.

Visually, we’re going to look at this matrix multiplication. This portion of the loop corresponds to a matrix-vector multiplication. We’ll take a matrix and multiply it by a vector. How do we do this? Every column gets multiplied by the vector and then summed. We’re going to sum along columns.

This 0 and 3 gets multiplied by the 3 and 7 and gets summed, and then the 1 and 2 gets multiplied by the 3 and 7 and gets summed. There is a multiply-accumulate associated with every single one of these entries in the matrix. We’ll draw out these four multiply-accumulates.

Dwarkesh Patel

Just to make sure I understand why there are four multiply-accumulates: each entry in the column that corresponds to the output vector is a dot product, and in this case it will be two multiplications and then the addition of those two multiplications. You’re accumulating...

Reiner Pope

Really there’s only one addition per dot product, but we like to start with zero.

Dwarkesh Patel

But it includes the initialization of zero.

Reiner Pope

Yeah. We want to have quadratically more compute. We have x times y as much compute as we had before. But we want to aim for having only x times as much communication. The intention is to get this advantage term going as y.

We’ve laid down the multiplications. We want to bring in a vector of size two, and that is already in line with our columns target. That’s fine. However, we need to manage the communication of this matrix, which exceeds our budget of x.

The idea is that in an AI context, this matrix is going to stay fixed for a long period of time. We’ve got some register files sitting over here.  The amount of stuff coming out of this register file… this is the term that we want to go as x, in some sense. We don’t want to bring this full matrix in from the register file every cycle, because that would cost too much in terms of wiring from the register file.

Our key trick is that this matrix can be stored locally to the systolic array. We’ll store these numbers 0, 1, 2, and 3 in a gate called a register that physically stores these numbers, and we’re going to reuse these numbers over and over again for a large number of different vectors.

Dwarkesh Patel

The optimization here is that the nature of matrix multiplication is that you can store this square quadratic thing directly where the logic is happening, which has an extra dimension compared to the inputs that you keep swapping in and out.

Reiner Pope

That’s right.

Dwarkesh Patel

This is the nature of what a matrix multiplication is. You do a lot of multiplication to get one value out. A dot product is the result of a lot of multiplications. So that optimization means you can stuff a lot of multiplication in before you get some value out of it.

Reiner Pope

That’s right. Just to complete the picture of concretely how that looks: I swapped the 3 and the 2 here. Just like this 0 and 3 is going to multiply by the 3 and the 7, we’re going to form a dot product along columns here. We’re going to feed a 3 and a 7 in here. This feeds into this multiplication and also feeds into this multiplication. Likewise, the 3 feeds into here and also into here. Then we’re going to sum along here. Starting at the top of a column, we feed in 0s, and then coming out the bottom we get results.

Visually, there’s a dot product performed along columns in a matrix, and that maps exactly to what is done spatially in the systolic array. This is one dot product summed vertically, and this is a second dot product also summed vertically.

What is the data that needs to go into and out of the register file? We have x amount of data coming out on the output, and we also have x amount of data coming from the input. With respect to the input and output vectors at least, we’ve met our goal of having only x as much data going in and out of the register file.

This leaves open the question: I said the weight matrix is stored locally in the systolic array, so how did it get there in the first place? At some point, you need to boot your chip and populate this data, so where did that come from?

The trick is that we just do it very slowly. We very slowly trickle-feed it into the systolic array. The simplest strategy is that we run this daisy chain: feed a number in here, and on the next clock cycle it will move down to the next entry of the systolic array. We can do that in every column in parallel, this is also going to come from here, and that gives us another factor of approximately x units of bandwidth coming in.

Dwarkesh Patel

Would you mind repeating that sentence one more time?

Reiner Pope

We know that we’re going to be bringing numbers only rarely into the matrix. We just want to come up with any construction at all such that the amount of wiring that crosses the boundary of the systolic array is bounded to x and not go as xy.

A particularly simple strategy is that we bring a number into the top row of the systolic array in one clock cycle. Then, for y consecutive clock cycles, we bring in the top row every time and shift all the other rows down by one. This keeps the wiring that needs to come from this expensive register file only down to a factor of x rather than xy.

Dwarkesh Patel

I see. There are two questions in terms of communication: communication time and communication bandwidth. You’re saying that since we’re only going to be loading this in once, let’s minimize bandwidth, because bandwidth equals die area. We load it in slowly over smaller lanes because we’re just going to keep this value in there for a while.

Reiner Pope

Exactly.

Dwarkesh Patel

It’s interesting to me that when we were talking last time about inference across many chips, the big high-level thing we’re trying to optimize for is increasing the amount of compute per memory bandwidth, that is to say, per communication. Here also, we’re trying to increase the amount of actual multiplies or additions relative to transporting information from registers to the logic. In both cases, you’re trying to maximize compute relative to communication.

Reiner Pope

This shows up all the way up and down the stack. This is close to the bottom, to the gates. There’s a version that’s maybe even closer to the gates in the precision of the number format that you choose to use. We saw that same effect. There’s a squared versus linear term going on both purely in the precision of the ALU, but also in the size of the matrix.

This unit is the next bigger unit. We had the multiplication circuit, and on top of that we have a pretty large systolic array. I drew it as 2x2, but older TPUs were described as 128x128 of this circuit shown here. This ends up being the most efficient known circuit for implementing a matrix multiply.

Dwarkesh Patel

We’ve talked about how it seems obvious that you should try to maximize compute relative to communication. What are non-obvious trade-offs that keep you up at night, about whether you should do X or Y and it’s not obvious what the answer is?

Reiner Pope

Most of the decisions in chip design are sizing decisions. Already in what we’ve drawn so far… AI chips all have this circuit in them. They have a systolic array, and somewhere near it a register file providing inputs and outputs.

Even within this scope, the sizing questions you have are: how big should I make my systolic array, and how big should I make the register file? These two questions are coupled. One way to think of it is to set a budget for what percentage of chip area you want to spend on data movement. Maybe I say that I want this to be 10% and the systolic array to be 90%.

Then I can size my register file. Bigger register files are more flexible. They allow me to get more application-level performance out, but they take away from the area spent on the systolic array.

00:39:11 – Clock cycles and pipeline registers

Dwarkesh Patel

Where does the clock cycle of a chip come in? What determines what that is? And what is a clock cycle of a chip?

Reiner Pope

At baseline, it’s worth observing that chips are incredibly parallel. You’ve got 100 billion transistors in a chip. A key thing you need to do whenever you have massive parallelism is synchronize between the different parallel units.

In software, typically you have these very expensive synchronization methods like a mutex. One thread will finish what it’s doing, grab a lock stored somewhere in memory, and notify the other thread that it’s done. On chips, we take a very different approach. Every nanosecond or so, all circuitry in the chip will pause for a moment and synchronize. That is the clock cycle. The entire chip typically goes in lockstep to the next operation in one fell swoop.

What this looks like in circuitry is that the clock is mediated by registers, which are these storage devices we’ve drawn elsewhere. The way to think of it is: I have some storage holding a bit, which might be 0 or 1. Then I have some cloud of logic, which maybe is this systolic array or multiplier. I have a bunch of inputs feeding into this cloud of logic, and eventually there’s going to be some output register that it writes to.

There is a global clock signal driving all these registers. At a certain instance in time, when the clock strikes, whatever value happens to be on that wire at that instant is what gets stored.

The challenge is that I would like to have my clock speed run as fast as possible. If I run at two gigahertz, I get twice as many operations done per second as if I run at one gigahertz. But what that ends up meaning is that I’m very sensitive to the delay through this cloud of logic, because any computation happening in there needs to finish before the next clock cycle hits. A major point of optimization on any chip is to make this delay as short as possible.

Dwarkesh Patel

Interesting. The constraint here seems to be that if you add too much logic, you might risk missing the clock cycle. But if you don’t add enough, you’re leaving potential compute on the table. Is there ever a situation where you take a probabilistic chance that a computation finishes, or is it strictly that it either finishes by the clock cycle or it doesn’t?

Reiner Pope

In standard chip design, you margin it such that there is a probability, but it’s many standard deviations out. For all intents and purposes, it is a reliable part and will always meet the clock.

There are some weird exceptions, like clock domain crossings where you go from one clock to another. Then you actually do have to reason about this probability. But in the main path, you margin it such that you’ll get there 25% of the clock cycle in advance, making it very unlikely that it misses.

Dwarkesh Patel

Where the clocks synchronize, where the registers are, is this something you determine as a chip designer? Or is it an artifact where you want a certain sequence of logic, and the software you use to convert your Verilog into what you send to TSMC just determines that to make it work, you have to put a register here, here, and here, making sure no single step makes the whole chip’s clock cycle longer than it has to be?

Reiner Pope

Inserting them is actually a huge part of the work of designing a chip. It’s done by a combination of manual and automatic methods.

To show the very dumb version of what you can do here, you can take this logic and split it in half. Instead of just one cloud of logic, I can have two smaller clouds of logic that do the same thing, but split them up by a register. If you split it in the middle, you can hit twice the clock frequency. That’s great, you get twice the performance, but at the cost of an extra register, which means more storage.

Dwarkesh Patel

Stepping back, why do we need to synchronize the whole chip? If you imagine playing Factorio or something, there’s no global clock cycle. Things are just done when they’re done. There’s iron on the plate, and you can take it if you want.

Reiner Pope

Taking that analogy, the thing you need to be mindful of is if I have two different paths through some logic. Say I have to do computation f here and computation g here, and they’re going to meet for computation h.

There’s going to be manufacturing variance. In some chips f will take a little longer; in some chips g will take a little longer. If I have a signal propagating through, and the results from f and g have to meet up at h, what can go wrong is that f gets there early and it meets the previous value of g, or the next value of g.

Dwarkesh Patel

Ah. And h needs to know when to start, when the next iteration has…

Reiner Pope

Exactly.

Dwarkesh Patel

This explains why different chips made at the same process node, the same TSMC technology, can have different clock cycles. Two chips made at 3 nm might have different clock cycles based on whether they were able to optimize to ensure no single critical path is so long that it slows down the whole chip’s clock cycle.

Reiner Pope

That’s right. This optimization I showed here is called pipeline register insertion. We’ve inserted a register in the middle of the pipeline. This is a pure trade-off between clock speed and area. That is the easy case. There is a harder case too. I drew out a pipeline of logic, but in other cases you may have some calculation which actually feeds back in on itself. It runs some function f and then writes back to itself. For example, this might be an addition where you’re adding a number every clock cycle. This little circuit essentially sums all the numbers presented on different clock cycles.

The challenge is, if this plus takes too long, what can I do? If I try to put a pipeline register right in the middle of it, it changes the computation that’s done. Instead of forming a running sum of everything that comes in, I will actually have two different running sums. I’ll end up with a running sum of the even numbers and a running sum of the odd numbers. This constraint—where I have a loop in my logic, which all chips have somewhere—is the hardest thing to address and sets the clock cycle.

Dwarkesh Patel

I don’t understand why it would be a problem to have that. I’m not even sure what it would mean to have a register there. Is it a sort of atomic operation?

Reiner Pope

Well, plus is not really atomic.

Dwarkesh Patel

As you just demonstrated.

Reiner Pope

It took a whole lot of work to do a summation. You can take the early parts of that work, stick a register in the middle, and then take the late parts of that work.

Dwarkesh Patel

Okay. TSMC offers a PDK which specifies the primitives of logic they can grant you in the chip. It’s up to them to determine that no primitive is bigger than the clock cycle they’re hoping a process node targets. But other than that, can’t you just take all the primitives from TSMC and keep adding registers between them as much as needed until you get to your desired clock cycle?

Reiner Pope

As a logic designer, the chip architect sets the clock cycle. For example, the primitives you get from TSMC are on the order of AND gates or full adders. It depends a lot on voltage and which library you choose, but generally you can have about 10, 20, or 30 of these sequentially in a clock cycle. These primitives are very fast, maybe 10 picoseconds.

As a logic designer, in principle, if you just had a register and an AND gate in a loop, you could get an insanely fast clock speed, more than four, five, or six gigahertz. But if you take this really simple circuit and look at the area you’re spending here… This is called one gate equivalent in size, so unit of one in area. This thing is maybe a unit of eight in area.

Again, almost all your cost becomes synchronization or communication cost compared to the actual logic. This would be a case where you’ve gone too far. You’ve made your clock speed really fast at the cost of spending almost all of your area on pipeline registers.

Dwarkesh Patel

Interesting. So you’re hinting at a dynamic where you can have a really fast clock speed but you’re not getting that much work done. You can have low latency but low throughput.

Reiner Pope

It hurts your throughput, in fact, because the throughput of your chip is the product of how much you get done per clock cycle—which is based on area efficiency—times how many clocks you get per second.

Dwarkesh Patel

This is actually so similar to the thing we were discussing last time about batch size, where if you have a low batch size, any one user can receive their next token really fast, but the total number of tokens processed in, say, an hour will be lower than it could otherwise be.

Reiner Pope

Exactly. You get less parallelism out if you drive your clock speed up really high.

00:51:51 – FPGAs vs ASICs

Dwarkesh Patel

I remember talking to an FPGA engineer at Jane Street, Clark, who helped me prep for the previous interview we did together. He was explaining why they use FPGAs. I imagine that for high-frequency trading, throughput is less important than latency, so having very specific control over the clock cycle in a deterministic way is the most important thing. Maybe it’d be interesting to talk about why you can’t just achieve that with an ASIC, or why you might use an FPGA to have deterministic clock cycles for high-frequency trading.

Reiner Pope

Let’s consider the business case for an FPGA versus an ASIC. FPGAs and ASICs use largely the same conceptual model. You have a series of gates built from small primitives—ANDs, ORs, XORs—connected together with wires running in a fixed clock cycle. Anything you can express in an FPGA you can express in an ASIC too. It will be about an order of magnitude cheaper and have better energy efficiency on an ASIC than an FPGA.

The trade-off is that the first FPGA costs you $10,000, whereas the first ASIC you make costs $30 million because it requires an entire tape-out. The business use case for an FPGA is when you want something that has very deterministic latency, fast runtime, and high parallelism, but you are going to change the workload frequently, maybe every month. You don’t want to pay that tape-out cost every time.

How does an FPGA actually emulate the ASIC programming model in a fixed piece of hardware? At its core, it has the two components we just talked about. It has registers as storage devices, and it has lookup tables (LUTs) which provide all of the gates.

Then there’s a third component. We have a swarm of these registers and LUTs, and they are connected by a big set of muxes. In front of every single one of these, we have a mux which selects an input from everywhere else. We have a whole bunch of different options feeding into all of these things.

What this allows is essentially that when I program my FPGA, I can take all of these components and superimpose a particular wiring which goes through this LUT, feed it into another LUT, send it to this register, and then feed it into another LUT, or something like that.

What I’ve drawn in orange is how you… FPGA means Field-Programmable Gate Array. The orange is what has been programmed in the field, whereas the white is all the wires that must exist in the FPGA in order to actually make the device in the first place.

Dwarkesh Patel

What does it mean to be programmed in the field?

Reiner Pope

Programmed in the field means the device is being deployed in a data center. It’s sitting out in the world, and then you can come and program it.

Dwarkesh Patel

Ah, not field as in like electric field. Field as in like out there in the world, ok.

If I look at how the field programming comes out of the first lookup table and goes into a second one, how does that work?

Reiner Pope

Where are the wires that make that happen? I got a little bit lazy in drawing all of these. Every single device here has a mux sitting in front of it, which can select from all of the nearby circuits that are available. The actual configuration of the FPGA amounts to the mux control. In this mux, we have the data inputs, and we have the control that selects.

There’s a little storage device sitting next to every single one of these muxes saying, “This is where you’re going to source your input from.” Programming it consists of configuring every single one of these muxes.

Dwarkesh Patel

That makes sense. What is happening inside of the lookup table?

Reiner Pope

The lookup table is also going to have a little bit of control telling it what to do. Its purpose is to configurably take the role of an AND gate, OR gate, XOR, or any of those different things. There are many ways you could consider doing that. The way it’s done in traditional FPGAs… A lookup table has four bits of input and one bit of output. How many different functions are there from four bits to one bit? There are 16 different functions.

You can tabulate this as 16 different numbers. You’ve got a table of 0111001, 16 entries. This table is stored in this blue configuration bit. It views these four bits as binary, looks up the relevant row of the table, and emits that bit. This is essentially a truth-table view of lookup tables.

Dwarkesh Patel

Okay, so if you think about an AND gate, OR gate, NOR gate, XOR gate, these all take as input…

Reiner Pope

Those are two-input functions. Sometimes we have a three-input function, like a three-way XOR, or a four-way XOR.

Dwarkesh Patel

In this case, does it just depend on how big it is?

Reiner Pope

The typical size for LUTs is four inputs. It’s sort of just a sweet spot. There’s another compute vs. communication trade-off here. If it has too few inputs, you need to use more LUTs.

Dwarkesh Patel

Basically the lookup table is a truth table. With a truth table, you can program in any gate you want. So instead of a lookup table, you can just think of it as a programmable gate.

Reiner Pope

That’s right. One of the things you can do here is you can see where the rule of thumb that an FPGA is an order of magnitude more expensive than an ASIC comes from. You count how many gates would be inside this lookup table.

We can view this lookup table essentially as one of these muxes. It has to select between 16 different values, so it’s a mux with n=16 options and p=1 bits. As we saw earlier, this circuit costs n times p many gates. So it costs np, which is 16, AND gates, and also 16 ORs.

Dwarkesh Patel

This circuit being the mux?

Reiner Pope

Exactly, the mux.

Dwarkesh Patel

The mux that goes into the lookup table?

Reiner Pope

The lookup table itself you can think of as being a big mux that selects from all 16 rows down to one output. That’s the lookup table.

Dwarkesh Patel

But the way you’ve drawn it here, there’s a mux and then a lookup table.

Reiner Pope

It’s muxes all the way down. There is a second mux that is inside here. This mux is this mux.

Dwarkesh Patel

And the other mux is just saying…

Reiner Pope

where it came from in this mess of gates.

Dwarkesh Patel

Right, and the second mux is, “Okay, now you have one value, but that value is still a four-bit value.”

Reiner Pope

Yeah, I’ve selected four bits from the soup. Then I use those four bits to select which entry in the lookup table I’m going to use.

Dwarkesh Patel

Suppose in the first mux you’re pulling from eight nearby registers as input. That’s a total of 32 bits going in. Out of that, four bits come out. Those four bits go into the second mux, which is inside the lookup table.

Reiner Pope

In this case, these registers are single-bit registers. If there are eight nearby registers and lookup tables, then I have eight bits total coming in nearby. I select from eight down to four different values. There are actually four different muxes, a little mux associated with each of these input bits. Each of them is selecting one out of eight.

Dwarkesh Patel

Where are those eight coming from?

Reiner Pope

Nearby registers and other LUTs.

Dwarkesh Patel

Each register is one bit.

Reiner Pope

Yes.

Dwarkesh Patel

I guess AMD or whoever makes these FPGAs still has to be opinionated about which registers are connected to which registers. You can program in the actual gates, but they add a wire in the connect… the communication topology, right?

Reiner Pope

You get flexibility at a local grain. There’s a nearby neighborhood you can select from, but for more coarse, long-distance connections, they form an opinion on that.

Dwarkesh Patel

And the reason it’s 10x slower is why?

Reiner Pope

If you look at the cost of building this lookup table, it’s 32 gates. It can give me the equivalent of—what’s an interesting thing I can do here—a four-way AND gate. A four-way AND means AND, AND, and then an AND of an AND. This is a circuit I could implement in an ASIC directly using three AND gates. Using a LUT, I can also implement it, but it’s going to take 32 gates instead of three.

Dwarkesh Patel

So the overhead is really coming from the fact that there’s a more concise way to describe a truth table than listing out every single possible combination of inputs, which is just to write out the gate.

Reiner Pope

Yes, to place down the polysilicon and the wires and so on.

01:03:25 – Cache vs scratchpad

Dwarkesh Patel

Interesting. One important point you made to me is that the reason they prefer FPGAs to CPUs is that they get deterministic clock cycles. They know when a packet will come in and go out. Why isn’t that a guarantee in CPUs?

Reiner Pope

You can actually design a CPU that has deterministic latency as well. In fact, the processors inside a lot of AI chips also have deterministic latency. Groq has advertised this. TPUs have that in the core as well.

The challenge is getting deterministic latency and high speed at the same time. Non-deterministic latency comes from specific design choices in a CPU. It’s actually possible to remove those design choices and make a CPU with deterministic latency, but those are not very attractive in the market, so people don’t make those CPUs anymore.

In some sense, deterministic latency is a simpler starting point, and some chip designers have added things in to make it non-deterministic. To take a concrete example, probably the most important source of non-determinism on a CPU is the CPU cache itself.

In a CPU, you have the CPU die itself, and then DDR memory off on the side. You have a cache system inside that remembers recent accesses to DDR and stores them. When I’m running through my CPU instructions, every time I have an instruction that accesses memory, it first checks if the data was stored in the cache. If not, it fetches it from DDR.

This is a huge optimization. The cache is two orders of magnitude faster than the DDR. If you never used the cache, basically all programs would run a hundred times slower. The presence of a cache is absolutely necessary for a CPU to run at a reasonable speed.

But whether or not you get a cache hit depends on the ambient environment of the CPU: what other programs are running, what has run recently, and what the random number generator inside the cache system is doing. That is a big source of non-determinism in the runtime of a CPU.

That is the memory system for a CPU. The big thing you can do differently is, instead of having the hardware say, “I’m going to read memory” and then the hardware decides whether or not it comes from the cache, you can bake this decision into software, a different design philosophy.

You see this in TPUs, for example. I’ll draw the same diagram, but I’ll call it a scratchpad. The main difference is… This would be a TPU, and you have HBM in this case rather than DDR, but it’s still an off-chip memory. Instead of the software saying “first access memory” and letting the hardware decide, you have one kind of instruction that goes to the scratchpad and a totally different kind of instruction that goes to HBM.

This style is generically known as scratchpad instead of cache. The key distinction is that you have one kind of instruction that says “read or write scratchpad,” and a totally different instruction that says “read or write HBM.”

Dwarkesh Patel

So scratchpad being the cache.

Reiner Pope

Yeah, this thing here is the scratchpad.

01:07:27 – Why CPU cores are much bigger than GPU cores

Dwarkesh Patel

Stepping way back: people say computers have the “von Neumann architecture”, where there’s this serial processing of information. Maybe it’s just because we’ve been talking about parallel accelerators, but the FPGA is super parallel. The AI accelerators, the TPUs, are super parallel. Even CPUs are super parallel if you think about all the cores they have. In what sense is modern hardware actually the von Neumann architecture? Is that actually a fair way to describe modern hardware?

Reiner Pope

I think it’s a fair way to describe CPUs. The amount of parallelism you get on a CPU is about 100 cores times maybe 16-way vector units, so about 1,000-way parallelism on a CPU.

Dwarkesh Patel

One question: there is a die being used for the CPU, and if there are fewer threads, just as a matter of transistor voltages switching on and off, is it that there’s literally one control flow—a small part of the die—where voltages are switching on and off?

How do you actually occupy the die area of a CPU…

Reiner Pope

If there are so few cores, what are you spending all of the die on?

Dwarkesh Patel

Yeah, what is happening there?

Reiner Pope

The cores are just much bigger and more complicated. We should compare a CPU core, which takes up one one-hundredth of the die, to a LUT. A LUT is only 16 gates. It’s clear why there are so many more LUTs in an FPGA than cores in a CPU.

But why are there more CUDA cores, for example, than CPU cores? What’s the difference between a CPU and a GPU? Inside the CPU, one big use of the area is the cache. Very little is actually the ALUs. Mostly it’s these register files rather than the logic units. Both of those have equivalents in a GPU, so that’s not a big difference.

But the thing that does not have an equivalent in a GPU is the branch predictor. There is a whole big area in the CPU which is just a bunch of predictors saying when the next branch will be and where the branch target is. Stripping a lot of that out, as well as making these register files tighter, drives a lot of the GPU gains over the CPU.

Dwarkesh Patel

What is the purpose of the branch predictor? To execute both branches at once, or what does it do?

Reiner Pope

The issue is that when I’ve got a series of instructions, if I have a branch, the actual step of processing an instruction takes a really long time. It takes maybe five nanoseconds.

The time to notice that I’ve got a branch, evaluate whether the Boolean is true, update the program counter to the new target, and then read from the instruction memory could take five nanoseconds to finish. So in reality, this may finish somewhere down here. I want to run a clock speed that is much faster than what five nanoseconds allows. Five nanoseconds is a 200 MHz clock speed. I would like to run at one or two gigahertz.

So I need to run other instructions while the branch is being evaluated. I just want to keep running the instructions that happen after me. But that might have been wrong. If the branch ended up being taken, then I need to know that instead of evaluating these instructions, I actually need to jump to wherever the target is and run those instructions instead. The purpose of the branch predictor is to predict, five cycles earlier, that a branch is going to happen, before you even get to that instruction.

01:12:00 – Brains vs chips

Dwarkesh Patel

If I think about how the brain works versus what you’re describing here, at a high level the differences might be that while you can do structured sparsity in these accelerators and save yourself some area that you would have otherwise had to dedicate to gates, in the brain there’s unstructured sparsity. Any neuron can connect to any other neuron, and not in ways where they have the column aligned.

Then there’s the fact that memory and compute are co-located. Although I guess you could say in a way the memory and compute are co-located on these dies too.

Reiner Pope

This is exactly the co-location, in some sense, of the memory and compute.

Dwarkesh Patel

So maybe that isn’t a big difference. Another big difference is that the clock cycle on the brain is much slower than on computers. Partly that’s to preserve energy, because the faster the clock cycle, the bigger the voltage needs to be in order for the signal to settle and to identify what state a transistor is in.

Reiner Pope

That’s right.

Dwarkesh Patel

I don’t know if you have any commentary on what the brain might be doing versus how these chips work.

Reiner Pope

Let’s take the clock speed one first. The clock speed is quite high on a chip because that drives higher throughput. When we compare a GPU running some workload, it’s running batch size 1,000. Whereas the brain is not running batch size 1,000, there’s only one of me.

You could imagine saying, “Take a GPU and instead of running at a gigahertz, run it at a megahertz,” and that would start to look a little more like the equivalent things you’re talking about in the brain. But in the way silicon works, that does not give you a 1,000x advantage in energy efficiency.

What it ends up looking like is you just run this circuit once to stabilization, and then it will sit idle for a long period of time. It doesn’t consume a lot of energy while it’s sitting idle because most of the energy is consumed in toggling bits from zero to one and back.

Let’s talk about the energy consumption of a circuit like this. The way to think of a bit being stored is that you’ve deposited some charge in a capacitor sitting somewhere in the chip implicitly. It becomes charged when the bit becomes a one, and then it becomes discharged when it next goes to a zero.

That cycle of charging the capacitor and then dumping that charge out to ground is where the energy is consumed. This is called the dynamic or switching power, and it’s most of the energy consumption of a chip. There is some other energy consumption just coming from the fact that insulators aren’t perfect, but we’ll discard that. Most of the energy consumption comes from toggling from zero to one and back to zero.

If you run a chip much slower and you only clock it once every thousand clock cycles, you will have 1,000 times fewer transitions. It will be about 1,000 times less energy consumption. But it’s not a substantial advantage in energy efficiency.

01:15:33 – A GPU is just a bunch of tiny TPUs

Dwarkesh Patel

Okay, so you described how a TPU works at a high level. What is the difference at a high level between how a GPU and a TPU work?

Reiner Pope

There is a high-level organization principle that is different, and then inside the cores things are different. Looking at the high level, we’ll take a GPU and a TPU and see what the top-level block structure looks like.

If you think of this as the whole chip in each case, the organization of the GPU is mostly a bunch of almost-identical units, which are the SMs. They’ve got an L2 memory in the middle, and then a bunch more of these SMs on the bottom. So there is a fairly regular grid of cores.

If we look at a TPU in comparison, you end up with much coarser-grained units of logic. You end up with just a few matrix units, which are the big systolic arrays. In the middle you’ve got some vector unit, and then you’ve got your matrix units at the bottom. These matrix units with a vector unit in the middle make up the whole TPU chip.

You can think of scaling this thing down into a really tiny unit with a smaller matrix unit and a smaller vector unit, and that is sort of what an SM is. From a very high-level point of view, the GPU has a lot of tiny TPUs tiled across the whole chip.

Dwarkesh Patel

Oh, interesting. You’re suggesting the tensor core within a streaming SM is analogous to an MXU?

Reiner Pope

Yeah, it’s all very similar.

Dwarkesh Patel

I see. If you had more lack of structure, having a bunch of tiny TPUs makes a lot of sense. Whereas if you just have huge matrix multiplications, you might want to avoid the cost of having individual SMs with their own registers and warp schedulers. Why not just make a huge thing and amortize those costs across the whole thing?

Reiner Pope

This shows up in how large you can grow things. We’ve seen this theme, especially with the systolic array, where a larger systolic array amortizes the register file costs better.

This design allows you to have larger systolic arrays, whereas the GPU design constrains you to having small units of everything. There is a trade-off, however. Because of this coarse-grained separation of things, you need to move a lot of data from the vector unit to the matrix units, through just two lines of perimeter here.

If you look at the equivalent thing in a GPU, you’ve got vector units everywhere, and you can move data through many different lines. The amount of data you can move between a vector unit and a matrix unit is actually much higher in a GPU than in a TPU. Instead of having to move all the data through just two lines, you’re moving it through 16 lines of wiring in a GPU.

Dwarkesh Patel

Right. But also you might have to move across less area.

Reiner Pope

Which is an energy saving as well. So if you can operate entirely within an SM, the data movement is much smaller. But the moment you want to operate across SMs, it becomes more complicated and expensive.

Dwarkesh Patel

So you don’t have to comment, but one might expect that a thing MatX might try to do is get the GPU-like smaller structure of systolic arrays surrounded by SRAM, but at the same time make it so that the things you need in an SM to support the CUDA architecture—which take a bunch of space—you might discard.

Reiner Pope

We’ve talked publicly about something we call a splittable systolic array, which in some sense you can think of as big systolic arrays that can be small systolic arrays too.

Dwarkesh Patel

Cool. Okay, I think that’s a good note to close on. Reiner, thank you so much.

Reiner Pope

Thanks, Dwarkesh.

To be clear, these people are obviously very competent and clever. But when you think of ASI, you don’t think of Trump, but more so.

The person who kept pressing this question was correctly pointing out that I basically defined intelligence as power. And by this definition, Stalin was the most intelligent person who ever lived.

Now, of course, you could change the definition of intelligence to something more like, manipulate abstract concepts and rotate shapes.

But notice that the most powerful people in the world do not max out this quantity. The correlation between extreme power and this kind of intelligence might be even weaker than the correlation between extreme power and height. The physicists are not running the world.

We tend to conflate power-seeking AI and superintelligent (in science and tech) AI. I’m not denying that AI can be power-seeking. Whatever skills and drives Donald Trump has could be embodied in a digital mind. I’m simply pointing out that the way AI systems are currently becoming smarter (by getting trained to be to be really good at specific economically valuable tasks like coding) is not that strongly correlated with power.

We often talk about power in this way that misunderstands how it is actually derived in our world. Our intuitions are primed by games like Diplomacy or Go, which are designed to isolate and reward a g loaded kind of strategic reasoning.

But in the real world, power is more the product of having the authority and trust to get lots of people to collaborate with you, rather than some galaxy brain scheming capability. Trump is not powerful because his brain, considered in isolation, is the most effective optimization engine on Earth. He is powerful because the government which hundreds of millions of people consider legitimate gives him a lot of authority.

A group versus individual level analysis is useful here. As Garett Jones has written a lot about, individual IQ is only modestly correlated with individual income, but national IQ is strongly correlated with national outcomes. This is because intelligence has a lot of spillover effects - smarter societies cooperate more, save more, and can coordinate to build things like space shuttles and semiconductors.

Richard Trevithick, who invented the high-pressure steam engine, died in poverty, buried in an unmarked pauper’s grave. But the fact that 18th and 19th century Britain had lots and lots of people like Trevithick contributed to Britain being able to set up a global empire and outcompete lots of backwards principalities around the world.

It seems to me that the right mental model is that automated firms will outcompete everyone else in normal capitalist ways, rather than a single AI outthinking everyone else.

On why pretraining runs fails

Had an interesting chat with someone on why pretraining runs often fail. It was very interesting to get a sense of all the tangible ways that things can get fucked, and why training is such a precarious operation. At a high level, breaking causality, and adding bias, seem to be key culprits.

Breaking causality:

• When you do expert routing, you first go through the router, which gives you a score of how much each token wants each expert. There’s two ways to proceed from here: 1. Token routing, where you read the scores from the token’s perspective, and allocate to each token’s top k experts. Problem is that you could end up with wildly unbalanced allocation across experts, which is terrible for performance. Alternatively, you could (and only in training) do expert choice, where you just split the tokens by which are more relatively preferred by each expert. This way you can enforce that each expert gets roughly the same number of tokens. But the big problem is that this breaks causality, because which expert token n gets allocated to may depend on which expert token n + k might be router to. And breaking causality is very bad, because you’re getting information in training (and updating based on it) that you wouldn’t see in deployment.

  • Rumor is that this explains why Llama 4 was underwhelming.

  • I guess you could do expert choice during prefill inference? But maybe it doesn’t work well in practice to allocate tokens to experts which would not have received that token in actual training.

  • Tbh I don’t fully understand why breaking causality is so bad. I understand you can’t see beyond causality in real inference. But why is this minor deviation such a big issue?

• Another thing that can break causality is token dropping. Where experts just ignore the tokens in the batch that they’re supposed to process, but which rank not so strongly, and cutting whom would spare going outside padding. This breaks causality cause a later token being more strongly matched to this expert might lead to an earlier token getting ignored.

  • Apparently this was an issue with Gemini 2 Pro.

Adding bias:

• Bias much worse than variance - variance can average out, but bias compounds

• Apparently the original GPT 4 training was slow and got initially fucked because of the following bug: they were using FP16 on their collectives like all-reduce. FP16 distributes its granularity according to logarithmic density - between 1 and 2, the mantissa bits carve the interval ~0.001 apart. But 1024 and up, the mantissa might be carving the interval by multiple whole number values. Suppose some collective involves adding 1 + 1 … 10,000 times - you could get in a situation where as soon as you get to 1024, you add 1, it goes to 1025, you round down to the nearest interval at 1024, add one again. And so the calculated value is 10x off the real value. Huge issue if you’re trying to sum many small gradients into a large accumulator. And imagine how hard the bug must have been to find!

Implications for AI training:

• Some of the people who think we can cure aging argue that there’s basically 5 different ways people die of old age (heart disease, cancer, etc), and that if we cure these 5 different diseases, then we’d basically have solved aging. You could ask a similar question about these failed pretraining runs - are there 5 different ways training runs fail, in which case once a lab figures out numerics and , you’ll just have smooth sailing, or will you keep seeing new bespoke issues emerge at each new level of scale? The person I talked to seemed to think the later - he pointed out that even within numerics, there’s so many ways you can fuck things up. And new ones will keep emerging at scale.

• Bearish on AI fully automating kernel writing anytime soon. Presumably this is because he thinks it’s more of an AGI complete problem than some give it credit for. There’s another school of thought that says, “Hey, which kernel gets attention or MLP to run fastest on this scaleup is a super verifiable domain, thus we can RL to superhuman performance easily.” But he says, it took Nvidia, which has the best kernel engineers in the world, a long time to optimize for Blackwell, which suggests that actually it’s quite hard, and might not be super easy to close the loop on.

• Sometimes people say inference for RL generation and inference for end user generation is basically the same. But this person pointed out that in RL inference, numerical drift between inference and training engine can cause these subtle off policy biases, which matter a ton for highest quality training. But are not an issue if just serving to users.

• Emphasized how important it is to have a disciplined process for amalgamating compute multipliers, because of the risks of stacking up bugs with subtle biases.

Pretraining parallelisms

Notes from an excellent lecture that Horace He gave my friends and me.

What made this lecture so good is that Horace built up the whole topic as a chain of problems and solutions: here’s what we want to do, here’s why it breaks, here’s how we fix it, here’s why that fix eventually breaks too. Most explanations just list out a hodge podge of different strategies, without ever connecting them to the problems they solve or explaining why you’d pick one over another.

• Equation for pretraining flops = 6ND. 2 FLOPs per parameter per token for the forward pass (multiply + add). Backward pass is 2× forward because you compute gradients w.r.t. both input matrices. So 2 + 4 = 6.

• Okay we can’t do all this on one GPU. So how do we split up this problem? The obvious solution is to do data parallel - where you copy the model weights across each GPU, and you just do a part of the batch on each GPU.

  • The obvious problem is that each GPU only has a limited amount of HBM - B300 is 288GB - and this is not enough to store the weights as models get bigger and bigger, much less their activations.

• Okay so next thing we try is fully sharded data parallel - each GPU only stores 1/N of the parameters of each layer - before processing each layer, you all-gather the full layer’s parameters from all GPUs (each GPU only stores 1/N of each layer). After processing, each GPU discards the gathered parameters.

  • It was emphasized that this is the go to default. And you only move on from this when having too many GPUs forces you to move on, for reasons explained later. The reason this is the default is that it’s trivial to overlap compute and communication time - that’s because the only thing being communicated is the weights, which are not dependent on what happened in the layer before, so you can start all gathering the next layer while you’re still computing this layer. Compare this against tensor or expert parallelism, which do need to share activations for one layer before you can process the next one. The problem with pipeline parallelism is bubbles as explained below.

  • From a comms volume perspective, FSDP looks insanely expensive at first — you all-gather every layer’s full weights across all GPUs, use them for one matmul, then throw them away. But this ignores what regular data parallelism already costs you - in regular DP, you still need to do an all reduce after every layer of the backwards pass in order to sync the batch’s gradients across all the GPUs. That all-reduce has comms volume of params × 2. FSDP adds all-gathers — one per layer in the forward pass, one per layer in the backward pass. But an all-gather is half the comms volume of an all-reduce. So naive FSDP comms volume ends up being # params * 4 (all gather forward and back, plus all reduce on back). You can do even better: since each gradient shard only needs to end up on the one GPU that owns it, replace the all-reduce with a reduce-scatter (which skips the final broadcast step). That gets you to params × 3 total — a 50% overhead over vanilla DP.

• So why can’t you always just do FSDP?

  • Comms crossover: You want your compute time to be greater than your comms time - you don’t want to be bottlenecked on comms. But since compute time for FSDP decreases as you increase the number of GPUs, and comms time does not, as you scale the number of GPUs on FSDP, your MFU can totally crater. When this happens, you need to add pipeline parallelism too.

    • Compute time = (6 * # tokens * active params) / (compute per GPU * number of GPUs)

      • This decreases as you increase number of GPUs

    • Comms time = (# total params * 3) / (nv link domain size * infiniband BW)

      • Comms time does not increase as you add more domains. This was really confusing to me. Each domain collectively holds all the parameters, and you need to sync gradients across domains after each layer of the backward pass. You’d think that adding more domains means more hops in the ring, so the all-reduce gets slower. But the standard ring algorithm splits the message into one chunk per participant. More domains means more hops, but proportionally smaller chunks per hop. (This breaks down when chunks get so small that per-hop latency dominates, at which point you switch to tree algorithms.)

        • Technically, you can do better than a naive single all reduce for the gradients between all the domains. You do a hierarchical collective to optimize comms time across multiple NVLink domains. Key thing to remember is that each GPU in the domain gets its own bandwidth access to infiniband. So you wanna use it all up since interconnect bandwidth is the bottleneck. You do this by trying to do as much as possible within a scaleup before you move out. So you do reduce scatter within a scale up to give each GPU the domain-level reduced gradients for a shard of the layer, then all reduce these shards across corresponding GPUs across domains, then all gather within a domain. This shifts the comms time line down, thus moving the crossover point to the right.

        • Made an animation to illustrate it using Cursor and Composer 2:

  • If you look at the equations, you can see that if you increase batch size, crossover point moves to right, and if you make the model more sparse, moves to the left.

  • Also why TPUs are better at FSDP - because more accelerators within a domain.

• Batch size floor: FSDP is data-parallel, so each GPU processes at least one sequence. Attention is computed within a sequence and can’t (easily) be split across GPUs. If your critical batch size is 10M tokens and sequence length is 10K, you only have 1K sequences — so you can’t scale beyond 1K GPUs with pure FSDP, even if you have plenty of comms bandwidth left.

• Problems with pipeline parallelism (the next addition you’d make to FSDP in order to deal with these issues):

  • The problem with pipeline parallelism is different - there you have bubbles that emerge from the fact that at the beginning of the batch, the GPUs dedicated to the final layers are not being used, and conversely at the end of the batch, the GPUs dedicated to the first layers are not being used. The reason you can’t overlap batches in training to solve pipeline bubbles is that you need to consolidate gradients and update the model before you process the next batch.

  • But also you’re adding architecture constraints - things like Kimi’s attention-to-residuals (where each block attends to all previous layers’ residuals) become very difficult when those residuals live on different pipeline stages. Similarly, interleaving sliding-window and global attention layers could cause load imbalance across stages. Dealing with all this slows down research iteration, which is the greatest sin you can commit.

The organizing question from my interview with Michael Nielson was, “How do we recognize scientific progress?” It’s especially relevant to thinking about what it would take for AI to close the RL verification loop on scientific discovery. But it’s also a surprisingly mysterious and elusive question when thinking about the history of human science.

Some people have this idea that AI is going to be disproportionately good at making scientific breakthroughs. The reason they think this is that 1. Science is ‘verifiable’, 2. AI is absolutely crushing domains that have a tight verification loop - coding, math, etc - because you can RL on these loops.

But the history of human science shows that the verification loop for theories can be on the order of decades and centuries, and even then experiments do not definitely rule out alternatives: Ancient Athenians dismissed Aristarchus (2nd century BC) on heliocentrism because it would imply stellar parallax. The first successful measurement of stellar parallax was in 1838, achieved by Friedrich Wilhelm Bessel.

What we know today as the better theory can often actually make worse predictions: it’s well known that Copernicus’s model of circular orbits around the sun was less accurate than Ptolemy’s geocentric model, which had accumulated millenia of correcting epicycles. What is not well known is that Copernicus’s theory wasn’t even simpler (Ptolemy’s model interpreted the true elliptical nature of orbits using an equant trick where other planets are not moving in uniform circular motion around Earth exactly, but rather an off center point. Copernicus didn’t like this, because it violated his Platonic heuristics - so he discarded the quant trick, which led to a less parsimonious model, since Copernicus had to add more epicycles and epicyclets to make up for it.)

So in what sense was it a better theory in 1543? In some sense, it wasn’t! You couldn’t have known ex ante that heliocentrism married with Kepler’s 3 laws (1619) is a much cleaner and more accurate theory, or that there’s a very beautiful unification of heliocentric orbits and terrestrial gravity (Newton in 1686).

There was one ex ante reason that you should have preferred Copernicus in 1543: his theory required retrograde motion as a natural consequence of his theory, whereas for Ptolemy it was an ad hoc addition. Even more impressively, his theory, developed in 1543, actually predicted the phases of Venus before they were observed by Galileo in 1610. But both of these things were also implied by Brahe’s model, which had set the sun to orbit the earth and then all the planets to orbit the sun.

Under a naive falsificationist framework, you’d have to wait until Stellar parallax was observed in 1838 to know that Brahe was wrong. But obviously the scientific community was able to make progress faster than this. There is some mixture of judgment and heuristics in the progress of science that we don’t even understand well enough to actually articulate, much less codify into an RL loop.

Or consider the case of the discovery of Neptune in 1846. Uranus deviated from its predicted Newtonian path. Le Verrier predicted that an unknown perturbing planet must exist, calculated its mass and orbit, and Neptune was found almost exactly where predicted.

But the Neptune story is symmetric to a failure case. Mercury had an anomalous precession, where the ellipse that shows its orbit would rotate 43 arcseconds more per century than should be implied by the impact of other planets using Newtonian mechanics. This led astronomers to speculate that there’s an unknown planet Vulcan within Mercury’s orbit. But it was resolved in 1915 with Einstein’s General Relativity.

A proper Newtonian would still proceed with the research agenda, but modify it as follows. First, you predict some unknown planet. If it can’t be found, you say it’s so small, it must require a bigger telescope, and you build a bigger telescope. And if you still can’t find it, maybe there’s a cloud of cosmic dust occluding it. If still not found, maybe the satellite’s instruments are being screwed by some unknown magnetic field, and you send a new satellite. At each of these steps, had you discovered a new planet, or some unknown cosmic dust, or some new magnetic field, that would have been a sensational victory for Newtonians.

Ex ante, this is not unreasonable to do! It is only after decades or maybe centuries of patchwork that we can then analyze, are we simply adding epicycles, or is this theoretical framework progressive, in that it makes predictions we wouldn’t otherwise be able to.

What do these examples illustrate? That ex ante it is almost impossible to determine which research programs are progressive (will predict and explain unanticipated new phenomenon) and which are regressive (need to be contorted repeatedly to accommodate seemingly disconfirming new phenomenon).

But the verification loop is often extremely long and weirdly hostile, and even then, experiments do not definitely rule out alternatives (see the discussion in the Nielson episode about how physicist contemporaneous with the 1880s Michelson-Morley experiments thought that it simply ruled out a particular theory of ether. Only Einstein made the full conceptual leap to discard the ether altogether).

This means that big conceptual breakthroughs cannot be easily verified. They are recognized decades or centuries later, when it turns out they were much more productive than the alternatives available. What this means for AI for science is that 1. You can’t easily train an RL loop for big conceptual breakthroughs.

And 2. the society of AI scientists will still need individual AI instances that have idiosyncratic biases and heuristics, and to pursue them unrelentingly for decades on end - for example, like the one Einstein had in insisting that there shouldn’t be some arbitrary inertial reference frame. There should be dedicated people to keep a bunch of dormant research agendas alive in case they turn out to be productive upon further investigation. To understand the kind of intransigent dedication to hypotheses that is needed to preserve correct scientific idea - even in the face of disconfirming evidence - consider the following story: In 1815, Prout hypothesized that the atomic weights of all pure chemical elements are whole numbers, because experimentally, most elements seem to come out like this. But there’s many anomalies - for example Chlorine’s atomic weight is measured at 35.5. And so Prout’s school claimed that maybe the chemical substance in which these elements appeared were impure. But there seemed to be no chemical reaction that could get rid of the impurities. And then they said, maybe it’s fractions of full atomic weights - but the closer you measure, the less natural the fractions seem to get - Chlorine goes from 35.5 to 35.46. It takes until almost a century later for people to realize that these measurements are showing multiple isotopes of the same element, which can be separated physically, but have no chemical distinguishing characteristics.

What I’m trying to say is that ex ante, one couldn’t have known which research program would be more productive. We need to invest in all of them concurrently. But that investment looks like a bunch of different individual scientists being super unreasonable and obstinate about propping up their preferred research agenda.

What does the parallel discovery of a deep idea like Darwinism tell us?

The Origin of Species was published in 1859. Principia Mathematica was published in 1687, two centuries earlier. Conceptually, it seems like natural selection is much simpler than the theory of gravity. A contemporary of Darwin’s, Thomas Huxley, read the Origin of Species and said, “How extremely stupid not to have thought of that!” Nobody ever said the same for not beating Newton to the Principia. I wonder if the reason this happened is that, while Darwin’s theory is conceptually simpler, it cannot be decisively tested. The evidence is circumstantial, retrospective, and cumulative. There’s no equivalent of Newton running the numbers on the moon’s orbital period and radius, and confirming that it corresponds to his equations.

Also you need this concept of deep time. Charles Lyell published the Principles of Geology in 1830, which gave Darwin the vast stretches of time that natural selection needed. And the fact that Darwin and Wallace basically arrived at evolution at the same time (and both credited Lyell’s contribution) does suggest that these underrated intellectual footholds were quite important (geology, paleontology of ancient extinct species which showed intermediate species (in some cases between apes and humans), biogeography from voyages and age of colonization, more sophisticated artificial selection like pigeon breeding). It’s interesting that an idea whose essence must have been obvious to herders and parents for thousands of years actually required many millennia of ancillary intuition pumps to fully spell out.

The pattern of parallel discovery in science and technology is very interesting, and seems to contradict this vibe that certain innovations could have happened earlier much earlier than they really did.

Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn.

Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second.

Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside.

Watch this one on YouTube so you can see the chalkboard.

And check out the flashcards I wrote to retain the insights.

Sponsors

• Cursor‘s agent SDK let me build a pipeline to generate flashcards for this episode. For each card, I had an agent read the transcript, ingest blackboard screenshots, generate an SVG visual, and run everything through a critic. A durable agent is much better at this kind of work than a chain of LLM calls, and Cursor’s SDK made it easy. Check out the cards at flashcards.dwarkesh.com and get started with the SDK at cursor.com/dwarkesh

• Jane Street gave me a real deep-dive tour of one of their datacenters. I got to ask a bunch of questions to Ron Minsky, who co-leads Jane Street’s tech group, and Dan Pontecorvo, who runs Jane Street’s physical engineering team. They were willing to literally pull up the floorboards and take out racks to explain how everything works. Check out the full tour at janestreet.com/dwarkesh

Timestamps

(00:00:00) – Basics of Go

(00:08:17) – Monte Carlo Tree Search

(00:32:04) – What the neural network does

(01:00:33) – Self-play

(01:25:38) – Alternative RL approaches

(01:45:47) – Why doesn’t MCTS work for LLMs

(02:01:09) – Off-policy training

(02:12:02) – RL is even more information inefficient than you thought

(02:22:16) – Automated AI researchers

Transcript

00:00:00 – Basics of Go

Dwarkesh Patel

Today, I’m here with Eric Jang, who was most recently vice president of AI at 1X Technologies, and before that, senior research scientist at what is now Google DeepMind Robotics. You’ve been on sabbatical for the last few months. One of the things you’ve been doing is rebuilding, improving, and hacking on AlphaGo.

Today, you’re going to explain building AlphaGo from scratch and what it tells us about the future of AI research and development. Before we get to that, why is AlphaGo interesting? Why is this the project you decided to do on sabbatical rather than just hanging out at the beach?

Eric Jang

I like making things, and AlphaGo and Go AI is one of those things that really got me into the field. When I saw the early breakthroughs on AlphaGo in 2014, 2015, 2016 and so forth, it was profound to see how smart AI systems could become and the computational complexity class they could tackle with deep learning. This is a problem that has long been understood to be intractable for search, and yet it was solved through deep learning. That was quite mysterious to me, and I’ve always wanted to understand that phenomenon a little better.

My training is in deep neural nets for robotics, where the decisions made by the neural networks are a bit more intuitive. But AlphaGo is a problem where the decisions are the result of a very, very deep search. It’s always been very mysterious to me how a ten-layer network can amortize the simulation of something so deep in the game tree.

Eric Jang

If you plot out how much compute it took to build various iterations of strong Go bots over the years, you can see that in 2020 there was an open-source project called KataGo by David Wu from Jane Street, which achieved a 40x reduction in the compute needed to train a really strong Go bot tabula rasa. I’m not certain if it’s stronger than AlphaGo Zero, AlphaZero, or MuZero, but it’s very strong, and this is what most Go practitioners today train against when they’re playing an AI.

Thanks to LLM coding, what took a whole team of research scientists at DeepMind and millions of dollars of research and compute can now be done for a few thousand dollars of rented compute.

Dwarkesh Patel

We should first discuss how Go works. How does the game work?

Eric Jang

Go is a very simple game that can be implemented quickly and easily on a computer. The objective is to put down black and white stones and try to occupy as much territory as possible. I might start by putting down a black stone. Black always goes first. Go ahead. The way you capture an opponent’s stones is that for every intersection, if you can surround all four of its neighbors with your stones, then it’s cut off from oxygen, if you will, and it’s a dead stone. Now I control these four stones as well as this empty intersection here.

There are slight variations between Chinese, Japanese, and what are called Tromp-Taylor rules. Tromp-Taylor rules are designed to be completely unambiguous, so this is what all Go AIs train and resolve against. In typical Go, when humans play, you’re actually not allowed to put this white stone down here. It would be instant suicide. In Tromp-Taylor, it’s actually fine. You put it down, and it immediately resolves to death, so the outcome is the same.

Let’s start over and play a few stones, and then I’ll explain some more. I’ll just start there.

Dwarkesh Patel

I’m basically playing randomly here, but I’m trying to get around your stones and see if I can surround them.

Eric Jang

This move exposes one empty neighbor for your white stone. It’s akin to a check in chess. If you don’t respond immediately by putting one here, then I can immediately capture this.

Dwarkesh Patel

I see. Because it’s the diagonals that determine whether you’re surrounded or not.

Eric Jang

The cross-section, not the diagonals. This one is surrounded on three sides, so you’re at threat of losing that stone if you don’t play one immediately there.

Now you can see that I’m starting to pressure you, because by putting a stone here, you’re forced to put one here.

Dwarkesh Patel

Otherwise, you would have this two-block to yourself.

Eric Jang

Yes. And if you think through what happens if you were to respond here, you can probably search into the future and deduce what I’ll do in response once you do that.

Dwarkesh Patel

You have a lot of confidence in my abilities, but I’m guessing you’d put the black here.

Eric Jang

That’s right, and then I would capture all three of these stones.

Dwarkesh Patel

So I should just assume that this little block is gone.

Eric Jang

Yes. In Go, it’s actually okay to let an opponent capture some stones if, for example, it lets you position to capture more stones somewhere else on the board. This is what makes Go a beautiful game: you can lose the battle but win the war. As the board size increases, the complexity of these micro versus macro dynamics gets more interesting.

Dwarkesh Patel

Presumably you’d put one here.

Eric Jang

So now I would capture this entire group, and this would be mine.

There’s one more case I want to demonstrate, which I actually had a bug in my code for recently. Let’s consider a formation like this, with other pieces on the board in play. Let’s talk about how the game ends. In this territory, who controls these areas? Is it white or black?

Dwarkesh Patel

White.

Eric Jang

It’s actually black, because I’ve surrounded this whole area. Assuming I have other black stones here, it’s very hard for you to break this out of the control of these stones.

Dwarkesh Patel

And when the final score is tallied, would these ones also count as being in...

Eric Jang

Great question. This is where different rule sets have different ways of scoring. We should talk about how you resolve scores between humans and how you resolve scores in computer Go, because there’s some ambiguity in how humans evaluate this.

Most humans would look at this board configuration and conclude that black has totally surrounded white, and white has no chance of life. We could play out more here, but at the end I would capture everything. However, if you have a way of breaking this formation and connecting white to something outside of it, then it can flip. This is where it’s a little bit hard for a computer to decide these kinds of things.

How do humans do it? It’s worth thinking about how humans resolve this, because this will map later to how we think about the deep neural network. Humans basically say, “I think the game is done,” and then you have to also say, “I think the game is done.” Then we’ll say, “I think these are my stones,” and you have to agree. If you don’t agree, we keep playing.

Essentially, once two humans—their so-called value function—agree on a consensus, then the Chinese rules resolve that. In Tromp-Taylor scoring, it’s perfectly unambiguous, so it can be decided algorithmically by a computer. If you have this at the endgame, the way you score it is that you first count how many stones you control, and that’s unambiguous.

Then you count how many empty intersections are not touched by your opponent’s stones. These intersections would not count for either player, because all of these intersections are connected to both white stones and black stones.

If this were like this, then white would get three points. This is a little odd because a human would know that white is actually losing these points. But Tromp-Taylor scoring would consider white to have all of these points as well as these points. So that is a very big difference in how computer Go scores things and how humans score things.

Dwarkesh Patel

How does the game end?

Eric Jang

The game ends when either a player chooses to resign or both players pass consecutively. Those are the rules.

Dwarkesh Patel

Now help me crack this with AI.

00:08:17 – Monte Carlo Tree Search

Dwarkesh Patel

Let’s understand how AlphaGo actually works and how somebody in the audience might be able to implement it.

Eric Jang

Let’s start with an intuition about the underlying search process used to make moves, and we’ll layer on ideas from deep learning to make it much more efficient and tractable.

Go is a game with just two players. We’re going to draw a person here, and we’re going to draw an AI here. Let’s say this person is playing black, so they go first. They go here. Now the AI is going to make a move based on what it sees here. There’s a question of how you encode these inputs into the AI. Maybe you could use ones and zeros, but you want to represent black, white, and empty. You would need at least three different values. Maybe you could use zeros, ones, and twos. The AI might see something like zero, zero, zero, zero, one. This is the input to the AI on its turn.

The AI can choose. Let’s just pick three possible random moves it can make, and I just drew these at random. Which move is best here? Well, we don’t know until the game ends. Go doesn’t have any kind of local reward for which move here is good. This is what makes Go a very difficult game. You don’t actually know who won until you really get to the end.

How deep is this tree? On a 19x19 Go board, there are roughly on the order of 361 moves on any given move, and of course, as it fills up, you have fewer moves. The number of steps in the game can be somewhere from 250 to 300 moves. Experts might decide to end the game well before that, but under Tromp-Taylor scoring, you actually have to play things all the way to the end. So this could be 300 moves, a depth of 300 in the tree.

If you keep on expanding possible moves—here the AI goes, then here the human would go, and so forth—you find that you end up with an enormous explosion in the possible game outcomes originating from just this one state. This is something on the order of 361³⁰⁰, which is far more than the number of atoms in the universe. Of course, there are redundancies and symmetries, so it’s not actually that, but if you were to do a naive tree with no merging of children, you end up with a tree about this big.

Dwarkesh Patel

What do you mean by “merging of children”?

Eric Jang

Let me use this board here. If we start here, and then you play here, and then I play here, and then you play here, that’s equivalent to me starting here, you playing here, me playing here, and then you playing here. Both arrived at the same spot, but through different paths. So this child node can be thought of as a shared ancestor.

Dwarkesh Patel

Got it. And I guess it starts at 361 but it decreases by one each time.

Eric Jang

Yes, the branching factor decreases by one each time. In any case, this is a very large tree. This is also why computer scientists for many years thought that Go was not a tractable problem this century. The amount of compute you would need to exhaustively search every possibility is just too large.

If you could, Go is actually a deterministic game. In any given state, you can compute the best possible strategy you can make in order to win the game. You can search all the possible futures where you win, and then just make sure you always stay in that set of futures.

AlphaGo’s core conceptual breakthrough was using neural nets to make this search problem tractable. Before we get into how neural networks are involved, let’s talk about how we can, assuming we had a powerful enough computer, search this tree to find the best move.

In the beginning, you’re not going to build out the whole tree, because storing it would be very expensive. Instead, you might interactively figure out which leaves of this tree are worthy of exploring and expanding into the future to see what else is there. There are some early algorithms in the bandit literature like UCB1, which is not exactly appropriate for a sequential game like Go, but very much inspired the action selection algorithm used in AlphaGo. UCB1 looks like: on every move, we’re going to take the best action—the argmax over a that maximizes Q of a, and I’ll explain what Q of a is in a moment—plus some sort of exploration bonus.

On every node, we’re going to track a few quantities. Let’s consider each of these a node. This is the root node where you’re making decisions from, and these are the children of the root node. Each node is basically a data structure that stores a visit count of this child node.

Dwarkesh Patel

How often the parent visited this node.

Eric Jang

Yes. One thing that’s easy to trip on if you come from robotics or other kinds of reinforcement learning is, where are the actions? I’m only talking about nodes. Nodes here represent states, and because this is a perfectly deterministic game with no randomness, you actually can just infer the action based on the child. If I go here, that implies an action, and this is the state we resolve to.

If you ask LLMs to vibe-code an MCTS implementation, they will most likely design the right data structure here. But it’s sort of a chef’s choice. You can rewrite the tree structure however you like. This is what Claude 4.6 wrote for me when I asked it, and it was a very reasonable choice.

Q_a represents the mean action value of this action. I’ll use a subscript a to denote that this corresponds to taking a specific action to get here from the root node. Taking a gets us to this node here. We’re also going to store the probability of taking this action

Dwarkesh Patel

Again, from the parent?

Eric Jang

From the parent, yes. What are the odds that we sample this one? This will become relevant later. We’ve talked about a deterministic tree for now, so I’ll bring probabilities into this later.

Finally, we have a dictionary of children, which is just more of these nodes in a classic linked list-style reference tree. This is the basic data structure to implement a tree.

In AlphaGo, they use a slightly different action-selection criterion called PUCT, short for Predicted Upper Confidence with Trees. When you select which child to take, you do argmax a of Q(s,a) plus a constant. The equation forms are pretty similar. These are both scoring criteria. You want to argmax this quantity and you want to argmax this quantity to determine which action to take.

Let’s break down the intuition of how you select actions here. Q(s,a) is the mean action value, so how good a given child is on average. If you actually knew the whole tree, this is all you need to select the best action. You don’t really need to do more than that. But if you’re interactively building this tree as you’re figuring out what the Q values should be, then occasionally you have to try some other actions as an explore-versus-exploit trade-off.

In both UCB and PUCT, there’s this term that basically rewards taking actions you haven’t taken before. As we mentioned, each node stores the visit count of taking that specific action. Everything is initialized to zero. For a given action, let’s call it action a, initially it’s zero. As n is increasing, if we’ve already made 10 action selections from that root node but we haven’t picked a yet, then this term starts to become quite large for a. Conversely, if we’ve chosen a 10 times out of 10, then this term is quite small. It diminishes very quickly. The same thing is true here.

Dwarkesh Patel

Just to make sure I’m understanding, let me put it in my own words. Let’s focus on UCB. Conceptually, you can think of it as two different things: the Q and the exploration term.

Let’s be clear about what Q is. Q is basically saying, once we do these rollouts—you’re actually running all these simulations—you go down the tree and figure out: if I end up at the terminal value of this tree, do I win this game or not? You average whether I win or not across all the leaves of this tree starting from this node. That average, you put in Q. So Q represents the probability that I’ll win this game starting at this node. That is your sort of exploit. It’s saying: I’ve run these simulations, and I think this is a good move or not.

The other term is saying: have I explored this branch enough yet relative to the other actions I could be exploring, or have already explored? If I haven’t explored this branch yet, maybe I think it has a low score, but I just haven’t explored that many leaves down this node in this tree. So I should try this even though Q, the exploit term, is telling me it’s not that valuable.

Because ln(n) grows slower than n, over time you will move from the argmax being dominated by the exploration term, which is the second term here, to the argmax being dominated by the Q term, which is when you’ve done enough simulations and are confident that this is the branch to go down.

Eric Jang

Yes, that’s right. The motivation for UCB was to come up with an algorithm where, if you don’t know the payoff of the different actions you can select, this strategy, given some exploration term here, bounds your regret in terms of how wrong you can possibly be.

I don’t know the proof. I also don’t know if this one is proved to have logarithmically or square-root-bounded regret, but I think the algorithm was derived to look something like this. You can tell these terms grow a little differently, and this is to account for the fact that Go has many more actions for any given move compared to your standard bandit problem.

One small clarification: you talked about simulations and probabilities. We should remember that Go fundamentally is a deterministic game. Where does the notion of probability come from here? If you had a very powerful computer, there are no probabilities. You can just compute the true average of the mean action value.

So where does the probability come in? In computer Go before AlphaGo, we’ve always done some sort of Monte Carlo method where we take the expected Q value averaged over a randomly selected tree. That randomly selected tree is where probabilities come in. The interpretation of Q is: what is the expected action value under the random distribution induced by some random search process?

Where does the random search process come in? That’s where P_a, of action, comes in. If we assume a naive algorithm where you have a uniform probability of taking any valid action, then this would just be one over the number of valid moves. You would be taking this average over a very diffuse tree. This is a valid integral, but it’s very slow because you’re going to consider a lot of trees that have very low value. It’s essentially almost like an importance sampling problem. Only a few actions and paths contribute high value, and almost everything else is low value. So that’s a tricky problem here.

This is the action selection criterion for how you decide which moves to go down. As you move down in tree search, you will eventually run into a node where it’s quite clear you’ve won or lost. At the very end of the game, when there are no valid moves left to play under Tromp-Taylor scoring, you can decide whether you won or lost. This is the final return of the whole game.

We can assign a value, U, to a terminal leaf node of the tree, but how do we assign values to the nodes prior to that, the parents? You take the mean action value, which is essentially your average. Suppose these were all leaf nodes. The mean action value of this node is just the average of whether you won or lost at the leaf nodes. Correspondingly, you can walk up the chain and say the mean action value of this node—let’s call it Q_b—is just a weighted average of these ones here.

The weighted average could depend on whether you have a different sampling distribution. But the basic intuition is that you want to resolve the game where you have a deterministic win or lose, and then you can go backwards—this is called the backup step—and assign values to these nodes or actions corresponding to the average over the final terminal leaf.

If you were to do this without neural networks, it would still be intractable. You would have trouble finding which actions to sample. A lot of the actions would contribute very low value, especially if you’re trying to fight your way out of a losing position. Only a few actions give you high value, so the search in practice is still very expensive. But the idea is that because Go follows a tree structure, you can inform a very good estimate of the value of this node based on the downstream values, assuming they’re all correct and you’ve searched deep enough.

Dwarkesh Patel

Your earlier explanation—about the sorts of states where it’s obvious to a human who’s going to win, but deterministically you still have to play it out—actually drew upon the intuition for why 1) the value function is both trainable and 2) necessary in order to learn this game effectively. Maybe it’s worth defining value in the first place.

Eric Jang

We talked about U being your final resolution of whether you won or lost. This is a terminal leaf node condition. Humans don’t play all the way to the leaves of the tree. They stop dozens of moves before, maybe even 100 moves before in high-level play.

How do they know? You can think about humans as implicitly having a neural network called a value function that takes in a board state and evaluates p(win). The human glances at the board and knows, “I’m probably going to lose.” They’re essentially running a neural network that looks at a board and implicitly amortizes a huge number of possible game playouts. They take that average and decide whether the board is winnable or not, and whether they should concede or keep playing.

This is remarkable. If you think about the beauty of something like this, a neural network in a human can somehow do all of this simulation at a glance. They just know within a few seconds, without actually playing every single game logically, based on crystallized knowledge and experience. They can do this. This gives us a hint that in games like Go, there are ways to radically speed up the search process. This is one of the fundamental intuitions behind why AlphaGo works. You can train a value function to look at a board and quickly resolve the game without playing out all of these trees to a very deep search depth.

Dwarkesh Patel

Makes sense. I will say for the audience, for previous episodes when I was prepping and it seemed relevant to understand how AlphaGo works, I would find it very confusing. But it’s the kind of thing where once you understand the problem in this way and then build the next few pieces, it is actually a lot more understandable and it makes a lot of sense. It’s okay to be confused right now, but it’s probably simpler to understand, by the end of this lecture, than you anticipate. I’ll just make that note for the audience.

Eric Jang

The important intuition at a high level—to step back about where we’re going with all this—is that classically, for games like Go, you could build a tree, but we don’t have computers powerful enough for that. Estimating the value of every action you could possibly take is also hard because you don’t know until the end of the game.

You could take averages by playing them to the end, but that’s also hard because you don’t know which actions to take to sample these averages. Conceptually, there are two problems: the breadth of the tree and the depth of the tree. AlphaGo gives us a way to shrink both of those to be tractable. That’s essentially the core idea behind it.

We take the idea that humans can glance at a board and instantly predict whether we win. That maybe gives us the opportunity to truncate how deep we search. We also know that humans can intuitively, at a glance, decide what moves might be good on a Go board. These are two things we can use deep neural networks for, to accelerate the search process.

Before we talk about neural nets, let’s go back to how this playout works. We’ve only talked about making one move. The AI looks at this encoded Go board. It has a tree. It searches deeply into the tree to find out which of its actions might be the best, takes that action, and goes back to the human. Now the human sees a Go board that looks like this, and they make their move. Maybe they put their stone here.

Now we go back to the AI, which now looks at a new encoded board. I’ve used 2 to denote the AI playing as white, 1 to denote the human playing as black, and 0 as empty. On the AI’s turn, it does the MCTS tree search all over again from scratch. It throws away the old tree that it searched last round. Now there’s a new root node and it begins to search anew.

You can basically think about MCTS as a search algorithm that decides which moves to play best, aided by neural networks, and it’s done on every move. Let’s talk about the neural network part of this.

Dwarkesh Patel

While you’re erasing, another thing that was important for me to understand was about the MCTS data structure with nodes and children of nodes. This is done per move and reinstantiated once a move is made. A human makes a move, then the AI looks at this and runs a bunch of simulations to figure out what move to make next.

A simulation is exploring one more node in the MCTS tree. Once you run 1,000 simulations, that informs the probability of what move to make next, as you’ll explain. That’s what you store. You choose the best move given those probabilities. You discard all of that, the next player makes a move, and you restart this process at the beginning of every move.

Eric Jang

Correct. One small addendum: you don’t discard all of that. You keep one thing behind that we’ll use later.

00:32:04 – What the neural network does

Eric Jang

Now that we have a basic intuition of how moves are made with search, we’re going to talk about how neural networks can speed this up by providing an analog to human intuition. There are two networks. There is the value network, which takes in a state and predicts, am I going to win or lose? It’s a binary classification problem. Then we have a policy network, which induces a distribution over good actions to take.

I’m going to draw a one-dimensional flattened move distribution, but this is really a square grid. These are the probability distributions over good actions. Both of these are categorical classification problems. You can train this like any classifier with deep learning, cross-entropy loss, that kind of stuff.

The specific architecture does not matter too much. I tried a few different architectures. Transformers work, ResNets work. For small data regimes, my experience is that ResNets still outperform transformers and give you more bang for the buck at lower budgets. But this may not always be true.

Dwarkesh Patel

Wait, why is that?

Eric Jang

They provide the inductive bias of local convolutions. Generally, transformers start to outperform residual convolutional networks when you want more global context.

One interesting finding from the KataGo paper was that they found it quite useful to pool together and aggregate global features throughout the network, to give the network a global sense of how to connect value from one side of the board to the other.

Dwarkesh Patel

What does it mean to aggregate global features?

Eric Jang

If you have a very large 19x19 Go board, and you’ve got some battles going on here and some battles going on there, when you pass this through a convolutional neural network, the receptive fields are going to be good at computing local things and making that invariant. But they won’t be able to connect these two features easily. They need to be pooled together and attend to each other somehow.

The argument for why transformers are good for computer vision tasks—vision transformers and so forth—is that because they have global attention across the whole thing, they can more easily draw these connections. But you do need more data there so that you can learn the invariant local features through data.

I’ve tried very hard to make transformers work for this problem because I was curious if transformers would present some sort of breakthrough in Go and just remove a lot of those tricks. But try as I might, I haven’t figured out a way to make transformers better than ResNets for now.

Dwarkesh Patel

One more tangential question. It makes sense why transformers, with their global pooling of information, would be better if you need to consider information that is not just spatially… CNNs give you a bias that the things next to you are especially relevant.

Eric Jang

And then they’re aggregated up slowly.

Dwarkesh Patel

For games where it isn’t that relevant what is happening locally—you just have to consider the whole thing—you’re saying transformers would work better. We’re talking about the spatial dimension. How about the temporal dimension? Right now we’re only considering the previous move because it is a deterministic full-information game.

What if it were something like poker or Diplomacy, where a bluff they made a while back is relevant to understanding now? And to decide your next move, you need to consider all those previous states. Would that change the consideration of what inductive bias and which architecture is most relevant?

Eric Jang

Great question. Go is a perfect information game. In perfect information games, there does exist a Nash equilibrium strategy for which you can do no worse than any other strategy.

If you know your opponent has a particular bias, like they love to play aggressively, you can in principle counter that specific strategy better than a Nash equilibrium policy. But to counter any given strategy, there exists a single Nash equilibrium that can be decided solely using the current state. That is a design choice AlphaGo made, which in hindsight turned out to work very well because the Nash equilibrium seems to be superhuman. No human strategy seems to be able to beat it.

There are variations of this where you would need to consider temporal history. This is a very exciting research area, and I’d encourage people to fork my repo and try it out. If you were to play 2v2 Go, you actually need to model your partner’s behavior. You may not have information on how they play, so you need to aggregate information on how they play so you can respond accordingly. These are situations where it’s no longer a perfect information game. In games of imperfect information or partial observability, you do need context to build a model. That’s a place where things can get very exciting in terms of self-play or Diplomacy-style games.

Returning to the neural network, the architecture again is not super important. You can get it to work with transformers. You can get it to work with ResNets. I found that for low-budget experiments, ResNets work a little better. You can also use Karpathy-style AutoResearch hyperparameter tuning to make your architecture pretty good. You don’t have to worry too much about that. You just need to set up the problem so that you have a target optimization.

We’re going to pick a somewhat arbitrary architecture that worked for what I did. But again, this part is not super important. You have your encoded board state and, similar to an RGB, we’re going to have three channels. One channel to encode black, one to encode white, and one to encode empties or a masked region if you want to train on multiple board sizes. I’m actually not going to talk about multiple board sizes for now. That’s a bit too complicated.

We have this two- or three-channel RGB-like image, and we feed it into a ResNet. Then we have two branching heads. One head predicts the value function, which is a single logit, let’s just call it R¹. Then we have the policy, which is R³⁶¹. This is the architecture. We’re going to train this to predict the outcomes of games given the board state, and we’re also going to train it to predict what are good moves.

The original AlphaGo paper, called AlphaGo Lee, initialized this network with a supervised learning dataset of expert human play. Later, they removed this restriction by having the model teach itself how to play well. I find it super nice for implementation, for your audience, to always initialize your experiments to something easy and get the problem working before trying to bite off the whole thing and learn tabula rasa.

In deep learning, initialization is everything. You always want to initialize your research project to something as close to success as possible, especially if you’re doing something new that you haven’t done before. Always pick something that works and then get it to do something better, rather than start from something that doesn’t work at all and try to make it work.

Under that philosophy, it’s a great idea to start with something that has a good initialization. We’re going to take human expert plays and train this model to predict good actions. We’re going to take all the moves in which an expert won and predict those actions. Regardless of board state, whether you won or lost, you’re going to predict the outcome.

You might be wondering, for some of the early boards where only one stone has been put down, how could you possibly know who the winner of this game is? Well, if you have hundreds of thousands of games, on average you’ll probably see that boards starting like this have half of the games branch off and win, and half branch off and lose. So that’ll actually be fine. When you train this model to predict those outcomes, the logit will sort of converge to 0.5. For these things, it’s expected that once you train the model, a starting board state will look like 0.5, and then as you progress towards the end of the game, the win probability will either go up or down.

This is your move number. As you get hundreds of steps into the game, it becomes much clearer who’s more likely to win or lose under your expert data distribution.

Dwarkesh Patel

I didn’t understand the significance of why this way of thinking about value is especially relevant to the expert data.

Eric Jang

It is not relevant to the expert data. It’s true for any data that you trained it on. If you were to learn tabula rasa, you would also expect this to fall out.

Imagine you’re vibe coding AlphaGo. You gather some expert datasets from KataGo online, or you have a dataset of human players, and you train this model. It turns out this model is already a pretty good Go player. It will most likely beat most human players. If you just take this policy recommendation and take the argmax over these probabilities—if you take the argmax and just take this action as your Go play—it’ll be a very fast Go player that doesn’t think in terms of reasoning steps. It just shoots from the hip, and it’ll be a very strong Go player. This is already quite miraculous if you think about how ten neural network layers, maybe under 3 million parameters, can already do something that impressive.

You can start this way. It’s important when implementing this to verify that this is probably true. It’s good to verify that your Go rules are implemented correctly and that you can run these simulations relatively quickly. Just as a checkpoint, you want to make sure you can actually do this basic step before you try to layer on more complex things like search.

But we can do a lot better than taking the raw neural network and playing the moves. So let’s apply the neural network to improve Monte Carlo tree search. We start with our root node, and we now have a four-step iterative process to do MCTS.

This tripped me up when I was first reading the paper and trying to understand it. Essentially what we’re going to do is choose a number of simulations, and this number varies. It can be somewhere between 200 and 2,048. I believe in the AlphaGo vs. Lee match, they used tens of thousands of simulations per move because they really wanted to boost the strength of the model as much as possible. But in training, you don’t actually need too many. KataGo, I think, uses something in this order as well.

Dwarkesh Patel

Do you know if they used… If you watch the documentary, they had a laptop out during the game. They didn’t use the laptop itself. It was on some—

Eric Jang

It was on some TPU pod, I think.

Dwarkesh Patel

Honestly, kind of unfair. Lee is not using 1E22 FLOPS to do a move.

Eric Jang

Fair enough. Interestingly, modern Go bots don’t need that much compute at test time. What we’ll find out, as we talk about how the MCTS policy improvement works, is that over time the raw network actually takes all of the burden of that big TPU pod and just pushes it into the network. You can do all of that work with one neural network forward pass. The TPU pod will always add the extra oomph on top, and that’s what they wanted for the match.

We’re going to pick this number of simulations thing, and for every simulation, we’re going to do several things simultaneously. We’re going to see which moves are the best in the current tree. We’re going to add extra leaves to the tree if we get to a point where we need to add a leaf, and we’re going to update the action values for the tree. Every simulation involves this four-step process: selection, expansion, evaluation, and backup.

At the beginning of our Monte Carlo tree search, our tree is very basic. It only has the root node, or our current board that our AI wants to play at. We’re basically going to select the best action for this. When this root node is created, we also know that we can evaluate it under our neural network and get the quantities V_θ, as well as our probability over actions. And I’m going to say root. For all of the actions here, we can create a bunch of children. In this case, I’m drawing a 3x3 board with one missing, so there are eight possible children associated with this root node. Each of these has an associated probability of taking that action, so there’s P₈, P₁, ... P₂, and so on.

So at the beginning of our Monte Carlo tree search, we have our root node, and we can initialize it with some children. The policy network evaluated on the root node gives us—on a 3x3 board with one existing stone placed—eight possible children that this AI could take. With each of the children, the policy network also gives us the probability of selecting that child. The first step is to do the selection of the tree. Again, this is a very shallow tree. All we have so far is essentially a tree of depth one.

Our first move is to select by maximizing, or argmaxing, the PUCT criterion, which is basically Q(s,a) + C_PUCT x P_a x (√N / (1 + N_a)). For each of these, N_a is zero for all the actions initially. N is zero. So we’re going to pick according to this. Initially, the chosen action is most likely going to be biased towards the highest-likelihood action, because these are uniform for every node.

Let’s suppose P₁ was the highest-probability node, so you selected this one here. Now you get to this node and you realize it’s not a leaf node. It’s not a terminal game, so you cannot resolve the final resolution. The next step is expansion. You will then run this board state through the policy network. Note that this is the AI’s move. The AI is making this move. When we expand this tree, we’re now thinking about what the human, or any opponent, might do.

When we evaluate the node here, we’re going to evaluate it from the perspective of this player. This node has possible actions that we could take, and we expand the leaf nodes here. For each of these nodes that we could arrive at, we’re going to now check how good they are. From here, the human could play here, here, or here. We’re going to store V_θ for each of these, so V_θ of Node₁’, V_θ of Node₁’’, and so on. We’re basically using our neural network to make an intuitive guess of how good this board is from the perspective of this player.

Fortunately, because it’s a zero-sum game, it’s easy to deduce that the value for this player at this step is just one minus the value from the other perspective. It’s easy to flip the search process depending on which player you’re at. This is the expansion step. You’ve taken a non-leaf node, expanded it, and evaluated the value. This is essentially a quick guess as to whether I’m going to win or not if I were to play to the end. You can almost think about V_θ as a shortcut for searching to the end of the tree, for any given simulation.

This is essentially the evaluation step. We’re evaluating the quality of each of these boards. In the original AlphaGo Lee, they did something kind of interesting. They took this value and averaged it with the value of a real Go playout. They actually played a real game from here all the way to the end. I’m just going to draw this squiggly line to indicate some path. They play this all the way to Tromp-Taylor resolution of a full board. So this is a zero or one. They took this value and averaged it with this one here. The formula was α x V_θ of some node plus (1 - α) of a truly randomly sampled playout.

You might be wondering how they play this out. It would be very costly to do another search on this playout, almost like a tree within a tree. They don’t do this. Instead, they just take the policy network and play it against itself. They use it as both players, and they just play it all the way to the end. This helps ground the estimates here in reality because you can get a single-sample estimate of whether you win or not. In the endgame, where the board is almost resolved, this becomes quite useful because the play according to the policy will most likely decide a pretty reasonable guess of the game, so you’re not facing a problem where this becomes untethered from reality.

It turns out this is totally unnecessary. In all subsequent papers after AlphaGo Lee, they got rid of it. In my implementation, I did the same, and it speeds things up a lot because you don’t have to roll these games out on every single simulation.

Dwarkesh Patel

Okay, just to reinforce my own understanding and re-explain it. For the audience, by the way, in case it’s not obvious, the P there in the select, that is the probability coming from the network in this case.

Eric Jang

Correct. The policy network here.

Dwarkesh Patel

Fundamentally, a simulation, just think of it as rolling out one more node in the search process.

Eric Jang

Almost. A simulation is easy to think about when the whole tree already exists. You just walk down the tree using the PUCT selection criterion, and then you keep going. In AlphaGo, the data structure is such that we begin with a tree that basically only has depth one—its only children—and you want to iteratively build out the tree as you’re also selecting actions down the tree. That’s the core thing here. Because Go is such a combinatorially complex game, you cannot afford to build the tree in advance and then search it. You must search while building the tree.

Let me finish up with the last step, which is the backup. Once you’ve scored these things, the Q value assigned to the node here for taking this action is just the average across your evaluated values. You take a running mean over all the simulations you’ve taken, averaging the values of the children nodes. That’s the backup step, and once you evaluate this, you can recursively go back up. If you know the action value of this node, you can then take the average on its parent, and so on.

You have this four-step process: 1) You’re choosing the best action you know of so far. 2) You may run into a node you haven’t been to before, so you need to grow the tree a bit. 3) You run it through the network to guess whether you’re going to win or not. 4) You walk all the way back up to the root node to update your values on what the best moves are.

As you do this iteratively, because you’re always selecting according to this criterion, you’re always going to be selecting the best action you think at any given branch. The final visit counts of how often you chose these things will reflect your correct policy distribution as induced through this search process. The visit count we stored in the node earlier actually becomes the vote for which action we should finally select.

As a test of understanding, it’s worth thinking a little bit about whether we could make this even simpler. Could we actually maybe even get rid of this one and still make the thing work? Recall that when you do an expansion and then an evaluation at this node, you are checking the win probability of each of the child nodes. So if this one is one and these are zero, you do know something about which action might be better to take. Why would you still need this, right? Why not just normalize the values into a distribution and call that your policy distribution?

This is fine. You can do this, and it probably does work. But in practice, having a single forward pass that gives you a pretty good guess is how the breadth is pruned out. There is a duality here. It would be weird if the policy recommended an action that disagreed with the value. If a policy said this was very high probability, but this one said it was a low value, then there’s something fundamentally wrong between your policy head and your value head. They are linked, and you probably could get rid of this if you came up with a different way to recover it from just the value evaluations.

Dwarkesh Patel

Right. Just to make sure I understand, the reason you don’t do that is so you don’t have to do 360 independent forward passes to say, “Okay, here’s the value of everything. Let’s argmax over it.” Instead, you can just do one forward pass and get the probabilities of all of them.

Eric Jang

You can usually batch these somewhat efficiently, so it probably isn’t a huge computational burden in practice. But yes, you would have to pass up to 361 boards into a single mini-batch update to evaluate all the values, then normalize them.

There’s actually a more important reason why we still do this, which is how Monte Carlo tree search is used to feed back on itself and recursively improve its own predictions and search capabilities. That’s where having this as an explicit entity you’re modeling, rather than an implicit normalization over your value, is a good idea.

Dwarkesh Patel

Makes sense. Okay.

Eric Jang

We talked about the simulations. What you end up with as you roll out the number of simulations is a tree that looks like… I’m drawing a very low-dimensional version of this. Of course, in the real game, it’s much more high-dimensional. You’ll end up with a tree structure that has a lot of leaves that terminate and are not visited again because their value is deemed to be too low. But along one path, there will be a set of actions with very high visit counts that gravitate towards that one set of decisions as you increase N.

This is the mental picture of what the tree in Monte Carlo tree search looks like. You should contrast this with an exhaustive tree like in tic-tac-toe, where there are nine actions, then eight, then seven and six, and so it’s a nine-factorial-sized tree. The Monte Carlo tree search in Go is very sparse. It only considers the paths that you’ve expanded children nodes on.

Now that we have the search algorithm that applies the value function as well as the policy function, we can talk about how the Monte Carlo tree search algorithm can act as an improvement operator on top of these guys here.

01:00:33 – Self-play

Eric Jang

We now talk about the RL part of how this thing gets stronger by playing itself. Let’s say we play a game. You make a move. The AI will compute the search, and this is the visit count distribution. Let’s say this is your initial policy recommendation at this node.

After MCTS, it gets more confident about one of these actions. Maybe the distribution looks a bit more peaky like this based on the search. Of course, you can tune the search process so that it ends up more diffuse, but that’s probably not a good idea. MCTS should get more confident about specific actions over others. It might place a lot of weight on other actions initially, and then as you increase the number of sims, it should converge to a very peaky distribution.

Let’s call this π. Let’s wrap this in an MCTS operator of (a | s). After applying the MCTS process, your policy recommended distribution looks like this. It’s a bit more peaky than the previous one. Then you take the argmax, or maybe you just sample from this. It doesn’t have to be the argmax. You make your move, throw away the tree, and begin anew on the next move. Again, you compute a new distribution. Initially, maybe your guess looks like this, and then you refine it through MCTS.

Dwarkesh Patel

There should be one more X on the board, right?

Eric Jang

I’m sorry, that’s correct, yes. Something that looks like this. On every move, you have your initial guess from your policy network. Then the search process, which combines your policy network and your value network, arrives at a more confident action that you take, and so on and so forth.

Then the game ends, and one person wins and one person loses. The beauty of how AlphaGo trains itself is that it can actually take this final search process—the outcome of the search process—and tell the policy network, “Hey, instead of having MCTS do all this legwork to arrive here, why don’t you just predict that from the get-go? Why don’t you not use this guess and predict this to begin with?” If you have this guess to begin with in your policy network, then MCTS has to do a lot less work to get things to work.

If we draw a test-time scaling plot… Let’s say this is the number of simulations. At zero simulations, your implicit win rate is here. If you just take this raw action, this is what your win rate is. Let’s say we increase the number of sims, maybe you have a win rate curve that looks like this. When you search for 1,000 simulation steps, that gets you to a policy here that gets you to here, which is great.

But if you were to distill this MCTS policy network back into your shoot-from-the-hip policy network, then you could actually start here. Let’s say this was zero, by distillation, if you then spend another 1,000 sim steps, you actually get to here. It’s almost as if you could amortize the first 1,000 steps into the policy network instead of the search process, then you could begin at a much better starting point and get a much better result for the number of sims that you play.

Dwarkesh Patel

The sigmoid type nature of test-time scaling as the number of simulations increases, the increase in win rate gets smaller. Is that true even for the distilled network? That is to say, is there some gain where we start from the distilled network and get these early gains again, or is that just inherent to the nature of MCTS?

Eric Jang

To be honest, I actually don’t know the test-time scaling behavior of MCTS simulations. I believe it might be quite sensitive to how strong this one is in practice. I’m just drawing a monotonically increasing function that gets to one.

Don’t pay too much attention to the shape of the curve. Just know that it’s monotonic with respect to sims. The idea of MCTS is very brilliant. We got something better by applying search. Now, on our next iteration of updating this network, we’re going to train it to approximate the outcome of 1,000 steps of search.

Instead of starting here, we get to have our neural network start here, and then the play gets stronger once we then apply another 1,000 steps on top of it. You can keep going. The training algorithm for AlphaGo is to basically take the games where you’ve applied the search on every move that the policy encountered—whether you won or lost, and that’s quite important—and just train the model to imitate the search process.

There’s an analogy to robotics, which is the DAgger algorithm. First I’m going to draw a schematic of the states: S₀, S₁, S₂, S₃. Let’s say we took a series of actions in an MDP to get a trajectory. These actions may be suboptimal. Maybe we lost at the end of this game. There is a family of algorithms that basically take trajectories and relabel the actions to better trajectories. Maybe a better action here would have been to take A₀’. A better action here would have been A₁’, and yet another one, A₂’, A₃’.

What MCTS is doing is saying: you played this game where you eventually lost, but on every single action, I’m going to give you a strictly better action that you should have taken instead. It does not guarantee that you are going to win, but it does guarantee that if you take these tuples as training data and retrain your policy network to predict these actions instead of the original ones, you’re going to do better.

This is very related to DAgger in robotics and imitation learning, where you want to collect an intervention here. Even if you’re in a not-great state—for example, a self-driving car that veers off the side of the road—there is still a valid action that corrects you and brings you back.

Dwarkesh Patel

Pedantic question, but is there a guarantee that MCTS must be better than the policy? For example, you could imagine early on in training, because MCTS is informed by the value network, that when the value network hasn’t been well-trained on finished games, MCTS is worse than a randomly initialized policy. Is it just a heuristic that MCTS is better than the policy, or is there some guarantee?

Eric Jang

In practice, it is a heuristic. It does work in practice, but let me illustrate an example where MCTS can give you a worse distribution than your policy network. This can often happen if your self-play algorithm has trained to a good point, but then somehow it collapses because it’s not trained on diverse data or something.

Let’s say we have a board state where the policy recommendations are very good. So Π_θ (a | s) is great. But somehow, maybe because we’re playing a lot of games where the bots just resign instead of playing all the way to the Tromp-Taylor resolution, they forget how to evaluate those late-stage plans. Like in the case we showed with the corner play, maybe 100% of our training data in our replay buffer has lost examples of how to evaluate the value function at those states.

You might end up in a scenario where your terminal value is very bad. If the terminal values of the leaves are not good, then this will propagate all the way up and cause your PUCT selection criteria and your backups to be off. And then you end up visiting a very different distribution than what your policy initially recommended.

Also, if your number of sims is low, you might have a variance issue where you just don’t explore enough. It’s only guaranteed to converge when you take N to infinity. Variance in your search process as well as inaccuracies in your evaluation can definitely screw with the quality of your policy recommendation. That’s why it’s not a guaranteed improvement.

That is why I suspect AlphaGo Lee had the playouts to the end in their training algorithm, so that they could ground this thing in real playouts. In practice, what you could also do is, for 10% of the games, prevent the bots from resigning and just say, “Resolve it to the end.” That way you get some training data in your replay buffer to really resolve those late-stage playouts that normal human players would not play to.

If you assume that the value functions are correct, this is why MCTS gives you a better policy. It’s a very critical chain of assumptions. Assuming this is accurate, your search process should give you a better recommendation than your initial guess.

Dwarkesh Patel

If you have a cold-started policy—if you have an AlphaZero-type thing—really what’s happening for the first few epochs is that the policy is kind of useless. What you’re really just doing is, “Hey, let’s play full games, and once we have played full games, for the preceding moves, we’ll have labeled who won and who didn’t win.” The loss for AlphaZero has two components: how good is the policy relative to MCTS, and how good is the value prediction relative to who actually won the game from this move. You can think of this being applied to every single action or every single move.

Really what’s happening at the beginning of AlphaZero training is just that we’re trying to get the value function to actually predict who will win the game if you find yourself in this state and you’re this player. Functionally, that’s all that’s happening. Later on, once that’s well trained, now the policy is also improving.

Eric Jang

Correct. There’s one trick I found to be pretty useful. This is not a peer-reviewed claim, so take it with a grain of salt. I found it useful in my own implementation to do the following. You want to first make sure that this is good before you invest a lot of cycles doing MCTS. It doesn’t really make a lot of sense to do search on garbage value predictions. You want to start at a good place where this works.

AlphaGo Lee does a very good thing where it just takes human games, you train on them, and it just works. It totally works. You can also take an open-source Go bot, play it against itself, generate data, and that also works. If you have some offline dataset that has realistic, good play, you can easily learn the late-stage value functions pretty well. That’s what you need to start the search process.

Dwarkesh Patel

Sorry, can you just repeat that sentence one more time?

Eric Jang

Sure. It’s quite easy to evaluate a late-stage Go game. When almost all the pieces are on the board, it’s almost like a decidable problem because there’s lower and lower uncertainty as to the depth of the tree. So most games played to the end by reasonable people will be good training data to train a good value function at terminal parts of the tree.

Then, as you play more games, the search will back up good values into the intermediate nodes of the tree. As you increase the amount of data, your value head gets a good intuition of what is a healthy board state versus a not-healthy board state. Those are much more subtle to judge in the mid-game than at the beginning or the end.

The most difficult part to score is not the beginning, because the beginning is obviously 0.5, and at the end it’s pretty obvious who’s winning. The hard part that you want to learn in the value function is who is winning in the middle.

Dwarkesh Patel

So this is actually very analogous to TD learning.

Eric Jang

Yes. There’s a beautiful connection to TD learning that we can talk about in a bit, contrasting it with Monte Carlo tree search.

You first want to get good value functions, and expert data can give you a quick shortcut. I recommend for practitioners, just do that first to initialize to a good starting point. If you want to do the AlphaZero thing or KataGo tabula rasa learning, what you can try to do is play random games on a small board. Just take a random agent. If you play 50,000 games, you’ll actually learn a pretty good value function as well. On a 9x9 board, you can see enough of the common patterns with random play.

If you train on both 9x9 and 19x19 data—and KataGo proposed one of these architectures—there’s pretty good transfer learning from the value head evaluated at 9x9 to 19x19.

Dwarkesh Patel

Right, because this, unlike other games, has very much a sense… There’s not like a new kind of piece introduced when you increase the size or something.

Eric Jang

If we take it to its limit and consider a very tiny 4x4 Go board, if you play 50,000 games, you’re going to have a lot of end states that look like human play. It’s just Tic-Tac-Toe at that point.

If you broaden this a little bit to 5x5 or 9x9, it’s not unrealistic to imagine that purely random play will generate pretty reasonable-looking boards. You can score those pretty easily. That is what gives you the bootstrapping to then improve your policy with search.

But it’s very critical that MCTS has accurate value estimates. You need to ground the value. Ultimately, MCTS will fall apart if you don’t have a grounding function for the value.

Dwarkesh Patel

I’d be curious how much compute you save by training the value and policy on the same network. Because they share the same representations, how much more efficient is learning? We’ve just talked about how they’re making similar predictions, or they should be in line with each other. So I’d be curious if you’re halving the amount of compute you have to do by keeping them in the same network.

Eric Jang

AlphaGo Lee, the original AlphaGo paper, had two separate networks. In all subsequent papers, they merged them into two heads, and presumably this saves compute. But answering that question in a rigorous scientific way is a simple question that in practice, if you really want to chase the question down to its limit, takes quite a bit of work to really resolve.

Intuitively, yes, they share a lot of representations. As we mentioned, your policy network and your value network, when doing evaluations, should agree. There really should be this consistency between them.

Dwarkesh Patel

Tell me if this is the wrong way to think about it. When I learn how an LLM works and how simple RLVR is as an algorithm, I’m stunned by the kinds of things it can do. It can learn how to build very complicated code repositories simply from getting a yes or no.

Here, if you understand it more deeply, just predicting MCTS, AlphaGo seems less impressive in retrospect the more you understand it. You’re putting in a lot of bias by telling it how it should titrate exploration as things go on. You’re building this very explicit tree search for it. I don’t know if you share that intuition where the more you understand it, the less impressive the accomplishment in 2017 seems.

Eric Jang

I personally disagree. I think they’re profound for different reasons. I don’t understand the LLM RL enough to comment on it on your podcast. But why is AlphaGo a profound accomplishment? It’s worth stepping back a little bit. It is different from modern RL, and we can talk a little bit about some of the algorithmic choices there.

I think the most profound thing here is that a 10-layer neural network pass, basically 10 steps of reasoning… Of course, the reasoning is not just one trail of thought. It could be distributed representations and a lot of thoughts going on at the same time. But by construction, let’s say a 10-layer neural network can only do 10 sequential steps of thinking. 10 steps of neural network parallelized distributed-representation thinking is able to amortize and approximate to very high fidelity a nearly intractable search problem.

This was a breakthrough that I think most people don’t even fully comprehend today, how profound that accomplishment is. This is what also girds AlphaFold, for example, where you have a very, very difficult physical simulation process where you would need to roll out as so many microscale simulations, and yet 10 steps of a somewhat small neural network can somehow capture what feels like an NP-class problem into a single problem.

It actually makes me wonder if our understanding of problems like P=NP, or these fundamental computational hardness problems, is incomplete. Obviously, it’s not a proof of P=NP, but there’s something to it that is very disturbing, where what felt like a very hard problem can fall to a very simple macroscopic solution.

Dwarkesh Patel

That is a very interesting insight, that a lot of problems which are proven to be NP-hard—I don’t know if Go is proven to be NP-hard, but protein folding, et cetera—neural networks can solve. They’re NP-hard in the worst case, but we’re usually not concerned about the worst case. These problems usually have a lot of structure to them.

Eric Jang

I think the question we should be asking ourselves is about how we’ve been formulating solutions to NP-hard problems in worst-case complexity. I wouldn’t say this solves Go. It doesn’t give us an exact solution of the optimum, but in practice, it is extremely useful. The same thing has been shown in AlphaTensor and AlphaFold. Yes, there is a very hard problem that, in the worst case, seems intractable, and yet we’re able to make almost arbitrary amounts of progress.

In the limit, what might this look like? If you want to simulate something very complex like weather, or predict the future—do we live in a simulation or not—the computing resources you need to build a very complex simulation might be much smaller than you think, based on our ability to amortize a lot of that computation into the forward pass of a single network. To me, AlphaGo was the first paper that really showed this profound level of simulation being compressed into a small amount of compute.

Dwarkesh Patel

I feel totally not qualified on the computational complexity or the math to comment on this, but I wonder if there’s an important role of chaos here. What is the problem with weather, and why does it take 10x the amount of resources to predict weather a day out, and continually so for every additional day out? It’s because it’s a chaotic system, so small perturbations can totally change the final estimate as time goes on. I guess you would expect that for Go and protein folding as well.

Eric Jang

Here’s an analogy to weather that might be relevant in Go. Here’s our current board state. Given what we know about both players, what is the exact board state in the future? This is extremely sensitive to initial conditions. A single stone placed here can disrupt the entire prediction. This is hard. Intuitively, this is the chaotic problem.

Yet somehow, we can predict who’s going to win. This captures a lot of possibilities. There’s a more macroscopic quantity that we really care about, which is the average or expectation or some sort of global macrostructure over a lot of possible futures.

In weather, it could be the same thing. We don’t exactly care what the velocity of wind 6,000 feet above a specific latitude and longitude is. We care where the hurricane is, and things like that. In chaos, there’s the classic Lorenz attractor. If you start anywhere on the Lorenz attractor, you don’t know where you’re going to end up, but you do know that the thing looks like this. There’s a kind of beauty here. Sometimes we don’t necessarily care about the microscale things. We actually care about the macroscopic structure, and these things can be predictable.

Dwarkesh Patel

Contrast that to something like a hash function, which is also incredibly dependent on initial conditions but doesn’t have a macrostructure, or at least hopefully doesn’t, if the algorithms work.

Eric Jang

One would hope.

Dwarkesh Patel

There’s no equivalent of a value function or broadly how the weather is going to be that’s interesting there. It’s really just about what the board is going to look like exactly 100 moves from now exactly.

Eric Jang

Intuitively, that seems correct. This is also out of my area of expertise, but I find it interesting that the tools of cryptography and hashing have also not been able to prove that you cannot come up with fast approximations. You cannot come up with fast approximations. If they were able to do that, then you could prove P is not equal to NP.

Dwarkesh Patel

In fact, we know that there’s structure in many cryptographic protocols, obviously RSA cryptography. There is structure, and that structure is what quantum computers exploit to break them.

Reiner has a very interesting blog post, which we’ve talked about in the episode, where he points out that if you look at a high level at what cryptographic protocols look like and what neural networks look like, they’re extremely similar. You have sequential layers of jumbling information together. There’s a convergent evolution in the algorithms. In cryptography, you want the final state to be incredibly sensitive to initial conditions, so that it comes out looking jumbled if you change anything. In neural networks, you similarly want everything to be dependent on all the information, because you want to process all the information and consider how it relates to itself.

Eric Jang

You have the maximum power of a neural network at the edge of chaos. I think there are some research papers from Jascha Sohl-Dickstein on this. There’s something quite fundamental about chaos that’s not just hopeless noise. There’s something useful in chaotic systems, at least at that boundary. But this is just thinking about it as a philosophy. I don’t know the math well enough to comment on it.

We’ll talk about LLM RL in a little bit, because there’s some connection there, but let’s go back to MCTS. What is it doing? Crucially, it is not saying that we’re going to increase the probability of winning directly. It’s not going to upweight all actions that won and downweight all actions that didn’t win.

Importantly, what it is doing is saying: for every action we took, we did a pretty exhaustive search on MCTS to see if we could do better, and we’re going to make every action that we took better by having the policy network predict that outcome instead. This is a very nice idea because you have one supervision target for every single action. The variance of your learning signal is very low compared to the alternative naive RL thing.

Let’s consider a very naive algorithm that looks a lot more like modern LLM RL today, where we do something like take the winner of a self-play game and encourage it to do more of that.

01:25:38 – Alternative RL approaches

Eric Jang

It’s worth thinking a little bit about what alternatives we could use to train self-play agents instead of MCTS. We use a lot of LLM-style RL these days. Is that relevant? Could we do that instead?

Suppose we have a very naive algorithm where we take a league of agents of different checkpoints and play them against each other. For the games where a single player wins, we reinforce those actions up and retrain the policy network to imitate those winners, instead of the MCTS objective. Let’s say you have a chain of actions that led to a win, and you have a matchup between two agents that are basically the same. Assume policy a (Π_a) and policy b (Π_b) are evenly matched, so their true win rate is 50%.

Let’s say you play 100 games, and each game lasts 300 moves. You’re doing some sort of evolution strategy or some way to perturb these things to get them to do different things. Or maybe you don’t, and you just play them against each other to see that occasionally one might have a better strategy than the other. Let’s say policy a wins 51 games and policy b wins 49. This is just due to random luck, or maybe you perturbed policy a in some way that let it do this.

Just to have a very simple model, let’s pretend that for 50 of the games, they played exactly equally. In the one game where a won, it played slightly differently. It made one critical move that normally it would have done differently, but due to some exploration or random noise, it happened to make a smarter move than it did previously.

So you have one true supervision signal for your policy network, and then you have 99 games times 300 moves for which imitating those actions gives you exactly the same policy you had before. The scale of your variance is actually very bad, because you only have one label out of this enormous dataset of supervision actions where you want… Actually, let me clarify a little bit.

Dwarkesh Patel

We’re talking about how the good move, the out-of-distribution move, is a small fraction of all the moves that are played across all the games on which you’d want to train. This, of course, reminds me of how LLMs are trained with policy gradient methods. When Karpathy was on the podcast, he called it like “sucking supervision through a straw”. It’s interesting that this thing you’re saying—which would be intractable and prevents you from actually getting beyond a certain level in Go—is just by default how LLMs are trained?

Eric Jang

Right. This is not to say it doesn’t work. If you imagine increasing the number of games to millions of samples, you actually can get some meaningful supervision samples, so long as you find a way to mask out the supervision from these guys. This is where things start to get pretty related to RL in terms of advantage and baselines.

Let’s look at the gradient variance of a very naive approach—I’m going to call it gradient RL—that’s basically the sum of rewards. The sum of rewards is the return. In our naive setup here, we only have an indicator variable for the return, where either you won or lost. In the case where you lost, your gradient is zero, so you don’t train on those examples, and when you won, you try to predict those things. You can think about this setup as a special case of this general formula here.

The trouble here is that this is very high variance. When you multiply these terms out to compute the variance of the gradient, it’s equal to the expectation of… Just for simplicity, we can pretend this is on average zero or something if you’re centering it at no signal. The variance here basically means that you’re taking the square of this product term. You end up with a term that grows quadratically with T. When you have a setup like this, this thing acts as a coupling effect on top of these terms here.

Let’s actually map this to an LLM case, and we can answer why LLMs only do one-step RL instead of a multi-step RL scenario. In LLMs, you have a decoder that might predict some words like “hello world”. In current LLM RL, they treat this entire sequence as a single action, a_t, and big T is just one. It’s true that because of how transformers are formulated through the product of conditional probabilities, the log probability of the whole sequence is equal to the sum of the probabilities of individual tokens.

In this case, I would say something like log(hel) + log(lo) + log(world). This is true, and if this term were one, then they would be the same thing. However, when you’re sampling, if you have a reward term assigned to every specific token, now you have these interaction effects between the cross-multiplication of these terms and these terms. The problem becomes, how do you ascribe the credit associated with every episode to all these different terms?

Dwarkesh Patel

The thing I’m confused on is what it would even look like to do it that way in LLMs, because you only get a reward at the end of the episode.

Eric Jang

You could imagine a reward that says, “I’m going to give you some process supervision where you get a reward for each of these actions on every step.”

Dwarkesh Patel

Okay, so you’re saying if instead of doing it that way… Well, the way you’ve written it, it would be a sum at the end anyway, so they wouldn’t have to be multiplied. But you’re saying instead of doing it that way, you would just add up these process rewards at the end and then treat that as one single reward signal?

Eric Jang

Correct, for one single log prob action.

Dwarkesh Patel

But isn’t that how it’s written to begin with anyways? The sum of the rewards?

Eric Jang

The thing that’s a little bit hidden here in the math is that we’re assuming that when you decompose the problem into a multi-step problem, you’re now introducing correlations between your actions through the computation of this guy. If you separate these things out, this will magnify the variance of this one.

In the case where you don’t separate it out, if you just have T=1, you have a single estimate of log prob and a single estimate of reward. This term still shows up. In LLMs, the naive REINFORCE estimator looks a bit like the return of the single action… It looks kind of like this. This is the basic form here, but it’s still a contributor to variance. You want to make sure that, similar to how in this case we were training on a lot of neutral labels, you’re sort of penalizing the labels that don’t help and only rewarding the ones that actually make you better.

Intuitively, the analogy is: can we find a term in our training objective such that it’s actually discouraged from doing this, or these don’t have any effect on the gradient, and this has an effect on the gradient?

Dwarkesh Patel

If you applied that there, the only thing you could do is eliminate 49 of the games. At least the way you have it written there, it would be 51 times…

Eric Jang

Actually, the optimal case is to discard all of these moves, and only get a gradient on that single move that you got better on.

Dwarkesh Patel

How would you do that?

Eric Jang

This is a pretty tricky problem in practice. This is where advantage estimation happens in reinforcement learning. You want to subtract a term from your multiplier. Instead of an indicator function of one and zero, you want something that behaves like a zero for all of these guys, and then a one for these ones.

Dwarkesh Patel

You could do that if you can say, “Hey, I won this game, so this is slightly above baseline performance.”

Eric Jang

Well, you won a lot of games. But you don’t know which ones let you win because they were truly better versus winning by accident.

Dwarkesh Patel

How would you design a baseline where it’s truly better?

Eric Jang

This is where in RL people use things like TD learning to better approximate the quality function, the Q that we mentioned earlier. You can try to subtract that from your return.

Ideally, what you really want to do in RL is push up the actions that make you better than average and push down the actions that make you worse than average. They call this advantage. There are multiple ways to compute it. I highly recommend John Schulman’s “Generalized Advantage Estimation” paper as a good treatment on how to think about various ways to compute it. At the end of the day, you want to reduce variance by trying to make this smaller, so it doesn’t magnify the variance of this one.

Dwarkesh Patel

That makes sense. This requires you to have a very good estimate of what average performance from a state would look like. This gets us back to the value function thing we were talking about earlier.

Eric Jang

Keep in mind that this model-free RL setting is trying to solve a credit assignment problem where you don’t know which actions were actually good and which ones were bad. Monte Carlo tree search is doing something very fundamentally different. It’s not trying to do credit assignment on wins. It’s trying to improve the label for any given action you took. We can actually think about a completely different algorithm called neural fictitious self-play, which was used to great effect in systems like AlphaStar and OpenAI’s Dota.

Let me talk a little bit about how you can unify some of these RL ideas in the model-free setting as well as the self-play setting. What happens if you don’t have the ability to easily search a tree? In Go, it’s a perfectly observable game. You can easily construct a pretty deep tree that completely captures the game state. In a game like StarCraft where you don’t have complete control over the binary, it’s a little bit hard to do this, and I’m not even sure if it’s a deterministic game. That makes this difficult from a data structures perspective.

What is done instead is that the basic idea of supervising your actions with a better teacher is still there. We’re going to talk a little bit about how neural fictitious self-play works. It’s the same idea. We’re going to come up with better labels for each of the actions we took, just like in MCTS. But how do we derive the better labels? In MCTS, we perform search. Assuming we have a good value function, the search will give us a better result than our initial guess.

In a game where you can’t easily simulate a search process, what they do instead is train what’s known as a best response policy. You fix your opponent. Let’s say you’re currently training Π_a against a strong opponent, Π_b. In StarCraft, maybe these are the Zergs and you’re playing Protoss or something. You fix your opponent, and you treat this as a classic model-free RL algorithm, where your goal is just to beat this guy. Here you use your standard TD-learning-style tricks, or use PPO or any model-free RL algorithm to try to hill climb to winning against this player. You have a reward function where the return is one if it wins against Π_b, and zero otherwise. This is no longer a self-play problem. It’s just a fixed opponent, and you’re trying to maximize a score against them. You have a fixed environment where all you care about is beating this guy.

Once you have a good policy that you train with your favorite model-free RL algorithm—PPO or SAC or V-MPO—you now have a good policy that gives you a good label for what this one should do when playing against that player. When you train multiple best response policies, you can basically distill the RL algorithms into the labels for a given opponent. You might have a best response policy against π_b, and then maybe you have a league of opponents like π_b, π_c, π_d. You’re going to take the best response policy that you train against each of these fixed opponents, and supervise them with the label that this one would provide.

This is almost like a proxy for your MCTS teacher. Instead of an MCTS teacher, you use a model-free RL algorithm to find the best search action that you could do to beat your opponent. Then you’re distilling the policy here into what’s known as a mixed strategy, where it’s trying to average across all possible opponents you could play against. This is what gives you something that can do no worse than an averagely selected opponent from the league. This gets around the problem of having to derive a teaching signal from MCTS, but fundamentally it’s still about relabeling your states with better actions so that they improve your policy.

Dwarkesh Patel

Just to make sure I understand, if you win a game against this other policy, you reinforce all the actions on that trajectory.

Eric Jang

Yes. Here you can use a number of algorithms like PPO, V-MPO, or even Q-learning if you want. The specific algorithm here is usually a model-free thing because you don’t have search, but there’s an interesting connection between MCTS and Q-learning that I want to bring up.

In MCTS, you have a tree, and through the resolution of your value function at the approximate leaves of the tree, you can back up through many sequences and obtain some sort of mean value estimate. Your Q is derived from the average of a bunch of simulations.

In model-free algorithms, there’s often a component of estimating a Q value. Q values are often learned through TD learning, although in PPO, the way they do advantage estimation is not necessarily through a Bellman backup. In Q-learning, there’s this very cool trick where Q(s,a) is backed up as r plus some discount factor times the max over a, Q of your next step. Intuitively, if you have an MDP and this is terminal, it’s saying that the best action you can take at this state is equal to the reward you get for taking this action, plus the best that you can do at the next state.

There’s a recursive, dynamic-programming property of MDPs, and you can train neural networks to basically enforce this consistency. You can say, “Well, once I know the Q value of this action, I can then use that to compute something about the Q value for the next.”

Dwarkesh Patel

Earlier I was like, “Hey, why are we training policy? Why don’t we just train the value alone?” That is what this is.

Eric Jang

This is an algorithm for recovering value estimates of intermediate steps when you don’t have the ability to do forward search. You must collect a trajectory first of n steps before you’re able to do this trick. But the intuition is kind of the same. Knowing something about the Q value here can tell you something about the Q value here. And indeed, you can recover a policy from a Q value. You don’t need to explicitly model the policy distribution. You can recover the policy distribution by doing argmax over your Q values.

Q-learning, or approximate dynamic programming, propagates what you know about the future Qs backward like this. You can see that there’s a similar structure going on here, where in one case you’re planning over trajectories your agent hasn’t been to yet, whereas in the other you’re planning over trajectories your agent has visited.

Importantly, why was Q-learning a big deal? It’s because historically we just haven’t had the ability to do search on fairly high-dimensional problems like robotics or whatever. For a long time, we made the assumption that if we can’t model the dynamics with a world model, we’re going to instead just collect trajectories and then plan with respect to the only number that really matters, which is reward.

01:45:47 – Why doesn’t MCTS work for LLMs

Dwarkesh Patel

This is very interesting. To unify this with our discussion of LLMs, you don’t have Q values, but you’re doing this backwards learning where you find the trajectories that pass some unit test in some coding environment and then reinforce those trajectories.

There’s a huge difference between that and this forward approach with MCTS. The reason it’s much more preferable to do MCTS is because you can do it per move and make each move better, rather than having to learn per trajectory and hope, as Karpathy said, to learn this…

Eric Jang

Through a straw.

Dwarkesh Patel

Right, you get supervision through a straw. You basically just upgrade all the tokens in a trajectory that might or might not have been relevant to getting the answer right.

The reason you can do this much more sample-efficient, favorable thing with Go is that because MCTS works in Go, you basically know that if you just do the search locally here—and this search is truncated at the end by a value function that works even if you haven’t unfolded your whole trajectory—you can just say, “This is my new policy,” and improve in a more iterative, local way, rather than having to unfold all these trajectories.

Eric Jang

There was some research from Google in 2023 or 2024 where they tried to apply tree structures to reasoning. The jury is still out as to whether this can ever work. We probably will see a revisiting of this idea of forward search in the future.

But there are two things that make MCTS very simple for Go. Value estimation is concrete. You can determine it for real, and then you can use it to truncate depth, as you said. The breadth is also determined. What’s critical is that the action selection algorithm, where you iteratively visit and grow the tree, is well suited for the size and depth of problem that Go is.

But for something like LLM reasoning, PUCT might not be a good enough heuristic. It might be too greedy with local tokens, and it might only give you obvious thoughts that are correct but don’t really solve your final problem. I would say the jury is probably still out on what the final instantiation of reasoning for LLMs will look like. I wouldn’t rule out that this stuff could come back, but it’s a bit hard.

Dwarkesh Patel

Don’t LLMs natively learn to do MCTS, where they’ll try an approach and be like, “Oh, that doesn’t work. Let’s back up. Let’s try this other thing,” and then go in the direction that proves to be more fruitful?

Eric Jang

Certainly, LLMs manage to do something that looks like real human reasoning without having to do an explicit tree structure. That being said, I think the idea of doing forward search and simulation to get a better sense of what is valuable might make a comeback, even if not in exactly the same instantiation as AlphaGo.

Dwarkesh Patel

Just to make sure I understand the crux of it: the breadth comes from the number of legal actions being wider, and the depth comes from not being able to train a value function as easily, because…

Eric Jang

Here’s an example where LLMs break down. The C_PUCT rule involves the √N over 1+N_a. In an LLM, you’re most likely never going to sample the same child more than once. If you have multiple steps of thinking, because language is so broad and open-ended, a discrete set of actions is not really an appropriate choice for an LLM. Even though they’re discrete tokens, it’s just such a large number that this type of exploration heuristic is probably not the right thing to guide how to search down a tree.

Dwarkesh Patel

I guess the crux comes down to the fact that in Go, you know that MCTS is almost certainly better than your current policy, even though you haven’t explored the end of any trajectory. In normal reasoning for LLMs or robotics, there’s no way to just locally evaluate and improve your next move in a way that’s independent of actually solving the problem.

Eric Jang

“No way” is a strong word. Lots of people have thought about how to apply MCTS or its successors like MuZero to continuous control spaces, and I’m sure very cool research work is still ongoing to try to crack that problem. But yes, the apparent challenge right now is that most problems in much higher-dimensional action spaces, or something combinatorially much bigger like language, don’t seem as amenable to the discrete action-selection heuristics and game-evaluation type stuff that Go has.

That’s not to say the idea of thinking into the future along multiple parallel tracks might not give you some information about which way to search. If you think about mathematics, it often occupies more of a logical search procedure where you can back up and see which paths seem good or not. There’s more of a rigid structure there, whereas in a business negotiation it’s less of a tree and maybe something a bit different.

Dwarkesh Patel

We’re now seated, so I can ask you some more questions about AlphaGo and AI research more generally. In 2021, Andy Jones had a paper called “Scaling Scaling Laws with Board Games”, and he basically anticipated inference compute, or inference scaling, by showing that you can trade off test-time compute and training compute. That is to say, you can spend more compute searching through MCTS, and if you do that, you can get the equivalent performance to having spent more time training the model.

If you see this pattern, you might think that with LLMs you might do something similar in the future, and in fact that’s what ended up happening.

What is a fun exploration one could do now to explore other axes of scaling in toy settings, which will be important to understanding what AI development might be like in a few years?

Eric Jang

Test-time scaling and reasoning, and how they interact with model size, are quite profound when it comes to how much needs to actually be done as explicit search versus how much can be packed into the forward pass of a neural network. How does a forward pass of a neural network learn to do something that should be sequential and recursive? That’s quite interesting.

The Andy Jones “Scaling Scaling Laws with Board Games” paper is quite cool. There was another really nice result from that paper. Not only can you predict scaling laws of the LLM variety—where as you increase parameters you can decrease the amount of compute for search or vice versa—but he also showed that you can predict how much compute is needed to solve a larger version of the board game.

For example with Go, which can scale from 3x3 to an infinitely sized Go board, you might be able to revisit this question and try to reproduce whether this shows up. I actually started this project with the motivation of asking: does the “Bitter Lesson”, or our knowledge of scaling laws, allow us to execute a lot better on a compute-optimal Go bot? Can we build a strong Go bot without all of the KataGo tricks, just by really focusing on the Bitter Lesson and scaling laws?

I have not been successful so far, but usually when you want scaling laws to work, you want to be in the regime where the recipe already works and the data sets are good, rather than trying to figure out how to do scaling while also trying to figure out what the right data sets are.

The scientific understanding component in research often follows a step where you get something to work first. Then you use that system to collect data that then helps you build a mental model of how things work, such as scaling laws. Usually, if you want to build a strong Go bot using scaling laws, you actually have to make a strong Go bot first and then use the scaling laws to extrapolate a bit further into the future.

Dwarkesh Patel

Say more. Just so I understand, are you saying scaling laws did not work? There was no scaling-laws pattern that you could see in your Go bot?

Eric Jang

A mistake I made initially, when I had some bugs around how MCTS labeling was working, was that I would collect a bunch of data with an expert policy and then treat it as a supervised learning problem and try to identify scaling laws with expert data sets. You can indeed plot things that look like this, but if you’re in a regime where your policy is not working well, you might just be studying scaling laws on bad data.

One important implementation detail is that if you want to study a scaling-laws problem, you have to have a problem for which the data is good, the architecture is good, and there are no bugs, and then you solve it from there.

Ex ante, I wasn’t able to apply scaling laws to direct what to look at until I had the rest of the system working. This sounds obvious to researchers. Of course you want a working, bug-free system before you study scaling. But as advice for practitioners, on where I tripped up when I started this project, you don’t necessarily want to jump into the science of studying your man-made artifact before your man-made artifact is interesting enough to be studied.

Dwarkesh Patel

Speaking of compute, you can look at these charts of compute used to train the best AI model in the world over time going back 10 years. It’s a very smooth line in log space that grows exponentially year over year. Except there’s this huge aberration, and that aberration is AlphaGo Zero, which was trained on way more compute than any other AI model at the time. It was 3E23 FLOPS. It’s comparable to a frontier LLM, orders of magnitude off, but still.

The question is, especially with you being able to get something off the ground, did you train it on your own?

Eric Jang

I got a donation from Prime Intellect for about $10K. I spent maybe the first $4K doing exploratory research, and then about $3K on the final run. Some of it remained for serving the model.

Dwarkesh Patel

Is your sense that they did a bad job training it, if you can do it in $10K now?

Eric Jang

The compute required to be the first to do something is always much larger than the compute it takes to catch up. It’s the same story playing out in LLMs. Once someone else has done it, you can use tricks like distillation. You can use all sorts of crutches to bootstrap your way to success.

With my own bot that I’ve hosted online, I actually used best-response training against the KataGo models to get a strong level of performance. As of the time of recording, I’m validating whether I can do that first step, which is to do the tabula rasa play.

Importantly for research, you often want to start from a good init. The simple thing I did first was train best-response agents against KataGo. The AlphaZero team did not have any policy they could train against, because they were trying to do everything tabula rasa. Being the first to do it means you’re prioritizing getting the thing working rather than the most compute-efficient possible implementation.

This plays out in robotics as well. If you look at the frontier of large models trained for robotics, the scatter plot is all over the place, and there isn’t a very clean line the way there is for frontier LLMs. That’s because the folks training these models often are not at the scale where every FLOP counts and they need to squeeze out the performance of every single FLOP as the deciding factor in pre-training. Instead, their focus is more, “We want a certain capability to show up, so we optimize the training setup to make it easy to derive that capability.”

Once you have that capability… Invariably if you scale up the compute, you are forced to make it compute-efficient, because this is hundreds of millions of dollars we’re talking about. But in the past, when compute for experiments was more plentiful—or not accounted for in a way the researcher was really responsible for—you end up with people optimizing for things besides being on the compute-optimal Pareto frontier.

Dwarkesh Patel

I see, like speed or something?

Eric Jang

Yeah, time to result, or just getting it to work. The first AlphaGo probably had lots of compute, and they didn’t need to worry too much about making it the most compute-optimal thing.

Dwarkesh Patel

How much of the improvements to compute efficiency are methods that did not exist as of 2017 versus things which they could have done in 2017 but didn’t?

Eric Jang

Great question. Going into this project, I knew in the back of my mind that things always get easier to do over time. I wanted to see where Go was at, given that there hadn’t seemed to be any major open-source strong bot after KataGo in 2020.

Reading the KataGo paper, there were a lot of clever ideas. I was thinking, “Let’s see if the Bitter Lesson had happened, where a lot of these tricks just go away because Nvidia made faster GPUs. Roughly, where are we on that?”

Again, this is not a peer-reviewed claim. It’s just my preliminary vibe guess based on what I’ve seen with my own experiments. It seems like architecture choices don’t matter that much. Transformer versus ResNet… We’re at the speed of GPU where the size of the model is not so big that this really matters.

You can simplify the setup quite a lot. Instead of doing a distributed asynchronous RL setup with replay buffers and pushers and collectors, you can do a dumb synchronous thing where you collect, train a supervised learning model, and then collect again. There are opportunities to simplify infrastructure.

Nvidia GPUs have indeed gotten faster. Whereas KataGo was trained on V100s, you can train on half the number of desktop Blackwell GPUs and it still works. Some of the auxiliary supervision objectives that KataGo developed aren’t really necessary if you have a strong initialization. If you’re initializing best-response training against KataGo itself, your own model needs none of the tricks that KataGo needs. The core thing is getting as quickly as possible to some strong opponents. That matters a lot more than the specific architectural innovations.

But there are still some nice compute multipliers. I found that training on 9x9 boards was very nice for resolving endgame value functions. If you can co-train that on an architecture that can transfer between 9x9 and 19x19, you can really cut down the warm-start time to learn from scratch. AlphaGo Zero’s plot showed the first 30 hours or so spent basically catching up to the supervised learning baseline. You can cut down that time a lot by pre-training on a small board and then warm-starting that into your 19x19 board play.

There was some other stuff, like varying the number of sims between episodes. This turns out to be not that sensitive. You can fix it or increase it. It doesn’t matter too much. Anyway, it’s nice from a scientific perspective, revisiting an old paper and seeing what really matters.

02:01:09 – Off-policy training

Dwarkesh Patel

This is a tangential question, but why is it okay to have a buffer in AlphaGo? Every time I talk to an AI researcher, they’re telling me about how bad it is to be off-policy. But the way a naive implementation of AlphaGo Zero would work is that most of the moves in a given backward step, or in a batch of backward steps, would not be among the ones made by the most recently trained model. So why is that okay?

Eric Jang

Great question. This gets into the fundamental off-policy versus on-policy reinforcement learning questions. As you’ll recall, in MCTS, you take actions that you took, and you relabel them to take different actions on the same states.

The off-policy part comes in where, what if you’re relabeling states that your new policy would never visit? What’s the point? You’re wasting capacity. In the extreme limit, imagine the distribution of states in your training buffer are all states you would never visit. Then you’re supervising them to take good actions on states you would never achieve. Therefore, your policy can get really bad. This is where off-policy can really hurt AlphaGo.

However, if you interpret this from the DAgger perspective—which is basically a way to correct yourself back to the optimal trajectory given some data—what you want in an algorithm like this is to have mostly states that you would visit, but then a small or reasonable percentage of states in this high-dimensional tube around your optimal trajectories. Any of those states are given a supervision target to funnel you back into your optimal trajectory.

Maybe I can just draw real quickly here. In a DAgger-style setup, your optimal training data distribution is your optimal states and actions. You want to be in this state, you want to be in this state, you want to be in this state, and then you win here. These are your optimal policy actions. These are the things you definitely want to train on. But to make it robust to disturbances, you want to make sure that if you happen to drift off into some other states, you can funnel yourself back in.

Dwarkesh Patel

But why isn’t this a fully general argument for off-policy training?

Eric Jang

This is actually why you want to do off-policy training sometimes. You don’t want a compounding error where, if you make a mistake, you don’t have the data for how to return to your optimal distribution.

Optimal control doesn’t really say too much about how to not accidentally get here, because it’s making the assumption that once you learn the policy, you’re going to get to here. But in applications like robotics, a gust of wind blows you slightly off and now you need to correct. Or the friction on one of your tires is slightly lower than the other wheel, and now your car is drifting and you have to correct it. These kinds of things often happen in more real environments.

There was a funny quote about chess and Go. The problem with Go and chess is that the other player is always trying to do some shit. Things can drift off, and you always want to be able to correct back to your winning condition. Your replay buffer really should have the states your policy would visit, plus some distribution of states you might drift to, and then how to return back to your optimal states.

Now, if you take this to the extreme and you say, “We don’t have any of this data, and we’re just going to be labeling with MCTS states that are so far away from our optimal behavior, like this bag of states over here…” Well, now each of them gets an MCTS label, and your policy learns how to take the best possible action here, but you never get here. You’re training your model on states you would never reach. This is where off-policy can really hurt.

As part of this project, I did try an experiment where I took a bunch of trajectories, and to try to saturate the GPU as much as possible, I took random states from the dataset and reran MCTS on just those states. Instead of playing a whole game where I’m doing MCTS on every move, I ignore the causality of moves and pick random board states, and I label those with my current network. I might revisit old states I’ve labeled before and relabel them again with my current network.

In practice, this actually does work. You can take some states that are reasonable and constantly be relabeling them while training. This starts to converge on a very robotics-like setup, which is very common. You have your dataset of trajectories, and then you have something like a replay buffer pusher. These are off-policy offline trajectories. Your replay buffer pusher pushes transition tuples to the replay buffer, and then you have a job that’s continuously replanning what the best action you should have done instead of the action you took.

In robotics, it’s very common to minimize TD error. Your Bellman updater is constantly pulling things from here and trying to satisfy Q(s,a). From here you have your trainer, which is trying to fit the Q to the ^Qtarget. You can think about this as a sort of planner. You visit old states you’ve been to, and you take your current model and rethink. What could I have done better if I visited this? This is how off-policy robotic learning systems are usually trained. These days there’s a simpler recipe, but in the Google QT-Opt days, we did things like this.

Dwarkesh Patel

So what is the trainer?

Eric Jang

The trainer is where you try to minimize Q(s,a) and Q^target.

Dwarkesh Patel

Can you explain the whole setup again at a high level?

Eric Jang

You have your off-policy data that came from various policies. You’re constantly pushing transitions you saw before to a replay buffer.

Then you’ve got this thing called a Bellman updater, which basically replans: instead of this action, what action should I have taken at s to have a better value? The way you enforce that is by trying to minimize the TD error. Given this, you have s’. You compute Q(s’,a) and you find the action that should go with s’ that makes this Q value as high as possible.

Then you add that to the reward here, and that gives you your actual target. For this current s and a, your Q^target is this. Now you send back the Q^target to this transition. With this tuple, you pair a Q^target. Then on the trainer side, you simply use supervised learning and minimize your current network’s Q(s, a) with its target.

Dwarkesh Patel

Got it. So in the background you’re basically thinking through how valuable all these actions actually were.

Eric Jang

Yeah. In a more optimal policy where you’re trying to maximize this, what is the Q^target of this transition?

Dwarkesh Patel

It’s sort of like daydreaming.

Eric Jang

Exactly. You can think about it like you’re going back in hindsight. Given what I’ve seen in the historical buffer, was there a better action I could have taken?

The connection to Go here—I tried and it was moderately successful, but too complex to open source—was replacing this with an MCTS relabeler. Instead of doing this target network computation, you run MCTS on your transition. In this case, you have your state, your action, and whether you won or not at the game. Actually, you can just toss those two. You don’t care about these ones. You just take your state and plan MCTS to get your best policy, Π, on your current network. Not the network that took this action, but your current best policy network. You rerun your search offline on these transitions.

If these are transitions your policy can get to, this actually acts as a very nice stabilizing effect. Another benefit is that you can fully saturate your GPU better because you’re not blocking on the Go game to give you board states. You simply search across all board states at any depth in parallel. Here the trainer would just predict the MCTS label as best as possible.

This kind of works, and it is quite relevant in robotics where you have a lot of offline data and you can’t simulate things like MCTS. But in practice, it does run into the problem that if the current model is looking at states it would never reach, it’s wasting capacity. You have to be a little bit careful here.

Much of RL has converged to a much more on-policy setup where they don’t really try to directly train on off-policy data. At best, they use off-policy data as a way to reduce variance, but not to directly influence the objective.

Dwarkesh Patel

Why have they converged to that?

Eric Jang

It’s just more stable. You might use the off-policy Q as a way to do advantage computation, like Q minus the sum of Q. If there are N actions, this is your value, and these are your current Q values. Your advantage for that action is the average value minus your current one. People can try to estimate Q in an off-policy way and then just use advantage here.

If there’s a problem in these dynamics, it doesn’t blow up your loss as much. In robotics, there’s a convergence toward using off-policy data to shape your rewards but not actually be directly here.

02:12:02 – RL is even more information inefficient than you thought

Dwarkesh Patel

I’m reminded of our earlier conversation about why MCTS is so favorable compared to the REINFORCE or policy gradient thing LLMs do. This might be totally wrong, but I wrote a blog post a few months ago about how policy gradient RL is even more inefficient than you might think.

The inefficiency one thinks about naively is the fact that you have to roll out a whole trajectory to get any learning signal at all. As these trajectories become longer and longer—as an agent has to do two days’ worth of work to figure out if it did a project correctly, rather than just completing the next word in a sentence—the amount of information per FLOP decreases. You have to unroll two days’ worth of thinking to see if you implemented a feature correctly, so the amount of samples per FLOP has been decreasing.

You’re trying to maximize as you’re learning, bits per FLOP. You can think of bits per FLOP as samples per FLOP times bits per sample. What I mentioned a second ago is that the samples per FLOP go down as RL becomes more long-horizon.

But this kind of naive RL is also terrible from a bits-per-sample perspective, at least compared to supervised learning. Early on in training, let’s say you have a vocabulary size for an LLM that is 100K long, so there are 100K possible tokens that one could answer. You have a totally untrained model and a prompt like, “The sky is…” With supervised learning, the model would have some probability distribution over all the things it could say. There’s a label that says the term here is “blue,” and it would learn through cross-entropy loss exactly how far its distribution is from correctly saying “blue.”

If you were doing this through RL, the model would try, “The sky is halycon.” Nope, that’s wrong. “The sky is told.” Nope, that’s wrong. This is a totally untrained model. You would have to do this on the order of 100K times just to stumble on “blue” and get some learning signal. In the supervised learning regime, you have your distribution of probabilities, you get told that it’s “blue,” and you figure out how far off you were. The amount you learn is a function of your pass rate. The further away you are from “blue,” the more you learn to go toward “blue” using cross-entropy loss.

You can think of it as like your pass rate, your prior probability of having said “blue.” As a function of that, in supervised learning, through cross-entropy loss, you would learn negative log(p), p being pass rate, bits, once you get this label. Whereas in RL, if you’re just randomly guessing and seeing if it works, that’s basically just the entropy of a binary random variable.

Eric Jang

What’s also tough here is that the distribution you’re sampling under is your policy’s distribution. If your policy has no chance of sampling “blue,” then you will never get a signal.

Dwarkesh Patel

Exactly. That’s modeled by the fact that your probability of sampling “blue” is extremely low. If you sample it, you do learn as much as you would have in supervised learning. In all other cases—99.99% of the time in an untrained model—you’re learning incredibly little from seeing that “halycon” or “told” is not the correct word.

That’s what happens most of the time. You learn very little. If you put your pass rate on the X-axis and the bits you’re learning from a sample on the Y-axis, with 0%, 50%, and 100%, so at the end of training you’re here. If you have supervised learning, the negative log pass rate would look something like this. The entropy of a binary random variable would look like this, depending on whether you’re doing nats or bits. If you do bits it’s one here at the peak. This is like a coin flip. You learn the most from a coin flip. This is supervised learning. This is RL.

However, the problem is you spend most of training in this regime, in the low pass rate regime. How fast you’re learning is a function of how many bits per sample you’re getting, and you’re getting very little signal here. If you chart the pass rate on a log scale—where at the beginning of training with a vocab size of 100K the pass rate is 1/100,000, then 1/10,000, 1/1,000, 1/100—what this graph looks like here, supervised learning would look like this, and then RL would look like that.

Eric Jang

And arguably you spend all your time here, potentially never even getting a single success. It’s a depressing plot in the sense that once you’re here, it’s not at all obvious how you get to there. Once you’re here you have something, but you actually spend all the time here in many RL problems. There’s a question of how you initialize so you’re at least at a non-zero pass rate.

One more thing I’d like to add about bits per sample that’s very relevant to any machine learning problem is that there’s a connection to soft targets and distillation. If you have access to the logits, not just the one-hot token answer, if you have access to the soft targets the entropy of this distribution is far higher than the one-hot. There’s way more information in bits per sample in a soft label. That’s why distillation is so effective per sample. It’s giving you way more information per sample.

Dwarkesh Patel

I wonder what the equation would be, but obviously it’s…

Eric Jang

It would just be the entropy of this distribution. The entropy of this is zero. The entropy of this is the entropy equation.

This is also why AlphaGo is quite beautiful. In AlphaGo, you don’t train the policy network to imitate the MCTS action. You train it to imitate the MCTS distribution. Both of these are valid, and if you wanted to do a scientific experiment of how important this soft label is, dark knowledge distillation, you can run an experiment where you retrain the policy network on the action MCTS selected rather than the soft target.

Dwarkesh Patel

Earlier I was stumbling around. Intuitively, why is this ability to do iterative search—where you don’t necessarily need to be able to win the game in the beginning, you just need to be able to improve your current policy—so powerful a capability in learning compared to how LLMs currently learn RL? It’s exactly this thing of considering your pass rate of the entire trajectory. I just don’t know a formal way to think about this. Maybe you should help me out here.

Eric Jang

Why is AlphaGo an elegant RL algorithm? The major reason is that you never have to initialize at a zero percent success rate and solve the exploration problem of how to get to a non-zero success rate. This is what allows you to hill-climb this beautiful supervised learning signal.

If you look at the actual implementation of AlphaGo, every step of the way, there’s actually no TD error learning or dynamic programming, at least explicitly. It’s just supervised learning on a value classification as well as a policy KL minimization. It’s just a supervised learning problem on improved labels.

The training is very stable. You can train as big of a network as you want. You can retrain this on the data set. Everything will just go stably. The infrastructure is very simple to implement as well. You don’t need a complex distributed system to keep everything on policy.

At the end of the day, you’re just saying, “I have some improved labels. Let’s retrain my supervised model on these targets.” You’re always in this beautiful regime where you’re just trying to improve the policy, rather than escape local minima where every signal is flat all around you.

One way to draw the curve is if you draw the win rate of an MCTS policy versus the raw network… Let’s say this dotted line is the raw network. The MCTS policy looks like this. Every step of the way, this supervision signal is very clean. You’re never in a situation where the MCTS is giving you no signal, unless your MCTS distribution converges to exactly what your policy network predicts.

Dwarkesh Patel

That’s a great way to explain it. Maybe we sit down and I ask some questions about automated research.

02:22:16 – Automated AI researchers

Dwarkesh Patel

One thing I really wanted to talk to you about is that you did a bunch of the research for this project through this automated LLM coding assistant loop. There’s an idea that if you fully automated AI research, you could have some sort of singularity.

Obviously we’re not there yet, but to the extent that we have early indications of what this process might look like, I’m curious about your observations about what the AI is good at, what it’s not good at, what you think about this scenario’s likelihood eventually, and what thoughts you have about this in general.

Eric Jang

I think automated scientific research is one of the most exciting skills that the frontier labs are developing right now. It’s important for everyone who’s doing any kind of research to get a good intuition of what it can do now and what it can’t, and how the science process might work in the future once we have AIs automating a lot of this investigation.

In brief, I mostly used Opus 4.6 and 4.7 while working on this. What works is that the models can do a very good job of hyperparameter optimization. In the past, people would come up with a search space of hyperparameters like learning rate, weight decay, and maybe how many layers are in your network. They would do a grid search or a Bayesian hyperparameter optimization approach, and it would find some tuned parameters.

The really cool thing that automated coding can do now is search a much more open-ended set of problems. It can say, “I’ve identified that the gradients are small in this layer, so let me change it up here. Let me rewrite the code so the data loader has a new augmentation I came up with. Let’s try to find the best way to fit the constraints of the optimization problem.” You end up with this much more flexible, high-level, almost grad-student-like ability to just grind a performance metric.

This can squeeze out quite a lot of performance. On a fixed data set with a fixed time budget, you can improve perplexity by quite a lot on a classification problem like LLMs or Go. It is also fantastic now at basically executing any experiment. I have a Claude Skill that I wrote called Experiment where I give it a description of what I want it to plot. I just describe, “Here’s the x-axis I want, here’s the y-axis. Answer this question for me.” It’ll run off and do all the experiments, compile the plot, make a report, and suggest what might have caused it and so forth. That’s what works quite well today, and I think we can expect these abilities to get better in the future.

But it’s also useful to know what it’s not doing so well today. In the blog version of this tutorial, I have a plot of all the experiments I did grouped in a tree, where every node represents a failed, successful, or mixed experimental result. From there, it branches off into a child representing the follow-on experiment. Occasionally, I’ll rabbit-hole down a track like off-policy MCTS relabeling, do a few experiments, and then realize it’s probably not worth it. So then I’ll jump to a completely different track. I call these things rows.

What I find is that the current closed models the public can access today don’t seem to be that great at selecting what the next experiment should be in a given track. They don’t seem to be able to step back and do the lateral thinking of, “Wait a minute, this track doesn’t really make sense. Let’s go back to first principles and think about what the bottleneck might be, or what we are trying to achieve.” Often I had to catch infra bugs myself by prompting the right question to Claude to investigate what’s causing the discrepancy, and then it’ll answer the question.

With Mythos-class models or Mythos++ models coming online, maybe this just completely changes and these problems fall to improved scaling. But at the same time, I think there’s a rich opportunity to develop RL environments that might incentivize this kind of lateral thinking. One of the motivations for setting up this Go environment was that Go captures a lot of very interesting research problems, often overlapping with LLMs or robotics. Yet it’s very quick to verify. The outer loop is ultimately: does the agent do what I think it does? You can check the outcome of a Go game quite easily.

The inner loop involves all this research engineering around distributed systems, predicting whether an idea is going to work or not, and predicting the difference a particular modification to your training algorithm might make. I think there’s a rich library of subtasks and sub-environments that you can train an automated scientist to work on, with Go as a sort of outer verification loop. Once you acquire these skills, maybe you can apply them to other domains like biosciences or robotics.

Dwarkesh Patel

Or automating AI research.

Eric Jang

Or automating AI research.

Dwarkesh Patel

Which is the real crux, the scary/incredible thing of making AIs make future versions of AIs. You’re suggesting the outer loop here could just be your win rate against KataGo?

Eric Jang

That’s one of them. I think there are a lot of deeper questions that one could tackle. For example, let’s say you have an idea on how to improve a scaling-law compute multiplier. The outcome isn’t necessarily “achieve the best Go bot ever”. The outcome might just be, “Can I predict what the win rate of my Go bot will be?” Or, “Can I predict the scaling-law plots that emerge from my idea?” But then you can verify that you haven’t reward-hacked anything by using a very verifiable game like Go on the outer loop.

Dwarkesh Patel

I think there are a couple of interesting follow-on questions. There are questions on the inner loop and the outer loop. On the inner loop, there’s a question of how locally verifiable any modification you might make is. That is to say, would you know whether some idea you try out is actually an improvement or a degradation? Would you know if something isn’t working as a result of a bug, or is it the result of the idea itself being wrong?

Ilya was talking about how one of the things he thinks makes him a good researcher is that he has a strong belief in what the correct idea is. He’s able to persevere through bugs and know which things are bugs versus mistakes in the fundamental idea, based on his high-level belief that “this idea should work, so therefore there has to be a bug”, versus the other way around. Why don’t we start with that question? How locally verifiable are things which are good ideas?

Eric Jang

As in the case of the success story for deep learning, you can think about this as a decades-long idea that took a lot of faith to get it to work. This presents a very challenging long-horizon RL problem where every step of the way you have a committee telling you that this is a bad idea, and then ultimately you break through. How do you design RL environments that maybe give you some feedback earlier? I think this is a very tough open question that I don’t have an answer to.

Ultimately, to play a very strong Go bot, you probably did need to discover deep learning. Having a challenging game that cannot be cheated easily on the outer loop could be used as an outer-loop signal for something like discovering the principles of deep learning. Now, of course, to make it tractable—and this is where research taste really matters—you have to come up with ways to initialize your problem so that you don’t try to solve a very intractable problem. Maybe you can leverage LLMs as a universal grammar in the middle to give you some sort of local feedback.

The fact that LLMs are a universal grammar means that they can move at almost any level of the stack. They can think very locally, as well as step back and think in very broad steps. I think that’s where a lot of the lateral thinking ability of humans comes from: knowing when the track you’re pursuing or the objective you’re pursuing is not right, and you should be asking a different question.

Dwarkesh Patel

The other question is how stackable local improvements are in the attempt to get to a better result on the outer loop. I’ve heard rumors that at some AI labs, the thing that has gone wrong is that people will individually pursue good ideas, but those don’t end up stacking well, and so the training run fails because of some weird interaction between two seemingly good ideas. Having a single top-down vision of how things should work is very important. Having worked at different AI labs and also played around with parallel agents trying different ideas, what’s your sense of how parallelizable AI innovation is?

Eric Jang

Great question. I think the research taste for executing well on the Bitter Lesson is that you need to know how much the Bitter Lesson can buy you and how much is too much to ask for, at any given moment. Of course, in the fullness of time compute is the single most important determinant of how things work. It’s almost inevitable that as you scale up energy and compute and parameters, intelligence will just fall out of that. That’s super beautiful, super profound. No algorithmic detail really matters beyond that.

But in the present day, we don’t have infinite compute and parameters and arbitrarily good initialization, so we have to come up with heuristics that give us that. These heuristics are probably somewhat redundant. That’s probably why you see this effect where a lot of these compute multipliers don’t necessarily stack. They might have some correlated benefit. And then three years down the line, when the Nvidia GPUs have gotten even stronger, maybe they stack even less well.

Maybe at any given point in time, the benefit of any given compute multiplier is transitory, which is what I suspected with the KataGo paper. There were many algorithmic ideas applied, and then you can see that with modern Blackwell GPUs and Ada-class GPUs—which are much better than the V100-grade GPUs that that paper used—some of these algorithmic tricks to speed up convergence just don’t matter so much compared to something else. I think that’s a matter of taste in the present time.

Dwarkesh Patel

How about the outer loop? How verifiable for making AI smarter? With Go, you do have this outer loop of win rate against the best open-source model out there. Even there, as you were saying, there are other outer loops of whether you discovered a new phenomenon, which is actually very hard to…. If you didn’t know scaling laws were important… When were Chinchilla or Kaplan scaling laws released?

Eric Jang

2018.

Dwarkesh Patel

So if you’re back in 2015, there’s not an automated procedure one can easily imagine for knowing which paper is the scaling laws paper versus which is just another random plot. Even in the Go case, it’s a hard-to-verify outer loop, and the whole idea of an outer loop is to have some backstop on improvement, let alone for general AGI, where of course we have a bunch of these benchmarks.

But there’s a problem. We know the things we can measure, and we improve on the things we can measure. We care about this broader ability to do economically useful work, which is not super easy to measure, at least until you automate everything. There’s a question of how good the outer verification loop is for AI self-improvement, and does that matter?

Eric Jang

I’m going to give a non-rigorous argument, but one that I intuitively believe. DeepMind started with a focus on games. They used games as their outer loop, and then their researchers learned from the experience of solving games, and now they’re working on LLMs.

Presumably, there was some positive transfer from their time working on games and Atari and Go and StarCraft that now helps them make good LLMs. I assume that there’s positive transfer in some regard, whether it’s coding or general research ability or project management. All these things probably help them do well.

If that’s the case, why wouldn’t it also be true for automated AI researchers? They should be able to positively transfer experience tackling quick-to-verify, quick-to-iterate environments to something more ambitious and economically useful, like automating drug discovery.

Dwarkesh Patel

I don’t know. Isn’t the issue… Historically, until Gemini 3 or whatever a couple years ago, people were saying, “Look, Google isn’t catching up in LLMs because they’re too tied to the old approach.” There are gains, but there are also ways in which it actively hinders you. It’s not obvious to me.

Eric Jang

The jury’s still out. Let’s say currently Google’s doing quite well. Who knows if the initialization of training on games is ultimately going to hobble their ability to be the winner in the long term. It’s hard to say for sure.

Likewise, who knows if the seeming late start was really just them pre-training for longer on how to scale up TPUs. They invested all their tech tree in getting TPUs to be good, which seemed not that useful in the short term but in the long term it becomes… So it’s even hard for humans to reason about what the optimal research strategy should be, even with the data we have today.

Dwarkesh Patel

Okay, we should let people know how they can find out more about this project, whether to fork it themselves, or check out your blog post where you do an excellent job explaining many of these ideas. Where do people go next?

Eric Jang

My website is evjang.com. There’s a blog post that links to an interactive version of this tutorial. On my GitHub, which is the username ericjang, there’s an AutoGo repo that people can fork and reproduce the training results.

Dwarkesh Patel

I also highly recommend people check out this blog post, “As Rocks May Think.” We touched on some of the ideas in this conversation, but it’s this grander thesis of what happens when you have thinking as a primitive in—

Eric Jang

Computer science.

Dwarkesh Patel

Exactly. I highly recommend people check out that blog post as well.

Eric Jang

I encourage the audience to think about the relationship between thinking and Go via MCTS and search, and how it relates to LLMs. I think there’s something quite profound there, and probably underexplored just because Go has been relatively underexplored compared to the boom in LLMs.

It’s not to say that I think we should have trees in our LLMs, but there is some very interesting duality between them. You can actually do a lot of research on Go, MCTS, and reasoning with very small budgets. So that’s very exciting.

Dwarkesh Patel

Awesome, Eric. Thanks for doing this.

Eric Jang

Thank you. It’s an honor to be on the podcast.

He and collaborator Ali Akbari just published a paper that overturns a long-standing consensus about human evolution — that natural selection has been dormant in our species since the agricultural revolution.

By scaling ancient DNA sequencing and developing a new statistical method, they found that selection has actually sped up.

Selection went especially bonkers during the Bronze Age (around 3,000 years ago).

That’s when gene frequencies for everything from immune function to body fat to intelligence were most in flux.

Over the last 10,000 years, selection pushed the genetic predictor of cognitive performance up by roughly a full standard deviation — most of it between 4,000 and 2,000 years ago.

After we finished recording, David sketched out on a whiteboard his new heretical model about who the Neanderthals really were. Luckily, I took out my iPhone and managed to record it.

He thinks the standard story (that Neanderthals are some separate archaic lineage we interbred with a little) just doesn’t fit the evidence. Instead, he proposes that Neanderthals are essentially genetically-swamped modern humans.

A small population somewhere around the Caucasus invented Middle Stone Age technology roughly 300,000 years ago and expanded outward. The ones that moved into Europe interbred with local archaic humans, got genetically swamped, and became Neanderthals. The same expansion went into Africa, met much more diverged archaic Africans, and that mixture became us.

This means Neanderthals and modern humans share the same cultural ancestry — the only difference is which archaic humans they mixed with afterward.

David is a brilliant and rigorous scholar. It was a real delight to learn from him again.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Cursor was super useful as I prepped for this episode. Whenever I had a question, I’d have Cursor kick off a few different models simultaneously and then compare their responses. I found that this led to better results than I could get out of any individual LLM. If you’ve only used Cursor for coding, you should try using it for research. Check it out at cursor.com/dwarkesh

• Jane Street uses an internal currency called “hive bucks” to allocate compute through a real-time auction – and anyone can change anyone else’s bids or even kill their jobs! Everyone just trusts each other to act in the firm’s best interest, which is what lets the system work in the first place. If this weird and high-trust culture sounds like your kind of thing, Jane Street’s hiring at janestreet.com/dwarkesh

• Crusoe’s ML infra team built fastokens, an open-source tokenizer that delivers a ~9x speedup over Hugging Face and up to 40% faster time-to-first token – on real production workloads! Crusoe achieved these results by parallelizing things and using some clever engineering to handle duplicates without cross-thread coordination. Learn more at crusoe.ai/dwarkesh

Timestamps

(00:00:00) – Ancient DNA suggests strong selection over last 10,000 years

(00:15:45) – Natural selection intensified during the Bronze Age

(00:35:02) – Why didn’t evolution max out intelligence?

(00:57:21) – Evolution is limited by time, not population size

(01:09:02) – Why no farming before the Ice Age?

(01:17:13) – The Neanderthal puzzle David can’t stop thinking about

(01:54:10) – The methodology behind this breakthrough

Transcript

00:00:00 – Ancient DNA suggests strong selection over last 10,000 years

Dwarkesh Patel

I am back with David Reich, who is a professor of ancient DNA at Harvard. How do you describe what it is that you study?

David Reich

I’m a geneticist, and I work on human history and how ancient people relate to each other and people living today.

Dwarkesh Patel

We did an interview two years ago, which ended up being one of the most popular interviews I’ve ever done. I think people found it really compelling that there’s so much about human history we don’t know and are just learning about now as a result of the kinds of techniques your lab is using.

You have a new preprint that’s very exciting, and I wanted to talk to you about it. Can you give me a little bit of context on what we’re talking about today?

David Reich

The dream was that when this ancient DNA field started, more than 16 or 17 years ago, we were going to learn a lot about biology —about how people’s biology changed over time— by getting DNA out of ancient human remains and tracking changes over time. And that dream has really not been realized since the beginning of this field.

The field has been a big success with regard to learning about human history. It’s resulted in surprising findings about human migrations —people not being descended from the people who lived in the same place hundreds or thousands or tens of thousands of years before— and mixture being common in human history, and sex-biased processes being common. And there have been things that were not expected from archaeology. The field’s been a big success from that perspective, but what’s not been successful is learning about biology and biological change.

One big reason has been that the sample sizes have been too small. When you have a single person’s DNA, it provides a tremendous amount of information about history. That’s because when you look at one person’s DNA, it’s not a single person. It’s many people. It’s your two parents, your four grandparents, your eight great-grandparents, 16 great-great-grandparents, and so on. Going back in time, thousands, tens of thousands, even hundreds of thousands of ancestors are contributing to people today.

When you look at the DNA of a single person’s genome or a Neanderthal genome, you have effectively tens of thousands of ancestors all represented in your data. And you can position that individual exquisitely with respect to other people from whom you have data. But when you are interested in how a particular genetic variant—that affects something like your skin pigmentation, or your ability to digest cow’s milk into adulthood, or a behavioral trait—changes over time, a single person gives you only one sample, or maybe two samples: the one in their mother and the one in their father.

To get a high-resolution picture of how the frequency changes over time, you need very big sample sizes, truly very large numbers of people. We just didn’t have that until the last few years. What motivates the study we’re talking about today, and the work that hopefully a number of groups will be doing in the coming years, is the fact that we now finally have those numbers. We can do something with the data to see how frequency changes over time.

Dwarkesh Patel

Can I ask a question? I’ll be asking a lot of naive questions through the next few hours, but why are frequency changes especially interesting?

David Reich

What we’re interested in is using the experiment of nature that’s occurred in our history, over the last tens of thousands of years, to understand what’s biologically significant in our DNA. If there has been a change in environment that a population has experienced—for example, people shifted to agriculture, began living close to domesticated animals, or moved from a cold place to a warm place, or a low place to a high place—then there’s pressure on the population to adapt to these new stresses and new needs.

The way you’re going to detect that is by seeing that the frequency of a genetic variant—that for example might allow you to live at higher altitude, or that might nudge you to have a different behavioral pattern advantageous in the new situation—pushes systematically in some direction in a way that is enough for you to detect. It’s very hard to detect slight shifts in frequency by a few percent or ten percent unless you have a very big sample size. What we’re looking for are those changes in frequency that are too extreme to be due to chance. That will tell us there have been pushes against the biology as a result of the changes in environment that people have experienced.

Dwarkesh Patel

Interesting. What did you guys find?

David Reich

Seven years ago, Ali Akbari, who at the time was a postdoctoral scientist in my laboratory and a few years later became a permanent staff scientist, set out to use the data we were producing to learn about biological change over time. I think the reason he was interested in our laboratory rather than other places was that a focus of our lab has been generating truly large amounts of data from ancient humans. We’ve been trying to industrialize the process, make it very inexpensive, make it high quality, and generate large numbers of samples with lots of good data for this purpose.

There’s been this large amount of data we’ve generated. And it made it possible to conceive again of asking whether there have been frequency changes over time. The mainstream view in human evolution in the last several decades has been that natural selection has been pretty quiescent over the last several hundred thousand years of human history.

There are several lines of evidence that have been deployed to document this. One is that if you compare diverse populations from different continents around the world, for example Europeans and East Asians, and you look at mutations that differ in frequency between these groups—all mutations differ a little bit in frequency, sometimes a lot—you can say, “What are the most different mutations in terms of frequency between Europeans and East Asians?” And there are almost no genetic changes that are 100% different in frequency between Europeans and East Asians.

Europeans and East Asians descend from a common ancestral population 40,000 or 50,000 years ago that came out of Africa and the Middle East. This population had a set of gene frequencies, and these variants bopped around randomly—a process known as genetic drift—or perhaps under selection in one direction or another. The time that’s passed since 40,000 or 50,000 years ago is sufficiently small on an evolutionary timescale that there’s just not much genetic differentiation on average between these two groups.

However, if there’s been natural selection, for example to help people in one place digest alcohol better, or digest milk better, what you might expect is that there would be some mutation that would have rocketed up to very high frequency. Forty or fifty thousand years is a lot of time, it’s maybe 1,500-2,000 generations. That might easily be enough time to see a 100% difference in frequency. Yet you don’t see any more than what you would expect by chance.

This combination of things made it seem that selection has just been quiescent. Maybe a few hundred thousand years ago, the ancestral human population got to some kind of optimum, and after that there hasn’t been much genetic change in one way or the other.

There have been small amounts of natural selection, or selection to remove bad mutations that are constantly raining down on the genome, but not what we call directional selection. That would be newly arising mutations, or mutations being pushed in a systematic direction, to help the population get to a different adaptive set point more favorable for the conditions that population is living in.

We were able to partition how much of the changes in frequencies of all the mutations that we’re seeing in the DNA—we’re looking at about 10 million positions that vary—is due to directional selection (adaptation) versus other factors, especially genetic drift. And 98% of it is other factors, especially genetic drift. It’s overwhelmingly migrations and population structure causing fluctuations in frequency.

As a result, it’s super hard to detect the signals of adaptive natural selection because they’re a tiny fraction of the total frequency change. The vast majority of it are these migrations and mixtures. Nevertheless, there’s so much natural selection, as our study has shown, that it’s actually been rampant in the genome.

Dwarkesh Patel

Can I ask a clarifying question here? Why are we discounting population admixture or replacement as selection? If you think about it at a group level, if one population replaces another population, isn’t that selection?

I remember from the last episode you were explaining how there have been huge changes in what kinds of people are in a specific area. One population came in and replaced the previous one, and then a new population came in and replaced that one. To the extent that the genetics are relevant to why that population replaced the other one, why should that not count towards what we understand to be selection over the last 10,000 years?

David Reich

It could count, and may count, and probably should count in some respects. But it could also be that this population replacement is due to some cultural phenomenon —technology held by one of these groups and not others. And maybe there are some genetic mutations that are contributing to this. Who knows? It’s possible.

But what you’re seeing is a whole-genome shift. What we’re looking to see is whether there’s one place in the DNA that is driving the change in a way that’s different from the rest of the genome. From a statistical point of view, what happens at these times of migration is there are just huge fluctuations in frequencies. These are extremely uninformative times for detecting natural selection. The best moments to detect natural selection are when migrations and population admixtures are not happening for a few hundred years. During these times, you can actually see the mutation slowly blowing in one direction as a result.

The way we think about the history of Europe and the Middle East for the purpose of this study is as an archipelago of little populations in space and time, each pretty isolated from each other. You have a little population in Britain isolated for a few hundred years, or a little population in Hungary isolated for a few hundred years, between big events of migration and mixture. In each of those little experiments of nature, we can ask: does this mutation slightly increase in frequency? Does that same mutation slightly increase in frequency? If all the arrows point in the same direction, we win. They’re telling us that natural selection is occurring.

For example, 4,500 years ago in Europe, almost all mutations went through huge frequency changes. That’s not because of natural selection. It’s because of the steppe migration from north of the Black and Caspian Sea. 40-80% of the DNA becomes Yamnaya from steppe pastoralists. Their frequencies of mutations were different not because of selection necessarily, but just because they had evolved in different places for thousands and tens of thousands of years. When you look at the descendant populations, there are huge changes in frequency. What you need to do is see if natural selection is explaining a shift more than you would expect by chance.

Dwarkesh Patel

So you found these locations that seem to be under selection. I have another clarifying question. You say you found 3,800 locations which you’re 50% confident have been under selection in the last 10,000 years.

David Reich

It’s 7,200 where we’re 50% confident. We’re getting about 7,200 positions in the DNA that have 50% confidence of being real. Only half of those are real—we don’t know which ones—so 3,600 of them are real.

Dwarkesh Patel

Does that also mean that outside of those 7,200, you’re confident the other locations in the genome are not under selection?

David Reich

No. If you look at the 25% probability cutoff, there will be tens of thousands, and there will be many real ones there too. In fact, multiple analyses we do suggest that the genome is vibrating with natural selection.

There are all sorts of weaker effects that would be picked up in even larger studies than we’ve done. In fact, almost every position in the DNA is correlated to another position that is being dragged in one way or the other by natural selection. Instead of being quiescent, natural selection is everywhere. Even though it’s only 2% of the frequency change, it’s tugging the positions in one direction or the other everywhere.

So we analyzed these positions that we had identified, the hundreds of positions we were super confident about. We looked to see whether they were randomly distributed in the DNA or whether they had patterns. We looked at maybe 100 or so traits where there had been genome-wide association studies for all sorts of different traits, associated with immunity or autoimmunity or behavior or metabolism, and other things.

For each of these we could ask: do the genetic variations that are known to affect these traits from genome-wide association studies have an unusual number of genetic selection signals? What we found is that there was a vast enrichment, by about four or five-fold, for immune traits. There was a super concentration of selected signals in immune traits. We also saw a strong enrichment for metabolic traits—things that might impact obesity or fat traits or Type 2 diabetes—and almost no detectable enrichment, as far as we could tell, for behavioral or psychiatric traits.

Dwarkesh Patel

Just to make sure I understand. This is not to say that behavioral or psychiatric or cognitive traits are not under selection. It’s just that the individual sites where such traits are controlled are not especially likely to be among the locations you’ve identified as under selection.

David Reich

That’s exactly right. It might seem from the results of that analysis that immune traits are highly selected and that there’s been no selection for behavior in the last 18,000 years in this part of the world. But that’s a wrong conclusion, and we have evidence that it’s a wrong conclusion. There’s clear evidence of selection also on behavioral traits.

The reason we think we see much weaker signals for behavioral traits is that behavioral traits, we know from medical studies, are underpinned by much larger numbers of genes than immune traits, which are underpinned by relatively small numbers of genes of strong effect. Behavioral traits are shaped genetically by a very large number of genes of weak effect, and we just don’t have the statistical power to detect these very weak signals.

When we do an analysis looking at our very strong signals of selection, that collection of very strong results is very effectively querying the immune traits, but is not very effectively querying the behavioral traits. It may still be the case, and I guess it is, that immune traits are the most selected category. But it is not at all the case—and we can prove it’s not the case—that behavioral traits are not selected.

Dwarkesh Patel

Interesting.

00:15:45 – Natural selection intensified during the Bronze Age

David Reich

We’ve been able to prove that there are two ways to reconcile the previous observations with our new observations. Remember, the previous observation is that natural selection seems to have been quiescent over a timescale of hundreds of thousands or many tens of thousands of years. Reason? That you don’t see 100% difference in frequency variance across Europeans and East Asians.

Now we’re seeing hundreds of positions that are rocketing up in frequency with selection rates of 1% or more in a lot of cases. A 1% or more selection rate will mean a rapid doubling over periods of dozens of generations. Over the 1,500 or 2,000 generations separating Europeans and East Asians, shouldn’t you see many genetic variants that are 100% different in frequency across populations?

We were able to show that this is explained by at least two factors. One is that in this part of the world—Europe and the Middle East—we are actually in a period of accelerated natural selection. One way to see this is to look at the enrichment pattern we’re observing, where immune traits are unusually associated with these selection signals. We could compare the last 5,000 years of our time period, what’s called the Bronze Age and further onward, to the previous 5,000 years. What we see is that this intensification of selection around immune traits, and similarly the intensification around metabolic traits, has accelerated over this time period.

It’s not like natural selection has been at the same rate over all places and times. It’s increasing over the time period we’re analyzing. Plausibly the whole time period has increased compared to previous periods. We’re in a period of intensified selection. That’s not implausible, because this is a population that went through a huge shock in terms of the way people live and the culture. Almost everyone we’re analyzing are farmers or food producers in one way or another. Farming was invented for the first time anywhere in the world in the Middle East 11,000 or 12,000 years ago. The people who invented farming exploded into Europe after 8,500 years ago, spread across the continent, and expanded rapidly.

In the Bronze Age, there was an intensification of how people lived, with much higher population densities. People were living more and more next to their animals and getting their diseases, and exchanging their diseases with the animals and with each other. This is a period of rapid change in how people are living, resulting in different biological needs of this population. It’s not surprising, perhaps, that in the context of these dramatic changes, the biology of the population might not be ideally adapted.

There might be what some people call an evolutionary mismatch, where you take a genetic variation that evolved in hunter-gatherers and put it into farmers or pastoralists, and it’s not exactly right. What you’re seeing is the DNA of this population, which descended from hunter-gatherers only 10,000 years ago, reacting to the shock of having been moved into an agricultural, Bronze Age, high-population-density, urban environment. A hypothesis is that what we’re seeing is the adaptation that occurs as a result.

Dwarkesh Patel

In the paper you have many examples of this intensification of selection around the Bronze Age. It might be helpful to go through some of these.

David Reich

One of the things we do in this work is look carefully at many of these positions in the DNA. We actually have an internet browser called the AGES browser, which Ali and a colleague of his—who’s a co-author of our paper—built. It allows you to query each of these 10 million positions and see the trajectories at each position and the evidence for selection.

One of the things we see is that, while for the most part the signals of natural selection we detect are consistent with constant natural selection over time, in a handful of them we’re able to see that there’s been a reversal or a radical change in natural selection. Very often that occurs in the period between 5,000 to 2,000 years ago, which is the Bronze Age and the Iron Age, a period of rapid population growth and rapid movement to intensive use of many technologies that were not used that way before.

An example of this is the TYK2 genetic variant that is a major risk factor for severe tuberculosis, which is the most important infectious disease killer in the world today. If you look at this major risk factor for tuberculosis, this variant rockets up in frequency from 8,000 or 6,000 years ago to maybe 9% or 10% in this part of the world. Then it rockets down in frequency in the last 3,000 years. In both cases, there’s very clear evidence of natural selection, in the first case to increase in frequency, and in the next case to decrease in frequency.

A possible reason is the spread of tuberculosis. It maybe becomes endemic in the population 2,000 or 3,000 years ago. That’s potentially consistent with pathogen sequence data and other lines of evidence. And maybe this variant was protecting against something before then, but then tuberculosis became significant after that point, and it was so bad that it pushed in the opposite direction. That’s speculative.

Dwarkesh Patel

The thing it was protecting against was probably another disease?

David Reich

Maybe.

Dwarkesh Patel

One of the big takeaways for me from the paper was just that something weird happened in the Bronze Age. As you said, across trait after trait, the selection intensifies during the Bronze Age.

This makes sense for some things. For example, why do we see lactase persistence, where adults can process milk, intensified during this period? This is the time when we start using cattle not just for the meat, but also for milk and wool and other secondary products. So it makes sense why lactase persistence would matter more.

But then there are other things that seem like they should have been relevant since the dawn of agriculture. I forget the exact name of the allele, but was it FADS1, which helps convert plant fatty acids into long-chain fatty acids that your body needs? That’s obviously relevant when you move from a diet of meat as a hunter-gatherer to a diet of cereals.

That is also one I think you found was under especially high selection 5,000 to 3,000 years ago. So what’s going on? Why is the Bronze Age so special across all these different traits that you’re observing?

David Reich

So this FADS1/2 variant is a vegetarian/meat-eating adaptation. Already in work prior to this, Ian Mathieson, who worked with me in 2015, identified this as a very strongly selected variant. It’s actually ancient. You see copies in archaic humans too.

One of the findings of our paper is the ABO blood system. You get your blood typed as A, B, and O. The B variant has increased up to 10% at the expense of A, but previous work has shown that A and B were both already present in the ancestor of humans and gibbons and other apes. Some of these mutations have been going back and forth and fluctuating over different time periods.

But we’re talking about changes in the Bronze Age. The TYK2 variant for tuberculosis risk, a multiple sclerosis risk variant, inflected and increased in frequency before the Bronze Age, and then 2,000 or 3,000 years ago reversed in that period. There are differences in Northern Europe where this process is super strong, very strong positive selection, very strong negative selection. And then in Southern Europe, only a little bit, and not even very strong negative selection.

For haemochromatosis, which is pathogenic iron buildup that causes problems in Europe, that too has reversed around this period. In some of the complex traits that maybe we’ll talk about later, these traits too have periods of intensification of natural selection. For example, depigmentation: Europeans have gotten lighter skin over the last 10,000 years. You can see it in our data. The period of strongest depigmentation is between about 4,000 to 2,000 years ago, and then after that it’s much less.

This seems to be a very impactful, eventful, important period where a lot of the processes we are seeing become very powerful. It’s surprising on first principles. You might think, before you walked into this genetic data, that the big change is going to be starting to grow plants and maybe farm animals. That happens in the Neolithic, beginning 10,000-12,000 years ago, and spreads into Europe after 8,500 years ago. But actually, the intensification happens 5,000 years ago, 4,000 years ago.

It’s really interesting. This observation of that being an inflection point tells us something about when humans, at least in this part of the world, were wrenched into a way of living that was so different from how their hunter-gatherer ancestors lived that the organism had to adapt very strongly. It may be that the degree of that wrenching process moving into the Bronze Age was qualitatively greater than the degree of the wrenching process that happened from the initial transition to growing plants. That’s surprising, because our cartoon picture is that the big transition is farming. But the biological readout is saying our genome is reacting much more strongly to these events that happened 5,000 years ago.

Dwarkesh Patel

You did some work with Bhatia and many other colleagues in 2014 where you were looking at 20,000 or 30,000 African American genomes today. You were saying, “Look, there’s 80% West African DNA and then 20% European DNA. Can we look at their genomes today and see that their allele frequencies are much different than we’d expect from this admixture?” Correct me if I’m wrong, but you found that they weren’t.

That is to say, over 200 or 300 years of extremely intense environmental change—going from chattel slavery to a completely new environment—there’s no effect of natural selection. So we see episodes like this where we don’t see natural selection, but then the Bronze Age apparently must have had an even stronger effect, where the change in environment is even stronger than what we see from Africans in Africa being migrated to the New World and living under slavery.

David Reich

That may be the case. It also may be the case that that period is just too short to see much effect. In the Bhatia et al. paper, where we looked at about 30,000 African Americans, we looked to see whether—instead of the average percentage of around 80% West African ancestry—there were some places in the DNA with significantly more than 80%, or significantly less than 80%. That’s what you would expect if there were natural selection for some genetic variant from Europeans or from Africans.

We didn’t see any place in the DNA that was significantly different from what you would expect by chance. One possible explanation is just that there’s only a handful of generations, maybe five, over which natural selection would operate. So if the selection was 2% a generation, you would still only see a 10% compounded effect, and there’s just not enough time to detect it.

But the Bronze Age is not 300 years, it’s 3,000 years. It’s the power of compound interest, and you have enough time to begin to see a strong effect. This really, really does seem to be a very impactful time in terms of human history, and you can see it in our complex traits. Look at pigmentation, for example, which is the strongest signal of selection for a complex trait in our data set. You look at genetic mutations that are known to affect pigmentation. You add up their effect across all of the DNA, there’s dozens or hundreds of them. You look to see when natural selection is strongest, and the time period is really 2,000 to 4,000 years ago.

For some of these other traits as well, you see again that the time period over which selection is strongest is 2,000 to 4,000 years ago. For example, if you look at genetic variants that affect measures of cognitive performance, such as performance on intelligence tests in white British people today. This is of course a very strange trait to measure in the past because there were no intelligence tests and there was no school. But it is a predictor today, and you can look at how it’s changed in the past.

We see very strong natural selection for this combination of genetic variants that predicts people’s performance on IQ tests and is also highly correlated to the predictor of the number of years of school or the household wealth of people. All crazy traits in the past because there was no wealth in the past, there was no school in the past. But if you look at the predictors today, there is a strong movement in a systematic direction, a large effect, about a standard deviation on the scale of modern variation.

We can do this trick of looking to see whether there are periods of time when this natural selection has occurred more intensely or less intensely. We drag a 2,000-year window through our data, and we repeat our whole analysis, not on 18,000 years, but just on a short 2,000-year window. We can measure the strength of selection in each of these 2,000-year windows. What you see when you look at intelligence is that this maxes out in the Bronze Age, between 5,000 and 2,000 years ago.

The impact in the last 2,000 years is almost nothing. There’s no evidence of natural selection at all. Your bias coming into this, my bias perhaps, might be that if there’s any signal of natural selection on this trait at all, that it would be unusually strong in the last 2,000 years. Maybe this is a time of industrialization. Maybe this is a time of greater need for this particular trait. But in fact, there’s no evidence of natural selection at all in the last 2,000 years. There’s very strong evidence between 2,000 and 4,000 years ago, where instead of a one standard deviation strength of selection, it’s a two standard deviation strength, averaged over this time period.

Dwarkesh Patel

The standard deviation here is how much the polygenic score for the trait itself moves?

David Reich

How much the polygenic score for the trait moves over a 10,000-year period within a population that is held constant in terms of its ancestry. What we’re actually doing is looking in our data set at a heterogeneous group of people. There’s Southern Europeans and Northern Europeans and hunter-gatherers and farmers. At different times in the past, those groups are more or less represented.

The whole strength of the methodology Ali Akbari developed is that it corrects for that changing ancestry over time. Really what’s being asked here is that we’ve divided up our whole data set into an archipelago of little populations in different places in space and time. We’re asking in each place in space and time: a little pocket of people in Britain from 4,000 years ago to 3,500 years ago, a little pocket of people in Hungary, a little pocket of people in Italy from 2,000 years ago to 1,500 years ago. In each of these places, where the ancestry is relatively similar without being too disrupted in that short period by migrations, we watch to see if the genetic changes blow in the same direction. We’re measuring the strength of selection at each point in time after correcting for the big population changes that have occurred.

Dwarkesh Patel

The effect here is huge then. One standard deviation above the median would be somebody in the 85th percentile. You’re saying the effect of selection has been so strong that comparing 10,000 years ago to now, the median has gone to the 85th percentile. That’s just a huge effect over the last 10,000 years on something like intelligence or the thing that predicts household income.

Especially given that this is only 2% of the change in allele frequencies, and the 98% is coming from migration… It’s stupendous to think about what the impact of migration is, if this alone is driving a standard deviation change in these kinds of qualities, at least among the kind of variation we see in the world today.

David Reich

One thing you can see in the data is that the migration impact is huge. For example, if you look at the trajectory for measures of cognitive performance—scores on intelligence tests in white British people today—but you look at the predictor of that in people in ancient times, the estimate for the hunter-gatherers of Europe is three standard deviations below the modern mean. So that’s hugely different.

Then you see a huge jump from them to the farmers, who are at the mean, at zero. That’s migration. What you’re seeing is that those two groups had different set points for those traits. And then the steppe pastoralists have a lower set value.

You see huge fluctuations in the predictor of this trait over time. That doesn’t prove selection. That’s just migration. But what our test is telling you is: in addition to those fluctuations due to migration, is there a consistent effect of natural selection blowing the trait in the same direction over all places and times? That’s what we’re detecting.

00:35:02 – Why didn’t evolution max out intelligence?

Dwarkesh Patel

There’s this theory called the collective intelligence hypothesis, which is the idea that selection for intelligence has actually been in the opposite direction. As society has developed, there’s been more specialization, and if there’s more specialization, each person only needs to understand a smaller and smaller part of the world. Therefore, the ancients were actually much smarter than us, and we’ve evolved down in intelligence.

Your results seem to point in the opposite direction. Although there hasn’t been selection in the last 2,000 years as society has gotten more complicated, at least when society began, there was more need for the kind of thing that predicts intelligence today.

The reason that’s surprising is, if you think about hunter-gatherers—reading your colleague Joseph Henrich’s book—the amount of information they needed to hold onto and assess, everything from how to process food, to how to build shelters, fire, et cetera, compared to my world, where I just need to know how to set up mics and ask questions… It seems like the demands on intelligence should have been way higher in the ancestral environment. So it’s very surprising that the beginnings of civilization increased the selection on intelligence.

David Reich

This is the power of data. I think if you asked Joe prior to this work what the hunter-gatherer selection would be and where their set point for this particular trait would have been… I think he probably wouldn’t have made a very strong prediction, but he would have said, “Maybe you would have expected it to have a high predicted value of this trait because these people were really having to do a lot of things and figure a lot of stuff out. Maybe once you have more complex societies, there would be more of a collective brain, and maybe there’d be selection against this trait.” In fact, it’s the opposite in some ways.

It’s the power of data. It’s not what you expect.  It’s actually the value of data to try to make sense of all these things. It’s very interesting. The genetic predictor of intelligence, there are lots of things that are confusing about it, so it’s worth talking about. Or the genetic predictor of years of schooling, which is highly correlated to it and is measured even better.

If you look at the genetic predictor of years of schooling, there’s another amazing study from 2017 from a group in Iceland that looked at this measure over the last hundred years in Iceland. It looked at older people and younger people born more recently. There’s an estimated 0.1 standard deviation decrease in the genetic predictor of intelligence in Iceland just within one century. It’s an absolutely huge effect over a short period. This is selection against years of schooling. If I said intelligence, I didn’t mean to. It’s selection against the genetic predictors of the number of years of school.

One possible interpretation of this—hand-wavy—is that what’s being measured here is not selection for years of schooling or for real intelligence, but for another trait altogether that’s correlated to both of them. For example, the predictor of the number of years of schooling is very strongly correlated to the age at which women have their first kid. If you control for that, all of the signal of years of schooling goes away. So maybe what you’re measuring is women’s decision about when to have children.

If you have children earlier, you don’t go to school as much. If you have children later, you go to school more. Maybe it’s some kind of measurement of delaying gratification or putting things off or planning. The same trait is correlated to body mass index, to obesity, and to walking pace. So is this really intelligence as we think about it, or is it something else that manifests itself differently at different times in the past?

Dwarkesh Patel

Obviously, a trait like years of schooling was not itself a meaningful thing in the past. The underlying things for it seem to have been under strong selection. Whatever in the genome predicts years of schooling seems to have been under strong selection. How should we think about this? What’s the actual thing that’s changing in the genome?

David Reich

There are two things going on that you need to think about. Years of schooling is connected to so many other things genetically. If you look at the genetic predictor of years of schooling—this trait has been measured in millions of people now—it’s correlated to really surprising things. It’s correlated to the age at which women have their first kid. It’s correlated to people’s obesity. It’s correlated to people’s walking pace. It’s correlated to people’s household wealth. It’s correlated to a variety of other traits that seem quite different from it.

If you think you’re actually measuring the genetic prediction of intelligence, or actual studiousness, you should think again because there are many things that it’s correlated to. There seems to be some kind of general trait that you could maybe think of as executive function or a propensity to defer gratification—I’m just waving my hands—that is under selection. It pushes all these traits in the same direction one way or the other, and at different times in the past, it’s advantageous or disadvantageous.

When we found this signal of the genetic propensity to go to school for more years as it manifests itself in white British people today, we were incredulous. How could this be? Maybe this is a problem. So we did a few tests to try to figure out whether this was real. One of the tests we did was that we looked for a study where this measurement of the number of years of school was done not in Europeans, but in Chinese people in China. We looked at the effect size of many variants as they affected the number of years of school in China, and we saw whether they had a correlation to the trajectory of those same genetic variants in Europeans over the last 10,000 years.

These are two parts of the world where the populations have been essentially completely disconnected. There’s no way by chance that the trajectory in Europeans over the last 10,000 years would have anything to do with the effect on years of schooling in China today. But there’s actually a huge statistical correlation, a five or six standard deviation correlation between the effect size of variants on the number of years of school in China today and the trajectory in Europe. It’s just as strong, actually, as the effect size of variants in Europeans on years of school to the trajectory in Europeans. We just could not see a way this could happen by chance. Once we saw that, we felt quite convinced that this was a real signal and that somehow there has been natural selection to increase the genetic changes that today manifest themselves as predicting more years of schooling.

Dwarkesh Patel

Just to make sure I understood, you’re looking at this ancient DNA in Europe. You’re saying it seems to predict years of schooling for modern people in Europe, or at least selection on that ancient DNA seems to predict more years of schooling in modern Europe. You also find that the same variants predict more years of schooling for Chinese people in China. So this is not just some weird artifact from the way these GWAS were done in Europe. These parts of the genome seem to robustly predict the kind of thing that actually leads to more years of schooling, at least in people today.

David Reich

Correct.

Dwarkesh Patel

Stepping back, I want to understand what this tells us about what actually changed in our environments over the last 18,000 years. We talked a little about what happened after the Bronze Age. We were talking about this during the collective intelligence part of the conversation. It’s surprising to me that things like intelligence, or lack of schizophrenia—things that just seem robustly good—were not maxed out before the Bronze Age.

The diversity among different populations was so big that you have the European hunter-gatherers having three standard deviations less predicted value for what they would score on an intelligence test if it existed. But they were existing in the real world in a place where intelligence matters.

How can it be that this was not a trait… You just look at the human body or any animal, and evolution has been acting on it so strongly to make it functional for the things it needs to do. And this one thing, which seems so relevant—especially to what human hunter-gatherers needed to do—doesn’t seem to have been under that strong selection in the Mesolithic or Paleolithic eras?

David Reich

I think that’s a great question. As we talked about before, selection is very effective. It can move the mean value of traits within hundreds or thousands of years in one direction or the other if that’s adaptive in a particular environment. So you might wonder, isn’t intelligence good in all contexts and places in time? There are a number of ways to think about that.

First of all, we are speaking from the point of view of a society which intensely values this particular trait, the ability to score well on IQ tests or things like them, or to go to school for a long time. I think it’s unprecedented in human history that we live in a time like this. If you look at the Hebrew and Christian Bible, and you look at how much intelligence is valued, it’s basically not at all.

Dwarkesh Patel

But when the Bible was being written, especially the Old Testament, that’s exactly when selection for intelligence is at the highest point it’s apparently ever been.

David Reich

Exactly. But there it’s about strength or courage or religiosity. Those are the values. If you read Homer or the texts of other religions, it’s not intelligence. It’s beauty and other things. This value system which has a hyper-focus on smarts is not obviously a trait value that’s been common in the past. You might think that in certain communities there might be valuation of things that are more proximate to years of schooling. But really broadly, it’s not been a high value in the population.

Dwarkesh Patel

Obviously, the thing we care about is not direct performance on an IQ test, especially in the past. The thing I’m trying to understand better is intelligence more broadly. Maybe IQ-test intelligence is just not that correlated with, “Here is a new-world environment, go figure out how to process food there and make shelter and everything else.”

Your colleagues like Joseph Henrich have talked about how modern people underestimate the difficulty of doing this kind of thing with a small band of people. Maybe that’s not IQ-test intelligence, and that’s why we don’t see that strong a selection effect on this thing. But intuitively, regardless of the value system, it just seems very valuable to have this trait maxed out.

David Reich

I’m being very speculative. Let me give you two examples of how I’m thinking about this, not that I’m a particularly good authority on these things. As I mentioned, a lot of these traits, which are quite disparate, are highly correlated to each other. Obesity, years of schooling, walking pace, performance on IQ tests, household wealth, all these crazy traits seem to be governed to a substantial extent by a shared combination of genetic variants.

Let’s think about what this might mean. In Iceland in the last hundred years, there’s been selection against this combination of variants. One possible interpretation is that it’s basically selection for two ways of investing in your children: having many kids and not investing a lot in them, or having few kids and investing more in them. If you invest in deferring having kids, having more wealth, having more resources, and putting more into each kid, you’re going to have lower fertility and fewer kids. That’s going to result in lower fertility, but those kids might survive more and do better in society. Alternatively, you can just have as many kids as you can and invest less in them. They might individually have less good outcomes, but in a time of plenty—which is potentially Iceland in the 20th century—it might make sense to have more kids and invest less in them.

There’s a toggle between having more kids and investing less in them, and having fewer kids and investing more in excelling in various ways. You can imagine that at different times and in different places… In ecology, there are different ways. Mammals often invest a lot with a pregnancy and a small number of children, whereas fish will spawn huge numbers of offspring into the river, the great majority of whom will be eaten. But that is an effective way to produce offspring in certain conditions. So there will be a toggle depending on the environmental conditions back and forth between investing in large numbers of offspring with less investment, or smaller numbers of offspring with more investment. Maybe we’re just seeing that move back and forth over different places and times.

Similarly, for schizophrenia and bipolar disease, how could this ever be advantageous? Maybe what we’re seeing with these diseases is a readout of some spectrum of traits that in some contexts might be advantageous. Maybe being anxious, imaginative, or neurotic might be helpful in a shamanistic tradition or a religious tradition which values people who can have visions or be creative. Maybe these are subclinical versions of schizophrenia or bipolar disease that in certain times may be advantageous and in other times may be disadvantageous. You might just be seeing selection for different types of creativity or other thinking that can be valuable in different contexts.

I’m waving my hands here, but my sense is that these complex traits have not pushed in one direction because there are advantages to both ends of the spectrum, and there are multidimensional impacts of these different traits.

Dwarkesh Patel

Julian Jaynes has this famous theory in The Origin of Consciousness in the Breakdown of the Bicameral Mind. I’m butchering this, but fundamentally, the way I understand it is that up until Homer, basically everybody was schizophrenic. People genuinely thought that gods were real people that you were communicating with. His claim is that ancient texts seem to show people behaving in this way.

David Reich

You’re being asked to believe in visions. Even today, there’s valuation in some religious communities in communicating with God, having visions, and having supernatural communions. So I just don’t know.

But I think it’s super interesting to ask the question of why certain traits are not always advantageous. For schizophrenia and bipolar disease, there is a sense in which most of the mutations are disadvantageous. We can see that from the patterns of variation, where the variants that are risk factors tend to be low frequency and they tend to be small effects.

Dwarkesh Patel

So another trait you find under selection is the trend away from body fat since the agricultural revolution. Why is that?

David Reich

What you see is a reduction in the combination of genetic mutations that make you at risk for obesity, body mass index, and similarly very correlated to it, higher fat mass, higher waist-to-hip ratio, and higher type 2 diabetes risk. There is clear selection, by about a standard deviation on the scale of modern variation for these traits, reducing over the last 10,000 years in this part of the world.

What can be going on there? Why wasn’t there selection for this combination of traits before? There’s a longstanding idea known as the thrifty gene hypothesis. The idea is that once you have hunter-gatherer populations that move into a farming environment where there’s plentiful food, there is no longer a need to the same extent to be able to build up body fat to survive in times of stress, because there are more constant stores of food.

As a result, there will be natural selection against body fat once you move into an agricultural environment and into periods of food plenty. Maybe what you’re seeing is that this group of people in Europe and the Middle East over the last 10,000 years has moved into a period of relatively more stable food, where building up stores of fat is not as advantageous, and there’s been selection against this combination of traits. Europeans are actually relatively better protected genetically against type 2 diabetes than some other populations around the world, like African Americans and Native Americans, that have perhaps not been exposed to agriculture for as much time. So you may be seeing the effect of more exposure to more stable food accessibility.

Dwarkesh Patel

This is also another way in which the data go against a common story. The common story is that hunter-gatherers actually had much more stable diets because they were more varied, and they weren’t reliant on a single cereal or crop for their calories. If one game went away, they had other things they could scout for. They could move locations more easily because they weren’t tied down to the land. So they were more food-stable. But if there’s been selection against storage of body fat, that suggests that as unstable and as common as famines might have been in agricultural societies, it’s at least more stable than what the hunter-gatherers had.

David Reich

There’s a timescale issue. You’re absolutely right. As I understand it, I’m no anthropologist, when there’s a hunt in traditional societies or communities that hunt, people will often gorge themselves, eat a huge amount, build up a temporary store of fat, and then go multiple days without eating meat until the next hunt. There is this boom-and-bust access to high-value nutrition that is not true to the same extent in farming communities.

On the flip side, famines are something that occurs more commonly in agricultural societies, but the timescale and the tempo of them is very different from the hunting tempo. Maybe there’s a famine every three years. Indeed, if you look at the bones of farmers, at least in some communities, there’s more stress in them, maybe due to a famine every three years or every five years. But selection might not be acting on that three-year time period. Your fat store from the latest hunt is not going to carry you through to the famine three years later. Survival of famines is a different thing than building up body fat to be able to survive two weeks later.

Dwarkesh Patel

A random question I have. You were mentioning that compared to these other things which matter much more for fitness in the ancestral environment—the immune system, especially after the Bronze Age—all these other things have mattered more than intelligence. They’ve been under much more selective pressure than intelligence.

That makes you wonder whether there’s much more room at the top for intelligence. If humans had been selected especially for intelligence, they could have been much smarter. The reason that’s relevant is that we’re currently building AI systems, which we’re trying to make as smart as possible. In fact, the only goal of the training process is intelligence. We don’t have to worry about at the same time making their immune systems powerful—

David Reich

We have lots of energy to spend on it.

Dwarkesh Patel

And at the same time making sure they’re not schizophrenic. I guess we kind of do worry about that. But if intelligence has not been the dominant trait under selection for humans over the last 10, 20, or 100,000 years, does that mean there’s more room at the top for this trait?

David Reich

I think there’s more room at the top for a lot of these traits. You can move height extremely in one direction, much more than it is today. You can move any of these traits much more extreme in the other direction. There are probably very strong negatives to doing that. You’re probably sacrificing other things, and there are trade-offs. But it’s highly likely that if natural selection pushed any of these traits more in one direction than it is, the mean would move.

00:57:21 – Evolution is limited by time, not population size

Dwarkesh Patel

So all of this evolution since “Out of Africa” is acting on alleles that already existed in the pool of human variation from that first group we were talking about last time, on the order of 10,000 people, that exploded out of Africa. Is it surprising that across all these different traits, from cognitive profiles to disease resistance to height, that one pool of people contained so much latent variation that they could supply enough stretchiness to accommodate all of these different traits you’re studying now?

David Reich

That’s a rich question, and I think the human population has within it a tremendous amount of variation for complex traits. There’s a huge amount of variation that affects height. There’s a huge amount of variation that affects body mass index. If you take all these mutations and set them to the high-height variant, a person will be extremely tall, like as tall as a tall building. Of course, that will never happen.

But if you take all these variants that affect schizophrenia risk and you point them all in the same direction, there will be extreme risk or extreme protection for schizophrenia. For complex traits, ones underpinned by many mutations, all the variation already exists to move the population to a different adaptive set point that’s optimal in the environment it’s in.

If you push the population into a new environment, within hundreds or thousands of years, the population can rapidly move to a new adaptive set point. There are some unusual traits, like the ability to digest cow’s milk or protection against sickle cell anemia, that require a single very important mutation that may not yet exist in the population. You have to wait for the mutation to occur in some people. When the populations are relatively small, only 10,000 people, you might have to wait dozens or hundreds of generations for that mutation to arise. But when the populations are large, there’s no mutation limit anymore.

Every mutation that can occur does occur. There are eight billion people in the world. There are maybe 30 new mutations every generation, so that’s 240 billion new point mutations every generation. There are only three billion DNA bases in the genome, so every mutation that can occur does occur about 100 times every generation. We’re not mutation-limited anymore. The mutations can arise again. They do arise again. But when the population is only 10,000, you sometimes have to wait dozens or hundreds of generations for the new mutation to occur.

Dwarkesh Patel

How likely is it that the thing that changed with the Bronze Age is just that the human population was big enough? By 3000 BC, you go to a population of 50 million-ish people. The population is big enough, and the gene flow between different areas is high enough, such that things which don’t have an overwhelming selection coefficient, which aren’t overwhelmingly favored by evolution, are finally visible to selection.

David Reich

I think that’s not likely to be true, but it’s an extremely interesting thing to think about. Already when population sizes are on the order of a million or so, every mutation that can occur does occur within a few generations. That’s well before the Bronze Age if you take the population even of a place like Europe, but also of other places. Or maybe it’s at the dawn of the Bronze Age or the farming period. The question you’re asking is whether, when the population is small, maybe natural selection doesn’t work effectively.

A common thing people think about with natural selection, which is true, is that in small populations selection doesn’t work effectively. That’s because mutations bop around in frequency from generation to generation a lot in a small population, just randomly. If you have a population size of 1,000, mutations will bop around by a frequency of one over 1,000 every generation. If the selection coefficient is less than that, it will be drowned in the random bopping around of frequencies due to genetic drift. But that is already for a population of 1,000. A 0.1% selection coefficient is very weak. We’re talking about 1% effects, and that’s very strong. It will work very well even in a population of size 1,000 or 10,000.

If you are talking about mutations of the type that will start rising only in large populations but not small populations, those are selection coefficients on the scale of one over 10,000 or one over 100,000. Those will take 10,000 or 100,000 generations to rise in frequency, which is hundreds of thousands or millions of years. That’s not going to do anything over the timescale we’re talking about. There’s just a timescale issue. We’re talking about strong, measurable selection coefficients on the order of half a percent or more in this study. All of those are going to work in small populations or large populations. It’s not going to be affected by the population size.

Dwarkesh Patel

Interesting. You’re saying that more generally, once you hit a given threshold of population, the dominant factor is time span, not population size.

David Reich

Correct. It’s very interesting, and it’s actually not widely understood.

Dwarkesh Patel

Speaking of data contradicting what you might have otherwise assumed, one of the papers you sent me beforehand, Mallick 2016, found that there are no fixed differences between modern and archaic humans 50,000 years ago. We know this is the period in which the so-called cognitive revolution happened, and modernity started, and people are making art. Does this suggest that nothing biological changed to make modern humans modern? The thing that happened was some cultural change? How do we understand what this data tells us?

David Reich

Right. 100,000 to 50,000 years ago, there’s a quickening of the pace of change in culture. You see the first extensive representational art, bead necklaces, drawings on the wall, and a rapidly increasing pace of innovation in the types of tools that people use. The thought might be that there would have been some important genetic switch, a kind of important genetic change that occurred in the population and swept to high frequency that everybody soon had. That made it possible to do these things. Maybe some genes allowed people to have complex, representational language, for example.

One thing we did in 2016 in this paper by Swapan Mallick and colleagues was look across the DNA for places that might be expected to look like this, where nearly all people living today share a common ancestor maybe 100,000 or 200,000 years ago. We looked really hard, and right across all the DNA we could look at, we couldn’t find anything more recent than four or five hundred thousand years ago.

This is a crazy result because it looks like there are no key selective sweeps that have occurred in this period that are ancestral to everyone living today. We talked before about no selective sweeps between Europeans and East Asians, but there don’t even seem to be any selective sweeps shared between all humans in this really important period when a lot of evidence in the material culture record appears. It could be that there’s biological adaptation in this period, but it’s polygenic. There are lots of mutations that all shift in the same direction to help the population move to a new set point, but there’s no key biological change that rises to high frequency in this time.

Dwarkesh Patel

This group 50,000 years ago, are they the ancestors of everybody out of Africa or also some Africans?

David Reich

This is 100,000 to 50,000 years ago. This is the population that’s ancestral to West Africans, to most East Africans, to all non-Africans.

There are a couple of populations in Africa that have substantial ancestry coming from more divergent groups. For example, Khoisan from Southern Africa or Central African rainforest hunter-gatherers have substantial fractions of their ancestry from groups that diverged maybe 200,000 years ago from the other lineages. But all of these groups today are able to go to college and do everything everybody else does. There is no evidence that there is any key mutation lacking in some groups that is not present in the others.

Dwarkesh Patel

The differences we see between different groups of people, especially if this group 50,000 to 100,000 years ago had a very small population size… I think last time we were discussing on the order of 10,000 people. So almost everybody in the world, or the variance we see between different humans today, was latent in this group.

I get your point that if you just stack up different things across the genome, stacking them up really has a big effect. But it’s interesting that we have so many different groups in the world today, and all that diversity comes from a very small population size.

David Reich

A lot of us in human genetics think that our population contains within it the clay that’s needed to make almost any trait. And that depending on environmental conditions or selection conditions, the mean value of these traits will move in different directions. There’s an empirical question about how much selection there’s been in different human populations over time.

One of the things this new work we’re involved in is showing is that at least in the last 18,000 years in this part of the world, there has been significant movement, at least for a handful of important traits. We looked at more than 500 traits. About 100 complex traits showed significant movement in a systematic direction over this time period. It really does seem that there is a response to the environments people are living in that has occurred over this period, and that is potentially stronger than in previous periods.

01:09:02 – Why no farming before the Ice Age?

Dwarkesh Patel

We were talking earlier about how there are no fixed differences between humans 30,000 years ago and humans today. So if there’s no genetic basis for the kind of thing that allowed humans to have more symbolic representation, have farming, et cetera—I think I asked you this question last time we talked, but especially with this context—why no farming before the Ice Age? Genetically we were there.

David Reich

That is such an interesting question. Genetically we’re there. The common ancestral population has all of the ingredients for farming 50,000 years ago. These people are distributed into different parts of the world: the Americas 15,000 years ago or whatever it is, New Guinea 40,000 years ago, East Asia, Europe, West Africa. No farming developed before 11,000 or 12,000 years ago. It only developed in the last 12,000 years, the period known as the Holocene, which is the end of the Ice Age.

If you talk to climate scientists and archaeologists—I keep asking people this question every time I meet someone who’s an expert in this—how can it be that farming develops in all these places? Are we really living in such an unusual time? People tell me, indeed, we’re living in a very unusual time on a scale of two million years. That is, 12,000 years ago we switched into this period of not just warmth, but climate stability.

It’s hard to believe that we’re living in such a special time. But if you look at data from the bottoms of ponds where you can measure the fluctuations of temperatures using isotopic signatures, apparently we’re in a period where it’s fluctuating a lot less year to year, 10 years to 10 years, and 100 years to 100 years. It’s a period of relative stability that we are miraculously living in.

When this period of relative stability happens, it follows that multiple groups independently turn to agriculture, even though they all have the same genetic complement that arose 50,000, 100,000, 200,000, 300,000 years ago. It’s a crazy observation that people just accept, but it’s unbelievable.

Dwarkesh Patel

Oh, so you increased the range there. You said 100,000, 200,000, 300,000 years ago. Based on the genetic differences between modern people and people from 300,000 years ago. Do you basically think they’re modern 300,000 years ago?

David Reich

I don’t know. This is actively what I’m thinking about all the time right now. There’s a big transformation in terms of the culture of humans 300,000 or 400,000 years ago: this invention of Levallois technology, the ability to make stone tools out of cores.

The Middle Stone Age Revolution, or the Middle Paleolithic Revolution depending on what you call it in Africa or Eurasia, is a new way of making stone tools that’s shared by Neanderthals and by modern humans, but is not shared in East or South Asia. It’s a big change, and it presumably involves a cognitive change in order to make this sort of technology.

Then there’s a further change to the Upper Paleolithic Later Stone Age, maybe 100,000 to 50,000 years ago, when there’s a second transition with a new type of tool making, but it’s not as revolutionary as the earlier one. So when the cognitive leap happens is unclear.

The diversification of the lineages leading to people living today, like Khoisan Southern Africans and rainforest hunter-gatherers, all occurs more on the timescale of 300,000 or 200,000 years. All of these people are capable of going to college and doing everything. So it’s not obvious that the cognitive toolkit, the behavioral toolkit, and the genetic abilities were not all in place 200,000 or 300,000 years ago, and that even Neanderthals had them. It’s not obvious that this was not the case.

I just don’t know. You distribute these people descended from this diversification that happened 200,000 or 300,000 years ago to different parts of the world, and then after 12,000 years ago, you start having agriculture popping up in different places. It’s an outstanding mystery of human history. I find it unbelievable that we live in a time period that climatologically is so unique on a scale of two million years, but my colleagues tell me it’s true.

Dwarkesh Patel

The climate thing seems surprising given there are so many different environments in which agriculture was independently developed. I understand that across environments the variance could have gone down. If it had only happened in one place at one time, I could have bought that explanation. But the fact that they’re making maize in the New World and they’ve got cereals in the Old World in very different environments makes it surprising.

David Reich

It’s very, very surprising. We accept it, but it’s a crazy observation that most normal people don’t realize.

The thing that basically everybody accepts is that the common ancestral population of almost everybody in the world, except for rainforest hunter-gatherers and Khoisan, is around 70,000 years ago. Everybody accepts that these people all have in place the cognitive, behavioral, and intellectual ingredients that are necessary for the farming revolution and building state societies. Because when these descendants get distributed to West Africa, East Africa, the Americas, Europe, South Asia, East Asia, New Guinea, and so on, their descendants all do this. They do it independently, semi-independently, or demonstrably completely independently in all these different parts of the world.

The cognitive resources for doing this must have all been in place, but it’s a very long fuse. It delays for 40,000 or 60,000 years in all these different places after the common ancestral population splits up, and then ignites into agriculture and all these other things after that point.

It’s a crazy claim. Then you could argue about whether the actual fuse is 300,000 years, from when Neanderthals separated and from when different lineages of extant modern humans separate, and that’s also plausible. It’s a crazy set of things that we’re being asked to believe.

Dwarkesh Patel

Is it possible that agriculture existed, but you didn’t have modern metallurgy or whatever it was that allowed populations to explode starting in 5000 BC with the Bronze Age? Population-wise, it doesn’t seem like much is happening from 10,000 BC to 5000 BC in the early Neolithic. Is it possible that they had farming but they didn’t have copper or tin, which you needed to go to the Middle East for, to develop a civilization that could make use of bronze at a large scale, and so they just disappeared from the historical record?

David Reich

I think we would see their archaeology. There are extraordinary developments in the Americas which are entirely Stone Age.

Dwarkesh Patel

You would see them today if they had completely vanished?

David Reich

Oh, yeah. We should go for a trip to Teotihuacán in Mexico. It’s so impressive. When I went there when I was 20, it was totally as impressive as ancient Egypt. It’s huge. It’s massive. It’s without metal.

Dwarkesh Patel

It’s even more impressive because it’s not only without metal, but without animals and without wheels, which is crazy. The marble is just hauled without wheels.

David Reich

Right. Take any person who has an old world superiority and take them to these places, and they will not have it anymore. It’s just extraordinary what’s in these places. These are people who separated 20,000 years ago at least from the ancestors of East Asians and 40,000 years ago from the ancestors of West Eurasians.

They just had the same biological and cultural shared toolkit from then, but there’s a long fuse delay until all this stuff happens. It’s an amazing thing, and we don’t question it.

01:17:13 – The Neanderthal puzzle David can’t stop thinking about

Dwarkesh Patel

What are other questions you are either investigating right now or want to investigate, these kinds of big picture questions of human history?

David Reich

I’m perplexed. I don’t know if we talked about it before, but I remain very confused about the relationships between archaic and modern humans.

We have genome sequences now from archaic humans who lived in Europe, West Eurasia, and Central Eurasia, and the Neanderthals. We have archaic sequences from these enigmatic Denisovans, who we now have a skeleton for since we last talked. There’s now a skull that’s been shown to be a Denisovan. We have data from lots of modern humans, and there are really big mysteries about the relationships amongst these groups.

Genetically, the Denisovans and the Neanderthals are sisters. They descend from a common ancestral population 500,000 or 600,000 years ago. That group descends 700,000 or 800,000 years ago from the common ancestors of modern humans. Genetically, the whole genome data says that Neanderthals and Denisovans are archaic humans from a common ancestral archaic population.

But there are so many things shared between Neanderthals and modern humans that don’t seem to be shared with East Asians. They both share Middle Stone Age stone tools, Levallois technology, this cognitively unique way of making stone tools that wasn’t used in East Asia. They both have the same mitochondrial DNA and Y chromosome sequence.

The Y chromosome sequence of Neanderthals and the mitochondrial DNA of Neanderthals, is actually modern human that came through interbreeding 200,000 or 300,000 years ago and then shot up to 100% frequency. Neanderthals and modern humans are both the product of mixture events that happened between archaic and modern humans 300,000 or 200,000 years ago, demonstrably through patterns of variation in ancient and modern DNA.

It feels that there’s something shared between Neanderthals and modern humans that’s not shared with Denisovans, even though the vote of the whole genome says that Denisovans and Neanderthals are related. One wonders whether there’s something connecting Neanderthals and modern humans that’s different from Denisovans, even though genome-wide, Denisovans and Neanderthals cluster. I’m thinking about that all the time now.

Dwarkesh Patel

Connecting them would be interbreeding events or being in the same place at the same time that we missed?

David Reich

There’s a known interbreeding event from the lineage leading to modern humans into Neanderthals, but it’s supposed to be only 5%.

I’m interested in the possibility that that 5% is actually a sign of something much more impactful, that somehow Neanderthals are in some sense deeply modern in some ways, and even though they get swamped by archaic genes, they actually have more of a modern impact than one would think. The Middle Stone Age and Middle Paleolithic Revolution that they share with modern humans is more fundamentally a part of who they are, in some sense, than we think.

Dwarkesh Patel

Interesting. Sorry, when was this interbreeding event?

David Reich

300,000 to 200,000 years ago.

Dwarkesh Patel

So the common ancestor between Neanderthals and most humans alive today is potentially more recent than the common ancestor between all humans alive today.

David Reich

Oh, for sure.

Dwarkesh Patel

Which is crazy.

David Reich

Well, the divergence to all the archaic humans, including Denisovans, is within human variation.

Dwarkesh Patel

Wait, what?

David Reich

Yes. The average time to the common ancestor of any two human genes is one or two million years ago. If you look at the copy of chromosome 3 you get from your mother and the copy of chromosome 3 you get from your father, the typical time they share a common ancestor is one or two million years ago. That’s before the split from Neanderthals and Denisovans. So there are many places in your DNA where you’re more closely related to a Neanderthal on your mother’s side than you are to your father.

Dwarkesh Patel

I’m sure there’s a simple explanation, but how?

David Reich

It’s the same reason that if you have a sister, in some places in your DNA you’re more closely related to her than you are to me because you share a parent. But in other places you’re more closely related to me than you are to your sister because you happen not to share the same DNA from your parents.

It’s just that the DNA we get from our common ancestral population was already quite variable 500,000 years ago, 700,000 years ago, a million years ago, and some of us descend from some of those ancestors and others descend from other of those ancestors. Neanderthals split from our lineage really close in time on human evolutionary timescales, such that in some places in our DNA we’re more closely related to Neanderthals than to each other.

Dwarkesh Patel

Interesting. What are the other big questions?

David Reich

That’s the main thing I’m thinking about a lot these days. I continue to be very obsessed with questions about the spread of human populations around the world and trying to reconstruct that with ancient DNA.

The thing I’ve been thinking about a lot recently is the possibility that maybe we’re not thinking in the right way about the relationship between archaic and modern humans. The standard model is one where Denisovans—these archaic humans that were found from ancient DNA—and Neanderthals descend from a common ancestral population 500,000 or 600,000 years ago, and these two separate earlier, maybe 700,000 to 800,000 years ago, from the ancestors of modern humans, people like us. That’s the big result of a lot of studies since 2010.

But there’s also evidence of an interbreeding event that happened maybe 200,000 to 300,000 years ago that resulted in modern humans contributing DNA to the ancestors of Neanderthals. So maybe 5% of the DNA of Neanderthals comes from this interbreeding event, and a lot of studies have shown this. I’m very interested in this because, from the archaeological record, Neanderthals and modern humans look quite similar to each other, much more similar to each other than a lot of them do to Denisovans, these archaic humans in East Asia.

For a lot of history, people have thought that Neanderthals are our sisters. But in 2010, the sequencing of the Denisovan genome made it very clear that on average, Denisovans are closer to Neanderthals than to modern humans. This was a very confusing result. Most people now think that Neanderthals and Denisovans descend from a common ancestral population that separated earlier from the ancestors of modern humans.

I’m interested in the possibility that the right way to think about Neanderthals is actually as somehow culturally modern humans, even though genetically they’re mostly Denisovans. The model I’m thinking about is motivated by this archaeological phenomenon known as the Middle Stone Age Revolution. If this is Africa and this is Europe, we know that the new way of making stone tools—with cores that were very carefully mined far away from the locations they were used, made out of high-quality stone like flint—starts being used 300,000 or 400,000 years ago, first in the Caucasus, places like Georgia today, or East Africa.

This way of making stone tools is quite revolutionary. It is known in Europe as the Middle Paleolithic and in Africa as the Middle Stone Age, and is associated with much more widespread use of fire and moving stone around at much further distances than before.

I’m interested in the idea that this is something shared between modern humans and Neanderthals. There’s somehow some shared cultural feature that’s absent in East Asia, and that might have a relationship in the genetic data and is somehow related to this 5% DNA.

The idea I’m interested in is the possibility that there is a population here that invents the Middle Stone Age and the Middle Paleolithic, sometimes called Levallois technology, and that people from this population expand into Europe and mix with the local archaic humans who are there. That is what this 5% interbreeding event is. It happens 200,000 to 300,000 years ago. It produces a group that, as it expands across this landscape in Europe, mostly picks up the local DNA and becomes mostly archaic genetically, but retains its modern human culture, the way of making stone tools and some of its traditions.

One of the things that’s super interesting about this is that if you actually look at the genetics, across the whole genome, Neanderthals and Denisovans cluster. But if you look at the mitochondrial DNA—which humans and Neanderthals get from their moms—Neanderthals and modern humans cluster. If you look at the mitochondrial DNA, Denisovans and modern humans share an ancestor well more than 700,000 or 800,000 years ago, as you’d expect from the history. If you look at the Y chromosome that you get from your dad, Denisovans and modern humans share an ancestor more than 700,000 or 800,000 years ago, which is consistent with this history.

But if you look at the Neanderthal mitochondrial DNA, it’s only 300,000 to 450,000 years. If you look at the Y chromosome, it’s only 300,000 to 450,000 years. What the current genetic work is asking us to believe is that even though this is only 5% of the whole genome, it introduces mitochondrial DNA and Y chromosomes, and they jump up to 100% frequency. It’s kind of a crazy claim because the probability of this occurring by chance is low, maybe 5% times 5%, a very small number. It’s what we actually all believe, but it’s a very surprising event. Somehow it’s accreted to all the findings in the literature so that we make ourselves believe this, but it seems unlikely on first principles that somehow only 5% would introduce both the Y chromosome and mitochondrial DNA.

And it really does look like this. There’s amazing data from a site in Spain that’s 300,000 to 400,000 years old, called Sima de los Huesos. They have a nuclear genome that looks Neanderthal-like for most of the genome, but their mitochondrial DNA and Y chromosome are Denisovan-like. So it really looks like there was a population related to modern humans that pushed into this Sima de los Huesos-like population, displaced its mitochondrial DNA and Y chromosome, but kept the rest of its genome. It really looks like something like this happened.

The idea I’m playing with—and probably it’s wrong, who knows—is that there’s a landscape… This is Europe and you can break up into a hundred or so demes, little areas. Modern humans get introduced at the bottom right corner, in the Middle East or somewhere, and they spread into Europe. As this population spreads, there’s a wave front of expansion, and they’re interacting with the local archaic humans.

The theory from simulations and studies of different species like mammals and birds shows that even if there’s a small amount of interbreeding—when there’s an invasion or a movement of expansion of one group into the territory occupied by another—there’s massive introgression of local genes. The pioneers at the wave front will sometimes interbreed with the local population. There are so many of them around that their DNA will get swamped by the local group, so by the time they make it to the other side, they’re largely local.

Maybe this is what we’re seeing. You have a modern human population that’s matrilineal, for example, where the transmission of making stone tools this way is happening from mother to child. That’s why they’re retaining their mitochondrial DNA, but by the time they get to the other end of Europe, they’re mostly local archaic. You end up with a 95% population replacement. This would explain why the mitochondrial DNA is shared between Neanderthals and modern humans, and it would also explain why the mixture proportion is only 5%.

The really interesting thing is that there’s other evidence from studies of modern humans showing that modern humans are also admixed. The right way to think about this is that modern humans are a mixture of two groups that diverged maybe 1.5 million years ago, and they come together 200,000 to 300,000 years ago with maybe 20% ancestry from this archaic African group and 80% ancestry from this early modern lineage. And that same group then mixes with Neanderthals, and it’s 5% modern and 95% local.

So you actually have this key population that makes the Middle Stone Age or Levallois technology. It appears and expands in all directions—into Europe and into Africa 200,000 to 300,000 years ago—bringing this technology, new ideas, and perhaps some genetic adaptations. It expands into archaic humans in Europe, mixes with the local population, and gets 95% replaced but still retains its cultural features and maybe some genetic features.

It expands in Africa too, but here it’s not 95% replaced, it’s only 20% replaced. Probably the reason is that this group is much more diverged. It’s 1.5 million years diverged rather than 700,000 to 800,000 years. As a result, there are many more genetic incompatibilities and barriers to gene flow. But there’s still a lot of mixing, maybe 20%, and we have evidence that this is a big mixture event.

So what you’re actually seeing is a modern human expansion both into Europe and into Africa. In one place, it forms Neanderthals. In another, it forms the ancestors of everybody living today. But all of these groups are descended from this key revolutionary event that happens here.

We often talk about the revolutionary events 50 to 100,000 years ago, the more symbolic behavior and so on, that first appears in Africa and the Middle East and spreads beyond. But there’s also this earlier event, and it is contemporaneous with the breakup of all the different groups in Africa today, the Khoisan Southern Africans and the Central African rainforest hunter-gatherers. One wonders whether this is an equally important formative event.

If that’s true, it makes you think of Neanderthals as somehow our cousins. Their shared Y chromosome, their shared mitochondrial DNA, they share the formation of this 200,000 or 300,000-year-old event, and their shared toolkit. Even though the genome is telling us they’re cousins of Denisovans, the correct way to think about them may, in an important sense, be as close cousins of modern humans.

Dwarkesh Patel

I have so many questions. Do you have 15 more minutes? First of all, what is going on with this group of archaic Africans 1.5 million years ago? Where in Africa are they, and what happens to the portion of them that don’t form modern humans? Do they survive?

David Reich

This is not from ancient DNA, but from analysis of modern DNA from different people, mostly in Africa, but also non-Africans. In multiple studies—there’s at least three, maybe four or five studies that I know about—they have looked at the patterns of variation in people today and say the data in modern people today, including in Africans, is not consistent with a homogeneous population.

It looks like a population that split well more than a million years ago into multiple groups—at least two, but maybe many—and then came together a few hundred thousand years ago. The papers have different models that they fit, but they all have this feature of a split-up more than a million years ago, and then on the order of a few hundred thousand years ago, a coming together and a remixture event forming the ancestors of anatomically modern humans.

Dwarkesh Patel

This includes the Khoisan and whatever other groups?

David Reich

Yes. All of these groups have this, maybe in slightly different proportions. So you ask, where are these people living? Who knows. In this scenario, the 80% is coming from the Caucasus or Northeast Africa, where the Middle Stone Age forms. It’s from this population that the Middle Stone Age comes. They mix with local groups, and who knows where they are: Southern Africa, Western Africa, Central Africa, Eastern Africa.

We don’t have any ancient DNA, but this is a very rich environment. People have been living there for seven million years at least, and there would have been different groups of people everywhere. Probably it’s not just two groups, it’s probably more.

The important theme here is there’s evidence of substructure that’s well more than a million years old. This place would have been a landscape full of archaic humans that would have been differently related to these expanding people and would have admixed with them when they came through.

Dwarkesh Patel

So with the Neanderthals, the first time around 300,000 years ago, our ancestors share culture with them. They share the Middle Stone Age technology, but they don’t replace the population. The technology spreads through culture, basically.

David Reich

It spreads through genes too. If you look at Yamnaya in India, there’s almost no Yamnaya ancestry in India. It’s just diluted down. As Yamnaya expanded into Central Asia and into Europe, it makes the Corded Ware. There’s a 25% dilution. It expands back across Central Asia. It goes through the Hindu Kush and gets into northern South Asia. It admixes more with local people. Today, the most Yamnaya ancestry you see in India is 20% or 10%. Most people have less than 10% or 5%.

There’s been a lot of mixture on the way, but it is the tracer dye. It tracks Indo-European languages, and important aspects of Indo-European culture are coming through Yamnaya. So if you know where to look, that tracer dye is only 10%, 5%, or 2% in some groups. But it’s the languages people speak, and important shared cultural elements, that connect them to people on the other side of the Indo-European-speaking world. So this 5%, you shouldn’t sneeze at it. That’s tracing something important in this model.

Dwarkesh Patel

I understand that if things are transmitted more through women… Sorry, let me back up. I don’t understand why the maternal mitochondrial DNA and the Y chromosome would be especially privileged as the spreading is happening. Can you explain that?

David Reich

The reason I’m talking about these matrilineal or patrilineal expansions is that I’m really troubled, and have been troubled for many years—especially in the last three or four years—by the fact that the mitochondrial DNA and Y chromosome cluster Neanderthals and modern humans, but the rest of the genome clusters Neanderthals and Denisovans.

This is a crazy result that is not seen in any other species. I’m very interested in patterns that would explain it. If you assume that there was a matrilineal or a patrilineal expansion—it could be either—then modern humans, when they were expanding across the landscape of Europe, retained their identity along one of the lines.

If it’s matrilineal, when they incorporate a male from the local community, he’s brought into the community, and the kids are raised based on the culture of the mothers. If it’s a patrilineal expansion and they incorporate a female from the community, she’s raised with the culture of the fathers. If that happens, it guarantees that one of these two parts of the genome looks like it does, because it’s a modern human expansion. If it’s patrilineal, it will retain the Y chromosome. If it’s matrilineal, it will retain the mitochondrial DNA. So it will solve one of your two problems.

Dwarkesh Patel

But not both.

David Reich

It won’t solve the other one, so you need to solve the other one. You can solve it either by natural selection or by social selection. By the way, patrilineality and matrilineality are the rule, not the exception, in human communities. Usually communities have continuity along the male or the female line. Usually it’s patrilineality, sometimes it’s matrilineality.

You can also have phenomena like social selection. It could be that once you have kids of someone whose father, for example, is from the outside community… Usually in most communities, females all reproduce. That’s typical today. Usually women have kids if they can. But men in traditional societies are actually very variable in their reproductive success. A large fraction of men never have kids. Then there’s a subset of men who have many kids with many women.

There’s competition among men for kids. In this context, where males are competing for access to females, female mate choice begins to be an important process. You have a phenomenon where if your dad is an archaic male, it could be the case that you’re not going to be as successful in the competition for local females as if your dad is a non-archaic male.

Some simple social phenomenon like that could explain the data, and we actually see this in human society. For example, if I remember right, in Central African rainforest hunter-gatherers, there’s different treatment of boys and girls depending on whether their dad or mom is one group or the other.

Dwarkesh Patel

I guess I don’t understand how the maternal… The group spreads, and it gets to the next front. They have kids. From the humans that have just entered, the kids will have the mitochondrial DNA from the humans. But from the existing people, they will have the mitochondrial DNA of the archaic humans. Why are the people with the archaic mitochondrial DNA not surviving?

David Reich

It’s a question. There are multiple possible explanations, but it’s much easier to explain that than both the mitochondrial DNA and the Y chromosome. One possibility is that the mitochondrial DNA was less biologically fit. Another possibility is that there’s social discrimination against people based on whether their parents are archaic or not, which is not at all surprising in a human context. It’s the weakest link in this argument. This argument’s probably wrong, but I’m just telling you what I’m thinking about.

Dwarkesh Patel

Okay, the Neanderthals. So 300,000 years ago our lineage interacts with them, but mostly their lineage survives, and there’s cultural and genetic diffusion. And then is it 70,000 years ago that we interact again?

David Reich

Yes.

Dwarkesh Patel

And they don’t survive.

David Reich

The genetic ancestry doesn’t survive.

Dwarkesh Patel

The genetic ancestry doesn’t survive. Presumably there was also other contact between 300,000 years ago and 70,000 years ago.

David Reich

Probably. But these are the ones we are detecting currently.

Dwarkesh Patel

Is it just contingent that one time there’s this kind of diffusion where most of the archaic genome survives, and the other time it’s total replacement?

David Reich

This is not at all surprising given the context. If you think about this model, this is 700,000 or 800,000 years ago. This is 300,000 years ago. So this is 400,000 years separated. You talked about the Bhatia paper with me earlier. That’s two populations 70,000 years separated. There are no biological incompatibilities between West Africans and Europeans. There’s no natural selection against biological incompatibilities.

We know when Neanderthals and modern humans met and mixed, there were biological incompatibilities. That was 700,000 years ago. As populations become further apart, biological incompatibilities rapidly develop, probably as the square of the separation distance, because you need pairs of interacting genes. Here, it would have been maybe only 400,000 years separated between this lineage and that lineage. But here, it’s 1.2 million years. That’s a lot. These are at the edge of not being able to produce children. These are quite different humans. These are actually three times closer than these.

If you look at mixtures of humans today, there are mixtures in Southern Africa of people who are half this distance. If you look at Khoisan and Bantu people mixing in Southern Africa, like the Xhosa, which is the population of Nelson Mandela, these are groups that are separated by almost 200,000 years, which is half of this. Totally compatible.

What you’re seeing is a group that’s actually completely permeable genetically, or nearly completely permeable. This other one almost certainly has substantial biological incompatibilities. Because 200,000 or 300,000 years later, we see interbreeding between Neanderthals and modern humans, or between Denisovans and modern humans, and there’s clear evidence of incompatibility at that point. But this would be even bigger.

What you would expect to see is that as this group spread, they would be moving into a territory full of archaic humans. There would be some interbreeding, but the kids would not be very fit. They would die off. There would be a lot of infertility. The barriers to gene flow and to interbreeding would be greater.

To me, it’s not at all surprising that as this group moves into Eurasia, you have Eurasian archaics—the ancestors of Denisovans—who are only 400,000 years diverged from these people over here. And then you have African archaics, and these are 1.2 million years diverged. They just don’t interbreed as much, and you don’t get as much gene flow.

But the key thing is the timing. It’s the same time. It really feels like the signature of an explosion of people from one place, interacting with people here and interacting with people there. It’s the same cultural or technological revolution impacting this place and that place, and creating populations that are impacted by this cultural revolution, which we know is the case because they share the same toolkit.

Some people argue that Levallois technology is independently invented. But it’s very similar, and this model would be a way that it could have the same origin. So there’s a culturally shared thread, this shared toolkit. There’s a mitochondrial DNA and Y chromosome thread. And then there is a shared timing thread, which is they both form by mixture.

Dwarkesh Patel

Because otherwise you’d have to believe that Neanderthals independently developed Stone Age tools.

David Reich

Yes, which is not inconceivable. But it’s a little bit like believing that farming independently developed in multiple parts of the world.

Dwarkesh Patel

Right. But it did.

David Reich

It did. So as I said, this is probably wrong. I’m trying to tell you that we don’t really know the world we live in. This is not obviously wrong. In fact, to me, this is much more plausible than the model we currently write down. It’s probably wrong, but it’s much more plausible. It explains many more things, and it’s no more complicated.

Dwarkesh Patel

Do you want to recapitulate the thing you were saying about the analogy to Ptolemy and the epicycles? I thought that was quite interesting.

David Reich

I think the model that we’ve put together collectively about the relationships between archaic and modern humans has accreted over time. There was this idea that modern humans are distinct and that Neanderthals and Denisovans are sisters of each other. Over time, we detected additional mixture events, like this modern human into Neanderthal, and then these other ones I didn’t even talk about, like a super-divergent lineage going into Denisovans and all this other stuff. We still say, “Oh, the whole genome says Neanderthals and Denisovans are sisters, so that’s the truth.”

We’ve patched it all together and gotten it all to work. You look at the mitochondrial DNA and the Y chromosome, and they have this odd pattern, and it’s improbable, but we can get that to work if we invoke natural selection, things like this. You patch it all together.

It reminds one of what happened in the ancient world, where there was this idea that the sun revolves around the Earth, but it doesn’t quite explain the movements of the planets properly. In order to get the movements of the planets to work right, Ptolemy and the astronomers made up these epicycles, these special extra rotations and movements to make everything work about right. It was such a convoluted model. When Copernicus and colleagues suggested instead that everything is revolving around the sun, it simplified things ever so much.

What was happening is that as astronomical information accumulated, it kept being contradictory to the standard model, but it could be made to work by proposing another complication and another complication and another complication. This is not as fantastic as proposing that everything revolves around the sun rather than the Earth, but it is much simpler. And it actually explains many things.

Dwarkesh Patel

What is counterintuitive or unexpected or hard to accept about this alternative model? What is the hesitation that people have for adopting this?

David Reich

I don’t know. Nobody’s thinking about this model right now. It just seems obviously a very natural model to me.

Dwarkesh Patel

The reason I ask is that Aristarchus, the ancient Greek, had the heliocentric theory because he had deduced how far the Earth is from the sun and noticed other things. But it was not adopted, because his fellow Athenians were like, “Look, if we believe that the Earth revolves around the sun, for it to be the case that we don’t see relative movement of the stars to the Earth, the only possible explanation is that the stars are so far away that it is just incomprehensible and implausible.” So the heliocentric theory was dismissed.

What I’m trying to ask is, what is the equivalent here of “for this to work, the stars have to be so far away that it’s inconceivable,” where actually the stars are so far away? Maybe we should adopt the implausible implication that this theory gives us.

David Reich

That’s a great question. I think we have to assume that there’s a linkage between the cultural transformations in Africa and Eurasia at this time, and that’s not something the community has really put together with the genetic data.

There’s this thread in the genetics about substructure in Africans, and then there’s this whole world based on ancient DNA, and they’ve never been put together. Nobody’s put together the now extensive work on modern human substructure with the extensive work based on ancient DNA of archaic human relationships to modern humans.

If you put them together, you realize they line up in terms of their time of substructuring. I don’t know if that’s improbable. It seems parsimonious to me.

Dwarkesh Patel

It also seems significant that different groups of humans at this time were capable of adopting Stone Age technology. Once one group had figured it out, the genetic difference between different human lineages was not so big that you could not show people how to use stone tools.

David Reich

Who knows? It could be that this was genetically driven. We talked before about the time to the common ancestor of human genes. There’s nothing at 100,000 years or 150,000 years, but there’s a lot at 400,000 or 500,000 years.

If that’s what happens, and you have a mutation that occurs in the Caucasus, somewhere in the Middle East, or Northeast Africa, there could be key genetic mutations that make people able to do this. Then this population expands. When it moves into Europe, it’s swamped by local genes, but there could be retention of those genes through selection as it expands.

Maybe what you’re seeing is that there are genetic developments. Most of the discussion has been focused on the 50,000 to 100,000-year event, which is anatomically modern human behavior. But a lot of archaeologists think this is an equally—if not more—profoundly significant event in many ways. Why is that not the event we should be talking about?

Dwarkesh Patel

You’re talking about how there are no fixed differences between modern humans and the humans 50,000 years ago. Do we know if there are any fixed differences between the people 50,000 years ago and the people 300,000 years ago?

David Reich

I think there are.

Dwarkesh Patel

Other than obviously these interbreedings.

David Reich

If you look at the genetic variation going back 300,000 or 400,000 years, there do begin to be places where all modern humans share common ancestry. That’s another way of saying there begin to be fixed differences at that time depth. That is where you start seeing evidence for possible fixed differences.

What’s happening, if everybody shares a common ancestor 400,000 or 500,000 years ago, is that there’s a single ancestor at that time. If you compared it to another population, they would descend from a different lineage, so any mutation that occurred ancestral to that single ancestor would be a fixed difference. This is the time at which you can begin to see fixed differences.

Dwarkesh Patel

But anatomically modern, cognitively modern humans exist by the beginning of the Middle Stone Age, before we’re breeding with this ancient group of Africans or breeding with Neanderthals.

David Reich

Anatomically modern humans occur exactly here. It’s the same moment. This is when they occur. The people who have skeletal features like ours, and Neanderthals, appear exactly then.

This is when it all happens. There is this disconnect between anatomically modern humans in the skeletal record and behaviorally modern humans, which is 50,000 to 100,000 years ago. Anatomically modern humans appear at this time, and recognizable Neanderthals appear roughly around this time, too.

Dwarkesh Patel

Interesting. But we don’t know what exactly happens, if anything, between 200,000 years ago and 50,000 years ago that goes from just anatomical modernity to behavioral modernity.

David Reich

My understanding is no. They’re busy making Levallois stone tools like Neanderthals for 200,000 years, and they are not more impressive than Neanderthals in any obvious way, as I understand it.

Then there begins to be in the archaeological record a quickening of behavioral traits, which could be not genetic at all, or could be genetic. There are lots of arguments about this. We were obsessed with intelligence earlier in our conversation. People are obsessed with art and these things that seem important to us but who knows what’s important?

Dwarkesh Patel

Interesting. Cool, thanks for the digression.

01:54:10 – The methodology behind this breakthrough

David Reich

The work that I’ve been involved in has consistently shown that I was wrong in my biases coming into the work, and I’ve really been almost traumatized by this. Again and again, I’ve come into a project with some kind of guess about what the data was showing, and then the data doesn’t show that.

For example, when I got involved in the Neanderthal genome project helping to analyze data looking at how archaic Neanderthals were related to modern humans, I was part of a group of scientists who had established that non-Africans were a simple subset of African variation and that there was no evidence at all of Neanderthal interbreeding into the ancestors of modern humans or other archaic interbreeding. Different analyses that I and many other people had done made it look like non-African variation was just a subset, a small sample of that in Africa, and that could have fully explained the data.

So when I was involved in analyzing the Neanderthal DNA sequences, what happened was I found this very strong evidence of Neanderthals being more closely related to non-Africans than to Africans. It was very surprising, and I thought it must be a mistake. I was quite incredulous. I thought it was unlikely to be true because other evidence that had been found before seemed to point in the other direction. So I spent several years trying to make these results go away, as did my colleagues, and we just couldn’t make the results go away. They just kept getting stronger.

And this experience working on natural selection was the same. What we were convinced of was that natural selection had been pretty quiescent in our species over the last several hundred thousand years. Therefore, if we look at patterns of variation in non-African people today, or in any people today, we should see not a lot of selection going on. Indeed, the first ancient DNA studies, beginning in 2015 with this paper that we were involved in with Ian Mathieson and colleagues, seemed to show relatively small numbers of genetic positions associated with natural selection.

In 2015, we analyzed data from about 200 Europeans and Middle Easterners to try to understand frequency changes over time. We compared those ancient people who were the sources of modern Europeans to people in Europe today, and we looked at frequency differences that were too extreme to be due to chance. We were very excited to find 12 positions that we were convinced were highly different in frequency between Europeans today and what we would expect, based on the history that we and others had identified as the history relating modern to ancient Europeans. Some of these were known and some of these were not known, and this was very exciting.

We hoped that as the numbers of samples would increase and we got higher resolution to be able to appreciate differences in frequencies over time, it would make it possible to detect far more. What was quite disappointing over the subsequent decade is that that didn’t happen. For example, the largest study of that type in 2024 by a group in Copenhagen analyzed much better data than we had in 2015 and found only 21 positions that were highly different in frequency across time. While that was exciting—it was almost twice as many as we had found in 2015—in a lot of ways it was disappointing because the sample size and data quality had gone up so much, and yet this is all that was found.

That suggested we might be hitting an asymptote and might not be able to get beyond where we currently were. This approach to learning about biology, which was very promising in theory, might not produce a high yield. Maybe natural selection was quiescent, and the reason we’re seeing so few changes is that there has not been a lot of adaptive directional selection. That was the situation we found ourselves in until just a few years ago when we carried out this study in our research group led by Ali Akbari.

What we did is we deployed a few innovations to try to improve our power to detect natural selection. One of them was that we just pumped a lot of data into the system, increasing the amount of data by about 14-fold. The main thing that we do in this study is report new data from about 10,000 individuals. This is a very big increase in the amount of data in the literature. The total dataset size of ancient individuals distributed over the last 18,000 years is about 16,000 people. This is a large dataset. It’s much larger than was previously possible, and when you have more data, you can estimate frequency changes with much more subtlety.

The data comes from only one part of the world, which is Europe and the Middle East. It’s not a more important part of the world than other places, but it’s the place where maybe 70-80% of the data in the ancient DNA literature so far comes from due to historical reasons. It provides us with a natural laboratory where we can see what happens to the genome in one place over time as environments change. It’s really interesting to imagine doing this type of analysis in other parts of the world, and the comparative analyses are super important and interesting, but this study right now is about this one place in the world where we have particularly fantastic data.

The other thing we did is that we developed an entirely new methodology that hadn’t been used in this area before. The methodology is based on a technique that had been developed for finding risk factors for a disease in medical studies. A simple way to explain it is that we ask how to predict the genetic type a person has based on their pattern of relatedness to other people. We have a dataset of about 16,000 ancient people, and 22,000 people if we include the modern people. Then we look at how closely related each of these 22,000 people are to each other, and we predict the genetic type at each position in the DNA—at 10 million positions—based on the pattern of relatedness to all of the other 22,000 people.

Then we ask if natural selection blowing the frequency of the mutation in the same direction in all geographic places and at all times predicts the data a little bit better than just knowing the relatedness to all the other samples in the database. We’re simply asking if the alternative hypothesis—that selection has been blowing in the same direction at all times—explains the data better. That’s a dumb assumption, because of course, the truth is that natural selection will have changed in frequency over time. But we’re just asking the simplest of questions: whether assuming a constant rate of selection explains the data more than not doing so.

Dwarkesh Patel

To summarize to make sure I’ve understood, you’re trying to make a model that predicts allele frequency changes over time. You have two different parts. One part is this genetic relatedness matrix, which captures how similar different genomes are to each other. That should capture the impact of different bottlenecks, of drift, of population admixtures, and all those things which affect the entire genome.

Then you have the separate thing, which is, if we look at specific locations, can we just say that, “Oh, this location has been selected at whatever coefficient over time”? And if we add some coefficient, does it become easier to predict the allele frequency changes than you would have just seen from this other artifact, which is just looking at, “Oh, if you look at the whole genome, are these guys in the same, have they gone through the same bottlenecks? Have they gone through the same drift,” etc.?

David Reich

That’s precisely right.

Dwarkesh Patel

Okay, what did we learn?

David Reich

When we analyzed the data this way, we looked at 10 million positions in the DNA across these 22,000 people, 16,000 of whom were ancient. We looked to see if there was more change in this consistent direction over time than you would expect by chance. When we analyzed the data, we found many hundreds of places in the DNA that were changing too much over time and in too consistent a way to be explained by chance.

There’s a bit of a statistical problem in figuring out how many there are because they’re so densely packed that they’re close to each other and interfering with each other. But when you try to piece them out and say, “Let’s count only one in each place in the DNA and blank out the others,” we find at least 479 positions that are all independently pushing in the same way. We are 99% confident that those positions are real. By another criteria of being more than 50% confident that they’re real, we think that about 3,800 positions are all pushing in the same direction. This is a crazy number of results given that in our previous work and other people’s work, there were at most a couple of dozen discoveries coming from a single scan.

So when we got this result, we were very surprised. We thought it must be wrong, and we spent the next couple of years trying to make the results go away, but they just kept getting stronger. We were trying to look for some independent type of evidence to tell us whether these positions were real. We stumbled on something really powerful for this purpose that had not been used in this way before. It relied on the fact that we had very large numbers of discoveries, many hundreds of discoveries or even thousands.

We took a completely independent dataset, which was the corpus of genome-wide association studies. These are studies that people have carried out in hundreds of thousands of people, looking for whether particular genetic mutations are more common in people with high blood pressure than with low blood pressure, or something like this. We took the UK Biobank, which is about 500,000 people from Great Britain who have been measured for hundreds of traits. The whole genomes of all these people have been sequenced. For each of these traits, we could look at whether each of these 10 million positions are connected to this trait in a convincing way. Out of 10 million positions, about 15%—about 1.5 million positions in the DNA—are predictive of at least one of these several hundred traits.

Then we could ask a question: is our natural selection signal, our statistic, related to whether a mutation causes high blood pressure or some other trait? We slid our statistic for natural selection upward, to a value of one, two, three, four, or five. As we did that, the enrichment for genetic mutations that affect traits got higher and higher. Whereas it was only 15% when we didn’t use our selection statistic, when we required the selection statistic to be above about five, there was about a five-fold enrichment for mutations that cause traits.

Dwarkesh Patel

Sorry, what is a selection statistic?

David Reich

This is the statistic we use to measure whether a mutation is changing over time significantly in a non-zero way. It can be approximately thought of as a normally distributed statistic, a Gaussian statistic, which is the number of standard deviations the statistical value is away from zero, where zero is no natural selection.

It’s not exactly that, but it’s close to that. If the statistic is above five, we see about a five-fold enrichment in mutations that affect a trait. Instead of 15% of the mutations that are at random affecting the trait, it’s 60% or 70% that are affecting the trait when we slide our statistic upward. This provides completely independent evidence that these sites are real, and as you slide above five, there’s no more enrichment. Our interpretation of these results—which we were able to validate and show made sense using computer simulations of our process—is that once you slide the statistic above five, essentially all the signals of natural selection are real.

Dwarkesh Patel

Okay, just to make sure I understand. You’re saying, in order to figure out what alleles have been under selection, your model assigns a statistic saying, “In order to explain why this allele has a specific frequency, we’re going to give it a selection statistic.” Independently, we run these studies on modern populations where we say, “If you look at height, eye color, intelligence, or whatever trait, what are the parts of the genome that are correlated with that trait?” The higher the statistic you give it in your study to explain allele frequency changes over time as a result of selection, the more probable it is that that region in the genome is associated with traits that have some functional thing we can measure.

David Reich

That’s exactly right. This is a brilliant idea that Ali had. It abandons the traditional approach of assigning statistical significance to mutations that cause a trait because we’re just using an external piece of information—the correlation to traits, measured in a completely different way—to read off the probability mutations are real.

We can ask how much enrichment for real signal is there given a particular selection statistic. If it’s halfway enriched to the plateau, we’re able to show the correct interpretation is that 50% of the mutations are really selected. If it’s three-quarters of the way toward the plateau, there’s a three-quarters probability that the mutation is real. If it’s 99% of the way to the plateau, there’s a 99% probability that it’s real. That gives us a calibrated estimate of the probability that a particular position is really under natural selection.

A major concern here is that what we’re actually seeing is not that these mutations are really under selection, but rather that both association to a disease and our selection signal are due to some third thing that’s causing both of them, which is a type of selection which is not what we’re after, not selection to adapt to new environments, but what’s called background selection: selection against newly arising bad mutations that are removed from the population that tend to be concentrated in genes. Genes are also the parts of the genome that tend to be associated to traits. This common process is causing both the enrichment for trait signals and is also causing the enrichment for selection signals that we’re observing. That’s the concern. We were super concerned about this.

So what we did is we repeated this enrichment analysis in slices of the DNA that all were affected to the same extent by background selection, by this rain of slightly bad mutations, and we get exactly the same pattern. We also repeated this experiment just using mutations of the same frequencies because there’s different statistical power to detect these signals at different frequencies. We see the same pattern where above a value of the selection statistic of around five, we get this plateau.

Dwarkesh Patel

The thing that changed that allowed you to increase the amount of sequences you’re generating by two orders of magnitude is just the statistical method you’re using to identify which part is human? Or what exactly changed in 2014 and since then?

David Reich

There’s been a whole series of improvements. The big ones have been the huge drop in sequencing cost, which made it possible to generate ancient DNA in the first place. The drop in cost has been a millionfold since the late 2000s, and another maybe one to two orders of magnitude from 2010 to today. That’s one big change.

Another change has been in-solution enrichment. It’s been this way of taking a sample that has very small percentages of human DNA, but then suddenly creating a process that will mean that the great majority of the sequences that one’s analyzing will be useful for analyses. The approach that we used was we took the DNA samples that we had, most of which were very low percentages of human DNA—less than 10%, often less than 1%—which is such a low proportion that it’s prohibitively expensive to sequence them and to just brute-force sequencing them given the technology that we had available at the time.

We took these samples and washed them over an artificially synthesized set of short DNA fragments that targeted positions of the DNA that we were interested in analyzing. This is more than a million positions that are highly variable in people, and we picked many of these to be biologically interesting. We had a whole set of known biological targets that affected traits in genome-wide association studies, which is the way that people look to see if there are particular genetic variants in modern people that have particular impacts in phenotypes and traits.

And so, what we did is we had this artificially synthesized set of DNA fragments that we washed our ancient sample over, and it bound the parts of the DNA that we targeted. The resulting sequence that we generated was very enriched for the parts of the genome that were informative about history. Even though only 10% or 1% of the DNA was human, it ended up that a very large fraction was from the parts of the genome that we were interested in, and it became economically efficient to do it.

Dwarkesh Patel

What was the other 99% of the DNA?

David Reich

It’s mostly microbial. It’s from bacteria and fungi that colonize a person’s body after they die. Depending on how they die, there’ll be more or less of these bacteria and fungi. When you typically sequence DNA from a person, it’ll just be full of microbial sequence. Sometimes the microbial sequence is very interesting; it might be pathogens that a person died of. There’s amazing work, for example, about different plagues of malaria and Black Death and hepatitis B and so on that have been obtained from the sequences of these pathogens in people’s teeth and other parts of their body when they died.

But we’re focusing here on the human DNA. This changed the amount of data that was possible to produce from tens per year to hundreds per year, and then we further roboticized and industrialized the process so that there were many hundreds or even thousands per year. Just in our laboratory, we’ve been generating genome-scale data from more than 5,000 individuals per year. I know this is true also of several other laboratories in the world now.

This huge jump in data, this semi-exponential or even super-exponential jump in some cases, has made it possible to ask and answer questions. While we were only on the order of 10 genome sequences from humans in 2010, this year it’s passed more than 20,000 reported sequences. There are several orders of magnitude increase, and the questions we were able to ask in 2014 are just not the same as the ones we can ask today.

Dwarkesh Patel

Awesome. Excellent. David, thanks for your time.

David Reich

Thank you, Dwarkesh.

It’s shocking how much you can deduce about what the labs are doing from a handful of equations, public API prices, and some chalk.

It’s a bit technical, but I encourage you to hang in there – it’s really worth it.

There are less than a handful of people in the world who understand the full stack of AI, from chip design to model architecture, as well as Reiner. It was a real delight to learn from him.

Wrote up some flashcards and practice problems to help myself retain what Reiner taught. Hope it’s helpful to you too!

Reiner is CEO of MatX, a new chip startup (full disclosure - I’m an angel investor). He was previously at Google, where he worked on software efficiency, compilers, and TPU architecture.

Highly recommend the scaling book he coauthored for further study.

Watch this one on YouTube so you can see the chalkboard.

Sponsors

• Jane Street needs constant access to incredibly low-latency compute. I recently asked one of their engineers, Clark, to talk me through how they meet these demands. Our conversation—which touched on everything from FPGAs to liquid cooling—was extremely helpful as I prepped to interview Reiner. You can watch the full discussion and explore Jane Street’s open roles at janestreet.com/dwarkesh

• Google’s Gemma 4 is the first open model that’s let me shut off the internet and create a fully disconnected “focus machine”. This is because Gemma is small enough to run on my laptop, but powerful enough to actually be useful. So, to prep for this interview, I downloaded Reiner’s scaling book, disconnected from wifi, and used Gemma to help me break down the material. Check it out at goo.gle/Gemma4

• Cursor helped me turn some notes I took on how gradients flow during large-scale pretraining into a great animation. At first, I wasn’t sure the best way to visualize the concept, but Cursor’s Composer 2 Fast model let me iterate on different ideas almost instantaneously. You can check out the animation in my recent blog post. And if you have something to visualize yourself, go to cursor.com/dwarkesh

Timestamps

(00:00:00) – How batch size affects token cost and speed

(00:32:09) – How MoE models are laid out across GPU racks

(00:47:12) – How pipeline parallelism spreads model layers across racks

(01:03:37) – Why Ilya said, “As we now know, pipelining is not wise.”

(01:18:59) – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal

(01:33:02) – Deducing long context memory costs from API pricing

(02:04:02) – Convergent evolution between neural nets and cryptography

Transcript

00:00:00 – How batch size affects token cost and speed

Dwarkesh Patel

Today, I’m interviewing Reiner Pope, who is the CEO of MatX, which is a new chip startup. Previously, he was doing TPU architecture and many other things at Google. This is a very different format from my usual interviews. This is going to be a blackboard lecture. We’re going to get up in a second. We in fact built this whole new studio with specifically this format in mind, so it’s a pleasure to get to inaugurate it with you.

We’re going to be talking about model architecture, ML infra, and many other things. The reason I think it’s an important topic is because once you understand how training and inference work in a cluster, a lot of things—about why AI is the way it is, why AI architectures are the way they are, why API prices are the way they are, and fundamentally why AI progress is the way it is—start making sense. You need to understand the details to get there, and you need a blackboard to understand the details. Reiner, thank you so much for doing this.

Reiner Pope

Very happy to be here.

Dwarkesh Patel

Full disclosure, I am an angel investor in MatX, but that’s unrelated to this podcast. Reiner, to kick us off I’ll ask this question. We have a couple of companies like Claude and Codex and Cursor offering something like Fast Mode, where for 6x the price, they’ll stream you tokens at 2.5x the speed. Mechanically, I’m curious what’s going on here. Why is it the case that you can pay more to get faster latency?

Two, could you keep going? Could you pay 100x more and somehow get much faster speeds? Three, could you go the other way? Could you have something like Claude Code “Slow Mode”, where if you are willing to wait for minutes on end, you could get even cheaper prices? Maybe this will help motivate the analysis that you’ll be doing through the lecture.

Reiner Pope

Great. To jump to the conclusion a little bit, the big effect is batch size. What we’re going to do now is quantify exactly what that looks like and what its implications are on latency and cost. There’s another effect, which you can call speculative decoding or multi-token prediction. We can maybe come back to that later, but the first thing that we’ll talk through is batch size.

What I’d like to introduce is the two principles of analysis. First, we’re going to look at a roofline analysis of how we run a transformer model on a cluster of chips. We’ll take a Blackwell NVL72 cluster, so a rack of 72 GPUs. The roofline analysis means we look at memory bandwidth and compute performance. The other side of that is that we’re going to look at just two simple factors of the model: the time to operate on the weights, and the time to operate on the context, the KV cache.

Let’s jump in. We’re going to try and estimate the time that it takes to run an inference of a certain shape. We’re not perfect here. We can’t exactly predict the time, so instead we’re going to approximate. We’re going to say that the time must be greater than or equal to a certain quantity. We’re going to consider two different aspects: the time it takes to do the memory fetches, and the time it takes to do the compute. It will turn out that this gives us very strong predictive power, even with a simple model.

One by one, what is the time that it takes to do the compute? There are really two things I need to do in the compute. I need to multiply by all of the active parameters, and then I need to do some work on the attention. Multiplying by all the active parameters, I have a certain batch size that I’m running, and I’ve got a number of active parameters in my model. Then I’m just going to divide this by the compute throughput, which is the FLOPs of the chip. This is a hardware concern.

This accounts for all of the compute time for all of the weight matrix multiplies. There’s a little caveat here. We’ve ignored the time to do any of the attention computation, but that in general will be quite small in comparison to this. So we’ll ignore this.

Dwarkesh Patel

I’ll just interrupt from time to time to ask some very naive questions or to clarify some basic points. For the audience, you’re not serving one user at a time. The batch refers to the fact that you’re serving many different users at the same time, and that’s a whole batch.

Reiner Pope

I can motivate the batch at least a little bit. We will see exactly why batch is such a favorable optimization. What will turn out to be the case is that if you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together. We’ll be able to see that quite explicitly.

Then, number of active parameters. If I look at, for example, a DeepSeek model, the DeepSeek V3 model has about 37 billion active parameters, and 700 billion total parameters. We’re focusing on just the ones that are active for a single AI token.

We’re modeling compute performance. I’m going to keep writing equals, but in all of these cases, you can think of this time as being at least this much, and maybe there will be some terms we ignored.

On the memory side, what do we need to do with memory? We need to fetch all of the weights, so there is some time to fetch the total number of parameters, not just the active parameters. There’s weight fetch time, and then in addition, there’s a KV cache fetch time. This actually depends on batch size. For every element of the batch, we have to fetch an entire context length worth of tokens, and there’s a size per token, bytes for one token. This is a model parameter.

Dwarkesh Patel

Maybe just backing up, let’s explain what the KV cache is real quick.

Reiner Pope

When I do a forward pass… Let me draw how the autoregressive inference works. This is during decode. If I have a bunch of text tokens… I’m drawing a tensor because ultimately the tokens are represented as a tensor in some embedding dimension. In this direction, I have the sequence length.

The work of running a decode is that I have to run each token through a whole bunch of matrix multiplies over a bunch of different layers. In general, I’m going to have to do that work over all of these tokens. But one step of decode is to produce just this one additional token up here.

What I’m going to do there is run a full forward pass of multiplying by all of the weight matrices in the entire model. But then I’ve got this attention mechanism where this token is looking at all of the past tokens, and what is it looking at specifically? It is looking at some internal representation that the model has produced of the tokens, and we call that the KV cache. This process of this single token attending to all of the history of tokens is attention. It is mostly dominated by memory fetches rather than matrix multiplies.

So we’ve got the amount of memory that we’re fetching shown over here, and then this is of course just divided by the memory bandwidth, so the memory bytes per second. In fact, these equations here are enough for us to now draw some fit lines. The things that we’d like to look at are sensitivity to batch, and then also, which we’ll draw separately, to context length. We said that the big effect you can get is some trade-off in latency versus cost in batch size.

Let’s draw them out. I think there are just really two graphs that we want to draw. We’ll first draw batch size versus time here. When we look at the shape of this, we’ve got a maximum of the sum and then another term. Let’s look at these terms one by one and how they scale: the time for compute and memory, and how they show up.

Let’s first look at this compute time. This is just purely linear in batch size with no offset, so it is some curve like this. This is t compute. On the memory side, we’ve got some portion here that is just this constant in some base offset here, which is the weight fetch. Finally, we have this term here, which is the KV fetch, which is pretty linear in batch size, and so it looks like that. The sum of this plus this maxed with this… Let’s at least first draw the sum. The two memory times in conjunction end up looking on this curved slope like this. Then the overall maximum is—I’ll draw a little thicker here—the maximum of these two curves.

What does this mean? This is a latency plot. If I grow my batch size, initially I get some not very strong dependence on batch size, so there is some lower bound on latency here. This already partially answers the question. For a given hardware configuration—and we can talk about varying the hardware configuration—there is a lower bound on latency. It is simply that I need to read all of my total parameters from memory into the chips, and that takes a certain amount of time. If I use all of my memory bandwidth, I can’t do any better than that.

Dwarkesh Patel

It seems like the way you’ve drawn the slopes for compute time and how the KV grows—and what implication the KV has on memory time—

Reiner Pope

What if this were above or below?

Dwarkesh Patel

Yeah, is that necessarily the case? If this is always true, then as batch size grows compute always dominates KV, which suggests that if you have a big enough batch size, maybe memory is never an issue.

Reiner Pope

This is really sensitive to the context length, so I think we should come back and explore this. As you vary the context length, the KV fetch time will go up and up, and that will cause a transition from compute-limited to memory-limited.

Dwarkesh Patel

Is there something especially significant about the slope being exactly the slope of the compute time?

Reiner Pope

Whenever we have balance points, it says that you’re getting it exactly right. For the particular context length where the slopes match, that says I am equally memory-bound and compute-bound, which is a really desirable place to be.

Dwarkesh Patel

This is a very simple algebra problem, but suppose the optimal is 100K context length, and you go to 200K context length. Does your MFU go down to 50%? Does it have a humongous impact on MFU to be slightly outside of the optimal context length range, the Goldilocks zone?

Reiner Pope

That’s right. That is true as modeled here. There is a key point here that I’m modeling the memory fetch as linear in context length. That depends on model architecture. It is true for all of the model architectures with dense attention. Sparse attention actually scales much better than that.

Dwarkesh Patel

Got it. Is sparse attention what everybody uses in practice?

Reiner Pope

I’m pretty excited about sparse attention. It’s hard to know what the labs are using. DeepSeek has published a sparse attention mechanism. I’ll just put a plug in that some of the DeepSeek papers that have published sparse attention end up putting a square root in this term.

So far, we’ve looked at the latency. It’s hard to read off cost from this. If I think about what cost means… To run this inference, I’m going to use the GPU for a certain number of seconds, like one millisecond or 20 milliseconds. I have to pay the rental time for that time. So it’s $2/hour per GPU or something like that.

That’s the cost of this inference, but how many tokens have I processed during that inference? That is the batch size. What we actually want to plot is the cost versus batch size, which is t over B versus batch size. This is the cost per token. We have to imagine dividing each of these three curves by B, so multiplying by this reciprocal. What we end up with there is… The compute curve was linear. We divide by B, and that makes it a constant here. This is t compute. The KV fetch was linear, and now it becomes a constant as well. Then the weight fetch was constant, and now we’ve divided by B, so it becomes this hyperbola.

Again, we’re going to compute the max of the sum. The sum of these two terms shifts the hyperbola up. The sum of the KV fetch and the weight fetch gives us a higher hyperbola that’s like this. Then we’re going to take the max with the compute here. We end up with this being the overall shape that we care about.

Again, we see some limiting behavior. The cost initially starts very high at a batch size of one. It almost goes to infinity because we’ve got so many weight fetches that are not amortized over a large batch size. But as we increase the batch size, the weight fetches become amortized over so many different batch elements that their cost grows very small, and eventually the compute time ends up driving the cost. So there is a limiting lower bound on cost, which is this line here.

Dwarkesh Patel

So Claude Code Slow or Codex Slow or whatever would just live on this line. It wouldn’t help much because you’re not able to amortize the KV values over a much bigger batch.

Reiner Pope

They’re unique per batch. The compute is also unique per batch. So what is the minimum work you can do per batch after amortizing everything else away?

Dwarkesh Patel

This point where you are no longer memory bandwidth bound, practically how big a batch do you need? How big are the batches practically for frontier models?

Reiner Pope

You can just solve for that. It’s not even particularly sensitive to model architecture. Let’s go ahead and do that.

What we’re talking about is when the memory time is equal to the compute time. That’s what that question is. Because we’re focused on what the batch size is—and really there’s a question of when the weights are amortized over the multiplies—I’m going to focus on comparing the weight fetch time to the weight multiply time. I’m going to disregard the KV fetch term just to simplify the analysis so we can get a clean answer out. We’re going to equate this portion with these two times.

Writing that out, we get N, number of total parameters, over memory bandwidth, is equal to batch size times number of active parameters divided by the compute performance. Looking over here, everything on the top are model parameters. Everything on the bottom are hardware parameters. It turns out to be nice to rearrange them such that we have the hardware parameters on one side.

This is equivalent to FLOPs over memory bandwidth being equal to batch size times number of active parameters, divided by the number of total parameters. This hardware parameter ends up being a dimensionless constant. If you look in terms of FLOPs… What are the dimensions of this? This is multiplies per second. This is bytes per second. So that’s not quite dimensionless. But what you do is you say, how many FP4 multiplies per second times the fact that each FP4 is half a byte. I can actually make this end up being dimensionless. On most GPUs, this ends up being somewhere around 300.

Dwarkesh Patel

Has that ratio changed over time as we’ve gone from model generation to model generation, where the FLOPs keep increasing?

Reiner Pope

This is a hardware parameter. To what extent has the hardware changed? From A100 to H100 to B100, the FLOPs have increased substantially, the memory bandwidth has also increased substantially, and it has remained reasonably stable.

We can express this one as well. This is a sparsity parameter. I might even phrase this slightly differently. Let’s solve for batch size in total. Moving this back over to the other side, we end up with batch size needs to be bigger than approximately 300 times sparsity. For example, in DeepSeek I activate 32 out of 256 experts, so this would be 8 for DeepSeek.

This actually gives you a ballpark which is remarkably accurate to practice. Generally, people will go a little bit larger than this. They don’t really want to be exactly at the balance point because real-world efficiencies aren’t as good as a roofline analysis would say. But take this and maybe double or triple it.

Dwarkesh Patel

Okay, so it’s two to three thousand tokens per batch. But then if you included the KV cache, the implication would be that the optimal batch size...

Reiner Pope

Should grow larger. We solved for the equivalence between when compute time is equal to memory time. If I add in something that consumes more memory bandwidth, then I have less available for the weight loads. I need to grow the memory bandwidth more, and therefore the batch size more.

Dwarkesh Patel

This seems incredibly small. This would be less than one sequence, right?

Reiner Pope

Keep in mind that I’m talking about the number of tokens that I’m generating one more token for. It’s actually 2,000 unique sequences.

Dwarkesh Patel

Got it. We’re just talking about a single forward pass on these sequences. You think of the batch as the number of sequences.

Reiner Pope

That’s right.

Dwarkesh Patel

If you’ve got a frontier model and you are actually doing inference, surely they must have more than 2,000 concurrent users. Is there any added latency from the fact that you need to have the whole batch fill up? Or if you have a reasonable amount of users, is it so unlikely that it would take you 100 milliseconds to fill up the next 2,000 slots?

Reiner Pope

The way to think about this is: when does the train depart, as a model? Let’s say I’ve picked a batch size that I’m going to run at. By the way, this intersection point is the same intersection point here. I pick this batch size, and I know that it’s going to take, for example, 20 milliseconds, which is a common place this ends up landing.

This is a timeline of what is running on the GPU. It’s going to start a new batch every 20 milliseconds regardless. You can think of this as a schedule for the train. A new train departs every 20 milliseconds. Any passengers who are ready board the train. If the train is full, they wait until the next train. If the train is not full, the train is going to go anyway.

In terms of what that means for queuing latency, the worst case is that a request arrives just after the train departed. It has to wait for the next train, so that’s up to 20 milliseconds, and then it has to wait for that train to complete. So the worst-case latency is 40 milliseconds.

Dwarkesh Patel

How is the 20 milliseconds derived?

Reiner Pope

It’s a rule of thumb, but where it comes from is not fully explained yet. So far we’ve focused on memory bandwidth and compute time. When we look at memory, the other consideration is that we want to use all of the memory capacity we have. Generally, we’re going to use all of that memory capacity to store the weights or the KVs. In the time of doing a forward pass, we want to read all of the memory capacity into the chip. That is capacity divided by bandwidth. That tends to be 20 milliseconds on many different generations of HBM.

Dwarkesh Patel

The units make sense. You would have a byte divided by bytes per second.

Reiner Pope

For example, on the Rubin generation, it is something like 288 gigabytes divided by 20 terabytes per second. This comes out to about 15 milliseconds.

Dwarkesh Patel

Let me make sure I understand what this is saying. I understand the unit analysis. What it’s saying is we can evacuate and replace the HBM in this amount of time. So we don’t want to be in a situation where the HBM is not big enough that we’re not actually able to write everything we want to it or take everything out of it. Or we don’t want to be in a situation where our ability to write back and forth is so small compared...

Reiner Pope

There are sort of two scenarios. Why don’t we pick a latency that is bigger than 15 milliseconds? If I think about what that means, it means I actually have time to read the HBM twice. By the way, most HBM accesses are reads, not writes. It’s almost all reads because the weight matrices are read-only, and almost all of the KV cache accesses are reads. In around 30 milliseconds, I can read all of HBM twice, but what’s the point of that? I don’t want to read the weight matrices twice. I don’t want to read the KVs twice.

Dwarkesh Patel

Makes a ton of sense. A couple of quick questions. If it is the case that the optimal batch size is something like 2,000, it’s totally dependent on the sparsity, not dependent on the model size or anything.

Reiner Pope

Sparsity shows up in model size, but beyond that, it only depends on sparsity, not on scale.

Dwarkesh Patel

That’s a very interesting result. One question is how much of a push towards centralization is it that you would have these economies of scale from inference for batching? But it seems like it’s not that big a deal. Is 2,000 users at the same time a lot? It doesn’t seem like a lot.

Reiner Pope

We can do a bit of analysis on this. You can think of it in terms of number of users, but a more productive way to think of it is in terms of tokens per second. What does this batch size mean in terms of tokens per second of the system?

Tokens per second is going to be equal to the batch size. We run a batch of tokens, and we do that every time interval, which is equal to the 15-millisecond or 20-millisecond number. This ends up being batch size times about 60, so 64 x B. This ends up being around 2,000 x 64, so 128,000 tokens per second. This is in more digestible units.

It’s hard to reason about concurrent users, but what is the global traffic for a system? When you look at some of the announcements, sometimes the API providers will brag about how much traffic they have. The numbers I remember from some announcements of Gemini last year were in the hundreds of millions of tokens per second worldwide. This is one-thousandth of that.

Dwarkesh Patel

Gemini is big. One-thousandth of Gemini is a lot. To actually be competitive at scale, you need to be able to serve at least one-thousandth of Gemini. That’s interesting.

The more sparsity you have, the less compute you need. It does seem that as batch sizes get bigger, compute ends up being the bottleneck, according to this analysis. Then the question is, how far can you take sparsity? As the sparsity ratio increases, as you have fewer active parameters relative to total parameters, how much is the performance of the model degrading? Is it degrading faster than you’re saving compute by increasing the sparsity factor?

Reiner Pope

You mean the quality of the model, rather than the speed of the model. Unfortunately, we’re not able to answer that analytically. That is an empirical question of model quality. The best I can do is pull up a paper and answer that empirically.

Dwarkesh Patel

Should we pull up the paper now?

Reiner Pope

This paper is “Unified Scaling Laws for Routed Language Models.” It’s a somewhat old paper by this stage, but one of the things they looked at is if I keep increasing sparsity, what is the model quality impact? This answer is very sensitive to the actual choice of mixture of experts. Mixture of experts has been around for a really long time, maybe even back in 2017, but the techniques have changed a lot. DeepSeek’s mixture of experts was a big change in how it worked. There have been older papers, like “GShard” and “Switch Transformer”. The actual empirical results are going to depend on all of that.

On one of the older techniques shown here, you can see if I hold constant the number of active parameters at a certain size, and then I increase the sparsity, which they call expert count, the quality keeps increasing. If you imagine drawing a horizontal line from 1.3B dense across, you end up seeing that, in this case, the 64-expert 370-million activated parameter model is as good as a dense 1.3-billion model.

Dwarkesh Patel

So in some sense, it’s actually not amazing returns where you need to increase total parameters a hundredfold to get the equivalent of 10x as many active parameters.

Reiner Pope

Actually even more so. It’s a huge increase in parameter count for a modest increase in efficiency.

Dwarkesh Patel

So in this case, actually it’s 4x?

Reiner Pope

64x for 4x.

Dwarkesh Patel

So while it is true that you get this benefit of being able to economize on your compute time if you increase sparsity, naively it would seem like a trade-off worth making. But if you’re decreasing this by 2x and then having this go up by 8x every time you double sparsity...

Reiner Pope

Is that good or bad, actually? Even from a memory point of view… Keep in mind you are doubling this portion of the memory fetches, which is amortized by batch. So just keep running a larger batch size. From the point of view of the analysis we’ve done here, this is a pure win. Keep doing it until you run out of available users, basically.

There’s this equivalence where if I have a lot of users, I can go to a much sparser model. From that point of view, it’s a reasonable trade-off. The other trade-off that shows up here is that it also consumes memory capacity. We’ve only reasoned about memory bandwidth here, but it also consumes memory capacity.

Dwarkesh Patel

I see. Let me make sure I understood. You’re saying we want to spend less time computing, therefore we do more sparsity. To make that work, we need bigger batch sizes. Which means we need more memory capacity to have more sparsity.

Reiner Pope

Maybe this would be a good point to talk about how a mixture of experts layer is typically laid out on a rack of GPUs.

00:32:09 – How MoE models are laid out across GPU racks

Dwarkesh Patel

Cool. Makes sense. Where were we?

Reiner Pope

Sparse mixture of experts. Maybe how we lay that out on a GPU.

Let’s zoom in on the mixture of experts layer first and draw what that looks like. Typically, we’ll have some kind of a router layer, which is making the decision of where we route the tokens to. We get tokens coming in here, they go through a router layer, and then we have a bunch of different experts. I’ll draw a few more to line some up.

The router will make a decision of which experts it’s going to route to, and it will be a small fraction of them, maybe 1 in 32. Maybe it will make a decision to route to this one, maybe this one, and maybe this one. Each expert itself is a normal MLP. It has an up projection and then a down projection with a nonlinearity in between.

Then finally, we do the inverse operation. Where we were broadcasting things out here, we’re going to bring them back in and sum them up. Bringing them in like this. Then finally, we have our residual connections. The token is also passed through here, and it gets added to the result of the MoE layer. This is a normal MoE layer.

What I want to talk through is how this is mapped to a GPU rack and what this means for communication, because I think this will start to show some of the limits of how sparse we can go. The standard practice here, and it is the best solution, is to use expert parallelism. That means different experts go on different GPUs. If we take something like a DeepSeek model, they have 256 experts. Let’s say we want to run that on a Blackwell rack. There are 72 GPUs.

We have a divisibility problem. This is not a power of two. We’ll just simplify and say we’re only going to use 64 of them. Just ignore the other eight. It’s not a big deal. So we have four experts per GPU. Very simple. For the sake of the diagram, actually let’s just say we have two experts per GPU. We end up just putting these GPU boundaries. Every pair of experts is on its own GPU.

Then we can look at the communication cost. We had some tokens stored centrally here. They get routed to all of these experts, and there is some communication cost paid here. There’s the same communication cost paid on the output. The hope is that this does not become communication limited.

Now what is the traffic pattern here? The traffic pattern here is that any GPU will be talking to any other GPU, depending on the decisions made by the model. This is an all-to-all traffic pattern.

Dwarkesh Patel

When you say any GPU in the pre-tense, the router is more than one GPU?

Reiner Pope

I drew this as one router. In reality, you would actually have many copies of the router, and you would have as many routers as GPUs, in fact.

Dwarkesh Patel

As the incoming traffic.

Reiner Pope

Yeah. These are 64 GPUs and these are 64 GPUs. It’s actually the same GPUs, we just draw them as separate because they’re serving different purposes. So at this point, any GPU can be sending to any other GPU.

This all-to-all pattern of communication that shows up and how the Blackwell racks are configured is a perfect fit for the communication pattern that the MoE actually wants to do. However, if you think maybe one rack is too slow and I want to do two racks, then I have this challenge that maybe I’ve got some sort of rack boundary drawn outside here like this, and I no longer have all-to-all communication between all the GPUs in two racks. The rack-to-rack communication ends up being a substantial bottleneck.

The fundamental thing here is that one rack bounds the size of an expert layer you can do. This has been part of what’s been driving towards larger and larger interconnect domains.

Dwarkesh Patel

Before we continue, it may be worth you explaining what exactly a rack is. The differences in bandwidth between a rack and within a rack, and the all-to-all versus not all-to-all nature of communication within versus outside.

Reiner Pope

This is a place where it starts to be very different between Nvidia, for example, and Google, and then others, including us. Generally, a rack is a physical structure. It’s a few meters tall, a meter or two wide, depending on configuration, and it stores some number of GPUs or XPUs, which is typically about 64.

What constrains it being a certain size is power delivery, weight, and cooling ability. It ends up being about this size in many cases because of these physical constraints. When I deploy a data center, a data center may have thousands of these racks. I’ve got one of these tall racks, it’s got a bunch of GPUs in it, and so on. And then I put another rack next to it.

Dwarkesh Patel

You make it sound so easy.

Reiner Pope

Right. I just drop them in. In Nvidia’s case, the communication topology… They actually put the GPUs on the outside of the rack, and then they put these switches on the inside of the rack. What this ends up being is that there’s a set of switches in here. These are the NV switches. Then they run a bunch of cables. Every single GPU has cables going to the switches in the middle. The switches have connections to all the GPUs. All of the GPUs can talk to all the other GPUs in just two hops: going to the switch, going to the other GPU.

Now, when I want to leave the rack, I end up going via a different path. The GPUs also have a much slower connectivity, which is typically about eight times slower. The green that I drew here in the GPU cases is the NVLink. More generally, it’s called the scale-up network. You will typically also have a scale-out network, which allows you to connect to some data center switch. All of the GPUs will have some connectivity up to some data center switch somewhere. This is the scale-out, and it tends to be about 8x slower in bandwidth.

The challenge, if you want to lay out a mixture of experts layer across two racks, is that half of the GPUs here are going to be wanting to talk to the GPUs here. On average, when I look at where the tokens on these GPUs want to go, half of the tokens want to go inside the rack. That’s great. They can use the fast scale-up network. But half the tokens are going to want to leave the rack and go to the other rack, and that’s not as good. They need to use a much slower network, and so that becomes the bottleneck on the all-to-all pattern.

A different choice would be, why don’t I have a big switch here and connect everything to a much bigger switch that actually combines the two racks together? There are many ideas in this direction, but in general, the reason you have this hierarchy of switches rather than one big switch is to manage the cabling congestion. You just need to run a large number of cables.

Dwarkesh Patel

Sorry, is that question you just asked basically, why isn’t it a bigger scale-up?

Reiner Pope

Exactly. Why not just have a million chips in scale-up or a thousand chips?

Dwarkesh Patel

What has changed that has allowed Nvidia to go from Hopper, which was 8, then Blackwell is 72, and now Rubin will be... is it 500 something?

Reiner Pope

Yeah, 500 and something.

Dwarkesh Patel

What has allowed that to happen?

Reiner Pope

From Hopper to Blackwell is mostly just the decision to switch from trays as the form factor to switching to racks as the form factor. That’s a product decision. There wasn’t a substantial technical barrier there.

Switching from 64 to 500 or so, there’s a bit of Jensen math there, but there is at least a genuine 4x increase, which is coming from a much more complicated and difficult rack design. That is actually a new physical design to run more cables.

Dwarkesh Patel

The cable complication is just the cost of figuring out which cable hops to which, or which signal goes from what to what?

Reiner Pope

Let’s zoom in on this and look at the wire density. I’ll draw this diagram just once more so we have a bit of a cleaner and larger version to work with.

Let’s say I have some switches in the middle. Initially, I’m going to start with just two GPUs on each side or two trays of GPUs on each side. Let’s say maybe each tray wants to have two cables coming out of it. I physically run vertical cables that look like this running out to the switches. Now if I want to double the number of GPUs in a rack, I need to run literally twice the density of cables. I need to run these as well.

Dwarkesh Patel

Extremely naive question. But if you look at a physical data center, it seems like there’s a lot of space within a rack. I don’t know. The cables are really big and...

Reiner Pope

There is space outside the rack. Inside the rack… As they become more optimized, these racks are very tight. There’s connector density going from the tray into the rack and the rack’s backplane, and the backplane itself has a really high density. There are other physical constraints including the bend radius of cables. You don’t want to snap them and so on.

Dwarkesh Patel

Okay, so it’s literally the physical space to put a cable that’s constraining it. I had no idea. Interesting. That seems surprising. The rack is so big and we can’t just stuff more cables in there.

Reiner Pope

Rack design is not my expertise, but when I talk to folks on what constraints they’re up against, it’s a combination of things. What are the big physical things you’re optimizing for? Space, weight of the rack. It’s actually really heavy, so you need enough metal to not sag and fall. But then you add more metal, and it’s heavier. Then power and cooling. All of those are competing. Modern racks are pushing all of those to very extreme physical limits.

Dwarkesh Patel

When was GPT-4 released again? Was it 2022 or 2023?

Reiner Pope

2023.

Dwarkesh Patel

Okay. And it was rumored to be over one trillion parameters. It seems like only now, within the last six months, have models been getting released that have significantly more parameters than the model released three years ago, when supposedly there should have been this scaling in the meantime.

Is the reason that we were just waiting for racks with enough memory to hold a five-trillion parameter model, along with its KV cache for enough users for a lot of sequences? Or if you’re doing RL, a similar consideration of actually holding the KV cache for the batch of problems you’re trying to solve.

If you look at Hopper, you had eight Hoppers, and I think that’s 640 gigabytes as of 2022. With Blackwell finally, which was deployed in…?

Reiner Pope

Very recently. Maybe last year.

Dwarkesh Patel

Last year. You finally have a scale-up on the order of 10-20 terabytes, which is enough for a 5T model plus KV cache.

Reiner Pope

Deploying in larger scale-up domains is a huge unlock. I’ve drawn here the Nvidia Blackwell deployment. The Google deployment has actually had very large scale-up domains for a long time.

Dwarkesh Patel

That also explains why Gemini seemed to be ahead. It just seems like Gemini has had successful pre-training for longer than some of the other labs.

Reiner Pope

Not having been there at the time, I’m not sure how much is coming from successfully deploying higher sparsity ratios, which it could be. It could also be a whole bunch of actual modeling things, specifically how you do the mixture of experts. We’ve seen the DeepSeek mixture of experts activate more experts, but finer-grained experts. That was a big innovation. I’m sure there are many other innovations on the model architecture as well as on the training data.

It’s hard to disentangle all of them, but what shows up in terms of the limits of what you can do is that the active parameters, as we saw, are limited by the compute cost, and the total parameters are limited by the scale-up size.

00:47:12 – How pipeline parallelism spreads model layers across racks

Dwarkesh Patel

When you’re operating within a single scale-up domain, is that a consideration specifically for either forward or backward, or specifically for prefill versus decode? Or is it preferred to always be within a scale-up whatever kind of workload you have, whether you’re doing a pre-training run, RL generation, or inference for users?

Reiner Pope

Really interesting. To answer that question, we’re going to need to talk about the communication patterns. We’ve talked about the mixture of experts communication pattern. That is this all-to-all. All-to-all very strongly favors full connectivity, which is what we’ve just shown here, and it favors being within one rack.

There are other kinds of parallelism besides expert parallelism, which we just showed here. In the literature is tensor parallelism. With the trend towards smaller experts, this has become much less relevant, so we can ignore that. But the other two things we have available are data parallelism and pipeline parallelism. They can be a much better fit for using multiple racks.

Let’s focus on pipeline parallelism specifically. This is one layer of MoE. I’m going to have a hundred more layers up above. I could decide at this point, for example, to move to a different rack, change rack. Now, is that going to become a communication bottleneck? We can actually solve for when this becomes a communication bottleneck. Before we do that algebraically, let’s visualize it out and sketch the path. We’re going to have another MoE layer, and another MoE layer here, and so on.

Let’s say I change rack here, and then some number of layers later, I change rack here as well. The methodology we’re going to use to determine whether we have a communication bottleneck at the point where we change rack is we’re going to compare the scale-out bandwidth requirements to the scale-up bandwidth requirements. Let’s write this. The hint is going to be that there’s a lot more sends here. We’re sending many things here, whereas we’re only sending one thing here, and we’re also maybe doing it many times. That’s what makes the difference.

Dwarkesh Patel

Can I try to guess? Just out of curiosity, to see if I’m actually understanding, it seems like you’re sending batch size into the rack.

Reiner Pope

In here? Yes.

Dwarkesh Patel

But the communication within the rack is batch size times number of GPUs.

Reiner Pope

Number of activated GPUs. I don’t send to this GPU at all. There’s an explosion from 1-3x larger here in this diagram. The key thing is that I didn’t even need to send to this GPU at all, and so that’s a big saving.

We’re going to talk through to what extent scale-up is a bottleneck over scale-out. We will directly jump to the ratio of the time spent on scale-up over the time spent on scale-out. This is the quantity we’re talking about.

The first consideration is that scale-up is 8x faster than scale-out generally. At a baseline, if the bandwidths were the same, we would have this 1/8, which is coming from bandwidth. But then we have some amount of expansion in how much data we’re sending. If one token comes in here, then this one token gets routed to, in the DeepSeek case maybe 32 experts or 16 experts. It gets routed to some number of experts. So this is the number of activated experts. This same thing applies on multiple different layers, so maybe I’m going to run two layers. There’s also multiple times the number of layers per stage.

Dwarkesh Patel

Don’t you need to multiply the whole thing by two for the all-to-all?

Reiner Pope

For the up and down. Yes, there’s a factor of two. Thank you.

What we would like is for the scale-up time to be greater than the scale-out time, because the scale-up time is the more important and precious resource. We would like this number to be greater than or equal to one. This really doesn’t seem hard. There’s just a factor of 8 that we need to overcome. So we need the product of these three things to be bigger than 8. Typically we have a fairly large number of activated experts. It could be 8 by itself. Then we can increase the number of layers per stage a lot until we satisfy this.

What this ends up looking like is that I can have an entire pipeline of racks where one rack does one layer, and then I move on to the next rack and do another layer, and then I move on to the next rack and do another layer.

Dwarkesh Patel

It’s interesting to me that the best parallelism strategy in practice ends up being one which physically resembles the actual architecture. It’s not some galaxy brain thing. It’s like, “Oh, we have experts, we’re going to put them on different GPUs, or we have different layers, we’re just going to put them on different racks.” I feel that’s interesting.

Reiner Pope

The cutting matches the model architecture.

Dwarkesh Patel

Exactly. It could have been something wackier with tensor parallelism and whatever.

Reiner Pope

The galaxy brain way to think of it is, what are all the different dimensions in which a model is scaled up? It is scaled up by layers, it is scaled up by the model dimension, it is scaled up by the DFF dimension, it is scaled up by the number of experts. Every single one of those numbers you can choose to cut along. If those numbers are big enough, it eventually becomes profitable to cut along there. We have selected two of them. The other two, in the way models are typically sized, are not profitable.

Dwarkesh Patel

So there’s a talk by Ilya where he says, “Today we know not to do pipeline parallelism.” And Horace He gave my friends and me… I hate that it sounds like a Dr. Seuss quote. But he gave us a lecture on these different kinds of parallelisms. He said the problem with pipeline parallelism is that, other than the bubbles, it creates these architectural constraints. Kimi, for example, has these residuals where attention attends to layers a few back, so it becomes hard to implement in this way.

Reiner Pope

I guess we didn’t fully articulate even what is the benefit that we’re getting from pipelining. These complexities are real. Pipelining is a massive hassle, but it does give you some benefits. You can then decide whether those benefits are worth the costs. It has some benefits in inference, maybe bigger benefits in training. In inference, what are we saving on? Are we saving on memory time or compute time? Not really. We’re just moving the memory time from one chip to another chip, or one rack to a different rack. There’s no actual benefit in runtime.

However, what we are saving on is memory capacity. If we think that the memory in a rack is a bottleneck, then there’s a constraint on how fast we can go. Pipelining allows us to massively reduce that bottleneck.

Dwarkesh Patel

The opposite connotation to this… Before this interview, I was chatting with Axel, who’s a GPU performance engineer at Jane Street. He was explaining that to do pipelining, you have to do micro-batches rather than full batches. If you do micro-batches, then you’re by definition not able to amortize loading the weights across all the users or all the sequences. The positive connotation of that is you don’t have to use as much memory. The negative connotation is that we can’t amortize loading the weights across all those users. Maybe it’s worth explaining why you have to do micro-batches.

Reiner Pope

Shall we draw the pipeline bubble? What is this micro-batching that shows up in pipeline parallelism?

I’ll focus on inference first. It’s a slightly simpler problem. I’m going to draw time, and then which rack we’re on. The idea is that maybe I’ll have four racks. I’ve got an inference that is going to step through these four racks in some time like this. This is inference number zero. It runs at a certain batch size and steps through all the pipeline stages like this.

Now, if we were to say, “Well, we’re going to run inference number one here,” this is clearly a massive waste. Like three-quarters of the time each of the racks is doing nothing. We don’t actually run inference one here, we run it as soon as we can, which is immediately after inference zero finishes. And then we keep going. If we hadn’t filled this in, we would call this the pipeline bubble. When I’ve drawn it in this inference context where we’re only going in a forwards pass, it’s obvious. Why would you do this stupid thing? In a training context, it’s maybe less obvious. But in the inference context, it’s really natural to make this change.

Dwarkesh Patel

Oh, interesting. This is sort of obvious, but the difference between micro-batch and batch doesn’t matter at all in inference because you can just call it whatever you want. It only matters in training because there is an optimal batch size.

Reiner Pope

Yes.

Dwarkesh Patel

Before you do a full backward step, you want to have accumulated all the sequences in that batch. If you want to do pipelining in training, in order to avoid that bubble, you need to—

Reiner Pope

Should we draw the training diagram with that? Let’s do that. This is the inference diagram, and I’ll call this forward so we don’t have the wrong thing showing up there. Let’s do the same thing for training now. We’ve got a forwards pass, but at some stage we’re going to have to transition to a backwards pass.

We’ll do some number of batches in the forwards pass, and then we’re going to transition to the backwards pass for everyone all in one go. The inference part is the same here, but then we do a hard stop at this point and transition everyone to the backwards pass, with similar numbering like this.

Dwarkesh Patel

It may be worth clarifying the reason there is that hard stop is because you want to do a whole batch at once for the backward step. And then there is an optimal size for how big that batch should be.

Reiner Pope

Smaller is always better, actually, is a way to put it. From an ML convergence rate perspective, smaller is always better because you’re getting the freshest information from the gradient descent.

Dwarkesh Patel

But from a total training time perspective?

Reiner Pope

From a total training time perspective, smaller is worse from a systems perspective. The optimum is the trade-off between those two.

So you pick a batch size, and for that batch size, you do some amount forwards and then some amount backwards. You asked why there is even a hard stop there. With pipeline parallelism, because you’ve got this idle time here which is the bubble, there are so many techniques in the literature for how to lay this out differently and avoid that. There are more complicated schemes called zero bubble or one-forward-one-backward, which interweave the forwards and the backwards in complicated ways.

Dwarkesh Patel

You can mine Bitcoin in that bubble.

Reiner Pope

Right. More usefully, you can do the weight gradient step, but you can also mine Bitcoin.

In inference, the effect of pipelining on anything you care about, like batch size or latency, is neutral. It doesn’t improve it, it doesn’t make it worse. If you look at the latency of this inference, running it if it were pipelined versus if it were all on one rack… If it were all on one rack, we would just slide all the boxes down and still put them in a row, and the latency would be the same.

Pipelining is neither better nor worse for latency. It does mean that you just use less memory capacity per rack. Because now instead of needing the whole model, you only need a quarter of the model, and you can expand.

Dwarkesh Patel

Makes a ton of sense. So it’s a no-brainer to use pipelining during inference, but there’s this harder trade-off during training.

Reiner Pope

Even in inference, in fact, it is not used a ton. It reduces your memory capacity requirements, but there’s actually a huge surplus. I think you were saying that a rack of Blackwell has many tens of terabytes. That’s much bigger than a trillion parameter model. A trillion parameter model only needs one terabyte, so it already fits. There’s not a huge benefit from pipelining because you’re reducing a number that’s already pretty small.

But it does say that theoretically, maybe you had too much memory there. You could have built different hardware that has less memory. If you were designing your hardware, you could say, “I didn’t need that much memory because I don’t need the weights to fit in one rack. I can fit the weights in eight racks, then I could have built hardware that didn’t have so much HBM per GPU.”

01:03:37 – Why Ilya said, “As we now know, pipelining is not wise.”

Dwarkesh Patel

Macro question: everybody’s talking about the memory wall right now. Memory is getting super expensive. There’s not enough memory. Smartphone volume will go down 30% because there’s not enough memory. This is shocking, Dylan said hyperscalers are spending 50% of their CapEx this year on memory.

Reiner Pope

That’s believable.

Dwarkesh Patel

What is hyperscaler CapEx? That’s high hundreds of billions, maybe a trillion, and they’re spending half of that on memory? That is a huge constraint. That’s why we’re not going to get new laptops and phones this year.

But at the same time, we have too much memory? People are willing to put too much memory into these systems. Why is Jensen shoving all this memory into these racks if you don’t need it?

Reiner Pope

In the equations we had here before we erased them, we were doing memory time, memory bandwidth and compute bandwidth. Let’s now start looking at memory capacity.

We’ll start off with memory capacity without even thinking about a parallelism scheme. The demand on memory is the number of total parameters. This is what we need to fit the weights in some system that we are using. Then we need to fit the KVs as well. KVs go as batch size times the length of the context times the bytes per token.

What I was arguing about in this context, and the case I was making for pipelining, is that there are some techniques that allow us to solve this. Let’s consider running this on some number of GPUs. We’re going to have one extent, which is E, the expert parallelism. When we had this sharding of an expert layer across many GPUs, to what extent do we do that? How many GPUs? We’re going to say that this is, for example 64. Then P is going to be the extent of pipelining. This is the number of racks, maybe we’ll pick 4 or something like that.

This is the total memory requirement across the system, but now I’m going to calculate a memory requirement per GPU. I’ll use a lowercase c_mem. Obviously, we just take all of these numbers and divide it by E and P. Really easy. It’s this N_total, plus the batch times length of context times bytes per token, all divided by E times P.

Why is this correct as divided this way? We knew that the parameters were perfectly divided amongst all the GPUs in a rack. The layers are perfectly divided amongst the different racks. So that works here. Somehow we’re going to arrange—I’ll hand-wave exactly how—the same perfect sharding of the contexts across GPUs in a rack, and then based on layer across racks.

Dwarkesh Patel

Sorry, 4 is the number of racks?

Reiner Pope

Yeah, for example.

This is the place where we actually need to go back and analyze this batch size B. You were making this comment that there’s micro-batching versus global batching. Let’s come back to this pipelining diagram here.

We’ve got one batch going forward here, and then as I drew it, it kind of just disappeared. That’s not really correct. If you think about how decode is working, I have a bunch of tokens that I have generated already. I do one forwards pass where I generate a new token, and then I write that to my KV cache. Then I do another forwards pass that generates the next token. I’m actually going to be running this batch zero in a loop. In fact, I go forwards. Once I finish, I can start the next iteration of the loop up here. We’ll just fill this in. We’ve got the two, three, two and three, and two and three.

Let’s split this batch. This batch will be the global batch size. B is going to be the number of micro-batches times the batch size per micro-batch. How many micro-batches do we need? The number of micro-batches in this diagram is 4: zero, one, two, three. The micro-batch size is still this 2000-ish number. Sorry, no, this is the 300 times sparsity.

Dwarkesh Patel

This is how big the train that takes off every 20 milliseconds is.

Reiner Pope

Right. This is going to be the 20-millisecond train. The global batch size is the number of micro-batches times the local batch size. Local batch size is set by this hardware parameter.

The number of micro-batches is as small as possible, such that we can wrap around and not leave any idle time. If we had fewer, we would have this idle time when we wrap around. You can visually see that it is equal to the number of pipeline stages. It’s a proof by visual here. It is 4, and it’s 4 this way as well. You can look and see that it goes along here, and then it wraps around to the number of pipeline stages.

Dwarkesh Patel

Sorry, very basic question. Is this what is actually done? A frontier model today will have pipelining during inference?

Reiner Pope

For sure during massive scale training this is done. It can be done for inference. I’m actually going to make the case for why it is less attractive. It is useful for weights, but not so useful for KVs.

The big challenge is... Let’s fill this in. The micro-batch size here ends up being equal to the number of pipeline stages. When we go back and substitute all of that into here, we get a number of pipeline stages times this little b showing up in here. When we factor this out, I’m going to split this plus into two terms.

We get the full division by E times P over here. We still have division by E times P over here, but the Ps cancel. What we find is that if you increase the number of pipeline stages, the memory footprint for the number of weights keeps going down and down and down, but the memory footprint for the number of activations stays constant. So it doesn’t actually work.

Most of your memory… Once you do enough pipelining—and it’s really not much, even two is often enough—this term becomes very small. The KV cache becomes the dominant term.

Dwarkesh Patel

I know this is wrong. I’m just trying to think about why my train of logic here is wrong. If you’re pipelining through many different stages, the KV values are not shared between layers. Why would it not help to be pipelining across multiple layers? Because then you don’t have to store...

Reiner Pope

You only need to store one layer rather than two layers of KVs. It helps from that perspective, you’re right. What’s competing with that, though, is that you need to be keeping all of the racks usefully busy at a time, so the number of sequences that are in flight simultaneously has gone up.

Dwarkesh Patel

Ah, that makes sense.

Reiner Pope

Those exactly cancel, and you end up not getting a saving per GPU.

Dwarkesh Patel

Right. This is going back fundamentally to the point of how you’re not able to amortize across KV caches.

Reiner Pope

First, we established you can’t amortize KV caches across batch size. Now we’re saying you also can’t shard it across pipeline stages. It sucks from both of those points of view.

Dwarkesh Patel

Interesting. So then what is done during inference?

Reiner Pope

The DeepSeek paper reports what they do, which is that they just do a lot of expert parallelism. In effect, you should increase your expert parallelism up to your scale-up domain size, and then do very little pipelining. Maybe none at all, maybe two, just enough to make the weight storage not too big of an issue.

Those are the only two parallelisms that really make sense. In the past, there was tensor parallelism, which was cutting up within an expert, but the experts are so small now that that is not a profitable optimization.

Dwarkesh Patel

Does that mean that frontier labs, when they’re doing inference, are just within a single scale-up?

Reiner Pope

Yes. You can look at how it depends on model size. You could have a very large model, one that exceeds the memory of a rack. There you should be doing a bit of pipelining. Maybe it’s extremely sparse, for example, and that would be a reason to do it.

Dwarkesh Patel

This goes back to the promise at the beginning of the lecture, which was this will actually tell you about AI progress as well. To the extent it is the case that model size scaling has been slow until recently…

Let me make sure I understand the claim. The claim would not be you could have trained across more racks. It was just that it would not have made sense before, we didn’t have the ability to do inference for a bigger model easily.

Reiner Pope

Actually, pipelining doesn’t help with context length. It totally helps with model size. Because of the ability to do pipelining, a rack at least should not be a constraint on your ability to fit the model parameters.

The other consideration you’re asking is, why hasn’t it scaled up more, and why did bigger scale-up domains help? We talked through one aspect of that, which is that it’s not because of memory capacity. We have a solution to the memory capacity at least with respect to model size, not with respect to KV cache size but at least with respect to model size. The other issue that shows up is latency.

Dwarkesh Patel

I was just about to ask, going from rack to rack, what is the latency cost per hop?

Reiner Pope

This is very much dependent on the hardware. I can’t say with a lot of authority. I think it’s probably on the order of a few milliseconds, but it could be off by an order of magnitude there.

Dwarkesh Patel

Is 4 a realistic number of how many pipelining stages you might have?

Reiner Pope

Yes.

Dwarkesh Patel

So that’s not that much.

Reiner Pope

On a small number of pipelining stages, this is not a huge latency impact.

Dwarkesh Patel

But I guess it’s 10 milliseconds per token.

Reiner Pope

That’s right.

Dwarkesh Patel

2 times 4-ish, or I don’t know how many you said… 10 milliseconds per token is actually a lot.

Reiner Pope

If it goes from 20 to 30, or something like that…

Just to chart the path that it goes through, here you’re going from your GPU or TPU to a network card, which then goes to a top-of-rack switch, and then hops over to the other rack and does the same thing in reverse. You have to sum up the latencies of these different things.

Dwarkesh Patel

Sorry, is this the same thing as the data center switch?

Reiner Pope

It may in fact go up to a data center switch and back. It depends on deployment configuration.

Dwarkesh Patel

Got it. And because it’s decode and sequential, they stack up across the stages. You can’t do them at the same time.

Reiner Pope

That’s right.

Dwarkesh Patel

This brings us back to the question then, is the size of the scale-up at all relevant to why AI model sizes have been what they have been over the last few years, whether through training or through inference?

Reiner Pope

We talked about latency of the hop. There is also just the t_mem latency. The memory time latency is actually massively improved by larger scale-up domains. I’ll recall t_mem down here. t_mem for the weights was equal to the number of total parameters divided by the memory bandwidth. Which memory bandwidth are we talking about here? Is it just one GPU? It is the number of GPUs that I can use in parallel to load these weights. I can’t use different pipeline stages in parallel because they’re not running at the same time, but I can use all the GPUs in my scale-up domain in parallel to load the weights. This is actually extremely effective. Basically, I end up with a term here, this memory bandwidth term itself is equal to scale-up size...

Dwarkesh Patel

Times memory bandwidth per GPU.

Reiner Pope

Yeah. Times GPU bandwidth. This term doesn’t increase a lot. It maybe increases 1.5 or 2x per generation, but this one increased by a factor of 8 from Hopper.

Dwarkesh Patel

So the reason the bigger scale-up matters, it’s not the memory capacity of the whole scale-up, but really the memory bandwidth.

Reiner Pope

Yeah. Pipelining totally solves the capacity problem, but scale-up size helps solve the bandwidth problem.

Dwarkesh Patel

And the bandwidth problem helps you do longer context lengths, which is more and more relevant as these models get more agentic.

Reiner Pope

It lets you just run the model at lower latency as a first thing. If I just do a very sparse model and it’s on a little H100 box, the latency will be really high.

01:18:59 – Because of RL, models may be over-trained 100x beyond Chinchilla-optimal

Dwarkesh Patel

A super tangential question. There’s Chinchilla scaling, which tells you how big a model should be relative to the amount of data you’re going to train it on. But now, obviously, you’re not just trying to optimize for the highest quality model you could get with training compute. You want the best results a user can get with a mixture of training and inference compute.

So there’s a question of how much you should over-train a model such that compute amortized over training and inference is minimized to get a certain performance. But now with RL, there’s another consideration which is, you’re going to do some amount of pre-training. That pre-training will be used both for RL generation and then for inference for the final user. By over-training here I mean that while it would have been more efficient just from a training compute perspective to have a bigger model that you train for less time because it can learn faster, maybe you get a smaller model, spend more compute training it than you otherwise would have, but now it’s cheaper to give it to users.

Let me make the question more concrete. Basically, how much more than Chinchilla optimal are models over-trained? And has that changed as a result of RL generation?

Reiner Pope

This is a place where we have to do a bit of guesswork because the updated scaling laws and the model traffic are not reported, so we have to guess there. One way to look at it…

Let me first just make a general heuristic claim. If I have some cost, and I’ve got a total cost which is a sum of cost A and cost B, like maybe this is the training cost and this is the inference cost, and I want to minimize this sum…

For many curves, the minimum tends to be where the costs are equalized. That’s something of a heuristic claim, but there are many examples where it’s true. Where one is 1/x and the other one is x, for example, they tend to be minimized at the point where they equal each other. It’s also true for e^x and e^-x and all kinds of other things. Basically, I’ve got some curve that’s going down, some other curve that’s going up, and they tend to be minimized at this equal point.

Heuristically, I will conjecture that that is true for the setup you described as well. Actually showing that would be true would require looking at the scaling laws and fitting these weird exponents, but things that follow power laws tend to have this property. So I’ll just make that claim and move on.

We’re going to say that we want to equalize the cost of training and the cost of inference. We can do all of it in general. The cost of pre-training, that’s the number of active params times the data on pre-training. There’s a factor of 6 out here, which is the number of FLOPs. There’s the famous 6ND formula. Then in RL, we have approximately the same thing. We’ve got the same number of active parameters, but now the amount of data is the RL data. There is this extra efficiency multiplier, or inefficiency...

Dwarkesh Patel

Which is the fact that you’re not training on all your rollouts.

Reiner Pope

Well, there’s that, and then the other, perhaps even bigger inefficiency is that this involves a substantial amount of decode. Often decode runs at less MFU than training.

Dwarkesh Patel

Okay. So if you’re doing a backward pass on every single generation in RL, it would be 6ND.

Reiner Pope

So this could be a smaller number, right?

Dwarkesh Patel

It would at least be two, because that’s the lower...

Reiner Pope

Somewhere in the range of two to six. We’ll say somewhere in the range of two to six and leave it at that. Then we can add in the inference cost. The inference cost is two, the number of active parameters times the data in inference.

Dwarkesh Patel

Sorry, I think the way I said it was super garbled. Just for the audience, forward plus backwards per parameter is 6. Forward alone is 2. That’s why RL, where you’re definitely going to generate all the trajectories but you might or might not train all the trajectories, is 2 to 6.

Reiner Pope

Yes. Thank you. And then inference is just 2. We’re going to solve for essentially equality of all three of these terms. That is the ballpark of where people are going to be. Labs have more information on what is productive in doing more RL, for example, versus doing more pre-training. I don’t have that information, but I think a good ballpark is a 33% split between each of them.

Dwarkesh Patel

I’m not sure I understand the intuition for that. Another naive model could have been that RL plus pre-training would be 50% and inference would be 50%.

Reiner Pope

That’s also a valid answer. Because this is heuristic, I can’t really argue for one versus the other. They don’t differ by that much. Thirty-three versus twenty-five is only a small factor off.

Let’s pick one of them. All equal seems simple enough, so we’re just going to solve for equality of them. It’s pretty straightforward. We can immediately see that the number of activated parameters totally disappears, so let’s factor that out. We’re going to just say that data in pre-training—I decided to do it your way, it’s a little bit nicer—plus... Oh, I didn’t have the inefficiency over here either. Data in pre-training plus some multiple of α times the data in RL is going to end up equal to some β times the data in inference.

Let’s just roughly size the α. This α is maybe somewhere in the range of 2 to 6. Over 6, from this term compared to this term. And then we’ve got an inefficiency term, which I would say is maybe in the range of 30%. So this alpha is going to be something like 1/10. And this β here is actually the same. It’s a third. It’s one third times 30%. So it also equals 1/10.

Dwarkesh Patel

If both of them are one in ten, that kind of implies that there’s never a backward pass on RL?

Reiner Pope

Yeah. Okay, we can make this 2/10. Make it a bit bigger. Just write it out once more, this is 2/10, this is 1/10.

The number of inference tokens you have is just a function of hundreds of millions of tokens per second times my model is deployed for two months before I ship to the next version. That should determine the number of tokens in RL and pre-training.

I guess we didn’t do the equivalence between pre-training and RL, so we’ll do that here. Data in pre-training should be equal to 2/10 data in RL for them to be cost equivalent. Sorry, 1/10. I got it backwards. We pay more cost when it’s inefficient, so this needs to be 1/10. Tracing this back… This thing ends up actually being, as written here… This is like 1.5, and this is one.

Dwarkesh Patel

Billions of dollars worth of compute just flowed in the other direction.

Reiner Pope

Right? I think if you do it with a spreadsheet and actually model it out, you might notice when the money’s going down the drain. All of these end up being close in, as modeled here. This 30% may have been a little bit too generous. So let’s say something like 1.5 here, and leave this as a one here.

I think at this point, you can almost read it off. The number of inference tokens should be about the same as the number of pre-training tokens, which should be about the same as the number of RL tokens, within factors that we’re not able to reason about.

Dwarkesh Patel

Sorry for making a basic algebra mistake. It seems like there should be fewer RL tokens than pre-training tokens?

Reiner Pope

That’s in general right. Because RL is less efficient in terms of machine time, if you’re trying to equalize the RL and pre-training time, then you should have fewer tokens in order to have the same wall time.

Dwarkesh Patel

This is all quite interesting. I never thought about it in terms of equalizing data.

Reiner Pope

I think starting with equalizing in cost is right, but depending on how you model the cost, this comes close to equalizing in data.

Dwarkesh Patel

So for GPT to be trained optimally, every single user who uses GPT-5, the total amount of tokens that they stream should equal the total amount that has gone into pre-training. And the total amount of tokens that have gone into pre-training is the sum of all human knowledge. Each model should generate the sum of human knowledge on the output that it gets on the input.

Reiner Pope

Yeah. Which way are people going to err? If you think that people’s power of prediction is not perfect, and also you run the risk that you make a model that is not a frontier model and then you just throw it away, then that changes the cost trade-off because there’s some probability that applies to the inference. And you should derate the inference tokens by some amount.

Dwarkesh Patel

Right. Can we back out how much more compute than Chinchilla optimal for a given sized model?

Reiner Pope

I think we just have to make some real-world assumptions here in order to do that.

The inference tokens, we should totally be able to count, right? Let’s say a few hundred million. Maybe it’s five hundred million tokens a second now, I don’t really know. Five hundred million tokens a second times. A model is deployed for two months before it becomes obsolete?

I can’t do this in my head. Can you type it into a computer?

Dwarkesh Patel

2.6 x 10¹⁵.

Reiner Pope

Okay. 2.6 x 10¹⁵. This number is probably too large because this is going to be multiple models in a family. Let’s make it 5x smaller or 10x smaller or something like that. So we’re estimating maybe fifty million tokens per second, per specific model. The model is live for two months. This comes out to around two hundred trillion tokens. And then we want to compare that to active parameters on a frontier model. I don’t actually know the latest rumors. Do you know?

Dwarkesh Patel

Somebody told me a hundred and fifty trillion.

Reiner Pope

Active parameters?

Dwarkesh Patel

Sorry, I meant tokens.

Reiner Pope

Trained on a hundred and fifty trillion tokens. Interesting.

Dwarkesh Patel

Which is similar.

Reiner Pope

That’s actually similar. So data on pre-training.

Dwarkesh Patel

This is not well-cited but it’s fine.

Reiner Pope

I think often the number of active parameters could be in the range of a hundred billion, something like that. Maybe a bit larger. So multiply by 20 to get the Chinchilla token count. So Chinchilla, D_Chinchilla, would be around two trillion. We see we’re about a hundred times larger than that.

Dwarkesh Patel

What does D_Chinchilla actually mean?

Reiner Pope

The token count for pre-training that the Chinchilla scaling law would recommend, I guess.

Dwarkesh Patel

Oh, I see. So how much is it over-trained? Got it.

Reiner Pope

The ratio of this two hundred trillion or a hundred trillion parameters over the Chinchilla optimal of two trillion, that’s the amount it’s over-trained. Which is a factor of a hundred over-trained.

Dwarkesh Patel

A hundred. So if you consider this right here, to the extent this is in the right ballpark, just by thinking about how you want everything to be equal in terms of compute… If OpenAI also realizes that and they’re serving a certain amount of tokens per second, that tells you how much data went into the pre-training of GPT-5. Even if it’s 50% off or something, it is wild that you can first-principles these kinds of numbers.

Reiner Pope

This is why you should just approximate everywhere, because there are big error bars on this. But it’s kind of empowering to just set A equal to B and figure it out.

01:33:02 – Deducing long context memory costs from API pricing

Dwarkesh Patel

That’s super cool. Okay, so in the spirit of trying to deduce things, we can publicly look up the API prices of these models, and maybe we can learn something from that.

First, with longer context, Gemini 3.1 is 50% more expensive if you go over 200k tokens than if you’re below 200k tokens. At a high level, I understand why that might be, but why specifically 50%?

Reiner Pope

Why specifically 50%? The high level, even in the first place, is that there is some amount of increasing cost with context length. We can bring that back up. That was the memory time versus the compute time.

We’ve put up these same equations from before, of the time for memory fetches which is the weights and the KV cache, and then the time for the compute which is just the matrix multiplications for the weights. I will also draw the cost curve, but this time I’ll do it as a function of context length instead of batch size. So this is the cost curve as a function of context length. We’ll draw the compute. The cost of the compute is actually constant as a function of context length. There’s no dependence here on context length. In reality, there is some dependence, but it is very mild, so we’ll ignore it. So this is the time for the compute.

Then we’ll also draw the dependence of the memory fetch on context length. This starts at a large number for the weights and then grows gradually with the context length. Maybe starting here, and then grow gradually with context length. And so, you take the maximum and you see there is this inflection point here.

So this is the cost that Gemini might be paying. And then you think, how might you put a pricing structure on top of that? You would like to ensure that no matter what the context length is, you are still profitable. So we’ve got a two-tier pricing structure. Maybe we’ve got something that looks like this up to some extent.

I think it says something about, given that the bump is at 200k, it probably means that this is somewhat aligned with this crossover point. Maybe not exactly aligned with it. We can actually probably even complete that calculation just to see where it lands out.

We can solve for the number of bytes per token if we make some assumptions about the number of active parameters. So solving for the number of bytes per token, we’re going to assume the point where we equalize the time of memory and the time of compute is at, let’s say, 200k tokens. So we equalize these two.

We’re also going to assume that the batch size is large enough that the memory time spent on weights is negligible. So we’ll forget about this, and we’ll focus on the actual memory time spent on KV cache. That ends up saying, copying this term over, batch times length of context times bytes per token over memory bandwidth is going to be equal to the number of activated params over FLOPs. And then we’re going to solve for bytes per token. Batch size was missing here. It shows up here, and then it cancels out by the time we get to here. And I dropped the length of context.

So we can plug in numbers. This is the reciprocal of the number that we saw before. This is 1/300, which is reasonably stable across many different hardware platforms. We conjecturally said that maybe the number of activated parameters is a hundred billion. The length of the context we said was 200k. Something is wrong here, though. Length of the context should be on the denominator, not the numerator.

Dwarkesh Patel

1667. Almost two kilobytes.

Reiner Pope

That is plausible, actually. You said around two kilobytes. Let’s just do a sanity check for what this could be. There are two mechanisms that people do attention with a small number of bytes per token. One is dense attention with a lot of reuse across layers. Character AI has a blog post talking about that, alternating long and short context. In the Character AI kind of model, which also showed up in the Gemma models, the global context—which is really what we’re talking about here—was shared across all the layers.

To get this to kilobytes, you could get that, for example, as a d_head of 128, which is typical. Then the number of bytes is typically the number of attention layers times two times d_head times the number of KV heads. This is the number of unique contexts per layer.

Do you share the context across many layers, or do you use it only once? In the Character AI-like models, this number is one. We said this is 128. This is a choice which typically ranges from one... Sorry, this is KV heads, I meant.

Dwarkesh Patel

The difference between a head and a KV head is that…?

Reiner Pope

The KV heads are the heads that are stored in memory, store the contents of the previous tokens. The Q heads are the retrieval heads. They’re only used temporarily and they’re used by the attending token. In this autoregressive context, I’ve got KV heads associated with all of the contexts, and then Q heads associated with this new token here.

Dwarkesh Patel

But this head, the 128.

Reiner Pope

Oh, sorry. This d-head is the dimension of the vector. The number of KV heads is typically in the range of 1 to 8. It is totally plausible to get this by, for example, having 8 KV heads and a d-head of 128. That gives you exactly this number. Or you could have fewer KV heads, but more layers.

This is one way to get there via dense attention. There’s also a way to get there via sparse attention, where you increase all of these numbers, but then you have a 1/sparsity term. I think this number is plausible, if maybe a little bit small.

Dwarkesh Patel

It’s funny that they would leak so much information through their API pricing.

Reiner Pope

I mean, you are incentivized to price close to your costs because otherwise someone could scoop you.

Dwarkesh Patel

Maybe we can learn something about the difference in input versus output prices, and what that tells us about decode versus prefill in these models. I think last I checked it’s 50% more expensive or something like that?

Reiner Pope

I don’t remember. What I’ve seen in the past is 3-5x more expensive.

Dwarkesh Patel

Okay, that makes more sense. So let’s say it’s 5x more expensive. This is the compute to process the next token in decode. Suppose you’re doing prefill, where you’re not just processing the most recent token, you’re processing all the tokens in parallel. I want to say that it would be this times length prefill?

Reiner Pope

Or length of the pass in general. If we can think of decode as being a pass with one, and then prefill being a pass with many.

Dwarkesh Patel

Okay. So maybe prefix? Okay, memory. You’re not storing the KV cache for the tokens that are the prefill tokens.

Reiner Pope

Let’s actually draw how prefill shows up here, if I may clarify. We do a bit of decode like this. We may actually come back and do more prefill. If you think this is a chat session, the user says something, the AI generates a response, and then the user says something else and we prefill this. Maybe this is the general case, rather than this.

Dwarkesh Patel

In fact, this is like you read a file or something.

Reiner Pope

Read a file or the AI is responding to a user input, tool call, or anything that’s not AI-generated.

Dwarkesh Patel

Okay, suppose we’re here. You will have calculated all of this previously. So just the KV of everything that came before. But what is the memory cost of this? Well, the memory bandwidth cost of this. If you’re doing flash attention, it would—

Reiner Pope

It’s basically temporary. It doesn’t even go to main memory. Just ignore that.

Dwarkesh Patel

Exactly. So then it would just be everything that came before. Is it not just that then?

Reiner Pope

There’s actually no adjustment at all to the memory time.

Dwarkesh Patel

Okay. Great. So it’s a very trivial change to accommodate. This term is making it 5x more expensive. Now, why would that be? What does that actually tell us? What variable does this help us clamp? The only thing that could have changed is that the compute is 5x more expensive as a result.

Reiner Pope

This is the time for one pass, but actually the amount of tokens is that much larger. We want the cost per token, in fact, or the time per token.

Dwarkesh Patel

I’m not sure I understood. This is for processing the next token in prefix?

Reiner Pope

Well, actually for processing the entire batch. At this cost, we have processed this many tokens, the length of prefill. Or I guess the length of the pass. Not this prefix, but it’s this cost.

Dwarkesh Patel

Okay. Let’s just do this pass. So this is 5x more expensive. Input is 5x more expensive.

Reiner Pope

Output is more expensive, in fact.

Dwarkesh Patel

Output is 5x more expensive.

Reiner Pope

The result we want to work towards is that prefill is compute-limited and decode is memory bandwidth-limited.

Dwarkesh Patel

Why don’t we do this? Why don’t we just chart it with len-pass on the X-axis and t on the Y-axis.

Reiner Pope

We want the cost per token, so it’ll be t over length of the pass. That’ll be right.

Dwarkesh Patel

I guess I’m getting confused by this. Len-pass is... It seems like this should be higher when you’re doing prefill.

Reiner Pope

Prefill has a bigger length pass. Yeah.

Dwarkesh Patel

But then why is it cheaper?

Reiner Pope

Why is the cost higher? It’s this division by length pass. This is going to divide out, but then all of this is going to divide by length of pass, and it’s going to make the memory costs cheaper.

Dwarkesh Patel

Okay. Let me think about this then. Basically we’ll have four different lines. Let’s do prefill first... Actually, let’s do decode first.

Reiner Pope

Length of the pass, when it’s one, that is decode. When it is bigger, that is prefill.

Dwarkesh Patel

Oh, okay. I see. That makes sense. Getting back to it. So t_compute, if you have basically just this divided by len-pass, so just this amount. This actually does not vary based on t, so it’ll just be some flat value like this. And this is t_compute. And this is—

Reiner Pope

That’s decode.

Dwarkesh Patel

Decode. Right. Now t_mem, we have this whole thing divided by len-pass. Well, it doesn’t really matter what’s up there, it’ll just be something that looks like this. Let’s say this is t_mem. This is decode again.

So as the length of the prefix goes up, or pass, your memory bandwidth time declines, and that means that to the extent that you were bottlenecked on memory bandwidth before, you can avoid being bottlenecked on memory bandwidth.

The fact that they are charging 5x less for prefill than decode does suggest that they are bottlenecked on memory bandwidth to quite a degree, such that for them at least—because t is equivalent to cost, it’s the cost of renting a compute—this would be at 1, and this would be at 5.

Reiner Pope

That’s right.

Dwarkesh Patel

So it is, in fact, tremendously memory bandwidth bottlenecked. The real graph looks something like that.

Reiner Pope

It still crosses, but yeah.

Dwarkesh Patel

Exactly. Let me do it this way. This is the gap on decode between the memory and the compute time. Okay, interesting.

Another interesting one would be why cache hits are so much cheaper. If I remember correctly, cache hits are like 10x… It’s more expensive to write to cache according to the pricing on all these models. But if you do hit a cache, it’s 10x. Presumably, this is the cost of keeping something in HBM rather than just evacuating it. But if you do keep it in HBM, then it’s cheaper to load again?

Reiner Pope

Right. There are two ways you can produce the KV cache for a token. You can just produce it from scratch by computing it from the underlying token IDs, which are tiny. Or you can previously have produced it and stored it in a memory somewhere.

The cost ratio is really talking about the ratio between those two mechanisms of producing it. A cache miss means you’ve deleted it from all your memories, and you have to recompute it from the tokens directly. You can even take that a step further and think about which memory tier you store it in. You could store it in HBM. There are other slower and cheaper memories than HBM, like DDR on your host or flash as well. One of the things you can do is a calculation of where it makes sense to be in each memory tier, and this is related to how long you’re going to store it for.

We want to look at the cost of storage in a few different memory tiers and also the cost of rematerialization. Remat means the cost to rebuild all of the KV cache from scratch after you deleted it, so we rematerialize it. Basically, this is going to cost the length of the context. Actually, we’ll look at the cost per token, so we don’t need to carry around this length of context everywhere.

To rematerialize one token of KV cache, I just need to run a forward pass on the whole model. This is going to be the compute time. I have to rerun the compute at whatever speed my GPU does it, and then I multiply it by my GPU dollars per second.

Dwarkesh Patel

Sorry, excuse a naive question. Why is there not a quadratic term?

Reiner Pope

There is a quadratic term. It shows up in the compute. As an approximation, I chose to remove it. I’ll just show you quickly what that looks like. If you look at the cost per token, or the number of FLOPs per token, there are the FLOPs that are coming from doing the weight matrix multiplies as a function of—

Dwarkesh Patel

Which is flat.

Reiner Pope

...context length. And then there is the number of multiplies that comes from doing the KV cache, which goes up linearly with the amount of stuff you attend to. The slope on this is so low that when you draw it like this, it’s very well approximated by a flat line. You start to notice the effect of the quadratic or the linear term up in the millions of tokens or so. So it’s just not super relevant.

Dwarkesh Patel

So what is the reason that there’s no company which has over a million token context length, if this is true?

Reiner Pope

There are two costs of long context. One is the memory bandwidth cost, which we’ve spent a lot of time analyzing. That’s this thing. The other one is the compute cost. The compute cost is almost always forced by fundamental principles to be a much smaller slope than the memory bandwidth cost. The primary things that limit you to really large contexts are memory bandwidth and memory capacity, which is exactly this effect.

Dwarkesh Patel

There’s this idea that Dario said on the podcast, and others have said, which is, “We don’t need continual learning for AGI, in-context learning is enough.” If you believe that, then you have to think that we have to get to a hundred-million-token context length to have an employee that is the equivalent of working with you for a month. Now, maybe that’s no longer true with sparse attention or something. But if you think that, then some ML infra thing would have to change to allow for a hundred million, like the memory bandwidth, to allow for a hundred-million-token context lengths.

Reiner Pope

Sparse attention gives you a get-out for sure, because you get this square root. It gives you a big improvement. But if you look at the history of context lengths of models, from earlier models like GPT-3, maybe to GPT-4—I don’t remember when the transition happened exactly—they shot up from about 8K to 100-200K. And then for the last year or two, they’ve all been hovering around there. I think that indicates that this is the reasonably balanced cost point, and going massively beyond that would be cost-prohibitive.

Dwarkesh Patel

Not because of the compute cost, because of the memory bandwidth...

Reiner Pope

Because of memory bandwidth cost, yeah. I actually don’t see a very good path to solving that. The HBM is where it is. It’s not getting hugely better.

Dwarkesh Patel

And why doesn’t sparse attention solve it?

Reiner Pope

Sparse attention is a big improvement. Maybe that is priced in already, perhaps. It’s not an infinite improvement because if you go too sparse, you lose too much quality.

The empirical result is that the context lengths haven’t been increasing that much. I think it’s because there is no solution to the memory wall here. Going too sparse just means you’re attending to a very small subset of the tokens, and the quality will get worse.

Dwarkesh Patel

Makes sense.

Reiner Pope

What is the cost of these different ways of resynthesizing the KV cache? Computing it from scratch is based on my GPU time. I have to do a certain amount of multiplies, of GPU time that I spend in order to produce it.

Storing in HBM. This really goes as my bytes per token. I need to just have some number of bytes per token, and then I need to store this in the HBM. It’s going to use up some of my HBM capacity. A way to think of this is that if I have too many of these things sitting in my HBM, if I fill up my HBM with just KV caches that I’m not using, I can’t use that GPU.

How do I price that? Maybe I say that the cost of it is proportional to the fraction of the HBM I’m using. There’s also times GPU dollars. Let’s just do one more memory tier and say store in DDR instead. The same kind of thing goes up for flash and for DDR.

I put these in the wrong columns. I meant to make two columns. The distinction I want to make is that there is the cost to retrieve, and then there’s a cost to hold on. This is a cost per second, whereas this is an instantaneous cost. Rematerialization has a cost to retrieve and has zero cost to store it because we’ve deleted it. This is the one that I put in the wrong location. This is actually the cost just to hold on, so I will rewrite it.

If we’re just storing it in HBM, it has this sort of cost profile. If we store in DDR, it’s actually going to take some time. We get the same thing here: bytes per token over DDR capacity times DDR cost per second. But now this has a cost to retrieve that is higher than the HBM because we need to copy it into the HBM. So this is bytes per token over DDR bandwidth. And then this consumes some amount of the DDR as well.

Dwarkesh Patel

And every scale-up has DDR and flash?

Reiner Pope

This is really a deployment question, so you can choose that. Nvidia does deploy in this form. It has both.

Dwarkesh Patel

Why isn’t the cost to retrieve HBM the bytes divided by memory bandwidth?

Reiner Pope

It depends what you define a retrieve to be. Here, I’m defining retrieve to be, move it into HBM so that you can start actually doing inference on it.

Dwarkesh Patel

Because if it’s already in HBM, you can be doing compute while you’re getting it from HBM to SRAM? Interesting.

Reiner Pope

Yeah, for example. These are three things, and I guess I ordered them wrong. In general, if you’re balancing two costs and you’ve got different tiers in the memory hierarchy, you should expect as this cost goes up, this cost should go down. You can kind of see where the zeros are. I should have ordered them this one first, this one second, and this one third.

If you’re going to hold onto it for a very short amount of time, then all of this is multiplied by the hold time. This one is, and so is this one.

Dwarkesh Patel

Interestingly, they have different prices to write for. Do you specify this in the API for five minutes versus an hour? Which suggests that the five minutes is HBM and the hour is DDR.

Reiner Pope

I think that’s a pretty good assumption. If you look at the numbers, it might also turn out that it’s one tier down, and it’s DDR versus flash.

Dwarkesh Patel

Interesting. I’ll look up the price difference. The base input tokens is $5 per million tokens.

Reiner Pope

Base, which means remat. This is $5.

Dwarkesh Patel

That’s $5 to “retrieve”. And then to write, presumably HBM, for five minutes is 6.25.

Reiner Pope

We might be able to determine which memory tier it is by the durations.

Dwarkesh Patel

Five minutes versus one hour.

Reiner Pope

Exactly. I think this will probably end up being the drain time of the memory tier that you’re in. What that means is, given that I know I’m going to be holding something for five minutes, I would like to pick a memory that I can read every five minutes. I can read the whole memory once per five minutes, ballpark. That is the drain time of the memory. So if I take the storage capacity over storage bandwidth, I would like this to be equal to five minutes.

We did this calculation for HBM. For HBM, we know that this number is 20 milliseconds. So HBM is much too small. DDR could be about an order of magnitude or two off from this, so this is probably on the order of seconds, like 1 to 10 seconds. I don’t have these numbers memorized, but generally, as you go to slower tiers, flash is plausibly on the order of one minute. And then spinning disk, which is massively different, is on the order of one hour. So this might actually identify the tiers of flash and spinning disk.

Dwarkesh Patel

Sorry, why is this the calculation? This is the storage capacity divided by the bandwidth?

Reiner Pope

You’ve got a bunch of different memory tiers, we’ve listed four of them. Your choice of which memory tier is about minimizing the cost. What fraction of the device are you using? You’re using some fraction of the device for holding onto it, and then you’re using some fraction of the device to retrieve it. Let’s say I’m using 10% of the device. And I want to equalize those two fractions. That’s a sign that I’ve hit the right thing.

Let’s say I’ve got some runtime here. I’m going to hold on for all of this time, so this is the time-hold. And then there’s going to be some amount of time here, which is time-retrieve. Basically to equalize these two costs, I want the retrieval time to be equal to the hold time times the fraction of capacity. Because this is the retrieval time, this is how many other things I can hold simultaneously.

Dwarkesh Patel

Basically, you want to store things in there for so long such that the amount of time it’s in there is the time to get all your things in there and out.

Reiner Pope

Yeah basically. I think that probably indicates that the two tiers are flash and spinning disk. I’m kind of shocked to see spinning disk being used at all, because it’s such an old technology.

Dwarkesh Patel

Interesting. It’s also crazy that it’s so slow that it takes an hour to load its full capacity to it in.

Reiner Pope

It’s a really unattractive technology but it’s useful in some places.

02:04:02 – Convergent evolution between neural nets and cryptography

Dwarkesh Patel

We’re sitting down because I want to ask you some questions that don’t need a blackboard. You have this extremely interesting blog post where you talk about how, at a high level, the architecture of different cryptographic protocols looks a lot like neural networks. There’s this convergent evolution where they both need to jumble information across all their inputs. For cryptographic protocols, it’s to make sure that each new input into a hash function will totally scramble what happens. For neural networks, of course, they need to consider how this piece of information changes what you should make of this other piece of information.

I thought that was an extremely interesting point. At a high level, in some sense they’re trying to do the inverse thing. Cryptographic protocols are trying to take information which has structure and make it look indistinguishable from randomness. Neural networks are trying to take things which look random—protein sequences, DNA, garbled text—and extract higher-level structure from it. They have similar high-level mechanisms, but they’re actually trying to do the opposite things. I wonder what you make of that.

Reiner Pope

I try to look for other examples where mixing and scrambling shows up as well. There’s almost a physical example where you’re making a cake and you want to stir the batter. Literally the idea to first stir it this way and then stir it this way is not too bad of an approach.

Beyond that, back to the digital world, there are some differences, and the one you call out is a pretty strong difference. The way it shows up, if you just randomly initialize a neural network, maybe it’s a reasonable cipher as well because the random initialization is going to jumble stuff in a complicated way. It may even do what you want. Who knows?

The thing that makes it interpretable is the gradient descent. You can differentiate a neural network and get a meaningful derivative. We do a lot of work to not overcomplicate the derivative, so the residual connection keeps it contained and simple. And so does the LayerNorm stuff that we do.

One of the biggest attacks against cryptographic ciphers is also to differentiate the cipher. Ciphers run in a different number field. They run in the field of two elements, so just binary, whereas neural nets run, in theory, in the field of real numbers. You have to differentiate with respect to binary numbers, but you can absolutely differentiate a cipher. This is called differential cryptanalysis.

Basically, what it says is that if you take a small difference of the input, it’s quite difficult to make the difference of the output be small. The whole job of a well-designed cipher is to make the difference in output very large. The distinction is that the optimization goals at that point are about complexifying. They don’t have the same residual connections, like LayerNorms.

Dwarkesh Patel

I guess a place where the two merge is backdoors. With a backdoor in an LLM, you’re trying to hide… Would you consider it an input? It’s not an input into the forward pass but it’s an input into the backward pass. You’re trying to hide an input into the backward pass.

Reiner Pope

This is an adversarial context? This is actually a place where you get exactly the avalanche property that ciphers have as well. Adversarial attacks on image classification models are about finding a very small perturbation of the image that totally changes the classification, totally changes the output. That is the common case in ciphers, whereas that’s the undesired case in neural nets.

Dwarkesh Patel

Ok, so I was asking you, have neural networks actually been used for cryptography? And we realized it may be better to just do this on the blackboard. Are they actually being used for cryptography?

Reiner Pope

Using neural nets for cryptography… In general, creating a new cipher is a very dangerous proposition. Almost all of them are broken. 99% of them are broken, so it’s probably a bad place to start. But the other direction has been, in at least one very clear case, quite productive.

There’s a construction that exists in ciphers and then was imported into neural nets called a Feistel cipher, or Feistel network. The idea is that you may have some function f which is not invertible, but you like the function because it does interesting things, like it does an MLP, for example. Or it mixes it in an interesting way.

You’d like to build something out of this that is invertible. The construction we’re going to make is going to be a two-input function rather than a one-input function. We’re going to apply f(x). We need to actually remember what x was, so we’re going to stick x over here so that we can work backwards, and then we also can’t drop y. We’re going to remember y, and we’re going to add them together to form this tuple.

The way to invert this, if you think I have this output and I want to recover x and y, I can easily recover x. That’s right there, I just read it off. To recover y, if this thing was called z, I can recover y by z minus f(x), because I’ve already recovered x. That means this construction is invertible.

This was used in ciphers a ton and still is used. It’s one of the main mechanisms of constructing ciphers. Often you want ciphers to be invertible, especially the layers of ciphers, because that has better cryptographic properties.

This has actually been ported over into neural nets. There’s a 2017 paper called RevNets, reversible networks. What it does is make the entire network invertible. You can apply it to any network, like a transformer network. I do a forwards pass, but then I can run the entire pass backwards as well. The whole neural network is invertible with exactly this construction.

This paper applied it to some layer, like a transformer layer, for example. We’ve got this function f, which is our transformer layer. Normally we would have just an input and then a residual connection coming out, and it gets added over here. Now, the variation of this is going to be we’ve got two inputs, x and y. x goes through the function, gets added to y, and then this becomes the new x, output x. Then this x becomes the output y.

Really what this is doing, if you think of two layers back, is the thing you mentioned before. It’s doing the residual connection from two layers back. This y came from the previous layer and was the residual connection there. Because of this construction, the whole thing is invertible.

Why do I care? What does invertible matter for? The big thing that it can be interesting for is training. If I think of a forward pass of training… Let’s say I have four layers and I run them in zero, one, two, three order. I have to write all of the activations to HBM. I get an HBM footprint here that is kind of linear in the number of layers.

This can actually be the largest memory footprint during training. This is normal training, and then I run the backwards pass and read it in reverse. The forward pass goes forward, and the backward pass goes backwards. I have to read them back out.

The idea of this RevNets paper is that because it’s invertible, I don’t need to store this at all. I can completely rematerialize it. I run my forwards pass, and then when I’m running my backwards pass, I’m simultaneously in lockstep undoing all of the forwards pass steps that I did in order to have the activations that I need here. This ends up being memory saving, which is a nice idea.

Dwarkesh Patel

Interesting. In some sense you’re spending more compute to save memory.

Reiner Pope

That’s right.

Dwarkesh Patel

Interesting. It’s the opposite of what you’re doing with the KV cache. With the KV cache, you’re spending more memory to save compute.

Reiner Pope

Yeah. Spending more memory to save compute is generally profitable given where hardwares are.

Dwarkesh Patel

Interesting. That was super fun. Reiner, thank you so much for doing it. I feel like it really vindicated the vision behind the studio and the blackboard.

Reiner Pope

Yeah.

Dwarkesh Patel

Cool, thanks so much for doing it.

Reiner Pope

Thanks.

I put out a blog prize to answer a couple of big questions I have about AI. The goal is really to find someone to hire as coresearcher. I have more questions of this variety, but I omitted them from that post’s list, because they don’t make it easy to judge submission quality. So I thought I’d post them here:

• 5 hyperscalers own 70+% of global AI compute, and much of that is actually reserved for the 3 member set of OpenAI/Ant/GDM. How worried should we be that AI use cases which are not building up to the singularity and the robot factories - aka normal people being more empowered, understanding the world better, being entertained, etc, is not the highest ROI activity for compute in the world. And given how valuable compute will be (whose opportunity cost increases in tandem with the quality of the AI models that run on it), will normal people basically get priced out of the benefits of AI? If we should be worried about this, how concretely should some kind of universal basic income/compute redistribution work? If not worried, what is the frame of this question missing?

• Data is arguably the main way that AI models have been getting better over the last few years. But I remain confused about what concretely these improvements have consisted of. To ask some sharper questions:

  • Clearly Anthropic (and now also OpenAI and GDM) have cracked something about making competent long horizon coding agents. What is it? Is it just stacking up more and more RL coding environments? Or is there something more particular behind this breakthrough?

  • Are models even getting more sample efficient (aka they learn more from each training sample) or have we just changed/expanded/improved the data input? The reason this question is important is because it tells us how fast deep learning progress will be in domains that actually do require sample efficiency (for example, robotics).

    • Models are very sample efficient in context, and the information in context can be used much more flexibly. But the attention “fast” weights consume a huge amount of memory in order to accommodate this faster learning. Why is there this memory/sample efficiency tradeoff?

      • If you look at the size of the KV cache for Llama 3 70B, it’s 320 KB / token. If you just divide the number of bits it takes to store Llama 3 weights by the number of tokens it was pre-trained on, then you get 0.075 bits / token. So there’s a 35 million fold difference in the amount of information per bit you’re storing.

• Let’s put frontier lab compute into 3 buckets: pretraining, RL generation, and inference. RL generation and inference look like very similar workloads. The big difference, of course, is that the model learns as a result of RL generation, but it doesn’t (at least currently) from inference. At the same time, the model actually does useful work during inference, but not during RL generation. Many people have pointed out it’s really weird that there’s a distinction between training and inference, and that in the limit it shouldn’t exist. How practically will these two workloads be merged? At a high level, one can imagine hiring an AI instance for a month-long work trial, getting it to do actual useful work for you during that time, and then sending a report card back to the model company. In fact, in a few years, maybe the only way that AI can continue to make progress is through this kind of on-the-job learning, because models will already have saturated anything that can be learned from contrived shorter-horizon RL environments.

• Does something Y2Key happen when most of the tokens on the internet (and presumably the ones future models will be trained on) are generated by other AIs? Has the relative value of pre-2023 internet datasets increased in any noticeable way?

• I wrote this in my continual learning blog post last June. Is this correct? Why might there not be a winner take all dynamic from continual learning?

  • “Even if there isn’t a software only singularity (with models rapidly building smarter and smarter successor systems), we might still see something that looks like a broadly deployed intelligence explosion. AIs will be getting broadly deployed through the economy, doing different jobs and learning while doing them in the way humans can. But unlike humans, these models can amalgamate their learnings across all their copies. So one AI is basically learning how to do every single job in the world. An AI that is capable of online learning might functionally become a superintelligence quite rapidly without any further algorithmic progress”

• A lot of economic analysis about the impact of AGI focus on human demand - will the economy shrink because our demands can be fulfilled much more cheaply, will it rise because AI will create new varieties of products, or maybe because the relational sector will grow? But all this analyses take as a given that the only demand that matters is the one originating from humans. How do we model the machine-only economy, where the demand originates from the AI’s themselves? And once we add this consideration to our economic analysis of the future, what changes?

I’m announcing a $20,000 blog prize in order to find people who will excel at researching and thinking through these problems. The not-so-secret point of this whole contest is so that I can hire a research collaborator to think through questions like this hand in hand with me. See more at the end.

Pick a question below, and spend no more than 1,000 words answering it. 1st, 2nd, and 3rd place will get $10,000, $6,000, and $4,000 respectively. I’ll publish the winning entry (and potentially the runner ups) on my blog. Please submit by May 10th, 11:59 PM PST.

Questions - choose one

• A couple years ago, there was this idea that AI progress might slow down as we make further progress into the RL regime. 1. Because as horizon lengths increase, the AI needs to do many days’ worth of work before we can even see if it did it right, so if we’re still in a naive policy gradient world, the reward signal / FLOP goes down, and 2. We’d crossed through many OOMs of RL compute from GPT 4 to o1 to o3, and it would not be feasible to replicate that many OOMs increase in compute immediately again. But AI progress seems to have been fast nonetheless - even potentially speeding up if rumors about Spud or Mythos are to be believed. What gives? What did that previous intuition pump that motivated longer timelines miss? Feel free to deny premise of question.

• What’s the most plausible story where foundation model companies actually start making money? If you consider each individual model as a company, then its profits may be able to pay back the training cost. But of course, if you don’t train a bigger, more expensive model immediately, then you stop making money after 3 months. So when does the profit start? Maybe at some point scaling will plateau, but if progress at the frontier has slowed down, then the combination of distillation and low switching costs (cloud margins result from high switching costs) makes it really easy for open source to catch up to the labs, eating into their margins. So how do the labs actually start making money?

• With OpenAI’s new raise at an $852B valuation, OpenAI Foundation’s stake is now worth $180B. Anthropic’s cofounders have pledged to donate 80% of their wealth. Nobody seems to have a concrete idea of how to deploy 100s of billions (soon trillions) of wealth productively to “make AI go well”. If you were in charge of the OpenAI Foundation right now, what exactly would you do? And when? It’s not enough to identify a cause you think is important, because that doesn’t answer the fundamental problem of how you convert money to impact. Identify the concrete strategy you recommend pursuing.

• What should countries which are not currently in the AI production chain (semis, energy, frontier models, robotics) do in order to not get totally sidestepped by transformative AI? If you’re the leader of India or Nigeria, what do you do right now?

Rules and tips

• Please don’t let a lack of domain expertise dissuade you from entering. I’m looking for someone who can ramp up fast on unfamiliar topics and think clearly.

• Each entrant may submit only once.

• You are still eligible for this essay competition even if you’re not interested in the researcher role. Nor does winning this competition guarantee that you will be offered the role.

• You’re welcome to use LLMs to help you research, but I specifically picked these questions because I’ve found LLM answers to them unsatisfying. On these kinds of ambiguous questions, LLMs are too all over the place. For example, they’ll identify 5 plausible answers but not have the context and taste to identify the crucial factor and iron out its implications.

• You only have 1000 words - make them count. People have the habit of spending the first paragraphs clearing their throat - avoid that.

Why am I hiring for a researcher?

I want my podcast/blog to move from just asking questions about AI to actually helping answer them. But there are too many important questions, and I need a collaborator to build up context on them all, to explore dozens of fractal sub-questions, to consider the rebuttals and syntheses, and to sharpen each others thinking.

The questions I want us to explore are very broad while at the same time requiring deep technical analysis across many domains to actually answer.

Why am I hiring this way?

Well, I could just put out a job ad for a researcher, but I’ll get 1,000 different resumes, and I’ll have no clue based on that information whether the applicant would be any good at synthesizing lots of technical arguments and information. So I thought, let’s just list out some questions where I genuinely don’t know the answer and would be keen to get some insight.

What this role looks like

• Ideally in person in San Francisco, but potentially open to remote.

• Will pay competitively

Submit here

If you have questions or comments, I’m hello@dwarkeshpatel.com.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Crusoe’s cloud runs on state-of-the-art Blackwell GPUs, with Vera Rubin deployment scheduled for later this year. But hardware is only part of the story—for inference, Crusoe’s MemoryAlloy tech implements a cluster-wide KV cache, delivering up to 10x faster TTFT and 5x better throughput than vLLM. Learn more at crusoe.ai/dwarkesh

• Cursor helped me build an AI co-researcher over the course of a weekend. Now I have an AI agent that I can collaborate with in Google Docs via inline comment threads! And while other agentic coding tools feel like a total black-box, Cursor let me stay on top of the full implementation. You can try my co-researcher out here, or get started on your own Cursor project today at cursor.com/dwarkesh

• Jane Street spent ~20,000 GPU hours training backdoors into 3 different language models, then challenged my audience to find the triggers. They received some clever solutions—like comparing the base and fine-tuned versions and extrapolating any differences to reveal the hidden backdoor—but no one was able to solve all 3. So if open problems like this excite you, Jane Street is hiring. Learn more at janestreet.com/dwarkesh

Timestamps

(00:00:00) – Is Nvidia’s biggest moat its grip on scarce supply chains?

(00:16:25) – Will TPUs break Nvidia’s hold on AI compute?

(00:41:06) – Why doesn’t Nvidia become a hyperscaler?

(00:57:36) – Should we be selling AI chips to China?

(01:35:06) – Why doesn’t Nvidia make multiple different chip architectures?

Transcript

00:00:00 – Is Nvidia’s biggest moat its grip on scarce supply chains?

Dwarkesh Patel

We’ve seen the valuations of a bunch of software companies crash because people are expecting AI to commoditize software. There’s a potentially naive way of thinking about things, which is: look, Nvidia sends a GDS2 file to TSMC. TSMC builds the logic dies, it builds the switches, then it packages them with the HBM that SK Hynix, Micron, and Samsung make. Then it sends it to an ODM in Taiwan where they assemble the racks. Nvidia is fundamentally making software that other people are manufacturing, and if software gets commoditized, does Nvidia get commoditized?

Jensen Huang

In the end, something has to transform electrons to tokens. The transformation of electrons to tokens and making those tokens more valuable over time is hard to completely commoditize. The transformation from electrons to tokens is such an incredible journey. Making that token is like making one molecule more valuable than another molecule, making one token more valuable than another. The amount of artistry, engineering, science, and invention that goes into making that token valuable, obviously we’re watching it happen in real time. The transformation, the manufacturing, all of the science that goes in there is far from deeply understood and the journey is far from over. I doubt that it will happen.

We’re going to make it more efficient, of course. The way that you framed the question is my mental model of our company. The input is electrons, the output is tokens. In the middle is Nvidia. Our job is to do as much as necessary and as little as possible to enable that transformation to be done at incredible capabilities. What I mean by “as little as possible,” whatever I don’t need to do, I partner with somebody and make it part of my ecosystem.

If you look at Nvidia today, we probably have the largest ecosystem of partners, both in the supply chain upstream and downstream, all of the computer companies, application developers, and model makers. AI is a five-layer cake, if you will. We have ecosystems across the entire five layers. We try to do as little as possible, but the part that we have to do, as it turns out, is insanely hard. I don’t think that gets commoditized.

In fact, I also don’t think the enterprise software companies, the tools makers… Most software companies today are tool makers. Some of them are not. Some of them are workflow codification systems. But for a lot of companies, they’re tool makers. For example, Excel is a tool, PowerPoint is a tool, Cadence makes tools, Synopsys makes tools. I actually see the opposite of what people see. I think the number of agents is going to grow exponentially, and the number of tool users is going to grow exponentially. It’s very likely that the number of instances of all these tools is going to skyrocket.

It’s very likely that the number of instances of Synopsys Design Compiler is going to skyrocket, along with the number of agents using the floor planners, our layout tools, and our design rule checkers. Today we’re limited by the number of engineers. Tomorrow, those engineers are going to be supported by a bunch of agents. We’re going to be exploring the design space like you’ve never seen before, and we’re going to use the tools that we use today.

I think tool use is going to cause the software companies to skyrocket. The reason why it hasn’t happened yet is because the agents aren’t good enough at using their tools yet. Either these companies are going to build the agents themselves, or agents are going to get good enough to be able to use those tools. I think it’s going to be a combination of both.

Dwarkesh Patel

I think in your latest filings, you had almost a $100 billion in purchase commitments with foundries, memory, and packaging. SemiAnalysis has reported that you will have $250 billion of these kinds of purchase commitments. One interpretation is that Nvidia’s moat is really that you’ve locked up many years of these scarce components. Somebody else might have an accelerator, but can they actually get the memory to build it? Can they actually get the logic to build it? Is this really Nvidia’s big moat for the next few years?

Jensen Huang

It’s one of the things that we can do that is hard for someone else to do. We’ve made enormous commitments upstream. Some of it is explicit, these commitments that you mentioned. Some of it is implicit. For example, a lot of the investments that are upstream are made by our supply chain because I said to the CEOs, “Let me tell you how big this industry is going to be, let me explain to you why, let me reason through it with you, and let me show you what I see.”

As a result of that process of informing, inspiring, and aligning with CEOs of all different industries upstream, they’re willing to make the investments. Why are they willing to make the investments for me and not someone else? The reason for that is because they know that I have the capacity to buy their supply and sell it through my downstream. The fact is that Nvidia’s downstream supply chain and our downstream demand is so large, they’re willing to make the investment upstream.

If you look at GTC, people are marveled by the scale of it and the people that go. It’s a full 360 degrees, the entire universe of AI all in one place. They’re all in one place because they need to see each other. I bring them together so that the downstream can see the upstream, the upstream can see the downstream, and all of them can see the advances in AI. Very importantly, they can all meet the AI natives, all the AI startups being built, and all the amazing things happening so they can see firsthand all the things that I tell them. I spend a lot of my time informing, directly or indirectly, our supply chain, partners, and ecosystem about the opportunity in front of us.

Some people always say, “Jensen, in most keynotes, it’s one announcement after another.” With our keynotes, there’s always a part of it that’s a little torturous in the sense that it almost comes across like education. In fact, that’s exactly on my mind. I need to make sure the entire supply chain, upstream and downstream, the ecosystem, understands what is coming at us, why it’s coming, when it’s coming, how big it’s going to be, and is able to reason about it systematically, just like I reason about it.

Regarding the moat as you describe it, we’re able to build for a future. If our next several years are a trillion dollars in scale, we have the supply chain to do it. Without our reach, the velocity of our business… Just as there’s cash flow, there’s supply chain flow, there’s churns. Nobody is going to build a supply chain for an architecture if the business churns are low. Our ability to sustain the scale is only because our downstream demand is so great. And they see it, they hear about it, they see it all coming. That allows us to do the things we’re able to do at the scale we do them.

Dwarkesh Patel

I do want to understand more concretely whether the upstream can keep up. For many years now, you guys have been 2x-ing revenue year over year. You’ve been more than tripling the amount of flops you’re providing to the world year over year.

Jensen Huang

And 2x-ing at this scale now is really incredible.

Dwarkesh Patel

Exactly. But then you look at logic. You’re the biggest customer on TSMC’s N3 node, and you’re one of the biggest on N2. AI as a whole this year is going to be sixty percent of N3. It’s going to be 86% next year, according to SemiAnalysis. How do you double if you’re the majority? And how do you do that year over year? Are we in a regime now where the growth rate in AI compute has to slow because of upstream? Do you see a way to get around this? How do we build 2x more fabs year over year, ultimately?

Jensen Huang

At some level, the instantaneous demand is greater than the supply upstream and downstream in the world. At any instant, we could be limited by the number of plumbers, which actually happens.

Dwarkesh Patel

The plumbers are invited to next year’s GTC.

Jensen Huang

By the way, great idea. But that’s a good condition. You want an industry where the instantaneous demand is greater than the total supply of the industry. The opposite is obviously less good. If we’re too far apart, if one particular component is too far away, the industry swarms it. For example, notice people aren’t talking very much about CoWoS anymore.

The reason for that is because for two years we swarmed the living daylights out of it. We doubled, doubled, doubled on several doubles. Now I think we’re in fairly good shape. TSMC now knows that CoWoS supply has to keep up with the rest of the logic demand and the memory demand. They’re scaling CoWoS and future packaging technologies at the same level as they scale logic. This is terrific, because for a long time, CoWoS and HBM memory were rather specialty. But they’re not specialties anymore. People now realize they’re mainstream computing technology.

Of course, we’re now much more able to influence a larger scope of our supply chain. At the beginning of the AI revolution, all the things that I say now, I was saying five years ago. Some people believed in it and invested in it, for example, Sanjay and the Micron team. I still remember the meeting really well where I was clear about exactly what was going to happen, why it was going to happen, and the predictions of today. They really doubled down on it. We partnered with them across LPDDR and HBM memories, and they really invested in it. It obviously has been tremendous for the company. Some people came a little bit later, but now they’re all here.

Each one of these bottlenecks gets a great deal of attention. Now we’re prefetching the bottlenecks years in advance. For example, the investments that we’ve done with Lumentum, Coherent, and the silicon photonics ecosystem over the last several years really reshaped the supply chain. We built up an entire supply chain around TSMC. We partnered with them on COUPE, invented a whole bunch of technology, and licensed those patents to the supply chain to keep it nice and open.

We’re preparing the supply chain through the invention of new technologies, new workflows, new testing equipment like double-sided probing, investing in companies, and helping them scale up their capacity. You can see that we’re trying to shape the ecosystem so that the supply chain is ready to support the scale.

Dwarkesh Patel

It seems like some bottlenecks are easier than others. Scaling up CoWoS versus scaling up—

Jensen Huang

I went to the hardest one, by the way.

Dwarkesh Patel

Which is?

Jensen Huang

Plumbers. Plumbers and electricians. This is one of the concerns that I have about the doomers describing the end of work and killing of jobs. If we discourage people from being software engineers, we’re going to run out of software engineers. The same prediction happened ten years ago. Some of the doomers were telling people, “Whatever you do, don’t be a radiologist.” You might hear some of those videos still on the web saying radiology is going to be the first career to go and the world is not going to need any more radiologists. Guess what we’re short of? Radiologists.

Dwarkesh Patel

Going back to this point about how some things you can scale, and other things… How do you actually manufacture 2x the amount of logic a year? Ultimately, memory and logic are bottlenecked by EUV. How do you get to 2x as many EUV machines year over year?

Jensen Huang

None of that is impossible to scale quickly. All of that is easy to do within two or three years. You just need a demand signal. Once you can build one, you can build ten, and once you can build ten, you can build a million. These things are not hard to replicate.

Dwarkesh Patel

How far down the supply chain do you go? Do you go to ASML and say, “Hey, if I look out three years from now, for Nvidia to be generating two trillion a year in revenue, we need way more EUV machines”?

Jensen Huang

Some of them I have to directly, some of them indirectly, and some of them… If I can convince TSMC, ASML will be convinced. We have to think about the critical pinch points. But if TSMC is convinced, you’ll have plenty of EUV machines in a few years.

My point is that none of the bottlenecks last longer than a couple of years, two, three years, none of them. Meanwhile, we’re improving computing efficiency by 10x 20x, and in the case of Hopper to Blackwell, 30x to 50x. We’re coming up with new algorithms because CUDA is so flexible. We’re developing all kinds of new techniques so that we drive efficiency in addition to increasing capacity. None of those things worry me. It’s the stuff that’s downstream from us. Energy policies that prevent energy from… You can’t create an industry without energy. You can’t create a whole new manufacturing industry without energy.

We want to reindustrialize the United States. We want to bring back chip manufacturing, computer manufacturing, and packaging. We want to build new things like EVs and robots. We want to build AI factories. You can’t build any of these things without energy, and those things take a long time. More chip capacity, that’s a 2-3 year problem. More CoWoS capacity, 2-3 year problem.

Dwarkesh Patel

Interesting. I feel like I have guests tell me the exact opposite thing sometimes. In this case, I just don’t have the technical knowledge to adjudicate.

Jensen Huang

The beautiful thing is you’re talking to the expert.

00:16:25 – Will TPUs break Nvidia’s hold on AI compute?

Dwarkesh Patel

True. I want to ask about your competitors. If you look at the TPU, arguably two out of the top three models in the world, Claude and Gemini, were trained on TPU. What does that mean for Nvidia going forward?

Jensen Huang

We build a very different thing. What Nvidia built is accelerated computing, not a tensor processing unit. Accelerated computing is used for all kinds of things: molecular dynamics, quantum chromodynamics, data processing, data frames, structured data, and unstructured data. It’s also used for fluid dynamics and particle physics. In addition, we use it for AI.

Accelerated computing is much more diverse. Although AI is the conversation today and is obviously very important and impactful, computing is much broader than that. Nvidia has reinvented the way computing is done, moving from general-purpose computing to accelerated computing. Our market reach is far greater than any TPU or ASIC can possibly have. If you look at our position, we’re the only company that accelerates applications of all kinds. We have a gigantic ecosystem. So all kinds of frameworks and algorithms run on Nvidia.

Because our computers are designed to be operated by other people, anyone who’s an operator can buy our systems. With most of these home-built systems, you have to be your own operator because they were never designed to be flexible enough for others to operate. Because anybody can operate our systems, we’re in every cloud, including Google, Amazon, Azure, and OCI.

If you want to operate it to rent, you better have a large ecosystem of customers in many industries to be the offtakers. If you want to operate it for yourself, we obviously have the ability to help you operate it yourself, like we did for Elon with xAI. And because we can enable operators in any company and any industry, you could use it to build a supercomputer for scientific research and drug discovery at Lilly. We can help them operate their own supercomputer and use it for the entire diversity of drug discovery and biological sciences that we accelerate.

There are just a whole bunch of applications that we can address that you can’t do with TPUs. Nvidia built CUDA to be a fantastic tensor processing unit as well, but it also handles every life cycle of data processing, computing, AI, and so on. Our market opportunity is just a lot larger, and our reach is a lot greater. Because we support every application in the world now, you can build Nvidia systems anywhere and know that there will be customers for it. It’s a very different thing.

Dwarkesh Patel

This is going to be a long question. You have spectacular revenue, and you’re not making $60 billion a quarter from pharma and quantum. You’re making it because AI is an unprecedented technology that is growing unprecedentedly fast.

The question then is what is best for AI specifically. I’m not in the details, but I talk to my AI researcher friends and they say, “Look, when I use a TPU, it’s this big systolic array that’s perfect for doing matrix multiplies, whereas a GPU is very flexible. It’s great when you have lots of branching or irregular memory access.”

But what is AI? It’s just these very predictable matrix multiplies again and again and again. You don’t have to give up any die area for warp schedulers or switches between threads and memory banks. And the TPU is really optimized for the bulk of this growth in revenue and use case for compute that is coming online right now. I wonder how you react to that.

Jensen Huang

Matrix multiplies are an important part of AI, but they’re not the only part. If you want to come up with a new attention mechanism, disaggregate in a different way, or invent a whole new type of architecture altogether—like a hybrid SSM—you want an architecture that’s generally programmable. If you want to create a model that fuses diffusion and autoregressive techniques, you want an architecture that’s just generally programmable. We run everything you can imagine. That’s the advantage. It allows for the invention of new algorithms a lot more easily, because it’s a programmable system.

The ability to invent new algorithms is really what makes AI advance so quickly. TPUs, like anything else, are impacted by Moore’s Law, which we know is increasing by about 25% per year. The only way to really get 10x or 100x leaps is to fundamentally change the algorithm and how it’s computed every single year.

That’s Nvidia’s fundamental advantage. The only reason we were able to make Blackwell to Hopper 50x… When I first announced Blackwell was going to be 35x more energy efficient than Hopper, nobody believed it. Then Dylan wrote an article saying I sandbagged, and it’s actually fifty times. You can’t reasonably do that with just Moore’s Law. The way we solve that problem is with new models, like MoEs, that are parallelized, disaggregated, and distributed across a computing system. Without the ability to really get down and come up with new kernels with CUDA, it’s really hard to do.

It’s the combination of the programmability of our architecture and the fact that Nvidia is an extreme co-design company. We can even offload some of the computation into the fabric itself, like NVLink, or into the network with Spectrum-X. We could affect change across the processors, the system, the fabric, the libraries, and the algorithm simultaneously. Without CUDA to do that, I wouldn’t even know where to start.

Dwarkesh Patel

This gets at an interesting question about Nvidia’s clientele. 60% of your revenue is coming from these big five hyperscalers. In a different era with different customers—let’s say professors running experiments—they need CUDA. They can’t use another accelerator. They just needed to run PyTorch with CUDA and have everything optimized.

But these hyperscalers have the resources to write their own kernels. In fact, they have to in order to get that last 5% of performance they need for their specific architecture. Anthropic and Google are mostly running their own accelerators or running TPUs and Trainium. But even OpenAI, using GPUs, has Triton because they need their own kernels. Down to CUDA C++, instead of using cuBLAS and NCCL, they’ve got their own stack which compiles to other accelerators as well. If most of your customers can and do make replacements for CUDA, to what extent is CUDA really the thing that is going to make frontier AI happen on Nvidia?

Jensen Huang

CUDA is a rich ecosystem. If you want to build on any computer first, building on CUDA first is incredibly smart. Because the ecosystem is so rich, we support every framework. If you want to create custom kernels… For example, we contribute enormously to Triton. So the back end of Triton has huge amounts of Nvidia technology.

We’re delighted to help every framework become as great as it can be. There are lots and lots of frameworks. There’s Triton, vLLM, SGLang, and more. Now there’s a whole bunch of new reinforcement learning frameworks coming out, like verl and NeMo RL. With post-training and reinforcement learning, that entire area is just exploding. So if you want to build on an architecture, building on CUDA makes the most sense because you know the ecosystem is great.

You know that if something happens, it’s more likely in your code and not in the mountain of code underneath. Don’t forget the amount of code you’re dealing with when building these systems. When something doesn’t work, was it you or was it the computer? You would like it to always be you and to be able to trust the computer. Obviously, we still have lots of bugs ourselves, but our system is so well wrung out that you can at least build on top of the foundation. That’s number one: the richness, programmability, and capability of the ecosystem.

The second thing is, if you’re a developer building anything at all, the single most important thing you want is an install base. You want the software you write to run on a whole bunch of other computers. You’re not building software just for yourself. You’re building it for your fleet or everybody else’s fleet because you’re a framework builder. Nvidia’s CUDA ecosystem is ultimately its great treasure.

We have several hundred million GPUs out there now. Every cloud has it. It goes back to the A10, A100, H100, H200, the L series, the P series. There’s a whole bunch of them. They’re in all kinds of sizes and shapes. If you’re a robotics company, you want that CUDA stack to actually run in the robot itself. We’re literally everywhere. The install base means that once you develop the software or the model, it’s going to be useful everywhere. That is just incredibly valuable.

Lastly, the fact that we’re in every single cloud makes us genuinely unique. If you’re an AI company or developer, you’re not exactly sure which cloud service provider you’re going to partner with or where you’d like to run it. We run everywhere, including on-prem for you if you like. The combination of the richness of the ecosystem, the expansiveness of the install base, and the versatility of where we are makes CUDA invaluable.

Dwarkesh Patel

That makes a lot of sense. I guess the thing I’m curious about is whether those advantages matter a lot to your main customers. There’s many people for whom they might matter. The kind of person who can actually build their own software stack makes up most of your revenue. Especially if you go to a world where AI is getting especially good at the things which have tight verification loops where you can RL on them…. This question of how do you write a kernel that does attention or MLP the most efficiently across a scale up? It’s a very verifiable sort of feedback loop.

Can all the hyperscalers write these custom kernels for themselves? Nvidia still has great price performance, so they might still prefer to use Nvidia. But then the question is, does it just become a question of who is offering the best specs, the best flops and memory bandwidth for a given dollar. Whereas historically Nvidia has just had, and still has, the best margins in all of AI across hardware and software, +70%, because of this CUDA moat. And the question is, can you sustain those margins if for most of your customers, they can actually afford to build, instead of the CUDA moat?

Jensen Huang

The number of engineers we have assigned to these AI labs is insane, working with them, optimizing their stack. The reason for that is because nobody knows our architecture better than we do. These architectures are not as general purpose as a CPU. A CPU is kind of like a Cadillac. It’s a nice cruiser. It never goes too fast. Everybody drives it pretty well. It’s got cruise control, and everything’s easy. But in a lot of ways, Nvidia’s GPUs, accelerators, are like F1 racers. I could imagine everybody’s able to drive it at a hundred miles an hour, but it takes quite a bit of expertise to be able to push it to the limit. We use a ton of AI to create the kernels that we have.

I’m pretty sure we’re going to still be needed for quite some time. Our expertise helps our AI lab partners to get another 2x out of their stack easily oftentimes. It’s not unusual that by the time we’re done optimizing their stack or optimizing a particular kernel, their model sped up by 3x, 2x, 50%. That’s a huge number, especially when you’re talking about the install base of the fleet that they have, of all the Hoppers and Blackwells that they have. When you increase it by a factor of two, that doubles the revenues. That directly translates to revenues.

Nvidia’s computing stack is the best performance per TCO in the world, bar none. Nobody can demonstrate to me that any single platform in the world today has a better performance-TCO ratio. Not one company. In fact, the benchmarks that are out there. Dylan’s InferenceMAX is sitting out there for everybody to use, and not one… TPU won’t come, Trainium won’t come.

I encourage them to use InferenceMAX and demonstrate their incredible inference cost. It’s really hard. Nobody wants to show up. MLPerf. I would welcome Trainium to demonstrate their 40% that they claim all the time. I would love to hear them demonstrate the cost advantage of TPUs. It makes no sense in my mind. It makes absolutely zero sense. On first principles, it makes no sense.

So I think the reason why we’re so successful is simply because our TCO is so great. Secondly, you say 60% of our customers are the top five, but most of that business is external. For example, most of Nvidia in AWS is for external customers, not internal use. Most of our customers at Azure, obviously all of our customers are external. All of our customers at OCI are external, not internal use. The reason why they favor us is because our reach is so great. We can bring them all of the great customers in the world. They’re all built on Nvidia. And the reason why all these companies are built on Nvidia is because our reach and our versatility is so great.

So I think the flywheel is really install base, the programmability of our architecture, the richness of our ecosystem, and the fact that there’s so many AI companies in the world. There’s tens of thousands of them now. If you were one of those AI startups, what architecture would you choose? You would choose an architecture that’s most abundant. We’re the most abundant in the world. You’d choose the one that has the largest installed base. We’re the largest install base. And you’d choose the one that has a rich ecosystem.

So that’s the flywheel. That’s the reason why, between the combination of: one, our perf per dollar is so great that they have the lowest cost tokens. Second, our perf per watt is the highest in the world. So if one of these companies, if our partners, built a one gigawatt data center, that one gigawatt data center better deliver the maximum amount of revenues and number of tokens, which directly translates to revenues. You want it to generate as many tokens as possible, maximize the revenues for that data center. We are the highest tokens per watt architecture in the world. Lastly, if your goal is to rent the infrastructure, we have the most customers in the world. So that’s the reason why the flywheel works.

Dwarkesh Patel

Interesting. I guess the question comes down to, what is the actual market structure here? Because even if there’s other companies… There could have been a world where there’s tens of thousands of AI companies that have roughly equal share of compute. But even through these five hyperscalers, really the people on Amazon using the compute are Anthropic, OpenAI, and these big foundation labs who can themselves afford and have the ability to make different accelerators work.

Jensen Huang

No, I think your premise is wrong.

Dwarkesh Patel

Maybe. But let me ask you a slightly different question.

Jensen Huang

Come back and make me correct your premise.

Dwarkesh Patel

Okay. Let me just ask you a different question.

Jensen Huang

But still make sure to make me come back and fix because it’s just too important to AI. It’s too important to the future of science. It’s too important to the future of the industry. That premise… Look —

Dwarkesh Patel

Let me just finish the question and then we can address it together.

Jensen Huang

Yeah.

Dwarkesh Patel

If all these things are true about price, performance, and performance per watt, et cetera, are true, why do you think it is the case that, say, Anthropic for example, just announced a couple days ago they have a multi-gigawatt deal with Broadcom and Google for TPUs and majority of their compute?

Obviously for Google, TPU is a majority of compute. So if I look at these big AI companies, it seems like a lot of their compute… There was some point where it’s all Nvidia and now it’s not. So I’m curious how to square, if these things are true on paper, why are they going with other accelerators?

Jensen Huang

Anthropic is a unique instance, not a trend. Without Anthropic, why would there be any TPU growth at all? It’s 100% Anthropic. Without Anthropic, why would there be Trainium growth at all? It’s 100% Anthropic. I think that’s fairly well known and well understood. It’s not that there’s an abundance of ASIC opportunities. There’s only one Anthropic.

Dwarkesh Patel

But OpenAI’s deals with AMD… They’re building their own Titan accelerator.

Jensen Huang

Yeah, but I think we could all acknowledge they’re vastly Nvidia. We’re going to still do a lot of work together. I’m not offended by other people using something else and trying things. If they don’t try these other things, how would they know how good ours is? Sometimes you’ve got to be reminded of it. We have to continuously earn the position that we’re in.

There are always big claims. Look at the number of ASICs that have been canceled. Just because you’re going to build an ASIC… You still have to build something better than Nvidia. It’s not that easy building something better than Nvidia. It’s not sensible, actually. Nvidia’s got to be missing something, seriously. Because of our scale, our velocity, we’re the only company in the world that’s cranking it out every single year. Big leaps, every single year.

Dwarkesh Patel

I guess their logic is, “Hey, it doesn’t need to be better. It just needs to be not more than 70% worse,” because they’re paying you 70% margins.

Jensen Huang

No, don’t forget, even in ASICs margins are really quite high. Nvidia’s margin is 70%, let’s say. But ASIC margins are 65%. What are you really saving?

Dwarkesh Patel

Oh, you mean from Broadcom or something like that?

Jensen Huang

Yeah, sure. You’ve got to pay somebody. I think the ASIC margins are incredibly good, from what I can tell. They believe it too. They’re quite proud of their incredible ASIC margins.

So, you asked the question why. A long time ago, we just didn’t have the ability to do it. At the time, I didn’t deeply internalize how difficult it would be to build a foundation AI lab like OpenAI and Anthropic, and the fact that they needed huge investments from the supplier themselves. We just weren’t in a position to make the multi-billion dollar investment into Anthropic so that they could use our compute. But Google and AWS were. They put in huge investments in the beginning so that Anthropic, in return, used their compute. We just weren’t in a position to do that at the time.

I would say my mistake is I didn’t deeply internalize that they really had no other options, that a VC would never put in $5-10 billion of investment into an AI lab with the hopes of it turning out to be Anthropic. So that was my miss. But even if I understood it, I don’t think we would’ve been in a position to do that at the time. But I’m not going to make that same mistake again.

I’m delighted to invest in OpenAI, and I’m delighted to help them scale, and I believe it’s essential to do so. And then, when I was able to, when Anthropic came to us, I’m delighted to be an investor, delighted to help them scale. We just weren’t, at the time, able to do it. If I could rewind everything—and Nvidia could have been as big back then as we are now—I would’ve been more than happy to do it.

00:41:06 – Why doesn’t Nvidia become a hyperscaler?

Dwarkesh Patel

This is actually quite interesting. For many years Nvidia has been the company in AI making money, making lots of money. Now you’re investing it. It’s been reported that you’ve done up to $30 billion in OpenAI and $10 billion in Anthropic. But now their valuations have increased, and I’m sure they’ll continue to increase.

So if over these many years you were giving them the compute, you saw where it was headed, and they were worth like one tenth what they’re worth now a couple years ago—or even a year ago in some cases and you had all this cash — there’s a world where either Nvidia themselves becomes a foundation lab, does a huge investment to make that possible, or has made the deals you’ve made now at current valuations much earlier on. And you had the cash to do it. So I am curious, actually, why not have done it earlier?

Jensen Huang

We did it as soon as we could have. We did it as soon as we could have, and if I could have, I would’ve done it even earlier. At the time that Anthropic needed us to do it, we just weren’t in a position to do it. It wasn’t in our sensibility to do so.

Dwarkesh Patel

How so? Was it like a cash thing?

Jensen Huang

Yeah, the level of investment. We had never invested outside the company at the time, and not that much. We didn’t realize we needed to. I always thought that they could just go raise from VCs, for God’s sakes, like all companies do. But what they were trying to do couldn’t have been done through VCs. What OpenAI wanted to do couldn’t have been done through VCs. I recognize that now. I didn’t know it then.

But that’s their genius. That’s why they’re smart. They realized then that they had to do something like that. And I’m delighted that they did. Even though we caused Anthropic to have to go to somebody else, I’m still happy that it happened. Anthropic’s existence is great for the world. I’m delighted for it.

Dwarkesh Patel

I guess you still are making a ton of money, and you’re making way more money quarter after quarter.

Jensen Huang

It’s still okay to have regrets.

Dwarkesh Patel

So the question still arises. Okay, now that we’re here and you have all this money that you keep making, what should Nvidia be doing with it? There’s one answer which is that there’s this whole middleman ecosystem that has popped up for converting CapEx into OpEx for these labs so that they can rent compute. Because the chips are really expensive, they make a lot of money over their lifetime because the AI models are getting better. So the value that they generate, their tokens, is increasing, but they’re expensive to set up. Nvidia has the money to do the CapEx. In fact, it’s been reported, you are backstopping CoreWeave up to $6.3 billion and have invested $2 billion.

Why doesn’t Nvidia become a cloud themselves? Why doesn’t it become a hyperscaler themselves and rent this compute out? You have all this cash to do it.

Jensen Huang

This is a philosophy of the company, and I think it’s wise. We should do as much as needed, as little as possible. What that means is, the work that we do with building our computing platform, if we don’t do it, I genuinely believe it doesn’t get done. If we didn’t take the risk that we take—if we didn’t build NVLink the way we built it, if we didn’t build the whole stack, if we didn’t create the ecosystem the way we did, if we didn’t dedicate ourselves to 20 years of CUDA while losing money most of that time—if we didn’t do it, nobody else would have done it.

If we didn’t create all the CUDA-X libraries so that they’re all domain-specific… A decade and a half ago, we pushed into domain-specific libraries because we realized that if we didn’t create these domain-specific libraries, whether it’s for ray tracing or image generation or even the early works of AI, these models, if we didn’t create them, for data processing, structured data processing, or vector data processing, if we didn’t create them, nobody would. I am completely certain of that. We created a library for computational lithography called cuLitho. If we didn’t create it, nobody would have. So accelerated computing wouldn’t advance the way it has if we didn’t do what we did.

So we should do that. We should dedicate our company, all of our might, wholeheartedly to go do that. However, the world has lots of clouds. If I didn’t do it, somebody would show up. So following the recipe, the philosophy, of doing as much as needed but as little as possible—as little as possible—that philosophy exists in our company today. Everything I do, I do it with that lens.

In the case of clouds, if we didn’t support CoreWeave to exist, these neoclouds, these AI clouds, wouldn’t exist. If we didn’t help CoreWeave exist, they would not exist. If we didn’t support Nscale, they wouldn’t be where they are today. If we didn’t support Nebius, they wouldn’t be what they are today. Now they’re doing fantastically.

Is that a business model [inaudible]? We should do as much as needed, as little as possible. So we invest in our ecosystem because I want our ecosystem to thrive. I want the architecture, and AI, to be able to connect with as many industries as possible, as many countries as possible, and make it possible for the planet to be built on AI and to be built on the American tech stack. That vision is exactly what we’re pursuing.

Now, one of the things that you mentioned… There are so many great, amazing foundation model companies, and we try to invest in all of them. This is another thing that we do. We don’t pick winners. We need to support everyone. It’s part of our joy of doing so. It’s imperative to our business. But we also go out of our way not to pick winners. So when I invest in one of them, I invest in all of them.

Dwarkesh Patel

Why do you go out of your way not to pick winners?

Jensen Huang

Because it’s not our job to, number one. Number two, when Nvidia first started, there were 60 3D graphics companies. We are the only one that survived. If you would have taken those 60 graphics companies and asked yourself which one was going to make it, Nvidia would be at the top of that list not to make it.

This is long before you, but Nvidia’s graphics architecture was precisely wrong. It’s not a little bit wrong. We created an architecture that was precisely wrong, and it was an impossible thing for developers to support. It was never going to make it. We reasoned about it from good first principles, but we ended up with the wrong solution. Everybody would have counted us out. And here we are.

So I have enough humility to recognize that. Don’t pick winners. Either let them all take care of themselves, or take care of all of them.

Dwarkesh Patel

One thing I didn’t understand is you said, “Look, we’re not prioritizing these neoclouds just because they are neoclouds and we want to prop them up.” But you also listed a bunch of neoclouds and said they wouldn’t exist if it wasn’t for NVIDIA. How are those two things compatible?

Jensen Huang

First of all, they need to want to exist, and they come to ask us for help. When they want to exist and they have a business plan, expertise, and the passion for it… They obviously have to have some capabilities themselves. But if, at the end of the day, they need some investment in order to get it off the ground, we would be there for them. But the sooner they get their flywheel going...

Your question was, “Do we want to be in the financing business?” The answer is no. There are people in the financing business, and we’d rather work with all the people in the financing business than be a financier ourselves. Our goal is to focus on what we do, keep our business model as simple as possible, and support our ecosystem.

When someone like OpenAI needs an investment of a $30 billion scale because it’s still before their IPO, and we deeply believe in them and I deeply believe that they’re going to be an… Well, they’re an extraordinary company already today. They’re going to be an incredible company. The world needs them to exist. The world wants them to exist. I want them to exist. They have the wind at their back. Let’s support them and let them scale. Those investments we’ll do because they need us to do it. But we’re not trying to do as much as possible. We’re trying to do as little as possible.

Dwarkesh Patel

This may be an obvious question, but we’ve lived many years in this situation where there’s a shortage of GPUs, and it’s grown now because models are getting better.

Jensen Huang

We have a shortage of GPUs.

Dwarkesh Patel

Yes. Nvidia is known for divvying up the scarce allocation, not just based on high bidder, but rather on, “Hey, we want to make sure that these neoclouds exist. Let’s give some to CoreWeave, let’s give some to Crusoe, let’s give some to Lambda.” Why is it good for Nvidia? First of all, would you agree with this characterization of fracturing the market?

Jensen Huang

No. No. Your premise is just wrong. We’re sufficiently mindful about these things. We’re very mindful about these things. First of all, if you don’t place a PO, all the talking in the world won’t make a difference. Until we get a PO, what are we going to do? So the first thing is, we work really hard with everybody to get a forecast done, because these things take a long time to build, and the data centers take a long time to build. We align ourselves with demand and supply and things like that through forecasting. Okay? That’s job number one.

Number two, we’ve tried to forecast with as many people as possible, but in the final analysis, you still have to place an order. Maybe, for whatever reason, you didn’t place your order. What can I do? At some point, first in, first out. But beyond that, if you’re not ready because your data center’s not ready, or certain components aren’t ready to enable you to stand up a data center, we might decide to serve another customer first. That’s just maximizing the throughput of our own factory. We might do some adjustments there.

Aside from that, the prioritization is first in, first out. You’ve got to place a PO. If you don’t place a PO… Now, of course, there are stories about that. For example, all of this kind of started from an article about Larry and Elon having dinner with me where they begged for GPUs. That never happened. We absolutely had dinner. We absolutely had dinner, and it was a wonderful dinner. At no time did they beg for GPUs. They just had to place an order. Once they place an order, we do our best to get the capacity to them. We’re not complicated.

Dwarkesh Patel

Okay. So it sounds like there’s a queue, and then based on whether your data center is ready and when you place a purchase order, you get them at a certain time. But it still doesn’t sound like the highest bidder just gets it. Is there a reason to do it…?

Jensen Huang

We never do that.

Dwarkesh Patel

Okay.

Jensen Huang

We never do.

Dwarkesh Patel

Why not just do high bidder?

Jensen Huang

Because it’s a bad business practice. You set your price and then people decide to buy it or not. I understand that others in the chip industry change their prices when demand is higher, but we just don’t. That’s just never been a practice of ours. You can count on us. I prefer to be dependable, to be the foundation of the industry. You don’t need to second-guess. If I quoted you a price, we quoted you a price. That’s it. If demand goes through the roof, so be it.

Dwarkesh Patel

On the other end, that’s why you have a productive relationship with TSMC, right?

Jensen Huang

Yeah, Nvidia’s been in business with them for, I guess, coming up on 30 years. Nvidia and TSMC don’t have a legal contract. There’s always some rough justice. Sometimes I’m right, sometimes I’m wrong. Sometimes I got a better deal, sometimes I got a worse deal. But overall, the relationship is incredible. I can completely trust them. I can completely depend on them.

One of the things you can count on with Nvidia is that this year, Vera Rubin is going to be incredible. Next year, Vera Rubin Ultra will come. The year after that, Feynman will come. And the year after that, I haven’t introduced the name yet. Every single year you can count on us. You’re going to have to go find another ASIC team in the world—pick your ASIC team—where you can say, “I can bet the farm, I can bet my entire business that you will be here for me every single year. Your token cost will decrease by an order of magnitude every single year. I can count on it like I can count on the clock.”

I just said something about TSMC. For no other foundry in history can you possibly say that. You can say that about Nvidia today. You can count on us every single year. If you would like to buy a billion dollars worth of AI factory compute, no problem. If you’d like to buy a hundred million dollars, no problem. You’d like to buy $10 million, or just one rack, not a problem. Or just one graphics card, okay, no problem. If you would like to place an order for a $100 billion of AI factory, no problem. We’re the only company in the world where you can say that today.

I can say that about TSMC as well. I want to buy one, buy 1 billion, no problem. We just have to go through the process of planning for it, and all the things that mature people do. So I think this ability for Nvidia to be the foundation of the world’s AI industry, this is a position that has taken us a couple of decades to arrive at. Enormous commitment, enormous dedication. The stability of our company, the consistency of our company, is really important.

00:57:36 – Should we be selling AI chips to China?

Dwarkesh Patel

Okay. I want to ask about China. I actually don’t know what I think about whether it’s good to sell chips to China or not, but I like to play devil’s advocate against my guests. So when Dario was on, who supports export controls, I asked him, why can’t America and China both have a country of geniuses in the datacenter? But since you’re on the opposite side, I’ll ask you in the opposite way.

One way to think about it is, Anthropic actually announced a couple days ago Mythos Preview. This model Mythos, they’re not even releasing publicly because they say it has such cyber-offensive capabilities that we don’t think the world is ready until we make sure these zero-days are patched up. But they say it found thousands of high-severity vulnerabilities across every major operating system, every browser. It found one in OpenBSD, which is this operating system that’s been specifically designed to not have zero days. It found one that’s existed for 27 years.

So if Chinese companies and Chinese labs and the Chinese government had access to the AI chips to train a model like Claude Mythos with these cyber-offensive capabilities and run millions of instances of it with more compute, the question is, is that a threat to American companies, to American national security?

Jensen Huang

First of all, Mythos was trained on fairly mundane capacity, and a fairly mundane amount of it. By an extraordinary company. The amount of capacity and the type of compute it was trained on is abundantly available in China. So you just have to first realize that chips exist in China.

They manufacture 60% of the world’s mainstream chips, maybe more. It’s a very large industry for them. They have some of the world’s greatest computer scientists. As you know, most of the AI researchers in all of these AI labs are Chinese. They have 50% of the world’s AI researchers. So the question is, considering all the assets they already have—they have an abundance of energy, they have plenty of chips, they’ve got most of the AI researchers—if you’re worried about them, what is the best way to create a safe world?

Victimizing them, turning them into an enemy, likely isn’t the best answer. They are an adversary. We want the United States to win. But I think having a dialogue and having research dialogue is probably the safest thing to do. This is an area that is glaringly missing because of our current attitude about China as an adversary. It is essential that our AI researchers and their AI researchers are actually talking. It is essential that we try to both agree on what not to use the AI for.

With respect to finding bugs in software, of course, that’s what AI is supposed to do. Is it going to find bugs in a lot of software? Of course. There are lots and lots of bugs. There are lots of bugs in the AI software. That’s what AI is supposed to do, and I’m delighted that AI has reached a level where it could help us be so much more productive.

One of the things that is underemphasized is the richness of the ecosystem around cybersecurity, AI cybersecurity and AI security and AI privacy and AI safety. There’s a whole ecosystem of AI startups that are trying to create this future for us, where you have one AI agent that’s incredible, surrounded by thousands of AI agents, keeping it safe, keeping it secure. That future surely is going to happen.

The idea that you’re going to have an AI agent running around with nobody watching after it is kind of insane. We know very well that this ecosystem needs to thrive. It turns out this ecosystem needs open source. This ecosystem needs open models. They need open stacks so that all of these AI researchers and all these great computer scientists can go build AI systems that are as formidable and can keep AI safe. So one of the things that we need to make sure that we do is we keep the open source ecosystem vibrant. That can’t be ignored. A lot of that is coming out of China. We ought to not suffocate that.

With respect to China, of course we want the United States to have as much computing as possible. We’re limited by energy, but we’ve got a lot of people working on that. We’ve got to not make energy a bottleneck for our country. But what we also want is to make sure that all the AI developers in the world are developing on the American tech stack, and making the contributions, the advancements of AI—especially when it’s open source—available to the American ecosystem. It would be extremely foolish to create two ecosystems: the open source ecosystem, and it only runs on a foreign tech stack, and a closed ecosystem that runs on the American tech stack. I think that would be a horrible outcome for the United States.

Dwarkesh Patel

Since there are a lot of things, let me just triage the response. I think the concern, going back to the flop difference in the hacking, is yes, they have compute, but there’s some estimates that because they’re at 7nm—they don’t have EUVs because of chip-making export controls—the amount of flops they’re able to actually produce, they have one tenth the amount of flops that the US has.

So with that, could they eventually train a model like Mythos? Yes. But the question is, because we have more flops, American labs are able to get to these levels of capabilities first. Because Anthropic got to it first, they say, “Okay, we’re going to hold onto it for a month while all these American companies, we’ll give them access to it. They’re going to patch up all their vulnerabilities, and now we release it.”

Furthermore, even if they train a model like this, the ability to deploy it at scale… If you had a cyber hacker, it’s much more dangerous if they have a million of them versus a thousand of them. So that inference compute really matters a lot. In fact, the fact that they have so many AI researchers who are so good is the thing that makes it so scary, because what is it that makes those engineer researchers more productive? It’s compute.

If you talk to any AI lab in America, they say the thing that’s bottlenecking them is compute. There are quotes from the DeepSeek founder, or Qwen leadership or whatever. They say the thing they’re bottlenecked on is compute. So then the question is, isn’t it better that we get American companies, because they have more compute, to get to the Mythos-level capabilities first, prepare our society for it, before China can get to it because, they have less compute?

Jensen Huang

We should always be first and we should always have more. But in order for that outcome you described to be true, you have to take it to the extremes. They have to have no compute. If they have some compute, the question is how much is needed?

The amount of compute they have in China is enormous. You’re talking about the country that is the second largest computing market in the world. If they want to aggregate their compute, they’ve got plenty of compute to aggregate.

Dwarkesh Patel

But is that true? People do these estimates and they’re like, “SMIC is actually behind on the process nodes.”

Jensen Huang

I’m about to tell you.

Dwarkesh Patel

Okay.

Jensen Huang

The amount of energy they have is incredible. Isn’t that right? AI is a parallel computing problem, isn’t it? Why can’t they just put 4x, 10x, as many chips together because energy’s free? They have so much energy. They have datacenters that are sitting completely empty, fully powered. You know they have ghost cities, they have ghost datacenters too. They have so much infrastructure capacity. If they wanted to, they just gang up more chips, even if they’re 7nm.

Their capacity of building chips is one of the largest in the world. The semiconductor industry knows that they monopolize mainstream chips. They have over-capacity, they have too much capacity. So the idea that China won’t be able to have AI chips is completely nonsense.

Now, of course, if you ask me, would the United States be further ahead if the entire world had no compute at all? But that’s just not an outcome. That’s not a scenario that’s true. They have plenty of compute already. The amount of threshold they need for the concern you’re worried about, they’ve already reached that threshold and beyond.

So I think you misunderstand that AI is a five-layer cake, and at the lowest layer is energy. When you have an abundance of energy, it makes up for chips. If you have an abundance of chips, it makes up for energy. For example, the United States is scarce on energy, which is the reason why Nvidia has to keep advancing our architecture and do this extreme co-design so that with the few chips that we ship—with the few chips, because the amount of energy is so limited—our throughput per watt is off the charts.

But if your amount of watts is completely abundant, it’s free, what do you care about performance per watt for? You get plenty. You can use old chips to do. So 7nm chips are essentially Hopper. The ability for Hopper… I’ve got to tell you, today’s models are largely trained on Hopper, Hopper generation. So 7nm chips are plenty good. The abundance of energy is their advantage.

Dwarkesh Patel

But then there’s a question of whether they can actually manufacture enough chips.

Jensen Huang

But they do. What’s the evidence? Huawei just had the largest single year in the history of their company.

Dwarkesh Patel

How many chips did they ship?

Jensen Huang

A ton. Millions. Millions is way more than Anthropic has.

Dwarkesh Patel

There’s a question of how much logic SMIC can chip, and there’s a question of how much memory—

Jensen Huang

I’m telling you what it is. They have plenty of logic, and they have plenty of HBM2 memory.

Dwarkesh Patel

Right. But as you know, the bottleneck often in training and doing inference on these models is the amount of bandwidth. So if you have HBM2… I don’t know the numbers offhand but versus the newest thing you have, there could be almost an order of magnitude difference in memory bandwidth, which is huge.

Jensen Huang

Huawei is a networking company.

Dwarkesh Patel

But that doesn’t change the fact that you need EUV for the most advanced HBM.

Jensen Huang

Not true. Not at all true. You could gang them together, just like we gang them together with NVL72. They’ve already demonstrated silicon photonics, connecting all of this compute together into one giant supercomputer. Your premise is just wrong.

The fact of the matter is, their AI development is going just fine. The best AI researchers in the world, because they’re limited in compute, they also come up with extremely smart algorithms. Remember, I just said that Moore’s law is advancing about 25% per year. However, through great computer science, we could still improve algorithm performance by 10x. What I’m saying is that great computer science is where the lever is.

There is no question, MoE is a great invention. There’s no question, all the incredible attention mechanisms reduce the amount of compute. We have got to acknowledge that most of the advances in AI came out of algorithm advances, not just the raw hardware. Now, if most advances came from algorithms and computer science and programming, tell me that their army of AI researchers is not their fundamental advantage. We see it. DeepSeek is not an inconsequential advance. The day that DeepSeek comes out on Huawei first, that is a horrible outcome for our nation.

Dwarkesh Patel

Why is that? Because currently you can have a model like DeepSeek that can run on any accelerator, if it’s open source. Why would that stop being the case in the future?

Jensen Huang

Suppose it doesn’t. Suppose it’s optimized for Huawei, suppose it’s optimized for their architecture. It would put ours at a disadvantage. You described a situation that I perceive to be good news. A company developed software, developed an AI model, and it runs best on the American tech stack. I saw that as good news. You set it up as a premise that it was bad news. I’m going to give you the bad news, that AI models around the world are developed and they run best on non-American hardware. That is bad news for us.

Dwarkesh Patel

I guess I just don’t see the evidence that there’s these huge disparities that would prevent you from switching accelerators. American labs are running their models across all the clouds, across all the different accelerators—

Jensen Huang

I am the evidence. You take a model that’s optimized for Nvidia and you try to run it on something else.

Dwarkesh Patel

But American labs do that.

Jensen Huang

And they don’t run better. Nvidia’s success is perfect evidence. The fact that AI models are created on our stack, run best on our stack, how is that illogical to understand?

Dwarkesh Patel

Anthropic’s models are run on GPUs, they’re run on Trainium, they’re run on TPUs.

Jensen Huang

A lot of work has to go into it to change. But go to the global south, go to the Middle East. Coming out of the box, if all of the AI models run best on somebody else’s tech stack, you’ve got to be arguing some ridiculous claim right now that that’s a good thing for the United States.

Dwarkesh Patel

But I guess I don’t understand the argument. Say Chinese companies get to the next Mythos first. They find all the security vulnerabilities in American software first, but they can do it on Nvidia hardware and they ship it to the global south. They do it on Nvidia hardware. How is that good? Okay, it runs on Nvidia hardware—

Jensen Huang

It’s not good. It’s not good.

Dwarkesh Patel

Right.

Jensen Huang

It’s not good. So let’s not let it happen.

Dwarkesh Patel

Why do you think it’s perfectly fungible, that if you didn’t ship them compute it would exactly be replaced by Huawei? They are behind, right? They have worse chips than you.

Jensen Huang

It’s completely… There’s evidence right now. Their chip industry’s gigantic.

Dwarkesh Patel

You can just look at the flop or bandwidth or memory comparisons between the H200 and the Huawei 910C. It’s like half to a third.

Jensen Huang

They use more of it. They use twice as many.

Dwarkesh Patel

It seems like your argument is they have all this energy that’s ready to go, right? And they need to fill it with chips.

Jensen Huang

And they’re good at manufacturing.

Dwarkesh Patel

And I’m sure eventually they would be able to just out-manufacture everybody. But there are these few critical years.

Jensen Huang

What is the critical year you’re talking about?

Dwarkesh Patel

These next few years. We’ve got these models that are going to be able to do all the cyber attacks.

Jensen Huang

In that case, if the next years are critical, then we have to make sure that all of the world’s AI models are built on the American tech stack, in these critical years.

Dwarkesh Patel

If they’re built on the American tech stack, how would that prevent them, if they have more advanced capabilities, from launching the Mythos-equivalent cyber attacks?

Jensen Huang

There’s no guarantee either way.

Dwarkesh Patel

But if you have it early, we can prepare for it.

Jensen Huang

Listen, why are you causing one layer of the AI industry to lose an entire market so that you could benefit another layer of the AI industry? There are five layers and every single layer has to succeed. The layer that has to succeed most is actually the AI applications. Why are you so fixated on that AI model? That one company? For what reason?

Dwarkesh Patel

Because those models make possible these incredibly offensive capabilities, and you need compute to run them.

Jensen Huang

The energy, the chips, and the ecosystem of AI researchers make it possible.

Dwarkesh Patel

Okay, stepping back, it has to be the case that China is able to build enough 7nm capacity. And remember, they’re still stuck on 7nm while you’ll move on to 3nm and then 2nm or 1.6nm with Feynman. So while you’re on 1.6nm, they’re still going to be on 7nm, and they have to produce enough of it to make up for the shortfall. They have so much energy that the more chips you give them, the more compute they’d have. So it comes out as a question of, ultimately they are getting more compute. Compute is an input to training and inference—

Jensen Huang

Listen, I just think you speak in absolutes. I think the United States ought to be ahead. The amount of compute in the United States is 100x more than anywhere else in the world. The United States ought to be ahead. Okay. The United States is ahead.

Nvidia builds the most advanced technologies. We make sure that the US labs are the first to hear about it and have the first chance to buy it. And if they don’t have enough money, we even invest in them. The United States ought to be ahead. We want to do everything we can to make sure the United States is ahead. Number one point, do you agree? We’re doing everything we can to do that.

Dwarkesh Patel

But how is shipping chips to China keeping the US ahead if they’re bottlenecked on compute?

Jensen Huang

No, no. We’ve got Vera Rubin for the United States. We have Vera Rubin for the United States. Now, am I in the United States? Do you consider me part of the United States?

Dwarkesh Patel

Yes.

Jensen Huang

Nvidia. You consider Nvidia a United States company? Okay. Number one, why is it that we don’t come up with a regulation that’s more balanced so that Nvidia can win around the world instead of giving up the world? Why would you want the United States to give up the world?

The chip industry is part of the American ecosystem. It’s part of American technology leadership. It’s part of the AI ecosystem. It’s part of AI leadership. Why is it that your policy, your philosophy, leads to the United States giving up a vast part of the world’s market?

Dwarkesh Patel

I guess the claim here is… Dario had this quote where he said that it’s like Boeing bragging that we’re selling North Korea nukes, but the missile casings are made by Boeing. And that’s somehow enabling the US technology stack. Fundamentally, you’re giving them this capability.

Jensen Huang

Comparing AI to anything that you just mentioned is lunacy.

Dwarkesh Patel

But AI is similar to enriched uranium, right? It can have positive uses, it can have negative uses. We still don’t want to send enriched uranium to other countries.

Jensen Huang

Who’s sending enriched—

Dwarkesh Patel

The analogy is that enriched uranium is like compute.

Jensen Huang

It’s a lousy analogy. It’s an illogical analogy.

Dwarkesh Patel

But if that compute can run a model that can do zero-day exploits against all American software, how is that not a weapon?

Jensen Huang

First of all, the way to solve that problem is to have dialogues with the researchers and dialogues with China, and dialogues with all the countries to make sure that people don’t use technology in that way. That’s a dialogue that has to happen. Okay? Number one.

Number two, we also need to make sure that the United States is ahead, that Vera Rubin, Blackwell, is available in the United States in abundance, mountains of it. Obviously, our results would show it. Abundance, tons of it. The amount of computing we have is great. We have amazing AI researchers here. It’s great. We ought to stay ahead.

However, we also have to recognize that AI is not just a model. AI is a five-layer cake. The AI industry matters across every single layer, and we want the United States to win at every single layer, including the chip layer. Conceding the entire market is not going to allow the United States to win the technology race long-term in the chip layer, in the computing stack. That is just a fact.

Dwarkesh Patel

I guess then the crux comes down to, how does selling them chips now help us win in the long term? Tesla sold extremely good electric vehicles to China for a long time. iPhones are sold in China, extremely good. They didn’t cause them lock-in. China will still make their version of EVs and they’re dominating. Their smartphones are dominating.

Jensen Huang

When we started the conversation today, you acknowledged that Nvidia’s position is very different. You used words like moat. The single most important thing to our company is the richness of our ecosystem, which is about developers. 50% of the AI developers are in China. The United States should not give that up.

Dwarkesh Patel

But we have a lot of Nvidia developers in the US, and that doesn’t prevent American labs from also being able to use other accelerators in the future. In fact, right now they’re using other accelerators as well, which is fine and great. I don’t see why that wouldn’t be the case in China as well, if you sell them Nvidia chips, just the same way that Google can use TPUs and Nvidia—

Jensen Huang

We have to keep innovating and, as you probably know, our share is growing, not decreasing. The premise that even if we competed in China, that we’re going to lose that market anyways… You’re not talking to somebody who woke up a loser. That loser attitude, that loser premise makes no sense to me.

We’re not a car. We are not a car. The fact that I can buy this car brand one day and use another car brand another day, easy. Computing is not like that. There’s a reason why the x86 deal exists. There’s a reason why ARM is so sticky. These ecosystems are hard to replace. It costs an enormous amount of time and energy, and most people don’t want to do it. So it’s our job to continue to nurture that ecosystem, to keep advancing the technology so that we can compete in the marketplace.

Conceding a marketplace based on the premise you described, I simply can’t acknowledge that. It makes no sense. Because I don’t think the United States is a loser. Our industry is not a loser. That losing proposition, that losing mindset, makes no sense to me.

Dwarkesh Patel

Okay. I’ll move on. I just want to make sure that—

Jensen Huang

You don’t have to move on. I’m enjoying it.

Dwarkesh Patel

Okay, great. Then I won’t. I appreciate that. But I think maybe the crux… and thanks for walking around the circles with me, because I think it helps bring out what the crux here is.

Jensen Huang

The crux is you’re going to extremes. Your argument starts from extremes. That if we give them any compute at all in this narrow moment, we will lose everything.

Dwarkesh Patel

No, I think what my argument is—

Jensen Huang

Those extremes, they’re childish.

Dwarkesh Patel

Let me just make my argument for myself. The idea is not that there is some key threshold of compute. It’s that any marginal compute is helpful. So if you have more compute, you can train a better model.

Jensen Huang

And I just want you to acknowledge that any marginal sales for the American technology industry is beneficial.

Dwarkesh Patel

I actually don’t… If the AI models that run on those chips are capable of cyber offensive capabilities, or the chips are training models with cyber capabilities and running more instances of those models, it is not a nuclear weapon, but it enables a weapon of a kind.

Jensen Huang

The logic that you use, you might as well say it to microprocessors and DRAMs. You might as well say it to electricity.

Dwarkesh Patel

But in fact we do have export controls on the technology that is relevant to making the most advanced DRAM. We have all kinds of export controls on China for all kinds of chip-making stuff.

Jensen Huang

We sell a lot of DRAM and CPUs into China, and I think it’s right.

Dwarkesh Patel

I guess this goes back to the fundamental question of, is AI different? If you have the kind of technology where they can find these zero-days in software, is that something where we want to minimize China’s ability to get there first, to deploy it widely?

Jensen Huang

We want the United States to be ahead. We can control that.

Dwarkesh Patel

How do we control that if the chips are already there and they’re using them to train that model?

Jensen Huang

We have tons of compute. We have tons of AI researchers. We’re racing as fast as we can.

Dwarkesh Patel

Again, we have more nuclear weapons than anybody else, but we don’t want to send enriched uranium anywhere.

Jensen Huang

We’re not enriched uranium. It’s a chip, and it’s a chip that they can make themselves.

Dwarkesh Patel

But there’s a reason they’re buying it from you. We have quotes from the founders of Chinese companies that say that they’re bottlenecked on compute.

Jensen Huang

Because our chips are better. On balance, our chips are better. There’s just no question about it. In the absence of our chip… Can you acknowledge that Huawei had a record year? Can you acknowledge that a whole bunch of chip companies have gone public? Can you acknowledge that?

Dwarkesh Patel

Yes.

Jensen Huang

Can you also acknowledge that we used to have a very large share in that market, and we no longer have a large share in that market? We can also acknowledge that China is about 40% of the world’s technology industry. To concede that market for the United States technology industry is a disservice to our country. It is a disservice to our national security. It is a disservice to our technology leadership, all for the benefit of one company. It makes no sense to me.

Dwarkesh Patel

I guess I’m confused. It feels like you’re making two different statements. One is that we’re going to win this competition with Huawei because our chips are going to be way better if we’re allowed to compete. Another is that they would be doing the same exact thing without us anyway. How can both of those things be true at the same time?

Jensen Huang

It’s obviously true. In the absence of a better choice, you’ll take the only choice you have. How is that illogical? It’s so logical.

Dwarkesh Patel

The reason they want Nvidia chips is that they’re better.

Jensen Huang

Yeah.

Dwarkesh Patel

Better is more compute. More compute means you can train a better model.

Jensen Huang

No, it’s just better. It’s better because it’s easier to program. We have a better ecosystem. But whatever the better is, whatever the better is… And of course we’re going to send them compute. So what? The fact of the matter is that we get to benefit. Don’t forget, we get the benefit of American technology leadership. We get the benefit of developers working on the American tech stack. We get the benefit, as those AI models diffuse out into the rest of the world, that the American tech stack is therefore the best for it. We can continue to advance and diffuse American technology. That, I believe, is a positive. It’s a very important part of American technology leadership.

Now, the policies that you’re advocating resulted in the American telecommunications industry being policied out of basically the world, to the point where we don’t control our own telecommunications anymore. I don’t see that as smart. It’s a little narrow-minded, and it led to unintended consequences that I’m describing to you right now that you seem to have a very hard time understanding.

Dwarkesh Patel

Okay, let’s just step back. It seems like the crux here is there’s a potential benefit and there’s a potential cost. What we’re trying to figure out is, is the benefit worth the cost? I guess I’m trying to get you to acknowledge the potential cost. Compute is an input to training powerful models. Powerful models do have powerful offensive capabilities, like cyber attacks. It is a good thing that American companies got to Mythos-level capabilities first, and then now they’re going to hold off on those capabilities so that the American companies and American government can make their software more protected before that level of capability was announced.

If China had had more compute or more crowd compute, if they could have made a Mythos-level model earlier and deployed it widely, that would have been very bad. One of the reasons that hasn’t happened is that we have more compute thanks to companies like Nvidia in America. That is a cost of sending it to China. So let’s leave the benefit aside for a second. Do you acknowledge that this is a potential cost?

Jensen Huang

I’ll also tell you the potential cost is we allow one of the most important layers of the AI stack, the chip layer, to concede an entire market—the second largest market in the world—so that they could develop scale, so that they could develop their own ecosystem, so that future AI models are optimized in a very different way than the American tech stack. As AI diffuses out into the rest of the world, their standards, their tech stack, will become superior to ours, because their models are open.

Dwarkesh Patel

I guess I just believe enough in Nvidia’s kernel engineers and CUDA engineers to think that they could optimize—

Jensen Huang

AI is more than kernel optimization, as you know.

Dwarkesh Patel

Of course, but there are so many things you can do, from distilling to a model that’s well-fit for your chips.

Jensen Huang

We’re going to do our best.

Dwarkesh Patel

You have all the software. It’s just hard to imagine that there’s a long-term lock-in to the Chinese ecosystem, even if they have a slightly better open source model for a while.

Jensen Huang

China is the largest contributor to open source software in the world. Fact. China’s the largest contributor to open models in the world. Fact. Today it’s built on the American tech stack, Nvidia’s. Fact.

All five layers of the tech stack for AI are important. The United States ought to go win all five of them. They’re all important. The one that is the most important, of course, is the AI application layer. The layer that diffuses into society, the one that uses it most will benefit from this industrial revolution most. But my point is that every layer has to succeed.

If we scare this country into thinking that AI is somehow a nuclear bomb, so that everybody hates AI and everybody’s afraid of AI, I don’t know how you’re helping the United States. You’re doing it a disservice. If we scare everybody out of doing software engineering jobs because it’s going to kill every software engineering job—and we don’t have any software engineers as a result of that—we’re doing a disservice to the United States.

If we scare everybody out of radiology so nobody wants to be a radiologist because computer vision is completely free and no AI is going to do a worse job than a radiologist, we misunderstand the difference between a job and a task. The job of a radiologist is patient care. The task is to read a scan. If we misunderstand that so profoundly and we scare everybody out of going to radiology school, we’re not going to have enough radiologists and good enough healthcare.

So I’m making the case that when you make a premise that is so extreme, everything goes from zero or infinity, we end up scaring people in a way that’s just not true. Life is not like that. Do we want the United States to be first? Of course we do. Do we need to be a leader in every layer of that stack? Of course we do. Of course we do. Today you’re talking about Mythos because Mythos is important. Sure. That’s fantastic.

But in a few years time, I’m making you the prediction that when we want the American tech stack, when we want American technology to be diffused around the world—out to India, out to the Middle East, out to Africa, out to Southeast Asia—when our country would like to export, because we would like to export our technology, we would like to export our standards, on that day, I want you and I to have that same conversation again. I will tell you exactly about today’s conversation, about how your policy and what you imagined literally caused the United States to concede the second largest market in the world for no good reason at all.

We shouldn’t concede it. If we lose it, we lose it. But why do we concede it? Now nobody is advocating an all or nothing. Nobody’s advocating all or nothing, meaning we ship everything to China at all times. Nobody’s advocating that. We should always have the best technology here. We should always have the most technology here, and the first. But we should also try to compete and win around the world. Both of those things can simultaneously happen. It requires some amount of nuance, some amount of maturity instead of absolutes. The world is just not absolutes.

Dwarkesh Patel

Okay. The argument hinges on this. They’ve built models that are specified for the best chips that they make in a few years. Those chips get exported around the world. That sets the standard. Because of EUV export controls, as we said, you’re going to move on to 1.6nm. They’re still going to be on 7nm, even after a few years from now.

It may make sense that domestically they would prefer, “Hey, we’ve got so much energy, we can manufacture at scale. We’ll still keep using 7nm.” But on the exporting thing, their 7nm chips have to be competitive against your 1.6nm chips. Their models have to be so far optimized for the 7nm that it’s better to run their models on 7nm than to run their models on your 1.6nm.

Jensen Huang

Can we just look at the facts then? Is Blackwell 50 times more advanced lithography than Hopper? Is it 50 times? Not even close. I just kept saying it over and over again. Moore’s Law is dead. Between Hopper and Blackwell, from the transistors themselves, call it 75%. It was three years apart, 75%. Blackwell is 50 times Hopper.

My point is, architecture matters. Computer science matters. Semiconductor physics matters as well, but computer science matters. The impact of AI largely comes from the computing stack, which is the reason why CUDA is so effective, which is the reason why CUDA is so beloved. It’s an ecosystem, a computing architecture that allows for so much flexibility that if you wanted to change an architecture completely—create something like MoE, create something like diffusion, create something that’s disaggregated—you could do so. It’s easy to do.

So the fact of the matter is, AI is about the stack above as much as it is about the architecture below. To the extent that we have architectures and software stacks that are optimized for our stack, for our ecosystem, it is obviously good, because we started the conversation today about how Nvidia’s ecosystem is so rich. Why do people always love programming CUDA first? They do. They do. So do the researchers in China.

But if we are forced to leave China, if we’re forced to leave China, first of all, it’s a policy mistake. Obviously it has backlash. It has turned out badly for the United States. It enabled, it accelerated their chip industry. It forced all of their AI ecosystem to focus on their internal architectures. It’s not too late, but nonetheless it has already happened.

You’re going to see in the future, they’re not stuck at 7nm, obviously. They’re good at manufacturing. They will continue to advance from 7nm and beyond. Now, is there a 10x difference between 5nm and 7nm? The answer is no. Architecture matters. Networking matters. That’s why Nvidia bought Mellanox. Networking matters. Energy matters. So all of that stuff matters. It’s not simplistic, like the way you’re trying to distill it.

01:35:06 – Why doesn’t Nvidia make multiple different chip architectures?

Dwarkesh Patel

We can move on from China, but that actually raises an interesting question. We were discussing earlier these bottlenecks at TSMC and memory and so forth.

So if we’re in this world where you’re already the majority of N3—and at some point you’ll be N2 and you’ll be a majority of that—do you see that you could go back to N7, the spare capacity at an older process node, and say, “Hey, the demand for AI is so great and our capacity to expand the leading edge is not meeting it, so we’re going to make a Hopper or Ampere, but with everything we know about numerics today and all the other improvements you described”? Do you see that world happening before 2030?

Jensen Huang

It’s not necessary to. The reason for that is because with every generation, the architecture is more than just the transistor scale. You’re doing so much engineering and packaging and stacking, and the numerics and the system architecture.

When you run out of capacity, to easily go back to another node… That’s a level of R&D that no one could afford. We could afford to lean forward. I don’t think we could afford to go back. Now, if the world simply says… If on that day, let’s do the thought experiment, on that day we go, “Listen, we’re just never going to have more capacity ever again.” Would I go back and use 7nm? In a heartbeat, of course I would.

Dwarkesh Patel

One question somebody I was talking to had is, why doesn’t Nvidia run multiple different chip projects at the same time with totally different architecture? So you could do something like a Cerebras-style wafer scale. You could do a Dojo-style huge package. You could do one without CUDA. You have the resources and the engineering talent to do all of these in parallel. So why put all the eggs in one basket, given who knows where AI might go and architectures might go?

Jensen Huang

Oh, we could. It’s just that we don’t have a better idea. We could do all of those things. It’s just not better. We simulate it all in our simulator, proveably worse. So we wouldn’t do it. We’re working on exactly the projects that we want to work on. If the workload were to change dramatically—and I don’t mean the algorithms, I actually mean the workload, and that depends on the shape of the market—we may decide to add other accelerators.

For example, recently we added Groq, and we’re going to fold Groq into our CUDA ecosystem. We’re doing that now because the value of tokens has gone up so high that you could have different pricing of tokens. Back in the old days, just a couple years ago, tokens were either free or barely expensive. But now you can have different customers, and those customers want different answers. Because the customers make so much money—for example, our software engineers—if I can give them much more responsive tokens so that they’re even more productive than they are today, I would pay for it.

But that market has only recently emerged. So I think we now have the ability to have the same model, based on the response time, have different segments. That’s the reason why we decided to expand the Pareto frontier and create a segment of inference that is faster response time, even though it’s lower throughput. Until now, higher throughput is always better. We think there could be a world where there could be very high ASP tokens, and even though the throughput is lower in the factory, the ASPs make up for it.

That’s the reason why we did it. But otherwise, from an architecture perspective, if I had more money, I would put more behind Nvidia’s architecture.

Dwarkesh Patel

I think this idea of extremely premium tokens and just the disaggregation of the inference market is a very interesting.

Jensen Huang

The segmentation of it.

Dwarkesh Patel

Yeah. Alright, final question. Suppose the deep learning revolution didn’t happen. What would Nvidia be doing? Obviously games, but given—

Jensen Huang

Accelerated computing, the same thing we’ve been doing all along. The premise of our company is that Moore’s law is going to… General purpose computing is good for a lot of things, but for a lot of computation it’s not ideal.

So we combined an architecture called a GPU, CUDA, to a CPU, so that we can accelerate the workload of the CPU. Different kernels of code or algorithms could be offloaded onto our GPU. As a result, you speed up an application by 100x, 200x. Where can you use that? Obviously engineering and science and physics, data processing, computer graphics, image generation, all kinds of things. Even if AI doesn’t exist today, Nvidia would be very, very large.

The reason for that is fairly fundamental, which is that the ability for general purpose computing to continue to scale has largely run its course. And the only way… Not the only way, but the way to do that is through domain-specific acceleration. One of the domains that we started with was computer graphics, but there are many other domains. There’s all kinds. Particle physics and fluids, structured data processing, all kinds of different types of algorithms that benefit from CUDA.

Our mission was really to bring accelerated computing to the world and advance the type of applications that general purpose computing can’t do, and scale to the level of capability that helps break through certain fields of science. Some of the early applications were molecular dynamics, seismic processing for energy discovery, image processing of course, all of those kinds of fields where general purpose computing is just simply too inefficient to do so.

If there were no AI, I would be very sad. But because of the advances that we made in computing, we democratized deep learning. We made it possible for any researcher, any scientist, anywhere, any student, to be able to access a PC or a GeForce add-in card and do amazing science. That fundamental promise hasn’t changed, not even a little bit.

If you watch GTC, there’s the whole beginning part of it. None of it’s AI. That whole part of it with computational lithography or our quantum chemistry work, data processing work, all of that stuff is unrelated to AI. And it’s still very important. I know that AI is very interesting and quite exciting, but there’s a lot of people doing a lot of very important work that’s not AI related, and tensors are not the only way that you compute it. We want to help everybody.

Dwarkesh Patel

Jensen, thank you so much.

Jensen Huang

You’re welcome. I enjoyed it.

Dwarkesh Patel

Me too.

Can distillation be stopped?

Can the frontier labs stop distillation? Because if they can’t, open source commoditizing models can catch up incredibly rapidly, making the long run business model for the labs less viable. Let’s say it takes 1T tokens from a frontier model to capture its juice (I have no idea if that’s correct, but let’s say). Even ignoring savings from caching, Opus 4.6 is $25/MTok. So $25 million for those 1T tokens. That’s nothing.

Labs are responding by hiding chain of thought. But there’s two problems with this solution:

• Chain of thought is not made of some fundamentally different kind of token. You can just instruct the model to not think first but just start solving the problem, or to write out its thinking somewhere else.

• Even if labs do figure out how to robustly hide chain of thought to train in the future, you can make reconstructing the chain of thought necessary to reproduce a decoded sequence as an RLVR target. Yes that costs more, but seems doable.

• Maybe most importantly, the real juice of these agentic models is their tool use (writing and updating files of code, running bash commands, etc). And if these things are done locally on the user’s computer, you can’t really hide them. And it seems like a hard lift to get users to migrate all their development workflows to a cloud that you fully control and hide visibility to, modulo a Claude agent input text prompt.

By the way, I learned about an interesting way companies which build products atop API access to AI models can basically distill these models, in a way that potentially makes the distilled models even better than the ones they’re actually built atop.

Suppose you’ve got a coding product. In order to build a feature, a user uses your product to query some frontier model API across 10+ back and forths. Once the user is satisfied with the end result, you have the end state that the user actually wanted - “the gold diff”. These coding product companies can now set the gold diff as the RL target for training their own models, where the model gets rewarded for producing outputs that look like what users eventually converged on, and penalized for producing the kinds of intermediate outputs that users kept rejecting or editing.

Mythos and the cybersecurity equilibrium

It seems like the key difference between Mythos and previous versions is that while previous versions could find individual vulnerabilities in the code (“Hey, there’s a missing bounds check here”), Mythos is long run agentic enough to rope 5 different vulnerabilities together which are all required in order to find an exploit (“Now I can execute arbitrary code, escalate privileges, etc”). To the extent that some discontinuity has been hit, it’s probably more the result of the combinatorial nature of cyberattacks rather than some off-trend increase in intelligence.

What does this mean for offense/defense? One way to look at it is that software is more secure today than it was 20 years ago, despite more and more human intelligence probing at public code, both white hat and black hat. If we get another influx of intelligence suddenly, why should the dynamic change?

In fact, we know that our foreign adversaries almost certainly have access to a bunch of critical zero days which they’re saving for a rainy day, or already using in inconspicuous ways. To the extent that Glasswing allows the whole industry to find a bunch of these latent exploits and patch them, shouldn’t we expect defense to have become much stronger relative to offense by the end of 26? Of course, this is thanks to the fact than American companies got there first and are cooperating with other companies and our government to patch things before our adversaries get to the same level.

One counterpoint I heard from a security expert is that there’s big difference between finding vulnerabilities and patching them - and AI is much better at the first than the later (people often talk about the offense/defense balance, but difficulty of finding versus patching vulnerabilities seems much more significant). In order to patch an issue, you have to find a fix that will not interfere with all the ways people use your software, and all the features which rely on weird bespoke behavior. XKCD has a nice comic illustrating how these kinds of issues come up:

Potential solutions, if it’s non-trivial to just push patches to every piece of software?

• TODO - I know nothing about formal verification of software - check out what a seL4 proof of some behavior might look like

• Use LLMs to rapidly port all C to Rust. Curious how easily Mythos can find vulnerabilities in memory safe languages.

In some sense, its good that Anthropic didn’t release this model publicly until critical IT could be patched up. In another sense, isn’t it a super bad precedent for private companies to be hoarding the ability to be able to break into any operating system and browser and device? One obvious question for Anthropic is why they didn’t just build some kind of classifier which would detect whether you’re using the model for cyberattack type stuff, and refuse requests if yes, and release that publicly.

• Patching your own software is isomorphic to finding bugs in someone else’s repo from the perspective of an LLM (and patching your own software is a frequent coding model use case).

• These kinds of classifiers can be easy to evade if you have enough expertise to break the problem of finding exploits down into smaller subproblems of finding vulnerabilities which each individually seem like sensibly good behavior to an LLM with no memory

Pipeline RL paper summary

As you keep RLing a model, not only does the average length of a response increase (since you’re basically training the model to think for longer before answering) but the variance in length also increases - sometimes you get an easy problem and you can immediately answer it - other times, you need to go think for 100k tokens.

This is a big problem for GPU utilization on training. Because you have to wait for all these stragglers to finish generating before you can start the next training step.

Okay one way you could get out of this conundrum is to just to just batch generation so that while stragglers keep going, you generate even more rollouts.

The problem is that there is an optimal batch size for each training step, so you’d need to split all these rollouts you made across lots of consecutive training steps.

But this takes you into the domain of offline RL, because your model is changing with each training step. And so you’re training your model on trajectories that were actually generated by an earlier model, which is not ideal.

Pipeline RL paper proposes the following fix: in flight weight weight updates - where you just sub out the generating model partway though these generating trajectories as soon as the new training step is done, so all the short trajectories, and a good chunk of the long trajectories, that the next training step will be trained on are generated by the most recent version of the model.

It’s especially relevant for closing the RL verification loop for scientific discovery.

But it’s also a surprisingly mysterious and elusive question when you look at the history of human science.

We approach this question stories like Einstein (who claimed that he hadn't even heard of the famous Michelson-Morley experiment, which is supposed to have motivated special relativity, until after he had come up with the theory), Darwin (why did it take till 1859 to lay out an idea whose essence every farmer since antiquity must have observed?), Prout (how do you recognize that isotopes exist if you cannot chemically separate them?), and many others.

The verification loop on scientific ideas is often extremely long and weirdly hostile. Ancient Athenians dismissed Aristarchus’s heliocentrism in the 3rd century BC because it would imply that the stars should shift in the sky as the Earth orbits the sun. The first successful measurement of stellar parallax was in 1838. That’s a 2,000-year verification loop.

But clearly human science is able to make progress faster than raw experimental falsification/verification would imply, and in cases where experiments are very ambiguous. How?

Michael has some very deep and provocative hypotheses about the nature of progress. One I found especially thought-provoking is that aliens will likely have a VERY different science + tech stack than us. Which contradicts the common sense picture of a linear tech tree that I was assuming. And has some interesting implications about how future civilizations might trade and cooperate with each other.

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Labelbox researchers built a new safety benchmark. Why? Well, current safety benchmarks claim that attacks on top models are successful only a few percent of the time, but the prompts in those benchmarks don’t reflect how real bad actors actually write. You can read Labelbox’s research here. If this could be useful for your work, reach out at labelbox.com/dwarkesh

• Mercury has an MCP that lets you give an LLM access to your full transaction history, including things like attached receipts and internal notes. I just used it to categorize my 2025 transactions, and it worked shockingly well. Modern functionality like this is exactly why I use Mercury. Learn more at mercury.com

• Jane Street’s ML engineers presented some of their GPU optimization workflows at GTC, showing how they use CUDA graphs, streams, and custom kernels to shave real time off their training runs. You can watch the full talk here. And they open-sourced all the relevant code here. If this kind of stuff excites you, Jane Street is hiring — learn more at janestreet.com/dwarkesh

Timestamps

00:00:00 – How scientific progress outpaces its verification loops

00:17:51 – Newton was the last of the magicians

00:23:26 – Why wasn’t natural selection obvious much earlier?

00:29:52 – Could gradient descent have discovered general relativity?

00:50:54 – Why aliens will have a different tech stack than us

01:15:26 – Are there infinitely many deep scientific principles left to discover?

01:26:25 – What drew Michael to quantum computing so early?

01:35:29 – Does science need a new way to assign credit?

01:43:57 – Prolificness versus depth

01:49:17 – What it takes to actually internalize what you learn

Transcript

00:00:00 – How scientific progress outpaces its verification loops

Dwarkesh Patel

Today, I’m speaking with Michael Nielsen. You have done many things. You’re one of the pioneers of quantum computing, wrote the main textbook in the field of the open science movement. You wrote a book about deep learning that Chris Olah and Greg Brockman credit with getting them into the field. More recently, you’re a research fellow at the Astera Institute and writing a book about religion, science, and technology.

I’m going to ask you about none of those things. The conversation I want to have today is, how do we recognize scientific progress? It’s especially relevant for AI because people are trying to close the RL verification loop on scientific discovery. What does it mean to close that loop? But in preparing for this interview, I’ve realized that it’s a more mysterious and elusive force, even in the history of human science, than I understood.

I think a good place to start will be Michelson-Morley and how special relativity is discovered, if it’s different from the story that you get off of YouTube videos. I will prompt you that way, and then we’ll go in there.

Michael Nielsen

Michelson-Morley is the famous result often presented as this experiment that was done in the 1880s that helped Einstein come up with the special theory of relativity a little bit later, changing the way we think about space and time and our fundamental conception of those things.

And there’s a big gap, I think, between the way Michelson and Morley and other people at the time thought about the experiment and certainly the way in which Einstein thought or did not think about the experiment. In actual fact, he stated later in his life he wasn’t even sure whether he was aware of the paper at the time. There’s a lot of evidence that he probably was aware of the paper at the time, but it actually wasn’t dispositive for his thinking at all. Something else completely was going on.

What Michelson and Morley thought they were doing was testing different theories of what was called the ether. If you go back to the 1600s, Robert Boyle introduced the idea of the ether. We know that sound is vibrations in the air. Boyle and other people got interested in the question of whether light is vibrations in something, and they couldn’t figure out what it was. Boyle did an experiment where he tested whether you could propagate light through a vacuum. He found that you could. You couldn’t do it with sound. He introduced this idea of the ether, and for the next two hundred or so years, people had all these conversations about what the ether was and what its nature was.

The Michelson and Morley experiment was really an experiment to test different theories of the ether against one another, in particular to find out whether or not there was a so-called ether wind. The idea was that the Earth is maybe passing through this ether wind. And if it is passing through the ether wind and you shoot a light beam parallel to the direction the ether wind is going in, it’ll get accelerated a little bit. If it’s being passed back in the opposite direction, it’ll get slowed down a little bit, and you should be able to see this in the results of interference experiments. What they found, much to their surprise, was that in fact there was no ether wind. That ruled out some theories of the ether, but not all, and Michelson certainly continued to believe in the ether.

Dwarkesh Patel

This is what was a shocking part of reading this story from the biography of Einstein that you recommended by... what was his first name?

Michael Nielsen

Abraham Pais.

Dwarkesh Patel

Abraham Pais. Subtle is the Lord. Also from Imre Lakatos, The Methodology of Scientific Research Programmes. The way it’s told is that Michelson-Morley proved that the ether did not exist. Therefore, it created a crisis in physics that Einstein solved with special relativity.

What you’re pointing out is he actually was trying to distinguish between many different theories of ether. If you’re in space or if you’re on Earth, it’s the same direction of ether, or maybe the ether wind is being carried around by the Earth, and so you can’t really experience it on Earth. But if you go to a high enough altitude, you might be able to experience it. In fact, Michelson’s experiments, the famous one is 1887, but he conducted these experiments for basically two decades.

Michael Nielsen

For longer than that. He conducted the first one in 1881, I think, but he continued to believe until he died. He died, I think it was 1929 or so. It was the late twenties. He was still doing experiments in the 1920s about whether or not the ether existed. So he continued to believe in the ether to the end of his life. I think the last public statement he made was a year or two before he died, and he basically still believed it at that point.

Dwarkesh Patel

In fact, there was another physicist, Miller, who kept doing these experiments in the 1920s. He thought that if he went to a high enough altitude, Mount Wilson in California… “Oh, I’m high enough that the ether winds are not being dragged by the Earth. And I’ve measured the effect of the ether.” Einstein hears about this and he says, and this is where you get the famous quote, “Subtle is the Lord, but malicious He is not.”

Anyways, I think the reason the story is interesting is for many different reasons. One of the ways in which the real history of science is different from this idea you get of the scientific method is that you really can’t apply falsification as easily as you might think. It’s not clear what is being falsified. Is it just another version of the theory of the ether that’s being falsified? Certainly you can’t induce the theory of special relativity from the fact that one version of the ether seems to be disconfirmed by these experiments.

Michael Nielsen

It certainly doesn’t show that ideas about falsification are wrong or falsified, but it does show that the most naive ideas… Things are often much more complicated than you think. Michelson did this experiment in 1881. He was a very young man, and then other people, I think Rayleigh was one of them, pointed out that there were some problems with the way he did it, so they had to redo it in 1887. At that point, a lot of the leading physicists of the day basically accepted this result, that there was no ether wind. But what to do about this?

Sure, maybe you falsified some theories of the ether. There are others that you haven’t falsified at all at this point, and people set to work on developing those. It is funny, people will phrase it as showing that the ether didn’t exist. Even just the word “the” there is a misnomer. You actually had a ton of different theories and a couple of leading contenders. So yes, there’s some version of falsification going on, but how you respond to this new experiment is very complicated. Certainly the leading physicists of the day responded by saying, “Okay, this gives us a lot of information about what the ether must be, but it doesn’t tell us that there is no ether.”

Dwarkesh Patel

In fact, Lorentz at the end of the 19th century, before Einstein, figures out the math of how you convert from one reference frame to another reference frame, and comes up with the Lorentz transformations, which is the basis of special relativity. But his interpretation is that you are converting from the ether reference frame to these non-privileged other reference frames if you’re moving relative to the ether.

His interpretation of length contraction and time dilation is that this is the effect of moving through the ether, and you have this pressure. This pressure is warping clocks. It’s warping measures of length. The interesting thing here is that experimentally you cannot distinguish Lorentz’s interpretation from special relativity.

Michael Nielsen

I think that’s a strong statement. Lorentz introduces this quantity called local time, which he regards as... My understanding is he’s not trying to give a physical interpretation of this, but it’s what Einstein would later just recognize as time in another inertial reference frame. He’s not trying to attribute much physical meaning to it. I think Poincaré gets much closer later on to realizing that this is the time that’s registered by clocks.

About forty-odd years later, people start doing these muon experiments where they see cosmic rays hit the top of the atmosphere. They produce a shower of muons, and you can look to see at different heights in the atmosphere how many of those muons remain. They decay over time, and a very strange thing happens, which is that they’re decaying way too slow. You expect they shouldn’t be able to last the whole way through the atmosphere at all. Their decay rate is too quick, if you were in a classical theory. But if in fact their time really has slowed down, it’s okay.

In fact, the measured decay rates in 1940—and there have since been more accurate experiments done—match exactly what you expect from special relativity. That’s the kind of thing where if Lorentz had been alive—he’d been dead ten or so years at that point—it seems quite likely that he would have tried to save his theory by patching it up yet again, but it would have been a massive setback. It starts to just look like time—this thing that Lorentz introduced as a mathematical convenience—that’s actually what time is, for the muons at least. Then there’s a whole bunch of other experiments that show this very similar phenomenon.

Dwarkesh Patel

When was that experiment done?

Michael Nielsen

That was, I think, 1940. It might have been published in 1941.

Dwarkesh Patel

Maybe to rephrase and change my claim: it’s not that you could not have distinguished them, but the scientific community adopted what we in retrospect consider the more correct interpretation before it was actually experimentally shown to be preferred. So there’s clearly some process that human science does which can distinguish different theories.

Michael Nielsen

Can I just interrupt? You used the word process, and it’s interesting to think about that term. Process carries connotations of something set in advance. It’s much more complicated in practice. You have people like Lorentz, who Einstein absolutely and utterly admired, and Poincaré, one of the greatest scientists who ever lived, and Michelson, another truly outstanding scientist, who never reconciled themselves.

It’s not as though there’s some standard procedure that we’re all using to reconcile these things. Great scientists can remain wrong for a very long time after the scientific community has broadly changed its opinion. But there’s no centralized authority or centralized method.

Dwarkesh Patel

That is the interesting thing. There’s progress even though it is hard to articulate the process by which it happens, the heuristics that are used.

You mentioned Poincaré. Lorentz has the math right, but the interpretation wrong. It seems like Poincaré had the opposite, where he understood that it’s hard to define simultaneity because it requires a circular definition with time, or velocity of something that might arrive at a midpoint together, but velocity is defined in terms of time. I find this interesting.

There are a couple of other examples we could call on. There is this phenomenon in the history of science where somebody asks the right question, but then they don’t clinch it. I’m curious what you think is happening in those cases.

Michael Nielsen

You actually do want to go case by case and try to understand. It’s not necessarily clear that they’re doing the same thing wrong in all of the cases. The Poincaré case is amazing. He seems to have understood the principle of relativity, the idea that the laws of physics are the same in all inertial reference frames. He seems to have understood that the speed of light is the same in all inertial reference frames. He doesn’t phrase it quite that way, but it is my understanding, though I don’t speak French.

These are basically the ideas that Einstein uses to deduce special relativity. But then he also has this additional misunderstanding where he thinks that length contraction is a dynamical effect, that somehow particles are being pushed together by some external force, something is going on dynamically. He doesn’t understand that it’s purely kinematics. That actually space and time are different from what we thought, and you need to fundamentally rethink those things.

It’s almost like he knew too much. He had almost too grand a vision in mind. Einstein subtracts from that and says, “No. Space and time are just different than what we thought, and here’s the correct picture.” There’s a paper in, I think it’s 1909, where Poincaré still has this dynamical picture of what’s going on with the length contraction. This is just not necessary. This is a mistake from the modern point of view.

Why is he doing this? Why is he clinging onto this idea? I don’t know. I’ve obviously never met the man. It would be fascinating to be able to talk it over and try and understand. His expertise seems to be getting in the way. He knows so much, he understands so much, and then he’s not able to let go of these things.

A really interesting fact is that a few years prior, in the 1890s, Einstein’s a teenager and he believes in the ether too. He knows about this stuff. But he’s not quite as attached as these older people were. Maybe they were a little bit prisoners of their own expertise. That’s my guess. Some historians of science would certainly disagree.

Dwarkesh Patel

Then there’s the obvious stories where Einstein himself later on is said to have not latched onto the correct interpretations of quantum mechanics or cosmology because of his own attachments.

Michael Nielsen

Yeah.

Dwarkesh Patel

Here’s the bigger question I have. The muon example is a great example of these long verification loops and how progress seems to happen in the scientific community faster than these verification loops imply. Maybe the clearest example is Aristarchus in the second century BC comes up with the idea of heliocentrism. The ancient Athenians dismiss it on the grounds that we should see as the Earth is moving around the Sun, if really the Sun is the center of the solar system, the stars move relative to the Earth. The only reason that would not be the case is the stars are so far away that you would not observe this.

And it’s only in 1838 that stellar parallax was actually measured. And so, we didn’t need to wait until 1838 to have heliocentrism. We didn’t need to wait for the experimental validation to understand that Copernicus is better in some way. In fact, when Copernicus first came up with his theories, it’s well known that the Ptolemaic model was more accurate because it had centuries of adding on these epicycles.

What’s maybe less well appreciated is that it was also in some sense simpler. Because Copernicus actually had to add extra epicycles. It had more epicycles than the Ptolemaic model because he had this bias that the Earth should go in a perfect circle in equal time. Anyway, I think this is an interesting story because it’s not a more accurate theory. It’s not a simpler theory. So how could you have known ex ante that Copernicus was correct and Ptolemy was not?

Michael Nielsen

Good question. I don’t entirely know the answer. I can give you a partial answer that I, centuries in the future, start to find very compelling. I’m sure it’s part of the historic story at least. One of the big shocks for Newton, he did understand Kepler’s laws of motion eventually, so you’re able to explain the motions of the planets in the sky. But he also, out of the same theory, his theory of gravitation, was able to explain terrestrial motion. He’s able to explain why objects move in parabolas on the Earth, and he’s able to explain the tides in terms of the moon and the sun’s gravitational effect on water on the Earth.

You have what seem like three very different disconnected phenomena all being explained by this one set of ideas. That starts to feel very compelling, at least to me. I think most people find that very satisfying once they eventually realize it.

00:17:51 – Newton was the last of the magicians

Dwarkesh Patel

Have you read the Keynes biography of Newton?

Michael Nielsen

He wrote an entire biography?

Dwarkesh Patel

No, the essay.

Michael Nielsen

Sure. I love that. This description of him as the last of the magicians is wonderful.

Dwarkesh Patel

In fact, I think it’s maybe worth superimposing. Or you should read out that one passage of the thing.

Michael Nielsen

Alright. It’s from a talk that he gave at Cambridge not long before he died. He’d acquired Newton’s papers somehow and gave a lecture twice about this, or his brother Jeffrey gave it the other time because he was too ill. There’s this wonderful, wonderful quote in the middle. The whole thing is really interesting, but I love this particular quote: “Newton was not the first of the age of reason. He was the last of the magicians, the last great mind which looked out on the visible and intellectual world with the same eyes as those who began to build our intellectual inheritance rather less than ten thousand years ago.”

This idea people have that Newton was the first modern scientist is somehow wrong. There’s some truth to it, but he really had this very different way of looking at the world that was part superstitious and part modern. It was a funny hybrid. He’s a transitional figure in some sense. That phrase, “the last of the magicians,” really points at something.

Dwarkesh Patel

The thing I’m very curious about with Newton is whether it was the same program, the same heuristics, the same biases that he applied to his alchemical work as he did to his understanding of astronomy. This is from the Keynes essay: “There was extreme method in his madness. All his unpublished works on esoteric and theological matters are marked by careful learning, accurate method, and extreme sobriety of statement. They are just as sane as the Principia if their whole matter and purpose were not magical. They were nearly all composed during the same 25 years of his mathematical studies.”

Clearly, there was some aesthetic that motivated people like Einstein to reject earlier ways of thinking and say, “No, the other is wrong, and there’s a better way to think about things.” The same is true with Newton. The question I have is whether similar heuristics toward parsimony, aesthetics, and so on, would be equally useful across time and across disciplines, or whether you need different heuristics. The reason that’s relevant is even if we can’t build a verification loop for science, maybe if the taste tests point in the same direction, you can at least encode that bias into the AIs. That would maybe be enough.

Michael Nielsen

The point is that where we always get bottlenecked is where the previous processes and heuristics don’t apply. That’s almost definitionally what causes the bottlenecks. Because people are smart, they know what has worked before. They study it. They apply the same kinds of things, so they don’t get stuck in the same places as before. They keep getting bottlenecked in different places. I’m overgeneralizing a bit, but I think it’s right.

If you’re attempting to reduce science to a process, you’re attempting to reduce it to something where there is just a method which you can apply, and you turn the crank and out pops insight. You can do a certain amount of that, but you’re going to get bottlenecked at the places where your existing method doesn’t apply. Definitionally, there’s no crank you can turn. You need a lot of people trying different ideas. The more difficult the idea is to have, the greater the bottleneck, but then also the greater the triumph.

Quantum mechanics is a great example of this. It’s such a shocking set of ideas. It’s such a shocking theory. The theory of evolution in some sense is also quite a shocking idea, not the principle of natural selection, but that it can explain so much. That’s a shocking idea.

00:23:26 – Why wasn’t natural selection obvious much earlier?

Dwarkesh Patel

Principia Mathematica is released in 1687. The Origin of Species is released in 1859. At least naively, it seems like Darwin’s theory of natural selection is conceptually easier than the theory of gravity.

I asked Terence Tao this question. There was this contemporaneous biologist with Darwin, Thomas Huxley, who read this and said, “How extremely stupid to not have thought of this.” Nobody ever reads the Principia Mathematica and thinks, “God, why didn’t I beat Newton to the punch here?” So what’s going on here? Why did Darwinism take so much longer?

Michael Nielsen

The idea must have been known to animal breeders for a long time at some level, or certainly large chunks of the idea were known, that artificial selection was a thing. In some sense, Darwin’s genius wasn’t in having that idea, it was understanding just how central it was to biology. You can go back and explain a tremendous amount about all the variety of what we see in the world with this as not necessarily the only principle, but certainly a core principle. He writes this wonderful book, The Origin of Species. It’s just so much evidence and so many examples, trying to tease this out and see what the implications are, and connecting it to as much else as he possibly can, to geology and all these other things.

That hard work—making the case that it’s actually relevant all across the biosphere—is what he’s doing there. He’s not just having the idea, he’s making a compelling case that it’s intertwined with absolutely everything else.

Dwarkesh Patel

The motivation for the question was Lucretius, this first-century Roman poet who has an idea that seems analogous to natural selection. It’s about species getting fitted more over time to their environments, or species losing fit to their environment. And so, why did this go nowhere for nineteen centuries?

Then I looked into it or, more accurately, asked LLMs what exactly Lucretius’s idea here was. It is extremely different from what real natural selection is. He thought there was this generative period in the past where all the species came about, and then there was this one-time filter which resulted in the species that are around today, and they became fit to the environment.

He did not have this idea that it is an ongoing gradual process or that there is a tree of life that connects all life forms on Earth together, which, by the way, is an incredibly weird fact that every single life form on Earth has a common ancestor.

Michael Nielsen

It’s not incredibly weird. If you think that the origin of life must have been very hard, that there’s a bottleneck there, then it’s not so surprising.

Dwarkesh Patel

There’s also this verification loop aspect where even if Newton might be harder in some sense, if you’ve clinched it, you can experimentally… I know “validate” is the wrong word philosophically, but you can give a lot of base points to the theory.

You can be like, “Okay, I have this idea of why things fall on Earth. I have this idea of why orbital periods for planets have a certain pattern. Let’s try it on the Moon, which orbits the Earth.” And in fact, it’s weird but the orbital period matches what my calculations imply.

Michael Nielsen

And the tides work correctly. It’s just amazing.

Dwarkesh Patel

Exactly. Whereas for Darwinism, it takes a ton of work for Darwin to compile all the cumulative evidence, but there’s no individual piece that is overwhelmingly powerful.

Michael Nielsen

And there’s a whole bunch of problems as well. He doesn’t really understand what the mechanism is. He doesn’t understand genes, all these things.

Dwarkesh Patel

The very interesting thing in the history of Darwinism is, this idea which theoretically you could come up with at any time, there is almost identical independent creation of that idea between Alfred Wallace and Charles Darwin. So much so that I think Wallace sends his manuscript to Darwin and is like, “What do you think of this idea?” And Darwin’s like, “Fuck.”

Michael Nielsen

I don’t think that’s an exact quote, but it’s pretty much correct.

Dwarkesh Patel

They end up presenting their ideas together in the spirit of sportsmanship. Why was this period in the 1850s or 1860s the right time for these ideas to form? You can come up with different ideas. One is geology. In the 1830s, Charles Lyell figures out that there’s been millions and billions of years of time that’s existed on Earth. The paleontology shows you that fossils have existed for that entire time. Life goes back a long way. In fact, you can even find fossils for intermediate species that show you the tree of life. Between humans and other apes as well, there’s intermediate humans.

There’s also the age of colonization, and we have all these voyages doing biogeography. That all must have been necessary. In fact, there’s a huge history of parallel innovation and discovery in the history of science. So maybe it is another piece of evidence that more had to be in place for a given idea to be discovered. Because if it’s not discovered for a long time and then spontaneously many different people are coming up with it, that shows you that the building blocks were in some sense necessary.

Michael Nielsen

This example of Lyell and other geologists in the early 1800s having this idea of deep time does seem to have been crucial. I know Darwin was very influenced by Lyell. If you don’t have at least tens or hundreds of millions of years, evolution starts to look like a non-starter.

In order to make it work on a timescale of 5,000 to 10,000 years or 6,000 years with Bishop Ussher you would need to see evolution occurring at a massive rate during human lifetimes, and we’re just not seeing that. That does seem to have been a blocker. To your question of what other blockers were there, were there any others? I don’t know.

Dwarkesh Patel

Or how much earlier could you, in principle, have come up with it if you were much smarter?

00:29:52 – Could gradient descent have discovered general relativity?

Michael Nielsen

Let’s go back and zoom out to your original question about the verification loop in AI. An example that should give you pause there is the big signature success so far, which is certainly AlphaFold. AlphaFold really isn’t about AI. A massive fraction of the success there is the Protein Data Bank. It’s X-ray diffraction, NMR, cryo-EM, and the several billion dollars that were spent obtaining those 180,000-odd protein structures.

It’s basically the story of how we spent many decades obtaining protein structure just by going out and looking very hard at the world experimentally, and then we fitted a nice model at the end of it, which was a tiny fraction of the entire investment. That’s a story of data acquisition principally. The AI bit is very impressive and quite remarkable, but it is only a small part of the total story.

Dwarkesh Patel

AlphaFold is very interesting, and philosophically I wonder what you think of it as a scientific theory or explanation. I guess over time the world is becoming harder to understand… As I’m saying things, because you’re such a careful speaker, I say a phrase and wonder if you’ll actually buy that premise.

But in some domains, we need to fit models to things rather than coming up with underlying principles that explain a broad range of phenomena. Compare the theory of general relativity, or any theory which just nets out to some equations, versus AlphaFold, which is encoding these different relationships between things we can’t even interpret over 100 million parameters.

Are those really the same thing? GR can predict things you could have never anticipated or it was never meant to do, like why Mercury’s orbit precesses. AlphaFold is not going to have that kind of explanatory reach. I want to get your reaction to that.

Michael Nielsen

I think it’s an incredibly interesting question. Maybe a really pivotal question. If you take a very classic point of view, you want these deep explanatory principles. You want as few free parameters as you possibly can. You want very simple models which explain a lot, and AlphaFold doesn’t look anything like that. You might just say, “It’s nice and maybe helpful as a model, but it’s not a scientific explanation.” That’s a conservative point of view, answer one to the question.

Answer two is to say maybe you shouldn’t think about AlphaFold as an explanation in the classic sense, but maybe it contains lots of little explanations inside it. Part of what you can get out of interpretability work is you can go into AlphaFold and start to extract certain things. Maybe by doing an archeology of AlphaFold, we can actually understand a great deal more about these principles. You can start to extract that a certain circuit does this interesting thing, and we learn from it.

I don’t know to what extent that’s been done with AlphaFold, but it’s been done a little bit with some of the chess models, like AlphaZero. There seem to be some strategies which were borrowed by Magnus Carlsen, which he seems to have just taken from AlphaZero. I don’t think there’s any public confirmation of this, but some experts have noticed that he changed his game quite radically after some public forensics were released on how AlphaZero worked. That’s an example where human beings are starting to extract meaning out of these models.

That leads to viewing the models as a potential source of explanations. You need to do more work because they’re not very legible up front, but you can potentially extract them. That’s an interesting intermediate situation where they’re not explanations themselves, but you can extract interesting explanations out of them and use them as a source.

The third and most interesting possibility is that they’re a new type of object. They should be taken very seriously as explanations, but where in the past we haven’t had the ability to really do anything with them, now we have interesting new actions we can do. We can merge them, we can distill them. It’s a big opportunity in the philosophy of science.

There’s an anticipation of this in some way. Some mathematicians and physicists work today… Historically, if you had a 100-page equation—which is the kind of thing that does come up—there’s just nothing you can do if it’s 1920. At that point, you give up on the problem. But today, with tools like Mathematica, you can just keep going. That’s an object now, a thing that you can work with. There are examples where people work with these things that formerly were regarded as too complicated, and sometimes they get simple answers out the end. That’s just an intermediate working state.

So I wonder if something similar is going to happen in this case, where you could take these models and use them in a similar way that people do with Mathematica, and take them seriously. They’re not explanations in the classic sense, but they’ll be something else which interesting operations can be done on.

Dwarkesh Patel

The thing I worry about is, suppose it’s 1500 and you’re training a model on… This is a weird history where we developed deep learning before we had cosmology. Suppose we live in that world. You’re observing how the stars don’t seem to move. The planets have all these weird behaviors. Then you train a model on that, and you do some kind of interp on it trying to figure out what the patterns are.

You’d just be able to keep building on Ptolemy’s model. You’d see there’s another epicycle we didn’t notice. Parameters X to Y encode this epicycle, parameters whatever encode the next epicycle. If you were just trying to figure out why the solar system is the way it is from observational data, you could just keep adding epicycles upon epicycles, but it really took one mind to integrate it all in and say, “Here’s what makes more sense overall.”

Michael Nielsen

This is to my point that we don’t really understand what to do with the models. We don’t have the verbs yet. It is certainly interesting to think about the question where you start to apply constraints to the models, essentially saying, “What’s the simplest possible explanation?” Or, “Can you simplify? Can you give me the 90/10 explanation?” And go further and further in boiling it down.

It might be that indeed they start out by providing a very, very complicated, many-parameter model. But you can just force the case, and basically that’s scaffolding, which maybe is the very early days of their attempt to understand something. They’re forced through that to a much more simple understanding.

Dwarkesh Patel

Sorry for misunderstanding, but it sounds like you’re saying maybe there’s some regularizer or some distillation you could do of a very complicated model that gets you to a truer, more parsimonious theory. Take Ptolemy versus Copernicus. You start off with lots of Ptolemy epicycles, and then you try to distill this model, and maybe it gets rid of some of the epicycles that are less and less necessary to get the mean squared error of the orbits to match.

But at some point it has to do this thing which is to switch two things. Locally, it actually doesn’t make things more accurate. It’s in a global sense that it’s a more progressive theory. There’s some process which obviously humanity did over its span, which did that regularization or did that swap. But with raw gradient descent, I don’t really feel like it would do that.

Michael Nielsen

Think about the example of going from Newtonian gravity to Einstein’s general theory of relativity. These are shockingly different theories, and the question is what causes that flip. As nearly as I understand the history, what goes on is Einstein develops special relativity and pretty much straight away he understands. It’s a very obvious observation. In special relativity, influences can’t propagate faster than the speed of light, and in Newtonian gravity, action is at a distance.

Straight away in special relativity, you could use Newtonian gravity to do faster-than-light signaling. You could send information backwards in time. You could do all kinds of crazy stuff. It’s not a big leap to realize we have a big problem here. That’s the forcing function there. You’ve realized that your old explanation is not sufficient. You need something new.

Then you’re going to start by doing the simplest possible stuff. It just turns out that a lot of that stuff doesn’t work very well, so you’re forced to go through these steps where gradually it gets more complicated, and it’s wrong in a variety of ways. The final theory appears shockingly simple and beautiful, but it’s gone through some somewhat ugly intermediate stages.

Dwarkesh Patel

If you’re thinking about what it looks like to have AI accelerate science, there’s one for well-understood domains where we just want local solutions, like how does this protein fold. We just train a raw model using gradient descent. Then there’s things like coming up with general relativity, where you couldn’t really just train on every single observation in the universe and hope that general relativity pops out.

What would it require? It also certainly wasn’t immediately discovered. It was decades of thought. You’d need independent research programs where people start off with these biases, where Einstein is initially motivated by this thought experiment of whether you can distinguish the effect of gravity from just being accelerated upwards. You just need different AI thinkers to start off with these initial biases and see what can germinate out of them. The verification loop for that might be quite long, but you just need to keep all those research programs alive at the same time.

Michael Nielsen

This point you make about keeping all the different research programs alive, I think that is very important and central. A great example is situations where the same answer has been correct in some circumstances and wrong in other circumstances.

The planet Uranus was not in quite the right spot, and people famously predicted the existence of Neptune on this basis. Wonderful, massive success for Newtonian gravity. The planet Mercury is not in quite the right spot. You predict the existence of some other distorting planet. It turns out that doesn’t exist. Actually, the reason Mercury is not in the right spot is because you need general relativity.

You’ve pursued very similar ideas, and it’s been very successful in one case, and it’s been completely and utterly unsuccessful in the other case. A priori, you can’t tell which of these is the thing to do, and you actually need to do both. This is certainly very true in the history of science.

This kind of diversity, where you just have lots of people go off and pursue lots of potentially promising ideas, you just need to support that for a long time. It’s hard to do that for a variety of reasons, but it does seem to be very, very important.

Dwarkesh Patel

This example of Uranus versus Mercury is very interesting. I think it illustrates the difficulty with falsificationism. The orbit of Uranus is in some sense falsifying Newtonian mechanics. But then you make some ancillary prediction that says, “Oh, the reason this is happening is there must be another planet which is perturbing Uranus’s orbit.” I think it’s Le Verrier in 1846. “Point a telescope in the right direction, you find Uranus.”

Michael Nielsen

Neptune.

Dwarkesh Patel

Sorry. Neptune, yes. But with Mercury, it’s observed that the ellipse which forms its orbit is rotating 43 arcseconds more every century than Newtonian mechanics would imply, so people say that there must be a planet inside Mercury’s orbit. They call it Vulcan and point the telescopes. It’s not there.

But if you’re a proper Newtonian, what you do is say, “Well, maybe there’s some cosmic dust that’s occluding this planet, or maybe the planet is so small we can’t see it, or let’s build an even more powerful telescope, or maybe there’s some magnetic field which is occluding our measurement.” At any one of these steps—

Michael Nielsen

And this happens over and over. There are just so many stories which are exactly like this. An example I love from the 1990s. Some people noticed that the Pioneer spacecraft weren’t quite where they were supposed to be.

You can get very excited about this. “Oh my goodness, general relativity is wrong. Maybe we’re going to discover the next theory of gravity.” Today the accepted explanation is that there’s just a slight asymmetry in the spacecraft. It turns out that the thermal radiation is slightly larger in one direction than the other, and that’s causing a tiny little acceleration towards the sun. Most of the time when there’s these apparent exceptions, it’s just something like that going on.

It’s very much like the Mercury-Vulcan case. But every once in a while, it’s not. A priori, you can’t distinguish these. Science is just full of these. It’s funny too, the way we tell the history of science, it sounds so simple. You just focus on the right exception and you realize that you need to throw out the old theory and lo and behold, your Nobel Prize awaits. But in fact, these exceptions are all over the place. 99.9% of the time, it just turns out to be some effect like this thermal acceleration in the case of the Pioneer spacecraft. Unfortunately, there’s a lot of selection bias going into those stories.

Dwarkesh Patel

The thing is there’s no ex ante heuristic which tells you which case you’re in. To spell out why I think this is important, some people have this idea that AI is going to make disproportionate progress towards science because it makes disproportionate progress towards domains where there’s tight verification loops. It’s really good at coding because you can run unit tests.

Science may be similar because you can run experiments. What that doesn’t appreciate is that there’s an infinite number of theories that are compatible with any given experiment. Over time, why we latch onto the one we think is more correct in retrospect is, as we’re discussing, hard to articulate.

Lakatos has all kinds of interesting examples in the book about these hostile verification loops that are extremely long-lasting. One he talks about is Prout. There’s this chemist in 1815 who hypothesizes that all atomic nuclei must have whole number weights. They’re basically all made of hydrogen. The reason he thinks this is because if you look at the measured weights of all elements, it does seem that almost all of them have whole number weights. But then there are some exceptions. For example, chlorine comes out at 35.5.

So then there’s all these ad hoc theories that people in this school keep coming up with, like, “Oh, maybe there’s chemical impurities.” But there’s no chemical reaction you can do which seems to get rid of this. Maybe it’s fractions of whole numbers, so 35.5 can be halves. But actually, if you measure chlorine even closer, it’s 35.46, so it’s getting further away from the correct fraction. Later on, what is discovered is what you’re actually measuring is different isotopes, which cannot be chemically distinguished. They can only be physically distinguished.

So you have 85 years before we realize what an isotope is, where the verification loop is actively hostile against the correct theory. You just need this remnant to be defending… There’s no ex ante reason it’s the preferred theory. As a community, we should just have people try to integrate new observations, even if they don’t seem to fit their school of thought, and hopefully enough of that happens… Anyways, I guess the thing I’m trying to articulate is the difficulty with automating science.

Michael Nielsen

The question is, where is the bottleneck at some level? Are we primarily bottlenecked on one type of thing, or are we bottlenecked on multiple types of things? Certainly, talking to structural biology people, they seem to think that AlphaFold was an enormous advance. It was a shock.

At some level, yes, AI can certainly help us speed up science. It is helping with a certain type of bottleneck. That doesn’t mean though, as you’re saying, that it’s necessarily going to help with all kinds of bottlenecks. I suppose the question you’re pointing at is, what are the types of bottlenecks that remain, and what are the prospects for getting past them?

Even in the case of coding, it’s really interesting talking to programmer friends. At the moment they’re all in this state of shock and high excitement, and they’re all over the place. You do wonder where the bottleneck is going to move to. Certainly, one thing that a lot of them seem to be bottlenecked on now is having interesting ideas, and in particular, having interesting design ideas. There’s not really a verification loop for knowing that a design idea is very interesting.

They’re no longer nearly as bottlenecked by their ability to produce code, but they are still bottlenecked by this other thing. Formerly, they weren’t bottlenecked on it because just writing code took so much of their time. They could have lots of ideas while they were taking three weeks to implement their prototype, and then they would implement the next version. Now they’re taking three hours to implement the prototype, and they don’t have as good ideas after that, from a design point of view.

00:50:54 – Why aliens will have a different tech stack than us

Dwarkesh Patel

You have a very interesting take. I think it was a footnote in one of your essays, and I couldn’t find it again, which was that it’s very possible that if we met aliens, they would have a totally different technological stack than us. That contradicts a common assumption I had that I never questioned, which is that science is this thing you do relatively early on in the history of civilization. You get to a point and you have a couple hundred years of just cranking through the basics, understanding how the universe works, and you’ve got it. You’ve got science. Then everybody would converge on the same “science.” I found that a very interesting idea, and I want you to say more about it.

Michael Nielsen

The idea there that I’m at least somewhat attached to is that the tech tree or the science and tech tree is probably much larger than we realize. We’re in this funny situation. People will sometimes talk about a theory of everything as a potential goal for physics, and then there’s this presumption that physics is done once you get there. Of course, this is not true at all.

If you think about computer science, computer science started in the 1930s when Turing and Church and so on laid down what the theory of everything was. They just said, “Here’s how computation works.” We’ve spent ninety-odd years since then exploring the consequences of that and gradually building up more and more interesting ideas. Those ideas, to some extent, you can regard as technology. But insofar as they’re discovered principles inside that theory of computation, I think they’re best regarded as science and in some cases, very fundamental science.

Ideas like public-key cryptography are incredibly deep, very non-obvious ideas which lay hidden already in the 1930s. My expectation is that there will be different ways of exploring this tech tree, and we’re still relatively low down. We’re still at the point where we’re just understanding these basic fundamental theories, and we haven’t yet explored them.

A thing which I think is quite fun is if you look at the phases of matter. When I was in school, we’d get taught that there are three phases of matter, or sometimes four or five, depending on what you included. As an adult, as a physicist, you start to realize we’ve been adding to this list. We’ve got superconductors and superfluids, and maybe different types of superconductors, and Bose-Einstein condensates, the quantum Hall systems, fractional quantum Hall systems, and so on. It’s starting to turn out there’s a lot of phases of matter to discover, and we’re going to discover a lot more of them. In fact, we’re going to be able to start to design them in some sense. We’ll still be subject to the laws of physics, but there is this tremendous freedom in there.

This looks to me like we’re down at the bottom of the tech tree. We’ve barely gotten started there, and I expect that to be the case broadly. Certainly, programming is a very natural place to look. The idea that we’ve discovered all the deep ideas in programming just seems obviously ludicrous. We keep discovering what seem like deep, new, fundamental ideas. We’re very limited. We’re basically slightly jumped-up chimpanzees, so we’re slow and it’s taking us time. But what do we look like another million years in the future, in terms of all the different ideas people have had around how to manipulate computers and information? I think we’re likely to discover that there are a lot of very deep ideas still to be discovered.

I think it was Knuth in the preface to The Art of Computer Programming who says something like it. He started this book back in the sixties. He talked to a mathematician who was a bit contemptuous and said, “Look, computer science isn’t really a thing yet. Come back to me when there’s a thousand deep theorems.” Knuth remarks, writing the preface decades later, “There clearly are a thousand deep theorems now.”

It’s really interesting to think what the long-term future is as you get higher and higher up in the tech tree, choices about which direction we go and how we choose to explore. It’s potentially the case that different civilizations or different choices mean we end up in different parts of that tree. In particular, there are just very basic things about how we’re very visual creatures, while certain other animals are much more aurally based. Does that bias the types of thoughts that you have? Then you extend it to much more exotic kinds of civilizations where maybe their biases in terms of how they perceive and manipulate the world are quite different than ours. That might make some significant changes in terms of how they do that exploration of the tech tree. It’s all speculation, obviously.

Dwarkesh Patel

This is such an interesting take. I want to better understand it. One way to understand it is that there might be some things which are so fundamental and have such a wide collision area against reality that they’re inevitably going to discover, like general relativity.

Michael Nielsen

Numbers. Numbers. Of all the intelligences in the Milky Way galaxy… Maybe that number is one. Well, actually, arguably we’ve already increased the number. But of all of those, what fraction have the concept of counting? It does seem very natural. What fraction have discovered the idea of some kind of decimal place system? Interesting question. Maybe we’re missing something really simple and obvious that’s actually way better than that.

What fraction got there immediately? What fraction had to go through some other intermediate state? What fraction uses linear representations versus a two-dimensional or a three-dimensional representation? I think the answers to these questions are just not at all obvious. It’s a lot of design freedom.

Dwarkesh Patel

On theoretical computer science, this is going to be extremely naive and arrogant, but I took Scott Aaronson’s class on complexity theory, and I was by far the worst student he’s ever had. What I remember is there was this period, in which you were one of the pioneers, where we figured out the class of problems that quantum computers can solve and how it relates to problems that classical computers can solve. It was groundbreaking. It’s crazy that this works. Since then… There’s literally this website called Complexity Zoo which lists out all the complexity classes. If you have this complexity class with this kind of oracle, it’s equivalent to this other class. It feels like we’re building out that taxonomy.

There are a couple ways to understand what you’re saying. One, maybe you disagree with me that this is actually what’s happened with this field. Another is that while that might happen to any one field, who would’ve thought in 1880 that computer science, other than Babbage, was going to be a thing in the first place? We’re underestimating how many more fields there could be. Or maybe you think both, or maybe a third secret thing. I’d be curious.

Michael Nielsen

A very common argument here is the low-hanging fruit argument. The argument that says there should be diminishing returns.

Dwarkesh Patel

In fact, empirically we see this. The amount of scientists in the world has exponentially increased.

Michael Nielsen

I think it’s worth thinking about why you expect diminishing returns and how well that argument actually applies in practice. An analogy I like is thinking about going to an event, like a wedding, and you go to the dessert buffet. They’ve put out thirty desserts. Naturally, what people do is the best desserts go first. We don’t quite have a well-ordered preference there, so maybe there’s some difference, but human beings are fairly similar, so the best desserts will go first. This is an argument for why you expect diminishing returns in a lot of different fields. If it’s relatively easy to see what’s available and people have similar preferences, then the best stuff goes first and it just gets worse and worse after that.

If you look at a very static snapshot in time of scientific progress, maybe there’s some truth to that. But if somebody is standing behind the dessert table and is replenishing and restocking the desserts and keeps adding new ones in, it may turn out that a little bit later, much better desserts appear, and you’re going to go and eat those instead.

Scientific progress has a little bit of that flavor. We go through these funny time periods. Computer science is a great example, where computer science basically arose as a side effect of some pretty abstruse questions in the philosophy of mathematics and logic. You’ve got these people trying to attack these rather esoteric questions that seem quite high up in exploration, and they discover this fundamental new field, and all of a sudden there’s an explosion there. The diminishing returns argument just didn’t apply there. We just weren’t able to see what was there.

This has been the case over and over again. New fields arrive and all of a sudden, and boom, it’s easy to make progress again. Young people flood in because you can be twenty-one and make major breakthroughs rather than having to spend twenty-five years mastering everything that’s been done before. It’s obviously very attractive. I’m not sure anybody understands very well the dynamics of that, or how to think about why the structure of knowledge is that way, where these new fields keep opening up. But it does seem empirically to be the case.

Dwarkesh Patel

Despite the fact that that is the case… Take deep learning. Obviously, this is an example of a new field where twenty-one-year-olds can make progress and it’s relatively new. Fifteen years or so since it got back into high gear. But already we’re in a stage where you need billions, tens of billions, or hundreds of billions of dollars to keep making progress at the frontier.

There are a couple ways to understand that. One is that it actually is harder than the kinds of things the ancients had to do, or is more intensive at least. Second is it might not have been, but because our civilizational resources are so large, the amount of people is so large, the amount of money is so large, we can basically make the kind of progress it would have taken the ancients forever to make almost immediately. We notice something is productive and immediately dump in all the resources. But it’s also weird that there’s not that many of them. I feel like deep learning is notable because it is one big exception to the fact that it’s hard to think of other examples.

Michael Nielsen

I think that’s a consequence of the architecture of attention. At any given time, there’s always a most successful thing. If deep learning wasn’t a thing, maybe you’d be talking about CRISPR. Maybe we wouldn’t think about solving the protein structure prediction problem as a success of AI. Maybe we would have figured out how to do it with curve fitting, more broadly construed, and we’d just be like, “Wow, that took a lot of computing resources.” But protein structure prediction might be an enormously important thing.

There is always our biggest thing. What you’re pointing at is more a consequence of the way in which attention gets centralized. It’s basically fashion, is what I’m saying. It’s not just fashion, but there is some dynamic there.

Dwarkesh Patel

There’s a very interesting and important implication of this idea. That the branching is so wide and so contingent and so path-dependent that different civilizations would stumble on entirely different technology stacks. There’s a very interesting implication that there will be gains from trade into the far, far future, which might actually be one of the most important facts about the far future in terms of how civilizations are set up, how they coordinate, and how they interface. There’s not this “go forth and exploit.” There are humongous gains to trade from adjacent colonies or whatever.

Michael Nielsen

Sort of. There’s a question of what’s actually hard. If it’s just the ideas, well, those spread relatively quickly. It’s relatively easy to share ideas. If it’s something more, it’s almost a Dan Wang kind of idea where there’s some notion of capacity. You need all the right techs, you need all the right manufacturing capacity, and so on.

So civilization A has a very different kind of manufacturing capacity, and it’s just not so easy to build in civilization B. Even if civilization B is ahead, I think that becomes true. There is a comparative advantage which is going to provide massive benefits to trade in both directions. Eventually, you expect some diffusion of innovation. It is funny to think about what the barriers are there.

A fun thought experiment I like to think about is GitHub but for aliens. Somebody presents you with all of the code from some alien civilization. I don’t even know what code means there, but their specification of algorithms. It would have many interesting new ideas in there, and it would take forever for human beings to dig through and try and extract all of those.

The origin of this for me was thinking about proteins in nature. We’ve been gifted this incredible variety of machines which we don’t really understand at all. We just have to go and try and understand them on a one-by-one basis. We’re still understanding hemoglobin and insulin and things like this. There are hundreds of millions of proteins known. So it is a little bit like that. We’ve been gifted by biology this immense library of machines, no doubt containing an enormous number of very interesting ideas, and we’re just at the very, very beginning of understanding it.

I suppose your point—I need to relabel your argument slightly—but you think of that as a gift from an alien civilization, which obviously it isn’t, but you think of it that way. And oh my goodness, there’s so much in there and we’re going to study it. Goodness knows how long we could continue to study it. There are tens of thousands of papers about hemoglobin and things like that, and we still don’t understand them, and yet we’re getting so much out of it. Just think about insulin alone. It’s such an important thing.

Dwarkesh Patel

That’s an incredibly useful intuition pump, that you have on Earth… I had Nick Lane on where he had this theory about how life emerged, but whatever theory you have, something like DNA has had four billion years. You have an alien civilization come here and be like, “There’s all these interesting things to learn about material science.”

Michael Nielsen

Think about kinesin walking along. We know almost nothing about these proteins, and yet the tiny few facts we do know are just incredible. The ribosome is another example, this miraculous sort of device, a little factory.

Dwarkesh Patel

All seeded by this particular chemistry on Earth with nucleic acids and carbon-based life forms. That chemistry gives rise to all of these interesting things which an alien civilization would find very interesting. That very seed, which must be one among trillions of possible seeds of general intellectual ideas, leads to all this fecundity. That’s a very interesting intuition pump.

I want to meditate on this “gains from trade” thing because I feel like there’s something very interesting about this idea that if you have this vision of how technology progresses and how it may be different in different civilizations, it actually has important implications about how different civilizations might interact with each other. The fact that there are going to be these huge gains from trade.

Michael Nielsen

It makes friendliness much more rewarding?

Dwarkesh Patel

Yes. That’s a very important observation.

Michael Nielsen

I hadn’t thought about that at all. That is a very interesting observation. It is funny. Comparative advantage is something that people love to invoke and it’s a very beautiful idea obviously. There are limits to it. It’s a special limited model.

Chimpanzees can do interesting things, but we don’t trade with them. I think it’s interesting to think about the reasons why. Part of it is just power, I think. Once there’s a sufficiently large power imbalance, very often—not always, but very often—groups of people seem to shift into this other mode where they just seek to dominate. Maybe there’s something special about human beings, but maybe it’s also a more general thing. You need all these special things to be true before groups will trade. It’s not necessarily obvious.

Dwarkesh Patel

I think the big thing going on here is one, transaction costs. Two, comparative advantage does not tell you that the terms on which the trade happens are above subsistence for any given producer. People often bring this up in the context of, “Well, humans will be employed even in a post-AGI world because of comparative advantage.”

There are five different ways that argument breaks down, but the easiest way to understand it is: why don’t we have horses all around on the roads? Because there’s some comparative advantage between cars and horses. One, there are huge transaction costs to building roads that are compatible with horses and cars at the same time. In a similar way, AI thinking at 1,000 times the speed that can shoot their latent states at each other is going to find it way more costly than the benefit, in terms of interacting with a human being in the supply chain.

Second, just because horses have a comparative advantage mathematically does not mean that it is worth paying $100,000 a year, or whatever it costs to sustain a horse in San Francisco. That subsistence isn’t going to be worth the benefit you get out of the horse.

Michael Nielsen

I do think it’s interesting, the sheer fact… My expectation and my intuition obviously differs a great deal from yours on this. Most parts of the tech tree are never going to be explored. There are just too many interesting ways of combining things. There are too many deep ideas waiting to be discovered, and not only we, but nobody ever is going to discover most of them. So choices about how to do the exploration actually matter quite a bit.

It’s something I really dislike about technological determinist arguments. I’m willing to buy it low enough down when progress is relatively simple. But higher up, you start to get to shape the way in which you do the exploration. And it’s interesting, we are starting to shape it in interesting ways.

There are various technologies that have been essentially banned. You think about DDT, chlorofluorocarbons, restrictions on the use of nuclear weapons, the Nuclear Non-Proliferation Treaty. Those kinds of things weren’t done before the fact, but they’re starting to get pretty close in some cases, where we just preemptively decide, “Oh, we’re not going to go down that path.” So that starts to look like a set of institutions where we are actually influencing how we explore the tech tree.

Dwarkesh Patel

On where you would see these gains from trade, obviously you’d see the most where it’s pure information that could be sent back and forth, because the information has this quality where it is expensive to produce, but cheap to verify and cheap to send. It’ll be interesting how much of future productivity can be distilled down to information.

Right now, it’s hard to do. If China’s really good at manufacturing something, there’s this process knowledge that’s in the heads of 100 million people involved in the manufacturing sector in China. But in the future, it might be easier if AIs are doing it.

Michael Nielsen

The question is to what extent our fabrication gets very uniform and gets really commoditized. 3D printers have been the next big thing for at least 20 years now. Why do they still not work all that well? Why are they still not at the center of manufacturing, and what comes after that? It is funny to look at the ribosome by contrast, which really is at the center of biology in a whole lot of really interesting ways.

Whether or not that’s the future of manufacturing is something very simple, where everything goes as throughput through a bioreactor or something like that. You send the information, and then you grow stuff, or you have some 3D printer that actually works. If they’re good enough, then it does become much more a pure information problem, and some of this process knowledge becomes much less important.

01:15:26 – Are there infinitely many deep scientific principles left to discover?

Dwarkesh Patel

Can I ask a very clumsily phrased question? There are these deep principles that we’ve discovered a couple of. One is this idea that if there’s a symmetry across a dimension, it corresponds to a conserved quantity. It’s a very deep idea. There’s another—which you’ve written a lot about, written a textbook about in fact—about ways to understand what kinds of things you can compute, what kinds of physical systems you can understand with other physical systems, what a universal computer looks like, et cetera.

Is your view that if you go down to this level of idea of Noether’s theorem or the Church-Turing principle, that there’s an infinite number of extremely deep such principles? Because I feel what makes them special is that they themselves encompass so many different possible ways the world could be. But no, the world has to be compatible with a couple of these very deep principles.

Michael Nielsen

I don’t know. All I have here is speculation and instinct. My instinct is that we keep finding very fundamental new things. It was quite formative for me to understand, as I gave the example before, these wonderful ideas of Church and Turing and these other people about universal programmable devices. Then you understand later, this also contains within it the ideas of public-key cryptography. Then you understand later, that also contains within it the ideas people refer to as cryptocurrency.

There’s a very deep set of ideas there about the ability to collectively maintain an agreed-upon ledger, which is built upon this. It’s taken many years to figure out the right canonical form of those. Just this fact that you keep finding what seem like deep new fundamental primitives has been a very important intuition pump for me. I’ve given that particular example, but I think you see that same pattern in a lot of different areas.

Dwarkesh Patel

What is your interpretation then of this empirical phenomenon where whatever input you consider into the scientific process or technological progress… Economists have studied this a million ways. It just seems to require a very consistent rate of X percent more researchers per year. There’s this famous paper from a couple years ago by Nicholas Bloom and others where they say, “How many people are working in the semiconductor industry, and how has it increased over time through the history of Moore’s law?” I think they find that Moore’s law means transistor density increases 40% a year, but to keep that going the number of scientists has increased 9% a year, in the semiconductor industry. They go through industry after industry with this observation.

Is your view that there are these deep ideas, but they keep getting harder to find? Or is there another way to think about what’s happening with these empirical observations?

Michael Nielsen

First of all, all of their examples are narrow. They pick a particular thing, and then they look at a particular metric. GPUs don’t show up there. All of a sudden you get this ability to parallelize, and that’s really interesting. There are a lot of external consequences. Basically they have these simple quantitative measures. They look at it in agricultural productivity. They look at it in a whole lot of different ways, but you do have to focus narrowly.

I’m certainly interested in the fact that new types of progress keep becoming possible. But I think even there, there does still seem to be some phenomenon of diminishing returns. Is that intrinsic? Is that something about the structure of the world? What is it? One thing which hasn’t changed that much is the individual minds which are doing this kind of work. Maybe those should be improved as well, or some feedback process going on there. Maybe that changes the nature of things.

I look at scientific progress up until, let’s say, 1700, and it was very slow, and also very irregular. You had the Ionians back five centuries before Christ doing these quite remarkable things, and so much knowledge would get lost, and then it would be rediscovered, and then it would be lost again. You’d have to say that progress was very slow. It’s partially just bound up with the fact that there were some very good ideas that we just didn’t have.

Even once you’ve had the ideas, you need to build institutions around them. You actually need to solve a whole lot of different problems about training, allocation of capital, and all these kinds of things. Even just basic security for researchers, so they’re not worried about the Inquisition or things like that. There are all these complicated problems. You solve all those complicated problems, and then all of a sudden, boom, there’s a massive burst of scientific progress.

If there’s some kind of stagnation, if you’re not changing those external circumstances, yes, you may start to get diminishing returns again. But that doesn’t mean there’s anything intrinsic about the situation. Maybe something external needs to change again. Obviously, a lot of people think AI is potentially going to be a driver. It certainly will at some level.

To that extent, you can think of a lot of modern scientific instrumentation as really, at some level, robots. What is the James Webb Space Telescope? It’s unconventional maybe to describe it as a robot, but it’s not completely unreasonable either. It is an example of a highly automated, very sophisticated system with electronically mediated sensors and actuators, where machine learning is being used to process the data. In that sense, we’re already starting to see that transition. We’ve been seeing it for decades.

Dwarkesh Patel

I have this “smoke a joint and take a puff” thought, which—

Michael Nielsen

I think we’ve had a few.

Dwarkesh Patel

I think we’re getting to that part of the conversation, and then you can help me get my foot out of my mouth and figure out a more concrete way to think about it. To your point that there was the Industrial Revolution, the Enlightenment, and now there’s AI, and each might be a different pace or a different way in which science happens. If you think about the pace of how fast such transitions have been happening, you can draw over the long span of human history this hyperbolic rate of growth that is increasing over time as well.

A hundred thousand years ago, you had the Stone Age. You go back even much further, how long have primates been around? It would be millions of years. A hundred thousand years ago, the Stone Age, then ten thousand years ago, the Agricultural Revolution, then three hundred years ago, the Industrial Revolution, each marked by this increase in the rate of exponential growth. Then people think it’s going to happen again with AI. But that would happen potentially even faster.

It would not have occurred to somebody at the beginning of the Industrial Revolution that the next demarcation in this trend will be artificial intelligence. So if things are getting faster, and it’s hard to anticipate what the next transition will be. I guess we just think of this singularity between now and AI as what distinguishes the past from the future. But applying the same heuristic that many people in the past should have had, maybe the “Intelligence Age” is also quite short and the next thing after that, we don’t even have the ontology to describe what it is, the future will not think of the past as pre-intelligent AI and post-AI.

Michael Nielsen

No, obviously we can’t prove this, but it certainly seems quite plausible. Part of the issue is just that the substrate we have available to conceive seems all wrong. You can’t speculate with a bunch of chimpanzees about what it would be to have language. Just to pick a major transition in the past, the transition itself is the thing. It seems likely.

If we’re talking about “taking a puff” kind of thoughts, I’m certainly amused by the idea that there’s going to be some transition involving artificial general intelligence using classical computers. But actually, there’ll be an interesting transition with quantum computers as well. They’re probably capable of a strictly larger class of potentially interesting computations. So maybe the character of AQGI, or whatever it should be called, is actually qualitatively different. So maybe there’s a brief period between those two things. As I say, this is just speculation, but it’s certainly amusing.

Dwarkesh Patel

Is there a reason to think that? From what I understand, for decades people like you have put pretty tight bounds on the kinds of things quantum computers are going to do. It’ll speed up search somewhat. The kinds of things it speeds up extremely, like Shor’s algorithm, it seems like… Again, maybe this is to your point that we can’t predict in advance what’s down the tech tree, but at least from here, it seems like you break encryption, but what else are you using Shor’s algorithm to do?

Michael Nielsen

We’ve only been thinking about it for 40 or so years. Not for very long, and we haven’t thought that hard about it as a civilization. Does it turn out that it’s very narrow? Maybe. Does it turn out that it’s very broad? That’s also a really radical expansion that seems distinctly possible. Keep in mind as well, we’ve been doing it without the benefit of having the devices. That’s a pretty big bottleneck to have.

Dwarkesh Patel

If you’re thinking about computer science in the 1700s and you’re like, “it can do AND/OR, what can come out of that?” You can’t anticipate Bitcoin. You can’t anticipate deep learning.

Michael Nielsen

Maybe you could if you were sufficiently bright, but it is a pretty hard situation.

01:26:25 – What drew Michael to quantum computing so early?

Dwarkesh Patel

What is your inside view, having been in and contributing to quantum information and quantum computing back in the ‘90s and 2000s? What is your telling of the history of what was the bottleneck? What was the key transition that made it a real field? How do you rank the contributions from Feynman to Deutsch to everybody else who came along?

Michael Nielsen

Let’s just focus on the question about what actually changed. Why was quantum computing not a thing in the 1950s? It could have been. Somebody like John von Neumann is a good example. He was absolutely pioneering computation. He also wrote a very important book about quantum mechanics and was deeply interested in it. He could have invented quantum computing at that time, and I think there were quite a number of people who potentially could have.

So why do we have these papers by people like Feynman and Deutsch in the ‘80s? Those are fairly regarded as the foundation of the field. There are some partial anticipations a little bit earlier, but they were nowhere near as comprehensive and nowhere near as deep. You should ask David. You can’t ask Feynman, unfortunately, but he’ll know much better than I do.

A couple things that I think are interesting. One is that computation became far more salient in the late ‘70s and early ‘80s. It just became a thing which many more people were interested in, partially for very banal reasons. You could go and buy a PC. You could buy an Apple II. You could buy a Commodore 64. You could buy all these kinds of things. It became apparent to people that these were very powerful devices, very interesting to think about.

At the same time, in the quantum case, that was also the time of the Paul trap and the ability to trap single ions. Up to that point, we hadn’t really had the ability to manipulate single quantum states. You got these two separate things that for historically contingent reasons had both matured around 1980 or so. Somebody like von Neumann could have had the idea earlier, but it is quite an interesting factor.

There’s a story about Richard Feynman. He went and got one of the first PCs around 1980 or 1981. He was apparently so excited with this device, he actually tripped and hurt himself quite badly carrying his brand-new computing device. That’s a very historically contingent coincidence, having somebody who’s very talented and understanding of quantum mechanics also just very excited about these new machines. It’s not so surprising perhaps that he’s thinking about it then. What similar story could you have told 10 years earlier? The conditions don’t exist for it. I mean, it’s quite a banal story, but…

Dwarkesh Patel

One of the things we were going to discuss was this idea you had about the market for follow-ups. I think this is the perfect story to discuss it for because you wrote the textbook about the field. “Mike and Ike” is the definitive textbook on quantum information. You presumably came in after Deutsch.

But you in the ‘90s somehow identified it as the thing that is worth following up on and building on. Instead of talking about it more abstractly, I’d love to just hear the firsthand story of how you knew that this is the thing to do. Of all the things that were happening in physics and computing, how did you decide you want to think about this problem?

Michael Nielsen

Richard Feynman writes this great paper in 1982. David Deutsch writes an absolutely fantastic paper in 1985 sketching out a lot of the fundamental ideas of quantum computing. I’m 11 in 1985. I’m not thinking about this. I’m playing soccer and doing whatever. But in 1992, I took a class on quantum mechanics that was really terrific, given by Gerard Milburn.

I just went and asked Gerard one day after the fifth lecture or something. I said, “Do you have any papers or whatever that you could give me?” He said, “Come by my office in a couple of days’ time.” I did, and he presented me with a giant stack of papers, which included the Deutsch paper, the Feynman paper, and a whole bunch of other very fundamental papers about quantum computing and quantum information at a time when essentially nobody in the world was working on it. He was. I think he wrote the very first paper that proposed a practical approach to quantum computing. It wasn’t very practical, but it was actually in a real system.

So in some sense, I’m benefiting from the taste of this other person. As soon as I read the papers… These are exciting papers. They’re asking very fundamental questions, and you realize I can make progress here. These are things that one could potentially work on.

Deutsch has this conjecture, or thesis or whatever you’d call it, that a universal model, a quantum Turing machine, should be capable of efficiently simulating any physical system at all. This is a very provocative idea. I think in that paper, he more or less claims that he’s proved it. I’m not sure everybody would agree with that. There are questions about whether or not you can simulate quantum field theory effectively. That kind of question is very interesting and very exciting. It’s obviously a fundamental question about the universe.

He has some wonderful ideas in there about quantum algorithms, where they come from, what they mean, and what they relate to the meaning of the wave function. Questions like this are still not agreed upon amongst physicists. There’s just some sense of, “Oh, I am in contact with something which is (A) deeply important, and (B) we as a civilization don’t have this.” Of course, you start to focus your attention a little bit there.

Dwarkesh Patel

I’m not sure I got the answer to the question…

Michael Nielsen

Maybe I misunderstood the question.

Dwarkesh Patel

Maybe I’ll explain the motivation first. In a previous conversation, we were discussing how you could have known in the 1940s that the Shannon theorems and Shannon’s way of thinking about a communication channel is a deep idea that goes beyond the problems with pulse-code modulation that Bell Labs was trying to solve at the time, and that it applies to everything from quantum mechanics to genetics to computer science.

One of the ideas you stated that we didn’t get a chance to talk about yet… Shannon published this paper. There are all these other papers, but there’s some market of follow-ups where people gravitate to and build upon Shannon’s work. How do they realize that that’s the thing to do, and how does that process happen? I guess you gave your local answer. You read these papers, and you immediately realized there’s work to be done here. There’s low-hanging fruit. There’s some deep provocative idea that I need to better understand, and I could tractably make progress on.

Michael Nielsen

To some extent, you’re saying, “Okay, I wanted to get into this game of contributing to humanity’s understanding of the universe,” and you are applying this low-hanging fruit algorithm. You’re like, “elative to my particular set of interests and abilities, where should I pick up my shovel and start digging?” There it was like, “Oh, this looks like quite a good place to start digging.” Different people, of course, chose very differently. It was a very unusual choice at the time. This was 1992. Very few people were thinking about that.

01:35:29 – Does science need a new way to assign credit?

Dwarkesh Patel

Fast-forwarding a bit, I don’t know how you think about your work on the open science movement now, but did it work? What does success there look like? What is the movement trying to accomplish?

Michael Nielsen

It’s interesting. You didn’t stop and define open science there, which 20 years ago you would have had to do. People recognize the phrase. People have some set of associations with it. Most often, they have a relatively simple set of associations. It means maybe something about making scientific papers open access. Very often they have some set of notions about also making code openly available or making data openly available.

Those are already very large successes of the open science movement, to make those salient issues. Those are issues on which people have opinions, and there are relatively common arguments. This is like the meme version: publicly funded science should be open science. That’s a distillation of a set of ideas which you might be able to contest. But if you can get people actually thinking about it and engaged with that kind of argument, that’s a very fundamental issue to be considering in the whole political economy of science.

If you go back three centuries, there was a very similar argument prosecuted, which is the question: do we publicly disclose our scientific results or not? If you look at people like Galileo and Kepler, the extent to which they publicly disclosed was done in a very odd way. Sometimes they did bizarre things where they published some of their results as anagrams. They’d find some discovery, write down the result in a sentence, scramble it, and publish that. Then if somebody else later made the same discovery, they would unscramble the anagram and say, “Oh, yeah, I actually did it first.” This is not an ideal foundation for a discovery system.

It took a very long time, over a century, I think, to obtain more or less the modern ideals, in which you disclose the knowledge in the form of a paper. There is an expectation of attribution, and a reputation economy gets built. “So-and-so did this work, so they deserve the credit for that,” and that’s the basis for their careers. This is the underlying political economy of science. That made a lot of sense when you have a printing press and the ability to do scientific journals.

Then you transition to this modern situation, where you can start to share a lot more. You can share your code, your data, your in-progress ideas. But there’s no direct credit associated to those. It’s not at all obvious how much reputation should be associated to them. That’s all constructed socially. Making it a live issue is a very important thing to have done. I view that as one of the main positive outcomes of work on open science.

I’ll give you a really practical example to illustrate the problem. For a long time in physics, there was a preprint culture in which people would upload preprints to the preprint archive, and in biology, this didn’t happen. There was no preprint culture. That’s changing now, but for a long time, this was the case. I used to amuse myself by asking physicists and biologists why this was the case.

What I would hear from biologists was they would say, “Biology is so much more competitive than physics that we need to protect our priority, so we can’t possibly upload to the archive. We have to just publish in journals.” Then I would sometimes hear from physicists, “Physics is so much more competitive than biology that we need to establish our priority by uploading as rapidly as possible to the preprint archive. We can’t possibly wait to do it with the journals.”

I think this emphasizes the extent to which this kind of attribution economy is just something we construct. It’s something we do by agreement. Any attempt to change that economy results in a different system by which we construct knowledge. There is this very fundamental set of problems around the political economy of science. We’ve got this collective project, and how we mediate it depends upon the economy we have around ideas.

Dwarkesh Patel

One of the things you’ve emphasized as a part of this project of open science, and we talked about it earlier, is collective science, or groups of people making progress on a problem where no individual understands all the logical and explanatory levels necessary to make a leap or a connection. Outside of mathematics, what is the best example of such a discovery?

Michael Nielsen

I’m not sure I have a well-ordering of them to give you a best. An example that I think is very interesting is the LHC, where it’s just this immensely complicated object. Years ago, I snuck into an accelerator physics conference. I didn’t know anything at all about accelerator physics, but I was just curious to see what they were talking about.

This particular group of people were experts on numerical methods, in particular on inverse methods. Inside these accelerators, you have these cascades. A particle will be massively accelerated, maybe it’ll be collided, and then you’ll get a shower of particles which decays and decays and decays. There’s just this incredible, consequential shower, which is ultimately what you see at the detector. Then you have to retroactively figure out what produced it. There are these very complicated inverse problems that need to be solved. You’ve got this final data, but you need to figure out what produced it, and that’s how you look for signatures of these.

Many of these people were incredibly deep experts on simulation methods for following particle tracks. This was really deep and difficult stuff. I was like, “Wow, you could spend a lifetime just learning how to do this and how to solve some of these inverse problems, and you would know very little about quantum field theory, detector physics, vacuum physics, or data processing, all these things that are absolutely essential to understanding, say, the Higgs boson”.

I don’t think it’s possible for one person to understand everything in depth. Lots of people broadly understand a lot of these ideas, but they don’t understand everything in the depth that is actually utilized. That’s why there are these papers with well over a thousand authors. Those people can talk to one another at a high level, but they don’t understand each other’s specialties in all that much depth. Things like detector physics, vacuum physics, solving inverse problems, this stuff is incredibly different from each other. To understand it in real detail is serious work.

01:43:57 – Prolificness versus depth

Dwarkesh Patel

How do you think about prolificness versus depth? Maybe Darwin’s an example of somebody who’s just gestating on something for many decades. There are other examples. Einstein during the year he comes up with special relativity is just doing a bunch of different things. And Pais talks about how they were all relevant to the eventual build-up.

Michael Nielsen

It’s something I stress about a lot. Sometimes I feel I’m too slow. It’s funny though, the Darwin example is really interesting. Prolific at what? God knows how many letters he wrote. It must have been an enormous number. So he was certainly very active.

There’s two types of work that tend to be involved in any kind of creative project. There’s routine stuff, and there you just want to avoid procrastination. You just want to ask, “How do I get good at this?” or “How do I outsource it?” and “How do I do it as rapidly as possible?” and just avoid getting into a situation where you’re prolonging it.

Then there’s high-variance stuff where you actually need to be willing to take a lot of time. You need to be willing to go to different places and talk to different people, where in any given instance, most of it is just not going to be an input. Somehow balancing those two things… I think a lot of people are very good at doing one or the other, but it’s almost like a personality trait which one you prefer. People tend to end up doing a lot of one and not enough of the other. So I certainly try and balance those two things.

Einstein is such an interesting example. 1905 is just this extraordinary year. You can delete special relativity entirely, and it’s an extraordinary year. You can delete special relativity, and you can delete the photoelectric effect for which he won the Nobel Prize, and it’s still an extraordinary year, plausibly a multi-Nobel-Prize-winning year. So what’s he doing? Maybe the answer is just that he’s smarter than the rest of us. There’s a lot of luck as well.

Certainly for myself anyway, trying to identify those things that are routine that I should get good at, and then just try to do them as quickly as possible. I think that’s yielded a certain amount of returns. But also being willing to bet a little bit more on myself on the variance side has also been very, very helpful. That’s really hard, because intrinsically you’re putting yourself in situations where you don’t know what the outcome is going to be. If you’re very driven to be productive, and actually mostly it’s not working over there, you think, “Let’s reduce this.” It doesn’t feel right.

When I worked in San Francisco, a practice I used to have each day was instead of taking the 15-minute walk to work, I would take the more beautiful 30-minute walk. Partially just because it was beautiful, but partially also as just a reminder that there are real benefits to not being efficient. But it’s not an answer to your question. Really, I think all I’m saying is I struggle a lot with the question.

Dwarkesh Patel

I think Dean Keith Simonton has this famous equal odds rule where he says the probability that any given thing you release—any paper, book, whatever—will be extremely important for a given person through their lifetime is not that different. What really determines in what era they are the most productive is how much they’re publishing. Any given thing has equal odds of being extremely important. I think some of the most successful creatives or scientists, they’re just doing a lot. Shakespeare was just publishing a lot.

Michael Nielsen

Of course, then there are counterexamples. Gödel published almost nothing. But broadly speaking, you need a very good reason to not do that. It’s funny, I’ve met a lot of people over the years who are clearly brilliant, and they’re just obsessed that they are going to work on the great project that makes them famous, and they never do anything. That seems connected. It’s a type of aversiveness. I think very often they just don’t want public judgment.

Something that I would love to see… There’s an awful lot of biographies and memoirs and histories of people who achieve a lot. I wish there was a very large number of biographies of people who are fantastically talented who just missed. I’ve known people who won gold medals at IMOs and things like that, who then tried to become mathematicians and failed. What happened? What was the reason? I suspect in many cases that’s actually more informative than anything else.

01:49:17 – What it takes to actually internalize what you learn

Dwarkesh Patel

You have this essay that I was reading before this interview about how you think about what the work you’re doing is. And “writer” doesn’t seem like the right label. As you say, was Charles Darwin a writer? What exactly is that label? I’m a podcaster. In a way, obviously our work is very different, but I also think a lot about what this work is and how I get better at it.

In particular, how can I make sure there’s some compounding between the different people I talk to on the podcast? I worry that instead of this compounding, I build up some understanding that’s somewhat superficial about a topic, and then it depreciates. I move down to the next topic, and it depreciates. There are a lot of podcasters in the world who will interview way more experts than I have, and I don’t think they’re much the wiser or more knowledgeable as a result. So it’s clearly possible to mess this up.

I wonder if you have thoughts or takes or advice on how one actually learns in a deeper way from this kind of work.

Michael Nielsen

It’s an incredibly complicated and rich question. It seems like the question is, how do you make it a higher-growth context? How do you make it a more demanding context? You can do that in relatively small ways that might yield compounding returns, or you can do something that is more radical. Maybe it means starting a parallel project in which you do something that is actually quite a bit different.

There is something really interesting about how being very demanding can simply change your response to something. Something that I would sometimes do with students and sometimes with myself, it was really aimed more at myself, was they would say some week, “I’m going to try and do this work over the coming week.” Then the next week would come by and they hadn’t solved the problem. If a million dollars had been at stake, would you have put the same effort in? And the answer is no, invariably. They’ve tried, but they haven’t really tried.

I think that’s a very familiar feeling for all of us. You could do a lot more if you had just the right demanding taskmaster standing by you and saying, “Look, you’re barely operating here.” I do wonder a little bit about what’s the demanding taskmaster? What can they ask you that is going to make your preparation way more intense?

Dwarkesh Patel

The most helpful thing honestly is… For some subjects it is very clear how I prep. I’m doing an upcoming episode on chip design with the founder of a company that does chip design, and he wrote a textbook on it. Yesterday I went over to his office, and we brainstormed five roofline analyses I can do. If I understand that, I have some good understanding.

The problem is with almost every other field, there’s not this curriculum. When I interviewed Ilya three, four years ago, it was: implement the transformer, and if you implement it, you have some nugget of understanding you have clamped down. With other fields, it’s just that I vaguely understand this. It’s not clamped. There’s no forcing function of “do this exercise, and if you do it, you will understand.”

Michael Nielsen

Really what you’re saying is you can do a good job at podcasting without actually attaining this kind of understanding, and that’s the problem from your point of view. You want to change your job description so that you are internalizing these chunks and just getting this kind of integration each time. It seems to me that what that means is you actually want to change the structure of the work output at some level.

There’s this terrible idea that lots of people have that they should be in flow all of the time. And as far as I can tell, high performers just don’t believe this at all. They’re in flow some of the time. You certainly see this with athletes. When they’re actually out there playing basketball or tennis, ideally they are in flow much of the time. But when they’re training they’re not. They’re stuck a lot of the time, or they’re doing things badly. I suppose I wonder what that looks like for you.

Dwarkesh Patel

That I would be extremely satisfied with. The problem is I just don’t know what the equivalent of doing 64 laps is. This is a thing you can change by choosing guests where there is a legible curriculum. So maybe it’s a mistake not to have done that. Also, there’s no real way to prep for Terence Tao. There’s no curriculum that’s a plausible one.

There are many failure modes, but one long-term dynamic I’m worried about is that you can have a good podcast and reach a local maximum, but for no particular guest or topic are you going deep enough. My model of learning is that if you don’t really understand the deeper mechanism, you’re just mapping inputs and outputs of a black box. That just fades incredibly fast or is not worth it in the first place. You just move on and it’s over. You need to build the intermediate connection.

AI in a weird way is really easy for that reason, because there is a clear thing you can do. Just implement it, and then you understand it. If I applied that criterion elsewhere, do I just not do history episodes?

Michael Nielsen

Exactly. Ada Palmer. Wonderful to talk to, incredibly interesting. But for you personally, what changed?

Dwarkesh Patel

There are some things I learned. If I had allocated more time, especially after the interview, to write up 2,000 words on everything I learned and how it connects to other things I know. Maybe that’s a thing worth doing, spreading out the episodes more and spending more time afterwards consolidating.

I would pay infinite amounts of money if there was somebody who was really good at coming up with the curriculum, the practice problems you need to do, and the exercise you need to do after the interview to clamp what you have learned.

Michael Nielsen

Have you tried doing that with somebody?

Dwarkesh Patel

It’s hard to find someone. I haven’t tried super hard, but isn’t it going to be tough to find somebody who could do that for every single kind of discipline? Maybe I should just hire different ones for different topics.

Michael Nielsen

Maybe. There’s something about, what problem are you solving for each episode? As far as I can tell, that’s the only way I really understand anything. I get interested in something. At first, I don’t even have a problem, but there’s just some sense that there’s some contribution to make here, and gradually you hone in, and there’s a problem.

Funnily enough, spending time stuck is incredibly important. That used to just be annoying. Now it seems like it’s maybe even the most important part of the whole process. That hard-won nature of it means that I internalize it afterwards. I’ve written 10,000-word essays in a couple of days, and I’ve written them in three months or six months. I feel like I didn’t learn very much from the ones that only took a couple of days. Whereas some of the ones that took three months, 15 years later, I’ll still remember.

Dwarkesh Patel

Can you describe outside of physics how you learn, of the ones that took three months?

Michael Nielsen

By far the most common thing is there’s always some creative artifact. Sometimes it’s a class. Sometimes it’s engagement with a group of people who are working on some collective creative artifact together. You might not even be aware of it, but you’re acting as an input to their creative ends in some way. Sometimes it’s an essay or a book or whatever.

It’s one of the reasons why I often quite enjoy doing podcasts. I said yes to come here partially because I know you ask unusually demanding questions. That’s an attempt to get this sort of perspective from a different kind of a forcing function. Trying to pick the most demanding creative context.

Dwarkesh Patel

For this interview, I went through three lectures of the Susskind special relativity book. The problem is that there’s almost no practice problems in it. So I hired a physicist friend. I haven’t done it yet, but for every lecture I want a bunch of practice problems to go through, and I’m planning on being appropriately humbled.

Michael Nielsen

How do you make it as jugular as possible? The higher you can raise the stakes, the better.

Dwarkesh Patel

The interview is in some sense high stakes, but also it doesn’t necessarily test deep understanding.

Michael Nielsen

I don’t think the interview is that high stakes. You’re not writing a book about special relativity, and you’re not trying to write a book that replaces whatever the existing standard textbook is. That’s a really high stake.

By the way, a phrase that I find particularly difficult. People will talk about “going deep” on a subject, and it turns out different people have different ideas of what this means. For some people it means they read a couple of blog posts. For some people it means they read a book about it. For some people it means they wrote a book about it. The standard you hold yourself to determines a lot about your ability to integrate knowledge in this way.

Dwarkesh Patel

I found that I’m in some sense able to move much faster on some things through the help of AI, but I don’t know if I’m learning better. I think it’s probably because… The hardest thing, the thing that is most demanding, is so aversive that you try to take any excuse you can to get out of it. Just having a back-and-forth conversation with an LLM where you gloss over…

Michael Nielsen

It’s entertaining but not necessarily anything else.

Dwarkesh Patel

It’s such an easy way to get out of the thing. In fact, it makes it easier because instead of doing some intermediate thinking, there’s always a next question you can ask a chatbot.

Michael Nielsen

Yeah. And it’s somewhat valuable. That’s part of the seductiveness, of course. It’s not actually useless. But it can substitute for actually doing the thing that maybe you should be doing. It’s interesting. To what extent should you be outsourcing that kind of stuff? It’s an interesting judgment call. There is a whole bunch of routine work that you want done. It’s low value for you, so if you can get a chatbot to do it, you may as well.

Somebody interviewed the pioneering computer scientist Alan Kay years ago, and he was asked what he thought about Linux. If I remember his answer correctly, he basically said, “It doesn’t have anything to do with computer science. It’s just a great big ball of mud. There are a few interesting ideas in there which are worth understanding, but mostly all you’re learning is stuff about Linux. You’re not actually learning anything which is transferable.” I thought that was very interesting.

There’s a certain kind of seductiveness to some things where it’s sort of a Rube Goldberg machine. You can just learn about all the bits, and it feels entertaining. But if you step back and think about what you’re actually doing here, it might not actually be meeting your objectives. Maybe you want to become a sysadmin, and learning Linux is a great use of your time. There’s no harm in that at all.

But if your objective is to understand the fundamentals of computing, it’s much less clear that that’s a good use of your time. It was certainly an answer I’ve thought a lot about, where for a certain type of mind, there is a seductiveness in just learning systems and confusing that with understanding.

Dwarkesh Patel

Okay, I’ll keep you updated on how this goes. I owe you a text within a month of some revamped learning system.

Michael Nielsen

I’d be really curious. It’s also true that tiny incremental improvements in this are just worth so much.

Dwarkesh Patel

It’s the main input into the podcast. It’s great that the bookshelves are fancy and I’ve got a blackboard or whatever, but really the thing that makes the podcast better is if I can improve the learning I do. So yes, it’s worth every morsel of improvement. All right, thanks for the therapy session. Great note to end on. Thanks, Michael.

Michael Nielsen

All right. Thanks, Dwarkesh.

People sometimes say that AI will make especially fast progress at scientific discovery because of tight verification loops.

But the story of how we discovered the shape of our solar system shows how the verification loop for correct ideas can be decades (or even millennia) long.

During this time, what we know today as the better theory can often actually make worse predictions (Copernicus's model of circular orbits around the sun was actually less accurate than Ptolemy's geocentric model).

And the reasons it survives this epistemic hell is some mixture of judgment and heuristics that we don’t even understand well enough to actually articulate, much less codify into an RL loop.

Hope you enjoy!

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Jane Street loves challenging my audience with different creative puzzles. One of my listeners, Shawn, solved Jane Street’s ResNet challenge and posted a great walk-through on X. If you want to try one of these puzzles yourself, there’s one live now at janestreet.com/dwarkesh.

• Labelbox can get you rubric-based evals, no matter your domain. These rubrics allow you to give your model feedback on all the dimensions you care about, so you can train how it thinks, not just what it thinks. Whatever you’re focused on—math, physics, finance, psychology or something else—Labelbox can help. Learn more at labelbox.com/dwarkesh.

• Mercury just released a new feature called Insights. Insights summarizes your money in and out, showing you your biggest transactions and calling out anything worth paying attention to. It’s a super low-friction way to stay on top of your business. Learn more at mercury.com/insights.

Timestamps

(00:00:00) – Kepler was a high temperature LLM

(00:11:44) – How would we know if there’s a new unifying concept within heaps of AI slop?

(00:26:10) – The deductive overhang

(00:30:31) – Selection bias in reported AI discoveries

(00:46:43) – AI makes papers richer and broader, but not deeper

(00:53:00) – If AI solves a problem, can humans get understanding out of it?

(00:59:20) – We need a semi-formal language for the way that scientists actually talk to each other

(01:09:48) – How Terry uses his time

(01:17:05) – Human-AI hybrids will dominate math for a lot longer

Transcript

00:00:00 – Kepler was a high temperature LLM

Dwarkesh Patel

Today, I’m chatting with Terence Tao, who needs no introduction. Terence, I want to begin by having you retell the story of how Kepler discovered the laws of planetary motion because I think this will be a great jumping off point to talk about AI for math.

Terence Tao

I’ve always had an amateur interest in astronomy. I’ve loved stories of how the early astronomers worked out the nature of the universe. Kepler was building on the work of Copernicus, who was himself building on the work of Aristarchus. Copernicus very famously proposed the heliocentric model, that instead of the planets and the Sun going around the Earth, the Sun was at the center of the solar system and the other planets were going around the Sun.

Copernicus proposed that the orbits of the planets were perfect circles. His theory fit the observations that the Greeks, the Arabs, and the Indians had worked out over centuries. Kepler learned about these theories in his studies, and he made this observation that the ratios of the size of the orbits that Copernicus predicted seemed to have some geometric meaning.

He started proposing that if you take the orbit of the Earth and you enclose it in a cube, the outer sphere that encloses the cube almost perfectly matched the orbit of Mars, and so forth. There were six planets known at the time and five gaps between them, and there were five perfect Platonic solids: the cube, the tetrahedron, icosahedron, octahedron, and dodecahedron.

So he had this theory, which he thought was absolutely beautiful, that you could inscribe these Platonic solids between the spheres of the planets. It seemed to fit, and it seemed to him that God’s design of the planets was matching this mathematical perfection of the Platonic solids.

He needed data to confirm this theory. At the time, there was only one really high-quality dataset in existence. Tycho Brahe, this very wealthy, eccentric Danish astronomer, had managed to convince the Danish government to fund this extremely expensive observatory. In fact, it was an entire island where he had taken decades of observations of all the planets, like Mars and Jupiter, at least every night for which the weather was clear, with the naked eye. He was the last of the naked-eye astronomers.

He had all this data which Kepler could use to confirm his theory. Kepler started working with Tycho, but Tycho was very jealous of the data. He only gave him little bits of it at a time. Kepler eventually just stole the data. He copied it and had to have a fight with Brahe’s descendants.

He did get the data, and then he worked out, to his disappointment, that his beautiful theory didn’t quite work. The data was off from his Platonic solid theory by 10% or something. He tried all kinds of fudges, moving the circles around, and it didn’t quite work. But he worked on this problem for years and years, and eventually, he figured out how to use the data to work out the actual orbits of the planets.

That was an incredibly clever, genius amount of data analysis. And then he worked out that the orbits were actually ellipses, not circles, which was shocking for him. So he worked out the two laws of planetary motion: the ellipses, and also that equal areas sweep out equal times.

Then ten years later, after collecting a lot of data—the furthest planets like Saturn and Jupiter were the hardest for him to work out—he finally worked out this third law, that the time it takes for a planet to complete its orbit was proportional to some power of the distance to the Sun. These are the three famous Kepler’s laws of motion. He had no explanation for them. It was all driven by experiment, and it took Newton a century later to give a theory that explained all three laws at once.

Dwarkesh Patel

The take I want to try on you is that Kepler was a high-temperature LLM. Newton comes up with this explanation of why the three laws of planetary motion must be true. Of course, the way that Kepler discovers the laws of planetary motion, or figures out the relative orbits of the different planets, is as you say a work of genius. But through his career, he’s just trying random relationships.

In fact, in the book in which he writes down the third law of planetary motion, it’s an aside on The Harmonics of the World, which is just a book about how all these different planets have these different harmonies. And the reason there’s so much famine and misery on Earth is because the Earth is mi-fa-mi, that’s the note of Earth. It’s all this random astrology, but in there is the cube-square law, which tells you what relationship the period has to a planet’s distance from the Sun. As you were detailing, if you add that to Newton’s F=ma and the equation for centripetal acceleration, you get the inverse-square law. And so Newton works that out.

But the reason I think this is an interesting story is that I feel LLMs can do the kind of thing of trying random relationships for twenty years, some of which make no sense, as long as there’s a verifiable data bank like Brahe’s dataset. “Ok, I’m going to try out random things about musical notes, Platonic objects, or different geometries, I have this bias that there’s some important thing about the geometry of these orbits.”

Then one thing works. As long as you can verify it, these empirical regularities can then drive actual deep scientific progress.

Terence Tao

Traditionally, when we talk about the history of science, idea generation has always been the prestige part of science. A scientific problem comes with many steps. You have to identify a problem, and then you have to identify a good, fruitful problem to work on. Then you need to collect data, figure out a strategy to analyze the data, and make a hypothesis. At this point, you need to propose a good hypothesis, and then you need to validate. Then you need to write things up and explain. There are a dozen different components.

The ones we celebrate are these eureka genius moments of idea generation. Kepler certainly had to cycle through many ideas, several of which didn’t work. I bet there were many that he didn’t even publish at all because they just didn’t fit. That’s an important part of the process, trying all kinds of random things and seeing if they worked.

But as you say, it has to be matched by an equal amount of verification, otherwise it’s slop. We celebrate Kepler, but we should also celebrate Brahe for his assiduous data collection, which was ten times more precise than any previous observation. That extra decimal point of accuracy was essential for Kepler to get his results. He was using Euclidean geometry and the most advanced mathematics he could use at the time to match his models with the data. All aspects had to be in play: the data, the theory, and the hypothesis generation.

I’m not sure nowadays that hypothesis generation is the bottleneck anymore. Science has changed in the century since. Classically, the two big paradigms for science were theory and experiment. Then in the 20th century, numerical simulation came along, so you can do computer simulations to test theories. Finally, in the late 20th century, we had big data. We had the era of data analysis.

A lot of new progress is actually driven now by analyzing massive datasets first. You collect large datasets and then draw patterns from them to deduce thoughts. This is a little bit different from how science used to work, where you make a few observations or have one out-of-the-blue idea, and then collect data to test your idea. That’s the classic scientific method. Now it’s almost reversed. You collect big data first, and then you try to get hypotheses from it.

Kepler was maybe one of the first early data scientists, but even he didn’t start with Tycho’s dataset and then analyze it. He had some preconceived theories first. It seems like this is less and less the way we make progress, just because the data is so much more massive and useful.

Dwarkesh Patel

Oh, interesting. I feel like the 20th-century science that you’re describing actually very well describes what happened with Kepler. He did have these ideas—1595 and ‘96 is where he comes up with the polygons and then the Platonic objects theory—but they were wrong. Then a few years later, he gets Brahe’s data, and it’s only after twenty years of trying random things that he gets this empirical regularity.

It actually feels a bit closer to Brahe’s data being analogous to some massive data bank of simulations, and now that you’ve got the data, you can keep trying random things. If it wasn’t for that, Kepler would be out there just writing books about harmonics and Platonic objects, and there would be nothing to actually verify against.

Terence Tao

The data was extremely important. The distinction I was trying to make was that traditionally, you make a hypothesis and then you test it against data. But now with machine learning, data analysis, and statistics, you can start with data and through statistics work out laws that were not present before.

Kepler’s third law is a little bit like this, except that instead of having the thousand data points that Brahe had, Kepler had six data points. For every planet, he knew the length of the orbit and the distance to the Sun. There were five or six data points, and he did what we would now call regression. He fit a curve to these six data points and got a square-cube law, which was amazing. But he was quite lucky that these six data points gave him the right conclusion. That’s not enough data to be really reliable.

There was a later astronomer, Johann Bode, who took the same data—the distances to the planets—and inspired by Kepler, he had a prediction that the distances to the planets formed a shifted geometric progression. He also fit a curve, except there was one point missing. There was a big gap between Mars and Jupiter. His law predicted that there was a missing planet. It was kind of a crank theory, except when Uranus was discovered by Herschel, the distance to Uranus fit exactly this pattern. Then Ceres was discovered in the asteroid belt, and it also fit the pattern. People got really excited that Bode had discovered this amazing new law of nature.

But then Neptune was discovered, and it was way off. Basically it was just a numerical fluke. There were six data points. Maybe one reason why Kepler didn’t highlight his third law as much as the first two laws is that instinctively, even though he didn’t have modern statistics, he kind of knew that with six data points, he had to be somewhat tentative with the conclusions.

00:11:44 – How would we know if there’s a new unifying concept within heaps of AI slop?

Dwarkesh Patel

To ask the question about the analogy more explicitly, does this analogy make sense if in the future we have smarter and smarter AIs? We’ll have millions of them, and they can go out and hunt for all these empirical irregularities. It sounds like you don’t think the bottleneck in science is finding more things that are the equivalent of the third law of planetary motion for each given field, so that later on somebody can say, “Oh, we need a way to explain this. Let’s work out the math. Here’s the inverse-square law of gravity.”

Terence Tao

I think AI has driven the cost of idea generation down to almost zero, in a very similar way to how the internet drove the cost of communication down to almost zero. It’s an amazing thing, but it doesn’t create abundance by itself. Now the bottleneck is different. We’re now in a situation where suddenly people can generate thousands of theories for a given scientific problem. Now we have to verify them, evaluate them. This is something which we have to change our structures of science to actually sort this out.

Traditionally, we build walls. In the past, before we had AI slop, we had amateur scientists have their own theories of the universe, many of which were of very little value. We built these peer review publication systems to filter out and try to isolate the high signal ideas to test.

But now that we can generate these possible explanations at massive scale, and some of them are good and a lot are terrible, human reviewers are already being overwhelmed. Many journals are reporting that AI-generated submissions are just flooding their submissions.

It’s great that we can generate all kinds of things now with AI, but it means that the rest of the aspects of science have to catch up: verification, validation, and assessing what ideas actually move the subject forward and which ones are dead ends or red herrings. That’s not something we know how to do at scale. For each individual paper, we can have a debate among scientists and get to a consensus in a few years. But when we’re generating a thousand of these every day, this doesn’t work.

Dwarkesh Patel

There’s this incredibly interesting question. If you have billions of AI scientists, not only how do you gauge which ones are real progress, but how do you... This is actually a question that human science has had to face and we’ve solved somehow, and I’m actually not sure how we solved this.

Let’s say in the 1940s, if you’re at Bell Labs and there are these new technologies coming out. Pulse-code modulation, how do you transfer signals? How do you digitize signals? How do you transfer them over analog wires? There are all these papers about the engineering constraints and the details, and then there’s one which comes up with the idea of the bit, which has implications across many different fields. You need some system which can then look at that and say, “Okay, we need to apply this to probability. We need to apply this to computer science,” et cetera.

In the future, the AIs are coming up with the next version of this unifying concept. How would you identify it among millions of papers that might actually constitute progress, but which have much less in terms of general unifying ideas?

Terence Tao

A lot of it’s the test of time. Many great ideas didn’t actually get a great reception at the time they were first proposed. It was only after some other scientists realized that they could take it further and apply them to their own... Deep learning itself was a niche area of AI for a long time. The idea of getting answers entirely through training on data and not through first principles reasoning was very controversial, and it just took a long time before it started bearing fruit.

You mentioned the bit. There were other proposals for computer architectures than the zero-one that is universal today. I think there were trits, three-valued logic. In an alternate universe, maybe a different paradigm would have shown up. The transformer, for example, is the foundation of all modern large language models, and it was the first deep learning architecture that really was sophisticated enough to capture language. But it didn’t have to be that way. There could’ve been some other architecture that was the first to do it and once that was adopted, it would become the standard.

One reason why it’s hard to assess whether a given idea is going to be fruitful is that it depends on the future. It depends also on the culture and society, which ones get adopted, which ones don’t. The base ten numeral system in mathematics is extremely useful, much better than the Roman numeral system, for instance. But again, there’s nothing special about ten. It’s a system that is useful for us because everyone else uses it. We’ve standardized it. We’ve built all our computers and our number representation systems around it, so we’re stuck with it now. Some people occasionally push for other systems than decimal, but there’s just too much inertia.

It’s not something where you can look at any given scientific achievement purely in isolation and give it an objective grade without being aware of the context both in the past and the future. So it may never be something that you can just reinforcement learn the same way that you can for much more localized problems.

Dwarkesh Patel

Often in the history of science when a new theory comes up that in retrospect we realize is correct, it seems to make implications that either make no sense because they’re wrong, and we realize later on why they’re wrong, or they’re correct but seem wildly implausible at the time.

As you talked about, Aristarchus had heliocentrism in the third century BC. The ancient Athenians were like, “This can’t be because if the earth is going around the sun, we should see the relative position of the stars change as we’re going around the sun, and the only way that wouldn’t be the case is if they’re so far away that you don’t notice any parallax,” which is actually the correct implication.

But there’s times when the implication is incorrect and we just need to graduate to a better level of understanding. Leibniz would chide Newton and disagree with Newton’s theory of gravity on the basis that it implied action at a distance, and they didn’t know the mechanism, and Newton himself was sort of stunned that inertial mass and gravitational mass were the same quantity. All these things later were resolved by Einstein. But it was still progress.

So the question for a system of peer review for AI would be: even if you can falsify a theory, how would you notice that it still constitutes progress relative to the thing before?

Terence Tao

Often, the ultimately correct theory initially is worse in many ways. Copernicus’s theory of the planets was less accurate than Ptolemy’s theory. Geocentrism had been developed for a millennium by that point, and they had made many tweaks and increasingly complicated ad hoc fixes to make it more and more accurate. Copernicus’s theory was a lot simpler but much less accurate. It was only Kepler that made it more accurate than Ptolemy’s theory.

Science is always a work in progress. When you only get part of the solution, it looks worse than a theory which is incorrect but somehow has been completed to the point where it kind of answers all the questions. As you say, Newton’s theory had big mysteries. They had the equivalence of mass and action at a distance, which were only resolved with a very conceptually different approach centuries afterwards.

Often progress has to be made not by adding more theories, but by deleting some assumptions that you have in your mind. One reason why geocentrism held on for so long is we had this idea that objects naturally want to stay at rest. This is the Aristotelian notion of physics, and so the idea that the Earth was moving… How come we weren’t all falling over? Once you have Newton’s laws of motion—an object in motion remains in motion and so forth—then it makes sense.

Conceptually, it’s a very big leap to realize that the Earth is in motion. It doesn’t feel like it’s in motion. The biggest advances, like Darwin’s theory of evolution, is the idea that species are not static. This is not obvious because you don’t see evolution in your lifetime. Well, now we actually can, but it seems permanent and static.

Right now we’re going through a cognitive version of the Copernican revolution, where we used to think that human intelligence is the center of the universe, and now we’re seeing that there are very different types of intelligence out there with very different strengths and weaknesses. Our assessment of which tasks require intelligence, which ones don’t, has to be reordered quite a bit.

Trying to fit AI into our theories of scientific progress and what is hard and what is easy, we’re struggling quite a lot. We have to ask questions that we’ve never really had to ask before. Or maybe the philosophers had, but now we all have to deal with it.

Dwarkesh Patel

This brings up a topic I’ve been very curious about. You mentioned Darwin’s theory of evolution. There’s this book, The Clockwork Universe by Edward Dolnick, which covers a lot of this era of history we’re talking about. He has this interesting observation in there. The Origin of Species was published in 1859. Principia Mathematica was published in 1687.

So The Origin of Species comes out two centuries after Principia. Conceptually, it seems like Darwin’s theory is simpler. There’s a contemporaneous biologist to Darwin, Thomas Huxley, who reads The Origin of Species and he says, “How stupid not to have thought of that.”

Nobody ever says that about Principia, chiding themselves for not having beaten Newton to gravity. So there’s a question of why did it take longer?

It seems like a big part of the reason is what you were saying. The evidence for natural selection is overwhelming in a certain sense, but it’s cumulative and retrospective, whereas Newton can just say, “Here are my equations. Let me see the moon’s orbital period and its distance, and if it lines up, then we’ve made progress.”

Lucretius actually had this idea that species adapted to their environment in the first century BC but nobody really talks about it until Darwin because Lucretius couldn’t run some experiment and force people to pay attention. I wonder if we’ll in retrospect end up seeing much more progress in domains which have this kind of tight data loop where you can verify them quite easily, even though they’re conceptually much more difficult.

Terence Tao

I think one aspect of science is that it’s not just creating a new theory and validating it, but communicating it to others. Darwin was an amazing science communicator. He wrote in English, in natural language. I’m speaking like a—

Dwarkesh Patel

No Lean.

Terence Tao

I have to get out of my technical mindset. He spoke in plain English, didn’t use equations, and he synthesized a lot of disparate facts. Little pieces of evolution had been worked out in the past, but he had this very compelling vision. Again, he was still missing things. He didn’t know the mechanism for heredity, he didn’t have DNA. But his writing style was persuasive, and that helped a lot.

Newton wrote in Latin. He had invented entire new areas of mathematics just to explain what he was doing. He was also from an era where scientists were much more secretive and competitive. Academia is still competitive, but it was even worse back in Newton’s day. He held back some of his best insights because he didn’t want his rivals to get any advantage. He was also a somewhat unpleasant person from what I gather. It was only a couple of decades after Newton when other scientists explained his work in much simpler terms that they became widespread.

The art of exposition and making a case and creating a narrative is also a very important part of science. If you have the data, it helps, but people need to be convinced, otherwise they will not push it further or take the initial investment to learn your theory and really explore it. That’s another thing which is really hard to reinforcement learn on. How can you score how persuasive you are? Well, there are entire marketing departments trying to do this. Maybe it’s good that AI is not yet optimized to be persuasive.

There’s a social aspect to science. Even though we pride ourselves on having an objective side to it, where there’s data and experiment and validation, we still have to tell stories and convince our fellow scientists. That’s a soft, squishy thing. It’s a combination of data and painting a narrative, and it’s a narrative of gaps.

Even with Darwin, as I said, there were pieces of his theory he could not explain. But he could still make a case that in the future, people would find transitional forms, that they would find the mechanism of inheritance, and they did. I don’t know how you can quantify that in such a precise way that you can start doing reinforcement learning. Maybe that will be forever the human side of science.

00:26:10 – The deductive overhang

Dwarkesh Patel

One takeaway I had from reading and watching your stuff on the cosmic distance ladder… By the way, I highly recommend people watch your series with 3Blue1Brown on the cosmic distance ladder. One takeaway was that the deductive overhang in many fields could be so much bigger than people realize. If you just had the right insight about how to study a problem, you might be surprised at how much more you could learn about the world.

I wonder if you think that’s a product of astronomy at the particular times in history that you’re studying. Or is it just that based on the data that is incident on the Earth right now, we could actually divine a lot more than we happen to know?

Terence Tao

Astronomy was one of the first sciences to really embrace data analysis and squeezing every last possible drop of information out of the information they had because data was the bottleneck. It still is the bottleneck. It’s really hard to collect astronomical data.

Astronomers are world-class in extracting all kinds of conclusions from little traces of data, almost like Sherlock. I hear that for a lot of quant hedge funds, their preferred hire is an astronomy PhD, actually. They are also very interested for other reasons in extracting signals from various random bits of data.

We do under-explore how to extract extra information from various signals. Just to pick one random study, I remember reading once that people were trying to measure how often scientists actually read the papers that they cite. How do you measure this? You could try to survey different scientists, but they had a clever trick.

Many citations have little typos, like a number is wrong or punctuation is almost wrong. They measured how often a typo got copied from one reference to the next, and they could infer whether an author was just copying and pasting a reference without actually checking it. From that, they were able to infer some measure of how much attention people were paying. So there are some clever tricks to extract…

These questions you posed earlier of how we can assess whether a scientific development is fruitful, interesting, or represents real progress… Maybe there are really useful metrics or footprints of this phenomenon in data. We can examine citations and how often something is mentioned in a conference. Maybe there’s a lot of sociology of science research to be done that could actually detect these things. Maybe we should get some astronomers on the case, actually.

00:30:31 – Selection bias in reported AI discoveries

Dwarkesh Patel

That brings us nicely to the progress that, from the outside, it seems like AI for math is making. You had a post recently where you pointed out that over the last few months, AI programs have solved fifty out of the eleven hundred odd Erdős problems. I don’t know if it’s still correct, but as of a month ago you said that there had been a pause because the low-hanging fruit had been picked.

First of all, I’m curious if that is still the case, that we have picked the low-hanging fruit and now we’re at this plateau currently.

Terence Tao

It does seem so. Fifty-odd problems have been solved with AI assistance, which is great, but there’s like six hundred to go. People are still chipping away at one or two of these right now.

We’re seeing a lot fewer pure AI solutions now where the AI just one-shots the problem. There was a month where that happened and that has stopped, not for lack of trying. I know of three separate attempts to get frontier model AIs to just attack every single one of the problems simultaneously. They pick out some minor observations, or maybe they find that some problem was already solved in the literature, but there hasn’t been any further purely AI-powered solution yet.

People are using AI a lot currently. Someone might use AI to generate a possible proof strategy, and then another person will use a separate AI tool to critique it, rewrite it, generate some numerical data for it, or do a literature survey. Some problems have been solved by an ongoing conversation between lots of humans and lots of AI tools. But it does seem like it was this one-off thing.

Maybe one analogy for these problems is that you’re in some sort of mountain range with all kinds of cliffs and walls. Maybe there’s a little wall which is three feet high, and one that’s six feet high, and then there’s fifteen feet high, and then there are some mile-high cliffs. You’re trying to climb as many of these cliffs as possible, but it’s in the dark. We don’t know which ones are tall, which ones are short. So we try to light some candles and make some maps, and slowly we figure out some of them are climbable. Some of them we can identify a partial track in the wall that you can reach first.

These AI tools, they’re like jumping machines that can jump two meters in the air, higher than any human. Sometimes they jump in the wrong direction, and sometimes they crash, but sometimes they can reach the tops of the lowest walls that we couldn’t reach before. We’ve just set them loose in this mountain range, hopping around. There was this exciting period where they could actually find all the low ones and reach them. Maybe the next time there’s a big advance in the models, they will try it again, and a few more will be breached.

But it’s a different style of doing mathematics. Normally we would hill climb, make little markers, and try to identify partial things. These tools either succeed or they fail. They’ve been really bad at creating partial progress or identifying intermediate stages that you should focus on first. Going back to this previous discussion, we don’t have a way of evaluating partial progress the same way we can evaluate a one-shot success or failure of solving a problem.

Dwarkesh Patel

There’s two different ways to think through what you’ve just said. One of them is more bearish on AI progress, and one of them is more bullish. The bearish one being, “Oh, they’re only getting to a certain height of wall, which is not as high as humans are reaching.”

The second is that they have this powerful property that once they achieve a certain waterline, they can fill every single problem that is available at that waterline, which we simply can’t do with humans. We can’t make a million copies of you and give each of them a million dollars of inference compute and have you do a hundred years of subjective time research on a million different problems at the same time.

But once AIs reach Terence Tao-level, they could do that. Once they reach intermediate levels, they could do the intermediate version of that. The same reason that we should be bearish now is the reason we should be especially bullish. Not even when they achieve superhuman intelligence, but just when they achieve human-level intelligence, because their human-level intelligence is qualitatively wider and more powerful than our human-level intelligence.

Terence Tao

I agree. They excel at breadth, and humans excel at depth, human experts at least. I think they’re very complementary. But our current way of doing math and science is focused on depth because that’s where human expertise is, because humans can’t do breadth. We have to redesign the way we do science to take full advantage of this breadth capability that we now have.

We should have a lot more effort in creating very broad classes of problems to work on rather than one or two really deep, important problems. We should still have the deep, important problems, and humans should still be working on them. But now we have this other way of doing science. We can explore entirely new fields of science by first getting these broad, moderately competent AIs to map it out and make all the easy observations. And then identify certain islands of difficulty, which human experts can then come and work on.

I see very much a future of very complementary science. Eventually, you would hope to get both breadth and depth and somehow get the best of both worlds. But we need practice with the breadth side. It’s too new. We don’t even have the paradigms to really take full advantage of it. But we will, and then science will be unrecognizable after that, I think.

Dwarkesh Patel

To this point about complementarity, programmers have noticed that they’re way more productive as a result of these AI tools. I don’t know if you as a mathematician feel the same way, but it does seem like one big difference between vibe coding and vibe researching is that with software, the whole point is to have some effect on the world through your work. If it leads to you better understanding a problem or coming up with some clean abstraction to embody in your code, that is instrumental to the end goal.

Whereas with research, the reason we care about solving the Millennium Prize Problems is that presumably that in the process of solving them, we discover new mathematical objects or new techniques that advance our civilization’s understanding of mathematics. So the proof is instrumental to the intermediate work. I don’t know if you agree with that dichotomy or if that in any way will explain the relative uplift we’ll see in software versus research.

Terence Tao

Certainly in math, the process is often more important than the problem itself. The problem is kind of a proxy for measuring progress. I think even in software, there are different types of software tasks. If you just create a webpage that does the same thing that a thousand other webpages do, there’s no skill to be learned. Well, there is still some skill maybe that the individual programmer could pick up. But for boilerplate-type code, it’s something that you should definitely offload to AI.

Sometimes once you make the code, you still have to maintain it. There are issues with upgrading it and making it compatible with other things. I’ve heard programmers report that even if an AI can create the first prototype of a tool, making it mesh with everything else and making it interact with the real world in the way they want is an ongoing process. If you don’t have the skills that you pick up from writing the code, that may impact your ability to maintain it down the road.

So yes, certainly mathematicians, we’ve used problems to build intuition and to train people to have a good idea of what’s true, what to expect, what is provable, and what is difficult. Just getting the answers right away may actually inhibit that process.

I made a distinction between theory and experiment before. In most sciences, there’s an equal division between the theoretical side and the experimental side. Math has been unique in that it’s almost entirely theoretical. We place a premium on trying to have coherent, clean theories of why things are true and false. We haven’t done many experiments as to, if we have two different ways to solve a problem, which is more effective. We have some intuition, but we haven’t done large-scale studies where we take a thousand problems and just test them.

But we can do that now. I think AI-type tools will actually revolutionize the experimental side of math, where you don’t care so much about individual problems and the process of solving them, but you want to gather large-scale data about what things work and what things don’t. The same way that if you’re a software company and you want to roll out a thousand pieces of software, you don’t really want to handcraft each one and learn lessons from each. You just want to find what workflows let you scale.

The idea of doing mathematics at scale is at its infancy. But that’s where AI is really going to revolutionize the subject.

Dwarkesh Patel

I feel like a big crux in these conversations about how good AI will be for science is, I think you said this, that they’re using existing techniques and modifying them. It would be interesting to understand how much progress one can make simply from using existing techniques.

If I looked at the top math journals, how many of the papers are coming up with a new technique, whatever that means, versus using existing techniques on new problems? What is the overhang? If you just applied every known technique to every open problem, would that constitute a humongous uplift in our civilization’s knowledge, or would that not be that impressive and useful?

Terence Tao

This is a great question, and we don’t have the data to fully answer it yet. Certainly, a lot of work that human mathematicians do… When you take a new problem, one of the first things we do is we look at all the standard things that have worked on similar problems in the past, and we try them one by one. Sometimes that works, and that’s still worth publishing because the question was important.

Sometimes they almost work, and you have to add one more wrinkle to it, and that’s also interesting. But the papers that go into the top journals are usually ones where the existing methods can kind of solve 80% of the problem, but then there is this 20% which is resistant and a new technique has to be invented to fill in the gaps.

It’s very rare now that a problem gets solved with no reliance on past literature, where all the ideas come out of nowhere. That was more common in the past, but math is so mature now that it’s just so much of a handicap to not use the literature first.

AI tools are getting really good at the first part of that, just trying all the standard techniques on a problem, often making fewer mistakes in applying them than humans. They still make mistakes, but I’ve tested these tools on little tasks that I can do, and sometimes they pick up errors that I make. Sometimes I pick up errors that they make. It’s about a tie right now.

But I haven’t yet seen them take the next step. When there are holes in the argument where none of the things are working, then what do you do? They can suggest random things, but often I find that trying to chase them down to make them work, and finding they don’t work, wastes more time than it saves.

I think some fraction of problems that we currently think are hard will fall from this method, especially the ones that haven’t received enough attention. With the Erdős problems, almost all of the 50 problems that were solved by AIs were ones for which there was basically no literature. Erdős posed the problem once or twice. Maybe some people tried it casually and couldn’t do it, but they never wrote up anything.

But it turned out that there was a solution, and it was just combining this one obscure technique that not many people know about with some other result in the literature. That’s the median level of what AI can accomplish, and that’s really great. It clears out 50 of these problems. So I think you will see some isolated successes.

But what we found… Some people have done large-scale sweeps of these Erdős problems. If you only focus on the success stories, the ones that get broadcast on social media, it looks amazing. All these problems that haven’t been solved for decades, now they’re falling. But whenever we do a systematic study, on any given problem an AI tool has a success rate of maybe 1% or 2%. It’s just that they can buy scale, and you just pick the winners. It looks great.

I think there’ll be a similar thing happening with the hundreds of really prestigious, difficult math problems out there. Some AI may get lucky and actually solve them, and there will be some backdoor to solve the problem that everyone else missed. That will get a lot of publicity. But then people will try these fancy tools on their own favorite problem, and they will again experience the 1% to 2% success rate.

There’ll be a lot of noise amongst the signal of when they’re working and when they’re not. It will be increasingly important to collect these really standardized datasets. There are efforts now to create a standard set of challenge problems for AIs to solve, and not just rely on the AI companies to only publish their wins and not disclose their negative results. That will maybe give more clarity as to where we’re actually at.

Dwarkesh Patel

Although I think it’s worth emphasizing how much progress in AI it constitutes already, to have models that are capable of applying some technique that nobody had written down as applicable to this particular problem.

Terence Tao

The progress is simultaneously amazing and disappointing. It is a very strange feeling to see these tools in action. But people also acclimatize really quickly.

I remember when Google’s web search came out 20 years ago. It just blew all the other searches out of the water. You’re getting relevant hits on the front page, exactly what you wanted. It was amazing, and then after a few years, you just took for granted that you could Google anything.

2026-level AI would be stunning in 2021. A lot of it—face recognition, natural speech, doing college-level math problems—we just take for granted now.

00:46:43 – AI makes papers richer and broader, but not deeper

Dwarkesh Patel

Speaking of 2026 AI, you made a prediction in 2023 that by 2026 it would be like a colleague in mathematics?

Terence Tao

A trustworthy co-author if used correctly.

Dwarkesh Patel

Which is looking pretty good in retrospect.

Terence Tao

Yeah, I’m pretty pleased.

Dwarkesh Patel

So let’s see if you can continue this streak. You personally are 2x more productive as a result of AI. What year would you say that?

Terence Tao

Productivity, I think, is not quite a one-dimensional quantity. I’m definitely noticing that the style in which I do mathematics is changing quite a bit, and the type of things I do. For example, my papers now have a lot more code, a lot more pictures, because it’s so easy to generate these things now. Some plot which would have taken me hours to do, now I can do in minutes. But in the past, I just wouldn’t have put the plot in my paper in the first place. I would just talk about it in words. So it’s hard to measure what 2x means.

On the one hand, I think the type of papers that I would write today, if I had to do them without AI assistance, would definitely take five times longer. But I would not write my papers that way.

Dwarkesh Patel

5x?

Terence Tao

Yeah, but these are auxiliary tasks. Things like doing a much deeper literature search or supplying a lot more numerics. They enrich the paper. The core of what I do, actually solving the most difficult part of a math problem, hasn’t changed too much. I still use pen and paper for that.

But there’s lots of silly things. I use an AI agent now to reformat. Sometimes if all my parentheses are not quite the right size, I used to manually change them by hand, and now I can get an AI agent to do all that quite nicely in the background.

They’ve really sped up lots of secondary tasks. They haven’t yet sped up the core thing that I do, but it’s allowed me to add more things to my papers. By the same token, if I were to write a paper I wrote in 2020 again—and not add all these extra features, but just have something of the same level of functionality—it actually hasn’t saved that much time, to be honest. It’s made the papers richer and broader, but not necessarily deeper.

Dwarkesh Patel

You made this distinction between artificial cleverness and artificial intelligence. I would like to better understand those concepts. What is an example of intelligence that is not just cleverness?

Terence Tao

Intelligence is famously hard to define. It’s one of these things that you know when you see it. But when I talk to someone and we’re trying to collaboratively solve a math problem together, there’s this conversation where neither of us knows how to solve the problem initially. One of us has some idea and it looks promising, so then we have some sort of prototype strategy. We test it, and it doesn’t work, but then we modify it. There’s adaptivity and continual improvement of the idea over time. Eventually, we’ve systematically mapped out what doesn’t work and what does work, and we can see a path forward, but it’s evolving with our discussion.

This isn’t quite what the AIs do. The AIs can mimic this a little bit. To go back to this analogy of these jumping robots, they can jump and fail, and jump and fail. But what they can’t do is jump a little bit, reach some handhold, stay there, pull other people up, and then try to jump from there. There isn’t this cumulative process which is built up interactively. It seems to be a lot more trial and error and just repetition: brute force. It scales, and it can work amazingly well in certain contexts. But this idea of building up cumulatively from partial progress is what’s still not quite there yet.

Dwarkesh Patel

Interesting. You’re saying if Gemini 3 or Claude 4.5, whatever, solves a problem, it is not the case that its own understanding of math has progressed.

Terence Tao

No.

Dwarkesh Patel

Or even if it works on a problem without solving it, it’s not that its own understanding of math has progressed.

Terence Tao

Yeah. You run a new session and it’s forgotten what it just did. It has no new skills to build on related problems. Maybe what you just did is 0.001% of the training data for the next generation. So maybe eventually some of it gets absorbed.

00:53:00 – If AI solves a problem, can humans get understanding out of it?

Dwarkesh Patel

One big question I have is how plausible is it that if we just keep training AIs—they get better and better at solving problems in Lean—that they will continue to solve more and more impressive problems, and then we will be surprised at how little insight we got from some Lean solution to proving the Riemann hypothesis or something.

Or do you think it is a necessary condition of solving the Riemann hypothesis, even by an AI that is doing it entirely in Lean, that the constructions and definitions created in the Lean program have to advance our understanding of mathematics? Or could it just be assembly code gobbledygook?

Terence Tao

We don’t know. Some problems have been basically solved by pure brute force. The four color theorem is a famous example. We have still not found a conceptually elegant proof of this theorem, and maybe we never will. Some problems may only be solvable by splitting into an enormous number of cases and doing brute force, uninsightful computer analysis on each case.

Part of the reason we prize problems like the Riemann hypothesis is that we’re pretty sure a new type of mathematics has to be created, or a new connection between two previously unconnected areas of mathematics has to be discovered to make this work. We don’t even know what the shape of the solution is, but it doesn’t feel like a problem that will be solved just by exhaustively checking cases.

Or it could be false actually. Okay, there is an unlikely scenario that the hypothesis is false, and you can just compute a zero off the line, and a massive computer calculation verifies it. That would be very disappointing. I do feel that fully autonomous, one-shot approaches are not the right approach for these problems. You’ll get a lot more mileage out of the interplay of humans collaborating with these tools.

I can see one of these problems being solved by smart humans assisted by extremely powerful AI tools. But the exact dynamic may be very different from what we envision right now. It could be a collaboration of a type that just doesn’t exist yet.

There may be a way to generate a million variants of the Riemann zeta function and do AI-assisted data analysis to discover some pattern connecting them that we didn’t know about before. This lets you transform the problem into a different area of mathematics. There could be all kinds of scenarios.

Dwarkesh Patel

Suppose the AI figures it out, and latent in the Lean is some brand-new construction which, if we realized its significance, we would be able to apply in all these different situations. How would we even recognize it?

Again, a very naive question, but if you come up with the equivalent of Descartes’ idea that you can have a coordinate system unifying algebra and geometry, in Lean code it would just look like R→R, and it wouldn’t look that significant. I’m sure there are other constructions which have this kind of property.

Terence Tao

The beauty of formalizing a proof in something like Lean is that you can take any piece of it and study it atomically. When I read a paper which solves some difficult problem, there’s often a big sequence of lemmas and theorems. Ideally, the author will talk their way through what’s important and what’s not. But sometimes they don’t reveal what steps were the important ones and which ones were just boilerplate, standard steps.

You can study each lemma in isolation. Some of them I can see look fairly standard and resemble something I’m familiar with. I’m pretty sure there’s nothing interesting going on there. But this other lemma, that’s something I haven’t seen before, and I can see why having this result would really help prove the main result. You can assess whether a step is really key to your argument or not, and Lean really facilitates that. The individual steps are identified really precisely.

I think in the future, there will be entire professions of mathematicians who might take a giant Lean-generated proof and do some ablation on it, trying to remove parts of it and find more elegant ways. They might get other AIs to do some reinforcement learning to make the proof more elegant, and maybe other AIs will grade whether this proof looks better or not.

One thing that will change quite a bit in the near future is how we write papers. Until recently, writing papers was the most time-consuming and expensive part of the job. So you did it very rarely. You only wrote up your results once all the other parts of your argument were checked out, because rewriting and refactoring was just a total pain. That’s become a lot easier now with modern AI tools. You don’t have to have just one version of your paper. Once you have one, people can generate hundreds more.

One giant messy Lean proof may not be very meaningful or understandable on its own, but other people can refactor it and do all kinds of things with it. We’ve seen this with the Erdős problem website. An AI will generate a proof, and here are 3,000 lines of code that verify the proof. Then people got other AIs to summarize the proof, and people write their own proofs.

There’s actually post-processing. Once you have one proof, we have a lot of tools now to deconstruct and interpret it. It’s a very nascent area of mathematics, but I’m not as worried about it. Some people are concerned about what happens if the Riemann hypothesis is proven with a completely incomprehensible proof. I think once you have the artifact of a proof, we can do a lot of analysis on it.

00:59:20 – We need a semi-formal language for the way that scientists actually talk to each other

Dwarkesh Patel

You posted recently that it would be helpful to have a formal or semi-formal language for mathematical strategies as opposed to just mathematical proofs, which is what Lean specializes in. I would love to learn more about what that would involve or look like.

Terence Tao

We don’t really know. We’ve been very lucky in mathematics that we have worked out the laws of logic and mathematics, but this is a fairly recent accomplishment. It was started by Euclid two millennia ago, but only in the early 20th century did we finally list out the axioms of mathematics, the standard axioms of what we call ZFC, the axioms of first-order logic, and what a proof is. This we’ve managed to automate and have a formal language for.

There could be some way to assess plausibility. You have a conjecture that something is true, you test a few examples, and it works out. How does this increase your confidence that the conjecture is true? We have a few sort of mathematical ways to model this, like Bayesian probability, for example. But you often have to set certain base assumptions, and there’s a lot of subjectivity still in these tasks.

This is more of a wish than a plan to develop these languages, but just seeing how successful having a formal framework in place, like Lean, has made deductive proofs so much easier to automate and train AI on… The bottleneck for using AI to create strategies and make conjectures is we have to rely on human experts and the test of time to validate whether something is plausible or not.

If there was some semi-formal framework where this could be done semi-automatically in a way that isn’t easily hackable... It’s really important with these formal proof assistants that there are no backdoors or exploits you can use to somehow get your certified proof without actually proving it, because reinforcement learning is just so good at finding these backdoors.

If there’s some framework that mimics how scientists talk to each other in a semi-formal way, using data and argument, but also constructing narratives... There’s some subjective aspect of science that we don’t know how to capture in a way that we can insert AI into it in any useful way. This is a future problem. There are research efforts to try to create automated conjectures, and maybe there are ways to benchmark these and simulate this, but it’s all very new science.

Dwarkesh Patel

Can you help me get some intuition? I have two sub-questions. One, it would be very helpful to have a specific example of what something like this would look like, the way scientists communicate that we can’t formalize yet.

Two, it seems almost definitionally paradoxical to say you’re building up some narrative or natural language explanation and then also having something which you could have formalized. I’m sure there’s some intuition behind where that overlap is, and I’d love to understand that better.

Terence Tao

An example of a conjecture: Gauss was interested in the prime numbers and created one of the first mathematical datasets. He just computed the first 100,000 prime numbers or so, hoping to find patterns. He did find a pattern, but maybe not the pattern he was expecting. He found a statistical pattern in the primes that if you count how many primes there are up to 100, 1,000, one million, and so forth, they get sparser and sparser, but the drop-off in the density was inversely proportional to the natural logarithm of the range of numbers.

So he conjectured what we now call the prime number theorem: the number of primes up to X is X divided by the natural log of X. He had no way to prove this. It was data-driven. This was a conjecture. It was revolutionary for its time because it was maybe the first really important conjecture of math that was statistical in nature. Normally you’re talking about a pattern, like maybe the spacing between the primes has a certain regularity. But this didn’t tell you exactly how many primes there were in any given range. It just gave you an approximation that got better and better as you went further and further out.

It started the field of what we call analytic number theory. It was the first in many conjectures like this, many of which got proved, which started consolidating the idea that the prime numbers didn’t really have a pattern, that they behaved like random sets of numbers with a certain density. They had some patterns, like they’re almost all odd. They’re also not actually random, they’re what’s called pseudo-random. There’s no random number generation involved in creating the prime numbers. But over time, it became more and more productive to think of the primes as if they were just generated by some god rolling dice all the time and creating this random set.

This allowed us to make all these other predictions. There’s a still-open conjecture in number theory called the twin prime conjecture, that there should be infinitely many pairs of primes that are twins just two apart, like 11 and 13. We can’t prove that, and there are good reasons why we can’t prove it. But because of this statistical random model of the primes, we are absolutely convinced it’s true. We know that if the primes were generated by flipping coins, we would just—by random chance like infinite monkeys at a typewriter—see twin primes appear over and over again.

We have over time developed this very accurate conceptual model of what the primes should behave like based on statistics and probability. It’s mostly heuristic and non-rigorous, but extremely accurate. The few times when we actually can prove things about the primes, it has matched up with the predictions of what we call the random model of the primes. We have this conjectural concept framework for understanding the primes that everyone believes in. It’s the same reason why we believe the Riemann hypothesis is true, and why we believe that cryptography based on the primes is mathematically secure. It’s all part of this belief.

In fact, one reason why we care about the Riemann hypothesis is that if the Riemann hypothesis failed, if we knew it was false, it would be a serious blow to this model. It would mean there’s a secret pattern to the primes that we were not aware of. I think we would very rapidly abandon any cryptography based on the primes, because if there was one pattern that we didn’t know about, there are probably more, and these patterns can lead to exploits in crypto. It would be a big shock. So we really want to make sure that doesn’t happen.

We’ve been convinced of things like the Riemann hypothesis over time. Some of it is experimental evidence, and some is that the few times we’ve been able to make theoretical results, they’ve always aligned. It is possible that the consensus is wrong and we’ve all just missed something very basic. There have been paradigm shifts in the past in scientific history. But we don’t really have a way of measuring this, partly because we don’t have enough data on how math or science develops. We have one timeline of history, and we have maybe 100 stories of turning points in history.

If we had access to a million alien civilizations, each with a different development of history and science in different orders, then maybe we’d actually have a decent shot at understanding how we measure what progress is and what is a good strategy. We could maybe start formalizing it and actually having a framework. Maybe what we need to do is start creating lots of mini-universes or simulations of AI solving very basic problems in arithmetic or whatever, but coming up with their own strategies for doing these things and having these little laboratories to test. There are people who investigate what’s the smallest neural network that can do 10-digit multiplication and things like that. I think we could learn a lot just from evolving small AIs on simple problems.

01:09:48 – How Terry uses his time

Dwarkesh Patel

You have to learn about new fields not only very rapidly, but deeply enough to contribute to the frontier. So in some sense, you’re also one of the world’s greatest autodidacts. What is your process of learning about a new subfield in math? What does that look like?

Terence Tao

We talked about depth and breadth before. It’s not a purely human-AI distinction. Humans also, I think it was Berlin who split them into hedgehogs and foxes. The hedgehog knows one thing very well, and a fox knows a little bit about everything. I definitely think of myself as a fox. I work with hedgehogs a lot, and sometimes I can be a hedgehog if need be.

I’ve always had a little bit of an obsessive streak. If there’s something I read about which I feel like I have the capability to understand, but I don’t understand why it works and there’s some magic in it… Someone was able to use a type of mathematics I’m not familiar with and get a result I would like to prove. I can’t do it myself, but they could do it by their method, and I want to find out what their trick was. It bugs me that someone else can do something I think I can do, but I can’t. I’ve always had that obsessive, completionist streak. I’ve had to wean myself off computer games because if I start a game, I want to play it to completion, through all the levels. That’s one way I learn new fields.

I collaborate with a lot of people who have taught me other types of mathematics. I just make friends with another mathematician working on another area of mathematics. I find their problems interesting, but they have to teach me some of the basic tricks, what’s known, and what’s not known. I learn a lot from that.

I found that writing about what I’ve learned helps. I have a blog where I sometimes record things I’ve learned. In the past when I was younger, I would learn something, do this cool trick, and say, “Okay, I’m going to remember this.” Then six months later, I’d forgotten it. I remember remembering it, but I can’t reconstruct my arguments. The first few times, it was so frustrating to have understood something and then lost it. I resolved I should always write down anything cool that I’ve learned. That’s part of how this blog came about.

Dwarkesh Patel

How long does it take you to write a blog post?

Terence Tao

It’s something I often do when I don’t want to do other work. There’s some referee report or something that feels slightly unpleasant for me to do at the time. Writing a blog feels creative and fun. It’s something I do for myself.

Depending on the topic, it could be a quick half an hour or several hours. Because it’s something I do voluntarily, time flies when I write these things down, as opposed to doing something I have to do for administrative reasons that is just drudgery. Those are tasks, by the way, that AI is really helping with nowadays.

Dwarkesh Patel

If civilization could from first principles decide how to use Terry Tao’s time, as a limited resource, what is the biggest difference? What if the veil of ignorance got to decide how to use Terry Tao’s time versus what it does now? This podcast wouldn’t be happening.

Terence Tao

As much as I complain about certain tasks that I don’t want to do, but have to do… As you get more senior in academia, you get more and more responsibilities, more committees, and whatever. I have also found that a lot of events I reluctantly went to because I was obliged to for one reason or another… Because it’s outside my comfort zone, it often results in interactions with people I wouldn’t normally talk to, like you for instance. I would learn interesting things and have interesting experiences. I would have opportunities to then network with other people that I never would have before.

So I do believe a lot in serendipity. I do optimize portions of my day where I schedule very carefully. But I am willing to leave some portions just to do something that is not my usual thing. Maybe it’ll be a waste of my time, but maybe I will learn something. More often than not, I get a positive experience that I wouldn’t have planned for.

So I believe a lot in serendipity. Maybe there’s a danger in modern societies, not just with AI, that we’ve become really good at optimizing everything. We’re not optimizing our own optimization. With COVID, for example, we switched a lot to remote meetings, so everything was scheduled. We kept busy in academia. We met almost the same number of people we met in person, but everything had to be planned in advance. What we lost out on was the casual knocking on a hallway door, just meeting someone while getting a coffee. Those serendipitous interactions may not seem optimal, but they are actually really important.

When I was a grad student, I would go to the library to look for a journal article. You had to physically check out the journal and read the article. You could browse through and sometimes the next article was also interesting. Sometimes it wasn’t, but you could accidentally find interesting things. That has basically been lost now. If you want to access an article, you just type it into a search engine or an AI, and you get exactly what you want instantly. But you don’t get the accidental things you might have found if you’d done it more inefficiently.

I spent a year once at the Institute for Advanced Study, which is a great place with no distractions. You’re there just to do research. The first few weeks you’re there, it’s great. You’re getting all these papers written up that you’ve been wanting to do for a long time. You think about problems for blocks of hours at a time. But I find if I stay there for more than several months, I run out of inspiration. I get bored. I surf the internet a lot more.

You actually do need a certain level of distraction in your life. It adds enough randomness and high temperature. I don’t know the optimal way to schedule my life. It just seems to work.

01:17:05 – Human-AI hybrids will dominate math for a lot longer

Dwarkesh Patel

I’m very curious when you expect AIs that can actually do frontier math at least as well as the best human mathematicians.

Terence Tao

In some ways, they’re already doing frontier math that is super intelligent that humans can’t do, but it’s a different frontier from what we’re used to. You could argue that calculators were doing frontier math that humans could not accomplish, but it was number crunching.

Dwarkesh Patel

But replacing Terry Tao completely.

Terence Tao

I mean, what do you want me for?

Dwarkesh Patel

You’ll just go on all the podcasts after.

Terence Tao

It might not be the right question to ask. I think within a decade, a lot of things that math students currently do—what we spend the bulk of our time doing and a lot of stuff we put in our papers today—can be done by AI. But we will find that that actually wasn’t the most important part of what we do.

A hundred years ago, a lot of mathematicians were just solving differential equations. Physicists needed some exact solution to some system, and they hired a mathematician to laboriously go through the calculus and work out the solution to this fluid equation, whatever. A lot of what a 19th-century mathematician would do, you could make a call to Mathematica, Wolfram Alpha, a computer algebra package, or now more recently to an AI, and it would just solve the problem in a few minutes. But we moved on. We worked on different types of problems after that.

Once computers came along—computers used to be human. People used to laboriously create log tables and work out primes as Gauss did, and that has all been outsourced to computers. But we moved on.

In genetics, to sequence the genome of a single organism, that was an entire PhD of a geneticist, carefully separating all the chromosomes and whatever. Now you can just spend $1,000 and send it to a sequencer and get it done. But genetics is not dead as a subject. You move to a different scale. Maybe you study whole ecosystems rather than individuals.

Dwarkesh Patel

I take your point but when is most mathematical progress, or almost all mathematical progress, happening by AI? If you find out this year a Millennium Prize Problem has been solved, you would put 95% odds that an AI did it autonomously. Surely there will be such a year.

Terence Tao

I guess I do believe that hybrid human plus AIs will dominate mathematics for a lot longer. It will depend. It will require some additional breakthroughs beyond what we already have, so it’s going to be stochastic. I think AIs currently are very good at certain things, but really terrible at others. While you can add more and more frameworks on top to reduce the error rates and make them work with each other a bit more, it feels like we don’t have all the ingredients to really have a truly satisfactory replacement for all intellectual tasks.

It is complementary currently. It’s not a replacement. Because current level AIs will accelerate science in so many ways, hopefully new discoveries and new breakthroughs will happen more quickly. It’s also possible that by destroying serendipity we actually inhibit certain types of progress. Anything is possible at this point. I think the world is very, very unpredictable at this point in time.

Dwarkesh Patel

What is your advice to somebody who would consider a career in math or is early in a career in math, especially in light of AI progress? How should they be thinking about their career differently, if at all, as a result of AI progress?

Terence Tao

We live in a time of change. As I said, we live in a particularly unpredictable era. Things that we’ve taken for granted for centuries may not hold anymore. The way we do everything, and not just mathematics, will change. In many ways, I would prefer the much more boring, quiet era where things are much the same as they were 10 years ago, 20 years ago. But I think one just has to embrace that there’s going to be a lot of change. The things that you study, some of them may become obsolete or revolutionized, but some things will be retained.

You always have to keep an eye on opportunities for things that you wouldn’t be able to do before. In math, you previously had to go through years and years of education and be a math PhD before you could contribute to the frontier of math research. But now it’s quite possible at the high school level, or whatever, that you could get involved in a math project and actually make a real contribution because of all these AI tools, Lean, and everything else.

There will be a lot of non-traditional opportunities to learn, so you need a very adaptable mindset. There will be room for pursuing things just for curiosity and for playing around. You still need to get your credentials. For a while it will still be important to go through traditional education and learn math and science the old-fashioned way. But you should also be open to very different ways of doing science, some of which don’t exist yet. It’s a scary time, but also very exciting.

Dwarkesh Patel

That’s a great note to close on. Terence, thanks so much.

Terence Tao

Pleasure.

And walks through the economics of labs, hyperscalers, foundries, and fab equipment manufacturers.

Learned a ton about every single level of the stack. Enjoy!

Watch on YouTube; listen on Apple Podcasts or Spotify.

Sponsors

• Mercury has already saved me a bunch of time this tax season. Last year, I used Mercury to request W-9s from all the contractors I worked with. Then, when it came time to issue 1099s this year, I literally just clicked a button and Mercury sent them out. Learn more at mercury.com.

• Labelbox noticed that even when voice models appear to take interruptions in stride, their performance degrades. To figure out why, they built a new evaluation pipeline called EchoChain. EchoChain diagnoses voice models’ specific failure modes, letting you understand what your model needs to truly handle interruptions. Check it out at labelbox.com/dwarkesh.

• Jane Street is basically a research lab with a trading desk attached – and their infrastructure backs this up. They’ve got tens of thousands of GPUs, hundreds of thousands of CPU cores, and exabytes of storage. This is what it takes to find subtle signals hidden deep within noisy market data. If this sounds interesting, you can explore open positions at janestreet.com/dwarkesh.

Timestamps

(00:00:00) – Why an H100 is worth more today than 3 years ago

(00:24:52) – Nvidia secured TSMC allocation early; Google is getting squeezed

(00:34:34) – ASML will be the #1 constraint for AI compute scaling by 2030

(00:55:47) – Can’t we just use TSMC’s older fabs?

(01:05:37) – When will China outscale the West in semis?

(01:16:01) – The enormous incoming memory crunch

(01:42:34) – Scaling power in the US will not be a problem

(01:54:44) – Space GPUs aren’t happening this decade

(02:14:07) – Why aren’t more hedge funds making the AGI trade?

(02:18:30) – Will TSMC kick Apple out from N2?

(02:24:16) – Robots and Taiwan risk

Transcript

00:00:00 – Why an H100 is worth more today than 3 years ago

Dwarkesh Patel

All right, this is the episode where my roommate teaches me semiconductors.

Dylan Patel

It’s also the send off for this current set.

Dwarkesh Patel

It is. After you use it, I’m like, “I can’t use this again. I gotta get out of here.”

Dylan Patel

No sloppy seconds for Dwarkesh.

Dwarkesh Patel

Dylan is the CEO of SemiAnalysis. Dylan, here’s the burning question I have for you. If you add up the big four—Amazon, Meta, Google, Microsoft—their combined forecasted CapEx this year that you published recently is $600 billion. Given yearly prices of renting that compute, that would be close to 50 gigawatts. Obviously, we’re not putting on 50 gigawatts this year, so presumably that’s paying for compute that is going to be coming online over the coming years. How should we think about the timeline around when that CapEx comes online?

Similar question for the labs. OpenAI just announced they raised $110 billion, and Anthropic just announced they raised $30 billion. If you look at the compute they have coming online this year—you should tell me how much it is, but is it on the order of another four gigawatts total? The cost to rent the compute that OpenAI and Anthropic will have this year to sustain their compute spend is $10 to $13 billion a gigawatt. Those individual raises alone are enough to cover their compute spend for the year. And this is not even including the revenue that they’re going to earn this year.

So help me understand: first, what is the timescale at which the Big Tech CapEx actually comes online? And second, what are the labs raising all this money for if the yearly price of a one-gigawatt data center is $13 billion?

Dylan Patel

So when you talk about the CapEx of these hyperscalers being on the order of $600 billion, and you look across the rest of the supply chain, it gets you to the order of a trillion dollars. A portion of this is immediately for compute going online this year: the chips and the other parts of CapEx that get paid this year. But there’s a lot of setup CapEx as well.

When we’re talking about 20 gigawatts of incremental added capacity this year in America, a portion of this is not spent this year. A portion of that CapEx was actually spent the prior year. When you look at Google having $180 billion, a big chunk of that is spent on turbine deposits for ‘28 and ‘29. A chunk of that is spent on data center construction for ‘27. A chunk of that is spent on power purchasing agreements, down payments, and all these other things they’re doing further out into the future so they can set up this super fast scaling. This applies to all the hyperscalers and other people in the supply chain.

So with roughly 20 gigawatts deployed this year, a big chunk is hyperscalers, and a chunk is not. For all of these companies, their biggest customers are Anthropic and OpenAI. Anthropic and OpenAI are at roughly two to two-and-a-half gigawatts right now, and they’re trying to scale much larger.

If you look at what Anthropic has done over the last few months, with $4 billion or $6 billion in revenue added, we can just draw a straight line and say they’ll add another $6 billion of revenue a month. People would argue that’s bearish, and that they should go faster. What that implies is they’re going to add $60 billion of revenue across the next ten months. At the current gross margins Anthropic had, as last reported by media, that would imply they have roughly $40 billion of compute spend for that inference, for that $60 billion of revenue.

That $40 billion of compute, at roughly $10 billion a gigawatt in rental costs, means they need to add four gigawatts of inference capacity just to grow revenue. That’s assuming their research and development training fleet stays flat. In a sense, Anthropic needs to get to well above five gigawatts by the end of this year. It’s going to be really tough for them to get there, but it’s possible.

Dwarkesh Patel

Can I ask a question about that? If Anthropic was not on track to have five gigawatts by the end of this year, but it needs that to serve both the revenue that’s gone crazier than expected—and maybe it’s going to be even more than that—plus the research and training to make sure its models are good enough for next year: Where is that capacity going to come from?

Dylan Patel

Dario, when he was on your podcast, was very conservative. He said, “I’m not going to go crazy on compute because if my revenue inflects at a different rate, at a different point… I don’t want to go bankrupt. I want to make sure that we’re being responsible with this scaling.” But in reality, he’s screwed the pooch compared to OpenAI, whose approach was, “Let’s just sign these crazy fucking deals.”

OpenAI has got way more access to compute than Anthropic by the end of the year. What does Anthropic have to do to get the compute? They have to go to lower-quality providers that they would not have gone to before. Anthropic historically had the best quality providers, like Google and Amazon, the biggest companies in the world. Now Microsoft is expanding across the supply chain, and they’re going to other newer players.

OpenAI has been a bit more aggressive on going to many players. Yes, they have tons of capacity from Microsoft, Google, and Amazon, but they also have tons with CoreWeave and Oracle. They’ve gone to random companies, or companies one would think are random, like SoftBank Energy, who has never built a data center in their life but is building data centers now for OpenAI. They’ve gone to many others, like NScale, to get capacity.

There’s this conundrum for Anthropic because they were so conservative on compute, because they didn’t want to go crazy. In some sense, a lot of the financial freakouts in the second half of last year were because, “OpenAI signed all these deals but they didn’t have the money to pay for them…” Okay, Oracle’s stock is going to tank, CoreWeave’s stock is going to tank. All these companies’ stocks tanked, and credit markets went crazy because people thought the end buyer couldn’t pay for this. Now it’s like, “Oh wait, they raised a ton of money. Okay, fine, they can pay for it.”

Anthropic was a lot more conservative. They were like, “We’ll sign contracts, but we’ll be principled. We’ll purposely undershoot what we think we can possibly do and be conservative because we don’t want to potentially go bankrupt.”

Dwarkesh Patel

The thing I want to understand is, what does it mean to have to acquire compute in a pinch? Is it that you have to go with neoclouds? Do they have worse compute? In what way is it worse?

Did you have to pay gross margins to a cloud provider that you wouldn’t have otherwise had to pay because they’re coming in at the last minute? Who built the spare capacity such that it’s available for Anthropic and OpenAI to get last minute?

What is the concrete advantage that OpenAI has gotten if they end up at similar compute numbers by 2027? Are they just going to end this year with different gigawatts? If so, how many gigawatts are Anthropic and OpenAI going to have by the end of this year?

Dylan Patel

To acquire excess compute, yes, there is capacity at hyperscalers. Not all contracts for compute are long-term, five-year deals. There’s compute from 2023 or 2024, or H100s from 2025, that were signed at shorter terms. The vast majority of OpenAI’s compute is signed on five-year deals, but there were many other customers that had one-year, two-year, three-year, or six-month deals, on demand.

As these contracts roll off, who is the participant in the market most willing to pay price? In this sense, we’ve seen H100 prices inflect a lot and go up. People are willing to sign long-term deals for above $2 even. I’ve seen deals where certain AI labs—I’m being a little bit vague here for a reason—have signed at as high as $2.40 for two to three years for H100s. If you think about the margin, it costs $1.40 to build Hopper, across five years. Now, two years in, you’re signing deals for two to three years at $2.40? Those margins are way higher.

Now you can crowd out all of these other suppliers, whether Amazon had these, or CoreWeave, or Together AI, or Nebius, or whoever it is. These neoclouds are the firms that had a higher percentage of Hopper in general because they were more aggressive on it. They also tended to sign shorter-term deals, not CoreWeave but the others. So if I want Hopper, there is some capacity out there.

Also, while most of the capacity at an Oracle or a CoreWeave is signed for a long-term deal in terms of Blackwell, anything that’s going online this quarter is already sold. In some cases, they’re not even hitting all the numbers they promised they would sell because there are some data center delays, not just those two, but Nebius, Microsoft, Amazon, and Google. But there are a lot of neoclouds, as well as some of the hyperscalers, who have capacity they’re building that they haven’t sold yet, or capacity they were going to allocate to some internal use that is not necessarily super AGI-focused, that they may now turn around and sell.

Or in the case of Anthropic, they don’t have to have all the compute directly. Amazon can have the compute and serve Bedrock, or Google can have the compute and serve Vertex, or Microsoft can have the compute and serve Foundry, and then do a revenue share with Anthropic, or vice versa.

Dwarkesh Patel

Basically, you’re saying Anthropic is having to pay either this 50% markup in the sense of the revenue share, or in the sense of last-minute spot compute that they wouldn’t have otherwise had to pay had they bought the compute early.

Dylan Patel

Right, there’s a trade-off there. But at the same time, for a solid four months, everyone was saying to OpenAI, “We’re not going to sign deals with you.” That sounds crazy, but it was because, “you don’t have the money.” Now everyone’s saying, “OpenAI, we believed you the whole time. We can sign any deal because you’ve raised all this money.” Anthropic is constrained in that sense. There are not that many incremental buyers of compute yet, because Anthropic hit the capability tier first where their revenue is mooning.

Dwarkesh Patel

That’s interesting. Otherwise you might think having the best model is an extremely depreciating asset, because three months later you don’t have the best model. But the reason it’s important is that you can sign these deals, lock in the compute in advance, and get better prices.

Maybe this is an obvious point. But at least until recently, people had made this huge point about the depreciation cycle of a GPU. The bears, the Michael Burrys or whoever, have said, “Look, people are saying four or five years for these GPUs. Maybe it’s because the technology is improving so fast, but it in fact makes sense to have two-year depreciation cycles for these GPUs,” which increases the reported amortized CapEx in a given year and makes it financially less lucrative to build all these clouds.

But in fact you’re pointing out that maybe the depreciation cycle is even longer than five years. If we’re using Hoppers—especially if AI really takes off and in 2030 we’re saying, “We have to get the seven-nanometer fabs up, we have to go back and turn on the A100s again”—then the depreciation cycle is actually incredibly long. I feel like that’s an interesting financial implication of what you’re saying.

Dylan Patel

There’s a few strings to pull on there. One is, what happens to depreciation of GPUs? I guess I didn’t answer your prior question, which is that I think Anthropic will be able to get to five gigawatts-ish, maybe a little bit more by the end of the year through themselves as well as their product being served through Bedrock, Vertex, or Foundry. I think they’ll be able to get to five or six gigawatts, which is way above their initial plans. OpenAI will be roughly the same, actually a little bit higher based on our numbers.

But anyway, the depreciation cycle of a GPU. Michael Burry was saying it’s three years or less. That’s sort of his argument. There are two lenses to look at this. Mechanically, there’s a TCO model, total cost of ownership of a GPU, where we project pricing out for GPUs and build up the total cost of a cluster. There are a number of costs: your data center cost, your networking cost, your smart hands and people in the data center swapping stuff out. There’s your spare parts, your actual chip cost, your server cost. All these various costs get lumped together. There’s some depreciation cycles on it, certain credit costs on it.

You build up to, “Hey, an H100 costs $1.40/hour to deploy at volume across five years if your depreciation is five years.” If you sign a deal at $2/hour for those five years, your gross margin is roughly 35%. It’s a little bit above that. If you sign it for $1.90, it’s 35% roughly. Then you assume at that fifth year, the GPU falls off a bus and is dead.

In some cases, the argument people are making is if you didn’t sign a long-term deal, because every two years NVIDIA is tripling or quadrupling the performance while only 2X-ing or 50% increasing the price… Then the price of an H100… Sure maybe the value in the market was $2 at 35% gross margins in 2024, but in 2026, when Blackwell is in super high volume and deploying millions a year, you’re actually now worth $1/hour. And when Rubin in ‘27 is in super high volume—even though it starts shipping this year, it’s super high volume next year—doing millions of chips a year deployed into clouds, you’ve got another 3X in performance, another 50% or 2X in price, then the Hopper is only worth $0.70/hour. So the price of a GPU would continue to fall. That’s one lens.

The other lens is, what is the utility you get out of the chip? If you could build infinite Rubin or infinite of the newest chip, then yes, that’s exactly what would happen. The price of a Hopper would fall at a spot or short-term contract rate as the new chips come out and the price per performance goes up. But because you are so limited on semiconductors and deployment timelines, what actually prices these chips is not the comparative thing I can buy today, but rather what is the value I can derive out of this chip today.

In that sense, let’s take GPT-5.4. GPT-5.4 is both way cheaper to run than GPT-4 and has fewer active parameters. It’s much smaller, in that sense of active parameter, because it’s a sparser MoE versus GPT-4 being a coarser MoE. There’s also been so many other advancements in training, RL, model architecture, and data qualities that have made GPT-5.4 way better than GPT-4. And it’s cheaper to serve. When you look at an H100, it can serve more tokens per GPU of 5.4 than if you had ran GPT-4 on it. So it’s producing more tokens of a model that is of higher quality.

What is the maximum TAM for GPT-4 tokens? Maybe it was a few billion dollars, maybe it was tens of billions of dollars. Adoption takes time. For GPT-5.4, that number is probably north of a hundred billion. But there’s an adoption lag, there’s competition, and there’s the constant improvements that everyone else is having. If improvements stopped here, the value of an H100 is now predicated on the value that GPT-5.4 can get out of it instead of the value that GPT-4 can get out of it. These labs are in a competitive environment, so their margins can’t go to infinity. You sort of have this dynamic that is quite interesting in that an H100 is worth more today than it was three years ago.

Dwarkesh Patel

That’s crazy. It’s also interesting from the perspective of just taking that forward. If we had actual AGI models developed, if we had a genuine human on a server… These are such hand wave-y numbers about how many flops the brain can do. But on a flop basis, an H100 is estimated to do 1e15, which is how much some people estimate the human brain does in flops. Obviously, in terms of memory, the human brain has way more. An H100 is 80 gigabytes, and the brain might have petabytes.

Dylan Patel

Oh, yeah, you’ve got petabytes? Name a petabyte of ones and zeros, bro. Name me a string.

Dwarkesh Patel

Well, this is actually the point.

Dylan Patel

No, we’ve just got the best sparse attention techniques ever.

Dwarkesh Patel

Genuinely though. In the amount of information that is compressed, it might be petabytes. The brain is an extremely sparse MoE. But anyways, imagine a human knowledge worker can produce six figures a year of value. If an H100 can produce something close to that, if we had actual humans on a server, the value of an H100 is such that it can repay itself in the course of a couple of months.

So when I interviewed Dario, the point I was trying to make is not that I think the singularity is two years away and therefore Dario desperately needs to buy more compute, although the revenue is certainly there that he needs to buy more compute. The point I was trying to make is that given what Dario seems to be saying—given his statements that we’re two years away from a data center of geniuses, and certainly not more than five years away, and a data center of geniuses should be earning trillions upon trillions of dollars of revenue—it just does not make sense why he keeps making these statements about being more conservative on compute or, to your point, being less aggressive than OpenAI on compute.

I guess that point got lost because then people were roasting me, saying, “Oh, this podcaster is trying to convince this multi-hundred billion dollar company CEO to YOLO it, bro.” I was just trying to say that internally, his statements are inconsistent. Anyway, it’s good to iron it out.

Dylan Patel

I think going back to the earlier view that if the models are so powerful, the value of a GPU goes up over time, right now only OpenAI and Anthropic have that viewpoint. But as we approach further out, everyone is going to be able to see that value skyrocket per GPU. So in that sense, you should commit now to compute.

Interestingly, in Anthropic fashion, there’s a bit of a meme that they have commitment issues and are sort of polyamorous. Not Dario, but this is a bit of a meme.

Dwarkesh Patel

Explains everything. By the way, there’s this interesting economic effect called Alchian-Allen, which is the idea that if you increase the fixed cost of different goods, one of which is higher quality and one which is lower quality, that will make people choose the higher quality good, on the margin.

To give a specific example, suppose the better-tasting apple costs two dollars and the shittier apple costs one dollar. Now suppose you put an import tariff on them. Now it’s $3 versus $2 for a great apple versus a medium apple.

Dylan Patel

Is that because they both increased by a dollar, or should it be a 50% increase?

Dwarkesh Patel

No, because they both increased by $1. The whole effect is that if there’s a fixed cost that is applied to both. Then the price difference between them, the ratio, changes. Previously, the more expensive one was 2X more expensive. Now it’s just 1.5X more expensive.

So I wonder if applied to AI that would mean that, if GPUs are going to get more expensive, there will be a fixed cost increase in the price of compute. As a result, that will push people to be willing to pay higher margins for slightly better models. Because the calculus is, I’m going to be paying all this money for the compute anyway. I might as well just pay slightly more to make sure it’s the very best model rather than a model that’s slightly worse.

Dylan Patel

So the Hopper went from $2 to $3. If a Hopper can make a million tokens of Opus and it can make two million tokens of Sonnet, the price differential between Opus and Sonnet has decreased because the price of the GPU has increased by a dollar from $2 to $3.

Interesting. I think that makes a ton of sense. We just see all of the volumes are on the best models today, all the revenue is on the best models today. In a compute-limited world, two things happen. One, companies that don’t have commitment issues and have these five-year contracts for compute have locked in a humongous margin advantage. They’ve locked in compute for five years at the price it transacted at two, three, or five years ago.

Whereas if you’re three years into that five-year contract and someone else’s two-year or three-year contract rolled off, and now they’re trying to buy that at modern pricing, when it’s priced to the value of models, the price is going to be up a lot more. So the person who committed early has better margins in general. The percentage of the market that is in long-term contracts is much larger than the percentage of the market in short-term contracts that can be this flex capacity you add at the last second.

At the same time, where does the margin go? Because models get more valuable, how much can the cloud players flex their pricing? If you look at CoreWeave, their average term duration is over three years right now. For ninety-eight percent plus of their compute, it’s over three years. They end up with this conundrum where they can’t actually flex price. But every year they’re adding incrementally way more capacity than they had previously.

This year alone, Meta’s adding as much capacity as they had in their entire fleet of compute and data centers for all purposes for serving WhatsApp, Instagram, and Facebook in 2022, and doing AI. They’re adding that alone this year.

In the same sense, you talk about Meta doing that, CoreWeave, Google, and Amazon, all these companies are adding insane amounts of compute year on year. That new compute gets transacted at the new price. In a sense, yes, you’ve locked in, as long as we’re in a takeoff. “Oh, OpenAI went from six hundred megawatts to two gigawatts last year, and from two gigawatts to six plus this year, and six to twelve next year.” The incremental added compute is where all the cost is, not the prior long-term contracts.

Then who holds the cards is the infra providers for charging margin. Now the cloud players, the neoclouds, or the hyperscalers can charge the margin. They can to some extent, but then as you go upstream to who has access to all the memory and logic capacity, it’s Nvidia for the most part. They’ve signed a lot of long-term contracts. They’ve got ninety billion dollars of long-term contracts today, and they’re negotiating three-year deals today with the memory vendors.

You’ve got Amazon and Google through Broadcom, Amazon directly, and AMD. These companies hold all the cards because they’ve secured the capacity. TSMC is not raising prices, but memory vendors are, to some extent, raising a lot of price. They’re going to double or triple price again, but then they’re also signing these long-term deals.

Who is able to accrue all the margin dollars is potentially the cloud, potentially the chip vendors, and the memory vendors, until TSMC or ASML break out and say, “No, we’re going to charge a lot more.” But at the same time, do the model vendors get to charge crazy margins? At least this year, we’re going to see margins for the model vendors go up a lot. Because they’re so capacity constrained, they have to destroy demand. There’s no way Anthropic can continue at the current pace without destroying demand.

00:24:52 – Nvidia secured TSMC allocation early; Google is getting squeezed

Dwarkesh Patel 1:20:33

Let’s get into logic and memory. How specifically has Nvidia been able to lock up so much of both? I think according to your numbers, by ‘27, Nvidia is going to have +70% of N3 wafer capacity, or around that area. I forget what the numbers were for memory at SK Hynix and Samsung and so forth.

Think about how the neocloud business works and how Nvidia works with that, or how the RL environment business works and how Anthropic works with that. In both those cases, Nvidia is purposely trying to fracture the complementary industry to make sure that they have as much leverage as possible. They’re giving allocation to random neoclouds to make sure that there’s not one person that has all the compute.

Similarly, Anthropic or OpenAI, when they’re working with the data providers, they say, “No, we’re going to just seed a huge industry of these things so that we’re not locked into any one supplier for data environments.”

And I wonder why on the 3 nm process—that’s going to be Trainium 3, that’s going to be TPU v7, other accelerators potentially—why is TSMC just giving it all up to Nvidia rather than trying to fracture the market?

Dylan Patel

There are a couple points here. On 3 nm, if we go back to last year, the vast majority of 3 nm was Apple. Apple is being moved to 2 nm. Memory prices are going up, so Apple’s volumes may go down. As memory prices go up, either they cut margin or they move on. There’s some time lag because they have long-term contracts, but Apple likely reduces demand or moves to 2 nm faster, where 2 nm is only capable of mobile chips today. In the future, AI chips will move there. So Apple has that.

Apple is also talking to third-party vendors because they’re getting squeezed out of TSMC a little bit. TSMC’s margins on high-performance computing—HPC, AI chips, et cetera—are higher than they are for mobile, because they have a bigger advantage in HPC than they do in mobile.

When you look at TSMC’s running calculus here, they’re actually providing really good allocations to companies that are doing CPUs. When you think about Amazon having Trainium and Graviton, both of those are on 3 nm, Graviton being their CPU, Trainium being their AI chip. TSMC is much more excited to give allocation to Graviton than they are to Trainium because they view the CPU business as more stable, long-term growth.

As a company that is conservative and doesn’t want to ride cycles of growth too hard, you actually want to allocate to the market that is more stable with a lower growth rate first before you allocate all the incremental capacity to the fast growth rate market. That is the case generally. Same for AMD. The allocations they get on their CPUs, TSMC is much more excited about those than they are for GPUs. Likewise for Amazon.

Nvidia is a bit unique because yes, they have CPUs, they make switches, they make networking, NVLink, InfiniBand, Ethernet, NICs. By and large, most of these things will be on 3 nm by the end of this year with the Rubin launch and all the chips in that family, the GPU being the most important one. Yet Nvidia is getting the majority of supply.

Part of this is because you look at the market and TSMC and others forecast market demand in many ways, but it’s also the market signal. The market signaled, “Hey, we need this much capacity next year. We need this much. We’ll sign non-cancelable, non-returnable. We may even pay deposits.” Nvidia just did it way earlier than Google or Amazon. In some cases, Google and Amazon had stumbling blocks. One of the chips got delayed slightly by a couple quarters. Trainium and all these sorts of things happened.

In that case, there was a huge sort of, “Well, these guys are delaying, but Nvidia is wanting more, more, more, more. And we are checking with the rest of the supply chain, is there enough capacity?” They’re going to all the PCB vendors and saying, “Is there enough PCB?” Victory Giant is one of the largest suppliers of PCBs to Nvidia, and they’re a Chinese company. All the PCBs come from China, or many of them. They’re like, “Do you have enough PCB capacity? Great. Hey memory vendors, who has all the memory capacity? Okay, Nvidia does. Great.”

When you look at who is AGI-pilled enough to buy compute on long timelines at levels that seem ridiculous to people who aren’t AGI-pilled—but nonetheless, they’re willing to pay a pretty good margin and sign it now because they view in the future that ratio is screwed up—the same thing happens with the supply chain for semiconductors. I don’t think Nvidia is quite AGI-pilled. Jensen doesn’t believe software is going to be fully automated and all these things.

Dwarkesh Patel

Accelerated computing, not AI chips, right?

Dylan Patel

It’s AI chips.

Dwarkesh Patel

But that’s what he calls it, right?

Dylan Patel

Yeah. I think it’s a broader term, AI is within that, but also physics modeling and simulations.

Dwarkesh Patel

But it’s like he’s not embracing the main use case.

Dylan Patel

I think he’s embracing it, but I just don’t think he’s AGI-pilled like Dario or Sam. But he’s still way, way more AGI-pilled than Google was in Q3 of last year, or Amazon was in Q3 of last year, and he saw way more demand.

The reason is pretty simple. You can see all the data center construction. He’s like, “Okay, I want to have this market share.” We have all the data centers tracked, and there’s a lot of data centers that could be one or the other. To some extent, Google and Amazon, Google especially, even though their TPU is just better for them to deploy, they have to deploy a crap load of GPUs because they don’t have enough TPUs to fill up their data centers. They can’t get them fabbed.

Dwarkesh Patel

I have a question about that. Google sold a million, was it the v7s?

Dylan Patel

Yes.

Dwarkesh Patel

—the Ironwoods to Anthropic, and you’re saying the big bottleneck right now, this year or next year, I guess going forward forever now, is going to be the logic and memory, the stuff it takes to build these chips. Google has DeepMind, the third prominent AI lab. If this is the big bottleneck, why would they sell it rather than just giving it to DeepMind?

Dylan Patel

This is again a problem of… DeepMind people were like, “This is insane. Why did we do this?” But Google Cloud people and Google executives saw a different thought process.

You and I know the compute team at Anthropic. Both of the main people came from Google. They saw this dislocation, they negotiated a deal, and they were able to get access to this compute before Google realized. The chain of events, at least from our data that we found, was in early Q3, over the course of six weeks, we saw capacity on TPUs go up by a significant amount. It went up multiple times in those six weeks.

There were multiple requests. Google even had to go to TSMC and explain to them why they needed this increase in capacity because it was so sudden. A lot of that capacity increase was for selling to Anthropic. Because Anthropic saw it before Google.

And then Google had Nano Banana and Gemini 3 which caused their user metrics to skyrocket. Then leadership at Google was like, “Oh.” Then they started making the statement that we have to double compute every six months, or whatever the exact number was.

They really woke up a lot more, and then they went to TSMC and said, “We want more. We want more.” TSMC replied, “Sorry guys, we’re sold out. We can maybe get 5-10% more for 2026, but really we’re going to work on 2027.”

There was this information asymmetry among the labs, in my mind. I don’t know exactly. It’s the narrative I’ve spun myself from seeing all the data in the supply chain on wafer orders and what’s going on with the data centers that Anthropic and Fluidstack signed.

It’s pretty clear to me that Google screwed up. You can see this from Google’s Gemini ARR. They had next to nothing in Q1 to Q3—in Q3 a little bit once they started inflecting. But in Q4 they reached $5 billion in revenue on an ARR basis. It’s clear Google didn’t see revenue skyrocket initially. In a sense, Anthropic had a little bit of commitment issues before their ARR exploded, even though they had far more information asymmetry and saw what was coming down the pipe. Google is going to be more conservative than Anthropic and Google had even less ARR. So they were just not willing to do it, and then they realized they should do it.

Since then, Google has gotten absurdly AGI-pilled in terms of what they’re doing. They bought an energy company. They’re putting deposits down for turbines. They’re buying a ridiculous percentage of powered land. They’re going to utilities and negotiating long-term agreements. They’re doing this on the data center and power side very aggressively. I think Google woke up towards the end of last year, but it took them some time.

Dwarkesh Patel

How many gigawatts do you think Google will have by the end of next year?

Dylan Patel

Buy my data.

Dwarkesh Patel

You charge for that kind of information.

Dylan Patel

Yes, yes.

00:34:34 – ASML will be the #1 constraint for AI compute scaling by 2030

I feel like every year the bottleneck for what is preventing us from scaling AI compute keeps changing. A couple years ago it was CoWoS. Last year it was power. You’ll tell me what the bottleneck is this year.

But I want to understand five years out, what will be the thing that is constraining us from deploying the singularity?

Dylan Patel

The biggest bottleneck is compute. For that, the longest lead time supply chains are not power or data centers. They’re actually the semiconductor supply chains themselves. It switches back from power and data centers as a major bottleneck to chips.

In the chip supply chain, there’s a number of different bottlenecks. There’s memory, logic wafers from TSMC, and the fabs themselves. Construction of the fabs takes two to three years, versus a data center which takes less than a year. We’ve seen Amazon build data centers in as fast as eight months. There’s a big difference in lead times because of the complexity of building the fab that actually makes the chips. The tools also have really long lead times.

The bottlenecks, as we’ve scaled, have shifted based on what the supply chain is currently not able to do. It was CoWoS, power, and data centers, but those were all shorter lead time items. CoWoS is a much simpler process of packaging chips together. Power and data centers are ultimately way simpler than the actual manufacturing of the chips. There’s been some sliding of capacity across mobile or PC to data center chips, which has been somewhat fungible.

Whereas CoWoS, power, and data centers have had to start anew as supply chains. But now there’s no more capacity for the mobile and PC industries—which used to be the majority of the semiconductor industry—to shift over to AI. Nvidia is now the largest customer at TSMC and SK Hynix, the largest memory manufacturer. It’s sort of impossible for the sliding of resources away from the common person’s PCs and smartphones to shift any more towards the AI chips. So now the question is how do we scale AI chip production? That’s the biggest bottleneck as we go to 2030.

Dwarkesh Patel

It would be very interesting if there’s an absolute gigawatt ceiling that you can project out to 2030 based just on “We can’t produce more than this many EUV machines.”

Dylan Patel

To scale compute further, there are different bottlenecks this year and next year, but ultimately by 2028 or 2029, the bottleneck falls to the lowest rung on the supply chain, which is ASML. ASML makes the world’s most complicated machine: an EUV tool. The selling price for those is $300-400 million. Currently, they can make about 70. Next year, they’ll get to 80. Even under very aggressive supply chain expansion, they only get to a little bit over 100 by the end of the decade.

What does that mean? They can make a hundred of these tools by the end of the decade, and 70 right now. How does that actually translate to AI compute? We see all these numbers from Sam Altman and many others across the supply chain: gigawatts, gigawatts, gigawatts. How many gigawatts are we adding? We see Elon saying a hundred gigawatts in space.

Dwarkesh Patel

A year.

Dylan Patel

A year. The problem with any of these numbers, or the challenge to these numbers, is actually not the power or the data center. We can dive into that, but it’s manufacturing the chips.

Take a gigawatt of Nvidia’s Rubin chips. Rubin is announced at GTC, I believe the week this podcast goes live. To make a gigawatt worth of data center capacity of Nvidia’s latest chip that they’re releasing towards the end of this year, you need a few different wafer technologies. You need about 55,000 wafers of 3 nm. You need about 6,000 wafers of 5 nm, and then you need about 170,000 wafers of DRAM memory.

Across these three different buckets, each requires different amounts of EUV. When you manufacture a wafer, there are thousands and thousands of process steps where you’re depositing material and removing them. But the key critical step—which at least in advanced logic is 30% of the cost of the chip—is something that doesn’t actually put anything on the wafer. You take the wafer, you deposit photoresist, which is a chemical that chemically changes when you expose it to light. Then you stick it into the EUV tool, which shines light at it in a certain way. It patterns it. There’s what’s called a mask, which is effectively a stencil for the design.

When you look at a leading-edge 3 nm wafer, it has 70 or so masks, 70 or so layers of lithography, but 20 of them are the most advanced EUV. If you need 55,000 wafers for a gigawatt, and you do 20 EUV passes per wafer, you can do the math. That’s 1.1 million passes of EUV for a single gigawatt. It’s pretty simple. Once you add the rest of the stuff, it ends up being 2 million, across 5 nm and all the memory. You’re at roughly 2 million EUV passes for a single gigawatt.

These tools are very complicated. When you think about what it’s doing across a wafer, it’s taking the wafer and scanning and stepping across. It does this dozens of times across the whole wafer. When you’re talking about how many EUV passes, that’s the entire wafer being exposed at a certain rate.

An EUV tool can do roughly 75 wafers per hour, and the tool is up roughly 90% of the time. In the end, you need about three and a half EUV tools to do the 2 million EUV wafer passes for the gigawatt. So three and a half EUV tools satisfies a gigawatt.

It’s funny to think about the numbers. What does a gigawatt cost? It costs roughly $50 billion. Whereas what do three and a half EUV tools cost? That’s $1.2 billion. It’s actually quite a lower number, which is interesting to think about. Fifty gigawatts of economic CapEx in the data center, and what gets built on top of that in terms of tokens is even larger. It might be $100 billion worth of AI value into the supply chain, held up by this $1.2 billion worth of tooling that simply cannot expand its supply chain quickly.

Dwarkesh Patel

You wrote an article recently saying over the last three years, TSMC has done $100 billion of CapEx. So it’s $30/$30/$40 billion. A small fraction of that is being used by Nvidia for the 3 nm, or previously 4 nm, that it’s using for its chips. What were its earnings last quarter? It was $40 billion. So $40 billion times four is $160 billion. Nvidia alone is turning some small fraction of $100 billion in CapEx, which is going to be depreciated over many years and not just this one year, into $160 billion in a single year.

That gets even more intense when you go down the supply chain to ASML, which is taking a billion dollars’ worth of machines to produce a gigawatt. Of course, those machines last for more than a year so it’s doing more than that.

Now I want to understand, how many such machines will there be by 2030, if you include not just the ones that are sold that year, but have been compiling over the previous years? What does that imply? Sam Altman says he wants to do a gigawatt a week in 2030. When you add up those numbers, is it compatible with that?

Dylan Patel

That’s completely compatible, if you think about it. TSMC and the entire ecosystem have something like 250 to 300 EUV tools already. Then you stack on 70 this year, 80 next year, growing to 100 by 2030. You’re at 700 EUV tools by the end of the decade. 700 EUV tools, at three and a half tools per gigawatt—assuming it’s all allocated to AI, which it’s not—gets you to 200 gigawatts worth of AI chips for the data centers to deploy.

Sam wants 52 gigawatts a year. He’s only taking 25% share then. Obviously, there’s some share given to mobile and PC, assuming we’re even allowed to have consumer goods still and we don’t get priced out of them. But roughly, he’s saying 25% market share of the total chips fabbed. That’s very reasonable given that this year alone, I think he’s going to have access to 25% of the Blackwell GPUs that are deployed. It’s not that crazy.

Dwarkesh Patel

When did ASML start shipping EUV tools, when 7 nm started? I don’t know when that was exactly. You’re saying in 2030, they’re going to be using machines that initially were shipped in 2020. So for ten years, you’re using the same most important machine in this most technologically advanced industry in the world? I find that surprising.

Dylan Patel

ASML’s been shipping EUV tools now for roughly a decade, but it only entered mass volume production around 2020. The tool’s not the same. Back then, the tools were even lower throughput. There are various specifications around them called overlay. I was mentioning you’re stacking layers on top of each other. You’ll do some EUV, you’ll do a bunch of different process steps—depositing stuff, etching stuff, cleaning the wafer—dozens of those steps before you do another EUV layer.

There’s a spec called overlay, which is: you did all this work, you drew these lines on the wafer, now I want to draw these dots. Let’s say I want to draw these dots to connect these lines of metal to holes, and then the next layer up is another set of lines going perpendicular, so now you’re connecting wires going perpendicular to each other. You have to be able to land them on top of each other. It’s called overlay.

Overlay is a spec that’s been improved rapidly by ASML. Wafer throughput has been improved rapidly by ASML. The price of the tool has gone up, but not as much as the capabilities of the tool. Initially, the EUV tools were $150 million. Over time, they’re now $400 million as I look out to 2028. But the capabilities of the tools have more than doubled as well, especially on throughput and overlay accuracy, which is the ability to accurately align the subsequent passes on top of each other even though you do tons of steps between.

ASML is improving super rapidly. It’s also noteworthy to say that ASML is maybe one of the most generous companies in the world. They have this linchpin thing. No one has anything competitive. Maybe China will have some EUV by the end of the decade, but no one else has anything even close to EUV, and yet they haven’t taken price and margins up like crazy. You go ask some other folks that we talk to all the time, like Leopold, and they’re like, “Let’s have the price go up.” Because they can. The margin is there. You can take the margin. Nvidia takes the margin. Memory players are taking the margin. But ASML has never raised the price more than they’ve increased the capability of the tool.

In a sense, they’ve always provided net benefit to their customers. It’s not that the tool is stagnant, it’s just that these tools are old. Yes, you can upgrade them some, and the new tools are coming. For simplicity’s sake, we’re ignoring the advances in overlay or throughput per tool for this podcast.

Dwarkesh Patel

You say we’re producing 60 of these machines this year and then 70, 80 over subsequent years. What would happen if ASML just decided to double its CapEx or triple its CapEx? What is preventing them from producing more than 100 in 2030? Why are you so confident that even five years out, you can be relatively sure what their production will be?

Dylan Patel

I think there are a couple factors here. ASML has not decided to just go YOLO, let’s expand capacity as fast as possible. In general, the semiconductor supply chain has not. It’s lived through the booms and busts, and we can talk a bit more about it. Basically some players have recently woken up, but in general no one really sees demand for 200 gigawatts a year of AI chips, or trillions of dollars of spend a year in the semiconductor supply chain. They’re not AI-pilled. They’re not AGI-pilled.

Dwarkesh Patel

We’re going to get to a trillion dollars this year.

Dylan Patel

Yeah, I feel you, but I’m saying no one really understands this in the supply chain. Constantly, we’re told our numbers are way too high, and then when they’re right, they’re like, “Oh, yeah, but your next year’s numbers are still too high.”

ASML’s tool has four major components. It has the source, which is made by Cymer in San Diego. It has the reticle stage, which is made in Wilmington, Connecticut. It has the wafer stage. It has the optics, the lenses and such. Those last two are made in Europe.

When you look at each of these four, they’re tremendously complex supply chains that, (A) they have not tried to expand massively, and (B) when they try to expand them, the time lag is quite long. Again, this is the most complicated machine that humans make, period, at any sort of volume.

Let’s talk about the source specifically. What does the source do? It drops these tin droplets. It hits it three subsequent times with a laser perfectly. The first one hits this tin droplet, it expands out. It hits it again, so it expands out to this perfect shape, and then it blasts it at super high power. The tin droplets get excited enough that they release EUV light, 13.5 nanometer, and then it’s in this thing that is collecting all the light and directing it into the lens stack.

Then you have the lens stack, which is Carl Zeiss, as you mentioned, and some other folks, but Zeiss being the most important part of it. They also have not tried to expand production capacity because they don’t see... They’re like, “We’re growing a lot because of AI. We’re growing from 60 to 100.” It’s like, “No, no, no. We need to go to a couple hundred, but it’s fine. Whatever.”

Each of these tools has, I think, 18 of these lenses, effectively. They are multilayer mirrors, which are perfect layers of molybdenum and ruthenium, if I recall correctly, stacked on top of each other in many layers, and then the light bounces off of it perfectly. When we think about a lens, it’s in a shape, and it focuses the light. This is like a mirror that’s also a lens, so it’s pretty complicated. Any defect in these super thinly deposited stacks will mess it up. Any curvature issues will mess it up.

There are a lot of challenges with scaling the production. It’s quite artisanal in this sense because you’re not making tens of thousands of these a year, you’re making hundreds, you’re making thousands. 60 tools a year, 18 of these per tool, you’re still in the hundreds, of tools, or you’re at the thousand number roughly for these lenses and projection optics.

Then you step forward to the reticle stage, which is also something really crazy. This thing moves at, I want to say, nine Gs. It will shift nine Gs because as you step across a wafer, the tool will go... The wafer stage is complementary. It’s the wafer part. You line these two things up. You’re taking all the light through the lenses that’s focused, and here’s the reticle, here’s the wafer. The reticle’s moving one direction, the wafer’s moving the other direction as it scans a 26x33 millimeter section of the wafer, and then it stops. It shifts over to another part of the wafer and does it again. It does that in just seconds. Each of them is moving at nine Gs in opposite directions.

Each of these things is a wonder and marvel of chemistry, fabrication, mechanical engineering, and optical engineering, because you have to align all these things and make sure they’re perfect. All of these things have crazy amounts of metrology because you have to perfectly test everything. If anything is messed up, the yield goes to zero, because this is such a finely tuned system.

By the way, it’s so large that you’re building it in the factory in Eindhoven, Netherlands, and they’re deconstructing it and shipping it on many planes to the customer site, and then you’re reassembling it there and testing it again. That process takes many, many months.

There are so many steps in the supply chain, whether it’s Zeiss making their lenses and projection optics or Cymer, which is an ASML-owned company, making the EUV source. Each of these has its own complex supply chain. ASML has commented that their supply chain has over ten thousand people in it.

Dwarkesh Patel

Like individual suppliers?

Dylan Patel

Yes. It might not be directly. It might be through Zeiss having so many suppliers and XYZ company having so many suppliers.

If you just think about it, you’re talking about two physically moving objects that are the size of a wafer, and it has to be accurate to the level of single-digit nanometers or even smaller because the entire system, the overlay, the layer-to-layer overlay variation, has to be on the order of 3 nanometers. If the overlay is 3 nms, that means each individual part, the accuracy of its physical movement has to be even less than that. It has to be sub-one nanometer in most cases, because the error of these things stack up. There’s no way to just snap your fingers and increase production.

Things as simple as power. The US going from zero percent power growth to two percent power growth, even though China’s already at thirty, was so hard for America to do. And that’s a really simple supply chain with very few people in it who make difficult things. There are probably 100,000 electricians and people who work in the electricity supply chain, or more, in the US?

When you look at ASML, they employ so few people. Carl Zeiss probably employs less than a thousand people working on this, and all of those people are super, super specialized. You can’t just train random people up for this in the snap of a finger. You can’t just get your entire supply chain to get galvanized.

Nvidia’s had to do a lot to get the entire supply chain to even deliver the capacity they’re going to make this year. When you go talk to Anthropic, they’re like, “We’re short of TPUs, we’re short of training, and we’re short of GPUs.” When you go talk to OpenAI, they’re like, “We’re short of these things.”

OpenAI and Anthropic know they need X. Nvidia is not quite as AGI-pilled. They’re building X - 1. You go down the supply chain, everyone’s doing X - 1. In some cases, they’re doing X ÷ 2, because they’re not AGI-pilled.

You end up with this time lag for the whip to react. The AI-pilledness and the desire to increase production takes so long. Once they finally understand that they need to increase production rapidly… They think they understand. They think AI means we have to go from 60 to 100, in addition to the tools getting better and faster, the source getting higher power from 500 watts to 1,000, and all these other aspects of the supply chain advancing technically and increasing production. They think they’re actually increasing production a lot.

But if you flow through the numbers… What does Elon want? He wants 100 gigawatts a year in space by 2028 or 2029. Sam Altman wants 52 gigawatts a year by the end of the decade. Anthropic probably needs the same, and Google needs that. You go across the supply chain, and it’s like, wait, no, the supply chain can’t possibly build enough capacity for everyone to get what they want on the side of compute.

00:55:47 – Can’t we just use TSMC’s older fabs?

Dwarkesh Patel

I feel like in the data center supply chain for the last few years, people have been making arguments like, “We are bottlenecked by this specific thing, therefore AI compute can’t scale more than X.” But as you’ve written about, if the grid is a bottleneck, then we just do behind the meter on the site, we do gas turbines, et cetera. If that doesn’t work, there are all these other alternatives that people fall back on.

I want to ask whether we can imagine a similar thing happening in the semiconductor supply chain. If EUV becomes a bottleneck, what if we just went back to 7 nm and did what China is doing currently, producing 7 nm chips with multi-patterning with DUV machines? If you look at a 7 nm chip like the A100, there’s been a lot of progress obviously from the A100 to the B100 or B200.

How much of that progress is just numerics? If you just hold FP16 constant from A100 to B100. The B100 is a little over one petaflop, and the A100 is like 300 teraflops.

Dylan Patel

Yeah, 312.

Dwarkesh Patel

Holding numerics constant, you have a 3x improvement from A100 to B100. Some of that is the process improvement, some of that is just the accelerator design improving, which we could replicate again in the future.

It seems there’s actually a very small effect from the process improving from 7nm to 4 nm. I don’t know the numbers offhand but let’s say there’s 150k wafers per month of 3 nm and eventually similar amounts for 2 nm. But then there’s a similar amount for 7 nm.

If you have all those old wafers and there’s maybe a 50% haircut because the bits per wafer area are 50% less or something, it doesn’t seem that bad to just bring on 7 nm wafers if that gives you another fifty or hundred gigawatts. Tell me why that’s naive.

Dylan Patel

We potentially do go crazy enough that this happens because we just need incremental compute, and the compute is worth the higher cost and power of these chips. But it’s also unlikely to a large extent because some of these are not fair comparisons.

For example, from A100, which is 312 teraflops, to Blackwell, which is 1,000 or 2,000 FP16, and then Rubin is 5,000 or so FP16… It’s not a fair comparison because these chips have vastly different design targets. With A100, Nvidia optimized for FP16 and BF16 numerics. When you look at Hopper, they didn’t care as much about that; they cared about FP8. When you look at Rubin, they don’t care about FP16 and BF16 so much, they care mostly about FP4 and FP6. Numerics are what they’ve designed their chip for.

Let’s say we make a new chip design on 7 nm, optimized for the numerics of the modern day. The performance difference is still going to be much larger than the FLOPS difference you mentioned. Often it’s easy to boil things down to FLOPS per watt or FLOPS per dollar, but that’s not a fair comparison.

Let’s look at Kimi K2.5 and DeepSeek. When you look at those two models and their performance on Hopper versus Blackwell on very optimized software, you get vastly different performance. Most of this is not attributed to FLOPS or numerics, because those models are actually eight-bit. So it’s not like Blackwells and Hopper are both optimized for eight-bit, and Blackwell is not really taking advantage of its four-bit there. The performance gulf is actually much larger.

Sure it’s one thing to shrink process technology and make the transistor smaller so each chip has X number of FLOPS, but you forget the big gating factor. These models don’t run on a single chip. They run on hundreds of chips at a time. If you look at DeepSeek’s production deployment, which is well over a year old now, they were running on 160 GPUs. That’s what they serve production traffic on. They split the model across 160 GPUs.

Every time you cross the barrier from one chip to another, there is an efficiency loss. You have to transmit over high-speed electrical SerDes, which brings a latency cost and a power cost. There are all these dynamics that hurt. As you shrink and shrink the process node, you’ve increased the amount of compute in a single chip. Now in-chip movement of data is at least tens of terabytes a second, if not hundreds of terabytes a second. Whereas between chips, you’re on the order of a terabyte a second.

Then you have this movement of data between chips that are super close to each other physically. You can only put so many chips close to each other physically, so you have to put chips in different racks. The movement of data between racks is on the order of hundreds of gigabits a second, 400 gig or 800 gig a second, so roughly 100 gigabytes a second.

So you have this huge ladder: on-chip communication is super fast, within the rack is an order of magnitude slower, and outside the rack is an order of magnitude lower than that. As you break the bounds of chips, you end up with a performance loss.

The reason I explain this is because when you look at Hopper versus Blackwell, even if both are using a rack’s worth of chips, Hopper is significantly slower. The amount of performance you have leveraged to the task within each domain—tens of terabytes a second of communication between these processing elements versus terabytes a second between these processing elements—is much, much higher and therefore the performance is much higher. When you look at inference at 100 tokens a second for DeepSeek and Kimi K2.5, the performance difference between Hopper and Blackwell is on the order of 20x.

It’s not 2x or 3x like the FLOPS performance difference indicates, even though those are on the same process node. There are just differences in networking technologies and what they’ve worked on. You can translate some of these back, but when you look at what they’re doing on 3 nm with Rubin, some of those things are simply not possible to do all the way back on A100, even if you make a new chip for 7 nm.

There are certain architectural improvements you can port and certain ones you cannot. The performance difference is not just going to be the difference in FLOPS. It’s in some senses cumulative between the difference in FLOPS per chip, networking speed between chips, how many FLOPS are on a chip versus a system, and memory bandwidth on a single chip versus an entire system. All of these things compound.

Dwarkesh Patel

Can I ask you a very naive question? The B200 now has two dies on a single chip, so you can get that bandwidth without having to go through NVLink or InfiniBand. Next year, Rubin Ultra will have four dies on one chip. What is preventing us from just doing that with an older… How many dies could you have on a single chip and still get these tens of terabytes a second?

Dylan Patel

Even within Blackwell, there are differences in performance when you’re communicating on the chip versus across the chips. Those bounds are obviously much smaller than when you’re going out of the entire chip. When you scale the number of chips up, there is some performance loss. It’s not perfect, but it is way better than different entire packages.

How large can advanced packaging scale? The way Nvidia is doing it is CoWoS. Google, Broadcom, MediaTek, and Amazon’s Trainium are all doing CoWoS. But actually you can go look back at what Tesla did with Dojo, which they cancelled and restarted. Dojo was a chip that was the size of an entire wafer. They had 25 chips on it. There were some tradeoffs. They couldn’t put HBM on it. But the positive side was that they had 25 chips on it. To date, it is still probably the best chip for running convolutional neural networks. It’s just not great at transformers because the shape of the chip, the memory, the arithmetic, and all these various specifications are just not well-suited for transformers. They’re well-suited for CNNs.

Dojo chips were optimized around that, and they made a bigger package. But as you make packages bigger and bigger, you have other constraints: networking speed, memory bandwidth, and cooling capabilities. All of these things start to rear their heads. It’s not simple. But yes, you will see a trend line of more chips on the package, and yes, you’re going to be able to do that on 7 nm.

In fact, that’s what Huawei did with their Ascend 910C or D. They initially put one, and then they did two. They’re focusing on scaling the packaging up because that is an area where they can advance faster than process technology where they can’t shrink. But at the end of the day, that’s something you can do on the leading-edge chips too. Anything you do on 7 nm, you can also probably do on 3 nm in terms of packaging.

01:05:37 – When will China outscale the West in semis?

Dwarkesh Patel

If we end up in this world in 2030 where the West has the most advanced process technology but has not ramped it up as much, whereas China… I don’t know if you think by 2030 they would have EUV and 2 nm or whatever. But they are semiconductor-pilled and they are producing in mass quantity.

Basically, I’m wondering what the year is where there’s a crossover, where our advantage in process technology has faded enough, and their advantage in scale has increased enough. And also, if their advantage in having one country with the entire supply chain indigenized—rather than having random suppliers in Germany and the Netherlands—would mean that China would be ahead in its ability to produce mass flops.

Dylan Patel

To date, China still does not have an entirely indigenized semiconductor supply chain.

Dwarkesh Patel

But would they in 2030?

Dylan Patel

By 2030, it’s possible that they do. But to date, all of China’s 7 nm and 14 nm capacity uses ASML DUV tools. The amount that they can import from ASML is large. But the vast majority of ASML’s revenue, especially on EUV all of it, is outside of China. The scale advantage is still in the favor of the West plus Taiwan, Japan, and Korea, et cetera.

Dwarkesh Patel

But they’re trying to make their own DUV and EUV tools, right?

Dylan Patel

They’re trying to do all these things. The question is how fast can they advance and scale up production as well as quality. To date, we haven’t seen that. Now I’m quite bullish that they’re going to be able to do these things over the next five to ten years. They will really scale up production and kick it into high gear. They have more engineers working on it and more desire to throw capital at the problem.

Dwarkesh Patel

So by 2030, will they have fully indigenized DUV?

Dylan Patel

I think for sure. DUV, yes.

Dwarkesh Patel

And fully indigenized EUV by 2030?

Dylan Patel

I think they’ll have working tools. I don’t think that they’ll be able to manufacture a bunch yet. There’s having it work, and then there’s production hell. ASML had EUV working in the early 2010s at some capacity. The tools were not accurate enough. They were not scaled for high-volume manufacturing or reliable enough. They had to ramp production, and that all took time.

Production hell takes time. That’s why it took another five to seven years to get EUV into mass production at a fab rather than just working in the lab.

Dwarkesh Patel

How many DUV tools do you think they’ll be able to manufacture in 2030?

Dylan Patel

ASML?

Dwarkesh Patel

No, China.

Dylan Patel

That’s a great question. It’s a bit of a challenge to look into this supply chain especially. We try really hard. In some instances, they’re buying stuff from Japanese vendors. If they want a fully indigenized supply chain, they need to not buy these lenses, projection optics, or stages from Japanese vendors. They need to build it internally.

It’s really tough to say where they’ll be able to get to. I honestly think it’s a shot in the dark. But it’s probably not unlikely that they’ll be able to do on the order of 100 DUV tools a year, whereas ASML is currently doing hundreds of DUV tools a year.

No company has a process node where they make a million wafers a month. Elon says he wants to do it and China is obviously going to do it. TSMC is trying to do that. The memory makers may get to a million wafers a month as well, but not in a single fab.

It’s mind-boggling to think of that scale, and challenging to see the supply chain galvanized for that. I don’t want to doubt China’s capability to scale.

Dwarkesh Patel

I guess this is an interesting question. I think at some point SemiAnalysis will do the deep dive on this. By when would indigenized Chinese production be bigger than the rest of the West combined. And put in the input of your model of when they’ll have DUV machines and EUV machines at scale?

Because there’s this question around if you have long timelines on AI—by long meaning 2035, which is not that long in the grand scheme of things—should you expect a world where China is dominating in semiconductors? It doesn’t get asked enough because if you’re in San Francisco, we’re thinking on timescales of weeks. If you’re outside of San Francisco, you’re not thinking about AGI at all.

What if we have AGI? What if you have this transformational thing that is commanding tens or hundreds of trillions of dollars of economic growth and token output, but it happens in 2035? What does that imply for the West versus China? SemiAnalysis has got to write the definitive model on this.

Dylan Patel

It’s really challenging when you move timescales out that far. What we tend to focus on is tracking every data center, every fab, and all the tools. We track where they’re going, but the time lags for these things are relatively short. We can only make reasonably accurate estimates for data center capacity based on land purchasing, permits, and turbine purchasing. We know where all these things are going, that’s the data we sell.

As you go out to 2035, things are just so radically different. Your error bars get so large it’s hard to make an estimate. But at the end of the day, if takeoff or timelines are slow enough, I don’t see why China wouldn’t be able to catch up drastically. In some sense, we’ve got this valley where, three to six months ago, or maybe even now, Chinese models are as competitive as they’ve ever been. I think Opus 4.6 and GPT 5.4 have really pulled away and made the gap a little bit bigger, but I’m sure some new Chinese models will come out.

As we move from selling tokens where they provide the entire reasoning chain, to selling automated white-collar work—an automated software engineer, you send them the request, they give you the result back, and there’s a bunch of thinking on the back end that they don’t show you—the ability to distill out of American models into Chinese models will be harder.

Second, look at the scale of the compute the labs have. OpenAI exited the year with roughly two gigawatts last year. Anthropic will get to two-plus gigawatts this year. By the end of next year, they’ll both be at ten gigawatts of capacity. China is not scaling their AI lab compute nearly as fast. At some point, when you can’t distill the learnings from these labs into the Chinese models, plus with this compute race that OpenAI, Anthropic, Google, and Meta are all racing on, they end up getting to a point where the model performance should start to diverge more.

Then look at all this CapEx being spent on data centers. Amazon is spending $200 billion, Google $180 billion. All these companies are spending hundreds of billions of dollars on CapEx. There’s nearly a trillion dollars of CapEx being invested in data centers in America this year, roughly. What’s the return on invested capital here? You and I would think the return on invested capital for data center CapEx is very high.

If we look at Anthropic’s revenues, in January they added $4 billion. In February, which was a shorter month, they added $6 billion. We’ll see what they can do in March and April, given that compute constraints are what’s bottlenecking their growth. The reliability of Claude is quite low because they’re so compute constrained. But if this continues, then the ROIC on these data centers is super high.

At some point, the US economy starts growing faster and faster over this year and next year because of all this CapEx, all the revenue these models are generating, and the downstream supply chain. China doesn’t have that yet. They have not built the scale of infrastructure to invest in models, get to the capabilities, and then deploy these models at such scale.

When you look at Anthropic, they’re at $20 billion ARR. The margins are sub-50 percent, at least as last reported by The Information. So that’s $13 or $14 billion of compute that it’s running on rental cost-wise, which is actually $50 billion worth of CapEx that someone laid out for Anthropic to generate their current revenue.

China has just not done this. If and when Anthropic 10Xs revenue again—and I think our answer would be when, not if—China doesn’t have the compute to deploy at that scale. So there is some sense that we’re in a fast takeoff. It’s not like we’re talking about a Dyson sphere by X date, it’s more like the revenue is compounding at such a rate that it does affect economic growth. The resources these labs are gathering are growing so fast. China hasn’t done that yet, so in that case, the US and the West are actually diverging.

The flip side is that these infrastructure investments have middling returns. Maybe they’re not as good as hoped. Maybe Google is wrong for wanting to take free cash flow to zero and spend $300 billion on CapEx next year. Maybe they’re just wrong and people on Wall Street who are bearish and people who don’t understand AI are correct. In that case, the US is building all this capacity but doesn’t get great returns. Meanwhile, China is able to build a fully vertical, indigenized supply chain, instead of the US/Japan/Korea/Taiwan/SE Asia/Europe countries together building this less vertical supply chain. In a sense, at some point China is able to scale past us if AI takes longer to get to certain capability levels than the vast majority of your guests on this podcast believe.

Dwarkesh Patel

It’s fast timelines, the US wins; long timelines, China wins.

Dylan Patel

Yeah but I don’t know what fast timelines means. I don’t think you have to believe in AGI to have the timelines where the US wins.

01:16:01 – The enormous incoming memory crunch

Dwarkesh Patel

Let’s go back to memory. I think people on Wall Street and people in the industry are understanding how big this is, but maybe generally people don’t understand what a big deal it is. So we’ve got this memory crunch, as you were talking about.

And earlier I was asking about, oh, could we solve for the EUV tool shortage by going back to seven nanometers? So let me ask a similar question about memory. HBM is made of DRAM, but has three to four times fewer bits per wafer area than the DRAM it’s made out of.

Is it possible that accelerators in the future could just use commodity DRAM and not HBM, so we can get much more capacity out of the DRAM we have? The reason I think this might be possible is, if we’re going to have agents that are just going off and doing work, and it’s not a synchronous chatbot application, then you don’t necessarily need extremely fast latency.

Maybe you can have lower bandwidth, because the reason you stack DRAM into HBM is for higher bandwidth. Is it possible to go to HBM accelerators and basically have the opposite of Claude Code Fast, like have Claude Slow?

Dylan Patel

At the end of the day, the incremental purchaser who’s willing to pay the highest price for tokens also ends up being the one that’s less price-sensitive. Compute should be allocated, in a capitalistic society, towards the goods that have the highest value, and the private market determines this by willingness to pay.

To some extent, Anthropic could actually release a slow mode. They could release Claude Slow Mode and increase tokens per dollar by a significant amount. They could probably reduce the price of Opus 4.6 by 4-5x and reduce the speed by maybe just 2x. The curve on inference throughput versus speed is already there just on HBM. And yet they don’t, because no one actually wants to use a slow model.

Furthermore, on these agentic tasks, it’s great that the model can run at a time horizon of hours. But if the model was running slower, those hours would become a day. Vice versa, if the model is running faster, those hours become an hour. No one really wants to move to a day-long wait period, because the highest-value tasks also have some time sensitivity to them.

I struggle to see… Yes, you could use regular DRAM. There are a couple of challenges with this. One of the core constraints of chips is that a chip is a certain size, and all of the I/O escapes on the edges. Often, the left and right of the chip are HBM—so the I/O from the chip to the HBM is on the sides—and then the top and bottom are I/O to other chips.

If you were to change from HBM to DDR, all of a sudden this I/O on the edge would have significantly less bandwidth, but significantly more capacity per chip. But the metric you actually care about is bandwidth per wafer, not bits per wafer.

Dwarkesh Patel

Because the thing that is constraining the FLOPS is just getting in and out the next matrix, and for that you just need more bandwidth.

Dylan Patel

Yeah, getting out the weights and getting in and out the KV cache. In many cases, these GPUs are not running at full memory capacity. It’s obviously a system design thing: model, hardware, and software co-design. You have to figure out how much KV cache you need, how much you keep on the chip, how much you offload to other chips and call when you need it for tool calling, and how many chips you parallelize this on.

Obviously, the search space for this is very broad, which is why we have InferenceX, an open-source model that searches all the optimal points on inference for a variety of different chips and models.

The point is, you’re not always necessarily constrained by memory capacity. You can be constrained by FLOPS, network bandwidth, memory bandwidth, or memory capacity. If you really simplify it down, there are four constraints, and each of these can break out into more.

If you switch to DDR, yes, you produce four times the bits per DRAM wafer, but all of a sudden the constraints shift a lot and your system design shifts. You go slower. Is the market smaller? Maybe. But also, all these FLOPS are wasted because they’re just sitting there waiting for memory. You don’t need all that capacity because you can’t really increase batch size because then the KV cache would take even longer to read.

Dwarkesh Patel

Makes sense. What is the bandwidth difference between HBM and normal DRAM?

Dylan Patel

An HBM4 stack—let’s talk about the stuff that’s in Rubin, because that’s what we’ve been indexing on—is 2048 bits across, connected in an area that’s 13 millimeters wide. It transfers memory at around 10 giga-transfers a second.

So a stack of HBM4 is 2048 bits on an area that’s roughly 11 to 13 millimeters wide. That’s the shoreline you’re taking on the chip. In that shoreline, you have 2048 bits transferring at 10 giga-transfers per second. You multiply those together and divide by eight, bits to a byte, and you’re at roughly 2.5 terabytes a second per HBM stack.

When you look at DDR, in that same area, it’s maybe 64 or 128 bits wide. That DDR5 is transferring at anywhere from 6.4 to maybe 8,000 giga-transfers a second. So your bandwidth is significantly lower. It’s 64 times 8,000 divided by eight, which puts you at 64 gigabytes a second. Even if you take a generous interpretation of 128 times 8 giga-transfers, you’re at 128 gigabytes a second for the same shoreline, versus 2.5 terabytes a second.

There’s an order of magnitude difference in bandwidth per edge area. If your chip is a square, or 26 by 33 millimeters—which is the maximum size for an individual die—you only have so much edge area. On the inside of that chip, you put all your compute. There are things you can do to try and change that, like more SRAM or more caching. But at the end of the day, you’re very constrained by bandwidth.

Dwarkesh Patel

Then there’s the question of where you can destroy demand to free up enough for AI. I guess the picture is especially bad because, as you’re saying, if it takes four times more wafer area to get the same byte, for HBM you have to destroy four times as much consumer demand for laptops and phones to free up one byte for AI.

What does this imply for the next year or two? Sorry for the run-on question, in your newsletter you said 30% of Big Tech’s CapEx in 2026 is going towards memory?

Dylan Patel

Yes.

Dwarkesh Patel

That’s insane, right? Of the $600 billion or whatever, 30% is going just to memory.

Dylan Patel

Yes. Obviously, there’s some level of margin stacking that Nvidia does, so you have to separate that out and apply their margin to the memory and the logic. But at the end of the day, a third of their CapEx is going to memory.

Dwarkesh Patel

That’s crazy. What should we expect over the next year or two as this memory crunch hits?

Dylan Patel

The memory crunch will continue to get harder, and prices will continue to go up. This affects different parts of the market differently. Are people going to hate AI more and more? Yes, because smartphones and PCs are not going to get incrementally better year on year. In fact, they’re going to get incrementally worse.

Dwarkesh Patel

If you look at the bill of materials for an iPhone, what fraction of it is the memory? How much more expensive does an iPhone get if the memory is two times more expensive?

Dylan Patel

I believe an iPhone has 12 gigabytes of memory. Each gig used to cost roughly $3-4, so that’s $50. But now the price of memory has tripled. Let’s say it’s $12 per gig for DDR. Now you’re talking about $150 versus $50.

That’s a $100 increase in cost for Apple. Apple has some margin, they’re not just going to eat the margin. NAND also has the same market dynamics, so in reality, it’s probably a $150 increase on the iPhone. So now that’s a $100 cost increase and that’s just on the DRAM. The NAND also has the same sort of market. So in fact it’s probably a $150 increase on the iPhone. Apple either has to pass that on to the consumer or eat it. I don’t see Apple reducing their margin too much, maybe they eat a little bit. But at the end of the day, that means the end consumer is paying $250 more for an iPhone.

Now that’s just on last year’s pricing versus today’s. There is some lag before Apple feels the heat because they tend to have long-term contracts for memory that last three months to a year. But at the end of the day, Apple gets hit pretty hard by this. They won’t really adjust until the next iPhone release.

But that’s the high end of the market, which is only a few hundred million phones a year. Apple sells two or three hundred million phones annually. The bulk of the market is mid-range and low-end. It used to be that 1.4 billion smartphones were sold a year. Now we’re at about 1.1 billion. Our projections are that we might drop to 800 million this year, and down to 500 or 600 million next year.

We look at data points out of China from some of our analysts in Asia, Singapore, Hong Kong, and Taiwan. They’ve been tracking this, and they see Xiaomi and Oppo cutting low-end and mid-range smartphone volumes by half.

Yes, it’s only a $150 BOM increase on a $1,000 iPhone where Apple has some larger margin. But for smaller phones, the percentage of the BOM that goes to memory and storage is much larger. And the margins are lower, so there’s less capacity to even eat the margins. And they have also generally tended not to do long-term agreements on memory.

Why this is a big deal is that if smartphone volumes halve, that drop will happen in the low and mid-range, not the high end. So it’s not like the bits released are halving. Currently, consumer devices account for more than half of memory demand. Even if you halve smartphone volumes, because of the shape of the halving, the low end gets cut by more than half, while the high end gets cut by less than half, because you and I will still buy the high-end phones that cost north of a thousand dollars. We’ll buy them even if they get a little bit more expensive. And Apple’s volumes will not go down as much as a low-end smartphone provider.

The same applies to PCs. What this does to the market is quite drastic. DRAM gets released and goes to AI chips, who are willing to do longer-term contracts and pay higher margins, because at the end of the day the margin they extract from the end user is much larger.

This probably leads to people hating AI even more. Today, you already see all the memes on PC subreddits and gaming PC Twitter. It’s cat dancing videos saying, “This is why memory prices have doubled and you can’t get a new gaming GPU or desktop.” It’s going to be even worse when memory prices double again, especially DRAM.

Another interesting dynamic is that it’s not just DRAM, it’s also NAND. NAND is also going up in price. Both of these markets have expanded capacity very slowly over the last few years, NAND almost zero. The percentage of NAND that goes to phones and PCs is larger than the percentage of DRAM that goes to phones and PCs.

As you destroy demand, mostly for DRAM purposes, you unlock more NAND that gets allocated and can go to other markets. The price increases of DRAM will be larger than those of NAND because you’ve released more from the consumer, and in fact, you’ve produced more memory for AI.

Dwarkesh Patel

Sorry, maybe you just explained it and I missed it. Is it because SSDs are being used in large quantities for data centers?

Dylan Patel

They are, but not in as large quantities as DRAM.

Dwarkesh Patel

Okay, so they will also increase because they’ll be using some quantity, but there’s not as much of a need as there is for HBM. Makes sense.

One thing I didn’t appreciate until I was reading some of your newsletters is that the same constraints preventing logic scaling over the next few years are quite similar to what’s preventing us from producing more memory wafers. In fact, literally the same exact machine, this EUV tool, is needed for memory. So I guess the question someone could ask right now is, why can’t we just make more memory?

Dylan Patel

The constraints, as I was mentioning earlier, are not necessarily EUV tools today or next year. They become that as we get to the latter part of the decade. Currently, the constraints are more that they physically just haven’t built fabs. Over the last three to four years, these vendors have not built new fabs because memory prices were really low. Their margins were low, and in fact, they were losing money in 2023 on memory. So they decided they weren’t building new fabs. The market slowly recovered over time but never really got amazing until last year.

In 2024, we were banging on the drums that reasoning means long context, which means a large KV cache, which means you need a lot of memory demand. We’ve been talking about that for a year and a half, two years. People who understand AI went really long on memory then. So you’ve seen that dynamic, but now it has finally played out in pricing.

It took so long for what was obvious: long context means the KV cache gets bigger, you need more memory. Half the cost of accelerators is memory. Of course they’re going to start going crazy on it. It took a year for that to actually reflect in memory prices. Once memory prices reflected that, it took another three to six months for the memory vendors to start building fabs. Those fabs take two years to build. So we won’t have really meaningful fabs to even put these tools in until late 2027 or 2028.

Instead, you’ve seen some really crazy stuff to get capacity. Micron bought a fab from a company in Taiwan that makes lagging-edge chips. Hynix and Samsung are doing some pretty crazy things to try and expand capacity at their existing fabs, which also have large knock-on effects in the economy.

So why can’t we build more capacity? There’s nowhere to put the tools. It’s not just EUV; there are other tools involved in DRAM and logic. In logic, for N3, about 28% of the cost of the final wafer is EUV. When you look at DRAM, it’s in the teens. It’s going up, but it’s a much smaller percentage of the cost. These other tools are also bottlenecks, although their supply chains are not as complex as ASML’s.

You see Applied Materials, Lam Research, and all these other companies expanding capacity a lot as well. But you don’t have anywhere to put the tool, because the most complex buildings people make are fabs, and fabs take two years to build.

Dwarkesh Patel

I interviewed Elon recently, and his whole plan is that they’re going to build this TeraFab and they’re going to build the clean rooms. I won’t even ask you about the dirty rooms thing, but let’s say they build the clean rooms.

I have a couple of questions. One, do you think this is the kind of thing that Elon Co. could build much faster than people conventionally build it? This is not about building the end tools. This is just about building the facility itself. How complicated is it to just build the clean room extremely fast? Is this something that Elon, with his “move fast” approach, could do much faster if that’s what we’re bottlenecked on this year or next year? Two, does that even matter if, in two years, your view is that we’re not bottlenecked on clean room space, but on the tooling?

Dylan Patel

As with any complex supply chain, it takes time, and constraints shift over time. Even if something is no longer a constraint, that doesn’t mean that market no longer has margin. For example, energy will not be a big bottleneck a couple of years from now, but that doesn’t mean energy isn’t growing super fast and there’s no margin there. It’s just not the key bottleneck. In the space of fabs, clean rooms are the biggest bottleneck this year and next year. As we get to 2028, 2029, 2030, there will still be constraints there.

The thing about Elon is he has a tremendous capability to garner physical resources and really smart people to build things. The way he recruits amazing people is by trying to build the craziest stuff. In the case of AI, that hasn’t really worked because everyone’s trying to build AGI. Everyone is very ambitious. But in the case of going to Mars, making rockets that land themselves, fully autonomous electric cars, or humanoid robots, these are methods of recruiting the people who think that’s the most important problem in the world to work on that problem, because he’s the only one trying really hard.

In the case of semiconductors, he stated he wants to make a fab that’s a million wafers per month. No one has a fab that big. It’s possible that he’s able to recruit a lot of really awesome people and get them on this crazy task of building a million wafers a month. Step one is to build the clean room, and that I think he probably can do. His mindset around deleting things, that it can be dirty, it’s fine, is probably not right. Actually I think it’s 100% not right. You need the fab to be very clean. All of the air in the fab gets replaced every three seconds, it’s that fast. There have to be so few particles.

But I think he can build the clean room. It’ll take a year or two. Initially, it won’t be super fast, but over time, he’ll get faster at it. The really complex part is actually developing a process technology and building wafers. I don’t think he can develop that quickly. That has a lot of built-up knowledge. The most complicated integration of very expensive tools and supply chains is done by TSMC, Intel, or Samsung. These two other companies aren’t even that great at it, and they’re tremendously complex.

Dwarkesh Patel

How surprised would you be if in 2030 there just happened to be some total disruption where we’re not using EUV? What if we’re using something that has much better effects, is much simpler to produce, and can be produced in much bigger quantities? I’m sure as an industry insider that sounds like a totally naive question, but do you see what I’m asking? What probability should we put on something coming totally out of left field to make all of this irrelevant?

Dylan Patel

Something that’s very simple and easy to scale, I assign a very, very low probability. There are a number of companies working on effectively particle accelerators or synchrotrons that generate light that’s either 13.5 nanometer, like EUV, or an even narrower wavelength, like X-ray at 7 nanometers, to then use in lithography tools. But those things are massive particle accelerators generating this light. It’s a very complicated thing to build.

There are a couple of companies and I think that could be a big disruption to the industry beyond EUV. But I don’t think we’re going to magically build something new that is direct write and super simple, and can be manufactured at huge volumes, although there are some attempts to do things like this.

Dwarkesh Patel

I ask because if you think about Elon’s companies in the past, rocketry was this thing that was thought to be—and is—incredibly complicated.

Dylan Patel

Look, I’m just a naive yapper compared to Elon. What have I built? So maybe it’s possible.

Dwarkesh Patel

In order to build more memory in the future, could we build 3D DRAM the way we do 3D NAND and then go back to DUV?

Dylan Patel

That is the hope currently. Everyone’s roadmap for 3D DRAM is that you’ll still use EUV because you want to have that tighter overlay. When you’re doing these subsequent processing steps, everything is vertically stacked and you have more layers on top of each other. You want the pitches to be tighter. So generally, people are still trying to do it with EUV.

But what 3D would do is change the calculation of how many bits a single EUV pass can make. That number would go up drastically if you go to 3D DRAM. That is the hope. Right now, everyone’s roadmap goes from the current 6F cell, to a 4F cell, and then finally 3D DRAM by the end of the decade or early next decade. There’s still a lot of R&D, manufacturing, and integration to be done. I wouldn’t call that out of the cards. I think it’s very likely going to happen.

It’s also going to require a huge retooling of fabs. The breakdown of tools in a fab will be very different. The lithography tool is actually the only thing that isn’t that different. But the number of them relative to different types of chemical vapor deposition, atomic layer deposition, dry etch, or different kinds of etch chambers with different chemistries… You have all these different tools for different process nodes. You can’t just convert a logic fab to a DRAM fab, or vice versa, or a NAND fab to a DRAM fab, in a short amount of time.

In the same way, existing DRAM fabs require a lot of retooling just to go from 1-alpha to 1-beta to 1-gamma process nodes, because they have to add DUV and change the chemistry stacks for when you’re using EUV in terms of deposition and etch. And the EUV tool has to be there. Furthermore, when you change to 3D DRAM, there’s going to be an even larger shift, so a lot of retooling of these fabs needs to happen.

That would be a big disruption. That would make EUV demand generally lower. But as we’ve seen across time, lithography demand as a percentage of wafer cost has trended up. Around the 2014 era, it was 17% of the wafer cost, and it’s gone to 30% over the last fifteen years. For DRAM, it was in the low to mid-teens, and now it’s trended toward the high teens. Before we get to 3D DRAM, it’ll likely cross into the 20% range. But then, if we get to 3D DRAM, the total end wafer cost as a percentage of EUV tanks again.

Dwarkesh Patel

I guess you care less about the percent of cost and more about how much it bottlenecks production.

Dylan Patel

Right, but the percentage of cost—

Dwarkesh Patel

It’s a proxy, yeah. If you’re Jensen or Sam Altman, or whoever stands to gain a lot from scaling up AI compute, there are these stories that they’d go to TSMC and say, “Why can’t we access Y and Z?” But I think the point you’re making is that it doesn’t really matter what TSMC does in some sense. In fact, even if you have Intel and Samsung building more foundries, in the long run, you’re going to be bottlenecked by ASML and other tool and material makers.

First, is that a correct interpretation? Second, should Silicon Valley people be going to the Netherlands right now to try to pitch ASML to make more tools so that in 2030 they can have more AI compute?

Dylan Patel

It’s a funny dynamic we saw in 2023, 2024, and 2025. People who saw the energy bottleneck before others asymmetrically went to Siemens, Mitsubishi, and of course GE Vernova, and bought up turbine capacity. Now they’re able to charge excess amounts for deploying these turbines in places because of energy.

In the same sense, this could be done for EUV, except ASML is not just going to trust any random bozo who wants to buy EUV tools. These turbines are much cheaper than EUV tools, and there’s many more of them produced. Especially once you get to industrial gas turbines, not just combined-cycle but the cheaper, smaller, less efficient ones, people put down deposits for these.

Someone could do this. Someone should go to the Netherlands and be like, “I’ll pay you a billion dollars. You give me the right to purchase ten EUV tools two years from now, and I’m first in line.” Then over those two years, you go around and wait for everyone to realize, “Oh crap, I don’t have enough EUV tools,” and you try to sell your option at some premium. All you’re effectively doing is saying, “ASML, you’re dumb. You weren’t making enough margin on these. I’m going to make a margin.” The question is, will ASML even agree to this? I don’t think so.

Dwarkesh Patel

There’s a world where they at least get the demand signal from that to increase production.

Dylan Patel

Potentially. I agree.

Dwarkesh Patel

But it sounds like you’re saying they couldn’t even increase production if they wanted to, given the supply chain.

Dylan Patel

Right. But that’s exactly the market in which… If they can’t increase production, just like TSMC cannot increase production that fast, and yet demand is mooning, then the obvious solution is to arbitrage this. You and I know demand is way higher than they’re projecting and their capability to build.

You arbitrage this by locking up the capacity, doing a forward contract, and then trying to sell it at a later date once other people realize everything is fucked and we don’t have enough capacity. Then you’ll have this insane margin that ASML and TSMC should have been charging. But the thing is, I don’t know if ASML and TSMC will ever agree to this.

01:42:34 – Scaling power in the US will not be a problem

Dwarkesh Patel

Let me ask you about power now. It sounds like you think power can be arbitrarily scaled.

Dylan Patel

Not arbitrarily, but yes.

Dwarkesh Patel

But beyond these numbers. If I’m remembering correctly, your blog post on how AI labs are increasing power implied that GE Vernova, Mitsubishi, and Siemens could produce 60 gigawatts a year in gas turbines. Then there are these other sources, but they’re less significant than the turbines.

Only a fraction of that goes to AI, I assume. If in 2030 we have enough logic and memory to do 200 gigawatts a year, do you just think that these things are on a path to ramp up to more than 200 gigawatts a year, or what do you see?

Dylan Patel

Right now we’re at 20 or 30. This is critical IT capacity, by the way, which is an important thing to mention. When I’m talking about these gigawatts, I’m talking about critical IT capacity. Server plugged in, that’s how much power it pulls. But there are losses along the chain. There is loss on transmission, conversion, cooling, et cetera. So you should gross this factor up from 20 gigawatts for this year, or 200 gigawatts by the end of the decade, to some number 20-30% higher.

Then you have capacity factors. Turbines don’t run at 100 percent. If you look at PJM, which I think is the largest grid in America—covering the Midwest and some of the Northeast area—in their models they want to have roughly 20 percent excess capacity. Within that 20 percent excess capacity, they’re running all the turbines at 90% because they are derated some for reliability, maintenance, and so on. In reality, the nameplate capacity for energy is always way higher than the actual end critical IT capacity because of all these factors.

But it’s not just turbines. If you were just making power from turbines, that’s simple, boring, and easy. Humans and capitalism are far more effective. The whole point of that blog was that, yes, there are only three people making combined-cycle gas turbines, but there’s so much more we can do. We can do aeroderivatives. We can take airplane engines and turn them into turbines. There are even new entrants in the market, like Boom Supersonic trying to do that and working with Crusoe.  Also there’s all the other ones like that already exist in the market.

There are also medium-speed reciprocating engines: engines that spin in circles, like a diesel engine. There are ten people who make engines that way. I’m from Georgia, and people used to be like, “Oh man, you got a Cummins engine in there,” regarding RAM trucks. Automobile manufacturing is going down, so these companies all have capacity and could scale and convert that for data center power. You stick all these reciprocating engines in. It’s not as clean as combined-cycle, but maybe you can convert them from diesel to gas if you want.

What about ship engines? All of these engines for massive cargo ships are great. Nebius is doing that for a Microsoft data center in New Jersey. They’re running ship engines to generate power. Bloom Energy is doing fuel cells. We’ve been very positive on them for a year and a half now because they have such a capability to increase their production. Their payback period for a production increase is very fast, even if the cost is a little bit higher than combined-cycle, which is the best for cost and efficiency.

Then there’s solar plus battery, which can come online as those cost curves continue to come down. There’s wind, where you might only expect 15 percent of the maximum power because things oscillate, but you add batteries. There are all these things.

The other thing is that the grid is scaled so we don’t cut off power at peak usage on the hottest day of the summer. But in reality, that’s a load spike that is 10-20% higher than the average. If you just put enough utility-scale batteries, or peaker plants that only run a small portion of the year—and those could be gas, industrial gas turbines, combined-cycle, batteries, or any of the other sources I mentioned—then all of a sudden you’ve unlocked 20% of the US grid for data centers. Most of the time that capacity is sitting idle. It’s really only there for that peak, which is just a few hours over a few days of the year. If you have enough capacity to absorb that peak load, then all of the sudden you’ve transferred it all.

Today, data centers are only 3-4% of the power of the US grid, and by 2028 they’ll be 10%. But if you can unlock 20% of the US grid like this, it’s not that crazy. The US grid is terawatt-level, not hundreds-of-gigawatts-level. So we can add a lot more energy.

I’m not saying it’s easy. These things are going to be hard. There’s a lot of hard engineering, risks people have to take, and new technologies people have to use. But Elon was the first to do this behind-the-meter gas, and since then we’ve seen an explosion of different things people are doing to get power.  They’re not easy, but people are gonna be able to do them. The supply chains are just way simpler than chips.

Dwarkesh Patel

Interesting. He made the point during the interview that for the specific blade for the specific turbine he was looking at, the lead times go out beyond 2030. Your point is that—

Dylan Patel

That’s great. There are so many other ways to make energy. Just be inefficient. It’s fine.

Dwarkesh Patel

Right now, combined-cycle gas turbines have CapEx of $1,500 per kilowatt. Are you saying it would make sense to have either technologies that are much more expensive than that, or other things are getting cheap enough to make it competitive?

Dylan Patel

Exactly. It can be as high as $3,500 per kilowatt. It could be twice as much as the cost of combined-cycle, and the total cost of the GPU on a TCO basis has only gone up a few cents per hour.

Because we’ve been talking about Hopper pricing, $1.40, let’s say the power price doubles. The Hopper that was $1.40 is now $1.50 in cost. I don’t care, because the models are improving so fast that the marginal utility of them is worth way more than that ten-cent increase in energy.

Dwarkesh Patel

So you’re saying 20 percent of the grid—the grid is about one terawatt—can just come online from utility-scale batteries, increasing what you’d be comfortable putting on the grid.

Dylan Patel

The regulatory mechanism there is not easy, by the way.

Dwarkesh Patel