I put out a blog prize to answer a couple of big questions I have about AI. The goal is really to find someone to hire as coresearcher. I have more questions of this variety, but I omitted them from that post’s list, because they don’t make it easy to judge submission quality. So I thought I’d post them here:

• 5 hyperscalers own 70+% of global AI compute, and much of that is actually reserved for the 3 member set of OpenAI/Ant/GDM. How worried should we be that AI use cases which are not building up to the singularity and the robot factories - aka normal people being more empowered, understanding the world better, being entertained, etc, is not the highest ROI activity for compute in the world. And given how valuable compute will be (whose opportunity cost increases in tandem with the quality of the AI models that run on it), will normal people basically get priced out of the benefits of AI? If we should be worried about this, how concretely should some kind of universal basic income/compute redistribution work? If not worried, what is the frame of this question missing?

• Data is arguably the main way that AI models have been getting better over the last few years. But I remain confused about what concretely these improvements have consisted of. To ask some sharper questions:

  • Clearly Anthropic (and now also OpenAI and GDM) have cracked something about making competent long horizon coding agents. What is it? Is it just stacking up more and more RL coding environments? Or is there something more particular behind this breakthrough?

  • Are models even getting more sample efficient (aka they learn more from each training sample) or have we just changed/expanded/improved the data input? The reason this question is important is because it tells us how fast deep learning progress will be in domains that actually do require sample efficiency (for example, robotics).

    • Models are very sample efficient in context, and the information in context can be used much more flexibly. But the attention “fast” weights consume a huge amount of memory in order to accommodate this faster learning. Why is there this memory/sample efficiency tradeoff?

      • If you look at the size of the KV cache for Llama 3 70B, it’s 320 KB / token. If you just divide the number of bits it takes to store Llama 3 weights by the number of tokens it was pre-trained on, then you get 0.075 bits / token. So there’s a 35 million fold difference in the amount of information per bit you’re storing.

• Let’s put frontier lab compute into 3 buckets: pretraining, RL generation, and inference. RL generation and inference look like very similar workloads. The big difference, of course, is that the model learns as a result of RL generation, but it doesn’t (at least currently) from inference. At the same time, the model actually does useful work during inference, but not during RL generation. Many people have pointed out it’s really weird that there’s a distinction between training and inference, and that in the limit it shouldn’t exist. How practically will these two workloads be merged? At a high level, one can imagine hiring an AI instance for a month-long work trial, getting it to do actual useful work for you during that time, and then sending a report card back to the model company. In fact, in a few years, maybe the only way that AI can continue to make progress is through this kind of on-the-job learning, because models will already have saturated anything that can be learned from contrived shorter-horizon RL environments.

• Does something Y2Key happen when most of the tokens on the internet (and presumably the ones future models will be trained on) are generated by other AIs? Has the relative value of pre-2023 internet datasets increased in any noticeable way?

• I wrote this in my continual learning blog post last June. Is this correct? Why might there not be a winner take all dynamic from continual learning?

  • “Even if there isn’t a software only singularity (with models rapidly building smarter and smarter successor systems), we might still see something that looks like a broadly deployed intelligence explosion. AIs will be getting broadly deployed through the economy, doing different jobs and learning while doing them in the way humans can. But unlike humans, these models can amalgamate their learnings across all their copies. So one AI is basically learning how to do every single job in the world. An AI that is capable of online learning might functionally become a superintelligence quite rapidly without any further algorithmic progress”

• A lot of economic analysis about the impact of AGI focus on human demand - will the economy shrink because our demands can be fulfilled much more cheaply, will it rise because AI will create new varieties of products, or maybe because the relational sector will grow? But all this analyses take as a given that the only demand that matters is the one originating from humans. How do we model the machine-only economy, where the demand originates from the AI’s themselves? And once we add this consideration to our economic analysis of the future, what changes?

The mistake of conflating intelligence and power

I had an interesting discussion recently. Someone asked me, what is intelligence? I said, the ability to achieve your goals across a wide range of domains. Okay, he says, then by that definition isn’t Donald Trump the intelligent person in the person, followed quickly by Xi Jinping and Vladimir Putin?

To be clear, these people are obviously very competent in certain ways. But when you think of ASI, you don’t think of Trump, but more so. The person who kept pressing this question was correctly pointing out that my definition of intelligence was basically power (after all, what is power if not the ability to achieve your goals across a wide range of domains?). If this is your definition of intelligence, then Stalin was the most intelligent person who ever lived.

Now, of course, you could change the definition of intelligence to something more like, comprehend and build atop abstract concepts. But notice that the most powerful people in the world do not max out this quantity. They’re above average in shape-rotation, but the correlation between extreme power and this kind of intelligence might be even weaker than the correlation between extreme power and height. The physicists are not running the world.

We tend to conflate power-seeking AI and superintelligent (in science and tech) AI. I’m not denying that AI can be power-seeking. Whatever skills and drives Donald Trump has could be embodied in a digital mind. I’m simply pointing out that the way we’re currently making AI systems smarter (training them to be really good coders, thought partners, and general coworkers) is not that strongly correlated with power.

We often talk about power in this way that misunderstands how it is actually derived in our world. Our intuitions are primed by games like Diplomacy or Go, which are designed to isolate and reward a g loaded kind of strategic reasoning. But in the real world, power is more the product of having the authority and trust to get lots of people to collaborate with you, rather than some galaxy brain scheming capability. Trump is not powerful because his brain, considered in isolation, is the most effective optimization engine on Earth. He is powerful because the government which hundreds of millions of people consider legitimate gives him a lot of power.

A group versus individual level analysis is useful here. As Garett Jones has written a lot about, individual IQ is only modestly correlated with individual income, but national IQ is strongly correlated with national outcomes. This is because intelligence has a lot of spillover effects - smarter societies cooperate more, save more, and can coordinate to build things like space shuttles and semiconductors. Richard Trevithick, who pioneered the high-pressure steam engine, died in poverty, buried in an unmarked pauper’s grave. But the fact that 18th and 19th century Britain had lots and lots of people like Trevithick contributed to Britain being able to set up a global empire and defeat lots of random kings and emperors around the world. George III himself didn’t need to be a genius — in fact he went mad halfway through his reign — but the country he sat atop still defeated Napoleon, conquered India, and built the world’s dominant navy. Similarly, even if some company’s AIs are just super obedient superintelligent coders and scientists, they could help the totally pedestrian human intelligences who have their reins (lab leaders, Presidents, some harder to imagine configuration of control) gain a lot of power. It seems to me that the right mental model is that more effective AI firms and countries will outcompete everyone else in normal capitalist ways, rather than a single AI outthinking everyone else.

RLVR might be disproportionately bad at science

Next two sections I’m writing up some threads that we explored in my interview with Michael Nielson. That episode was one of my favorite.

The organizing question from my interview with Nielson was, “How do we recognize scientific progress?” It’s especially relevant to thinking about what it would take for AI to close the RL verification loop on scientific discovery. But it’s also a surprisingly mysterious and elusive question when thinking about the history of human science.

Some people have this idea that AI is going to be disproportionately good at making scientific breakthroughs. The reason they think this is that 1. Science is ‘verifiable’, 2. AI is absolutely crushing domains that have a tight verification loop - coding, math, etc - because you can RL on these loops.

But the history of human science shows that the verification loop for theories can be on the order of decades and centuries, and even then experiments do not definitely rule out alternatives: Ancient Athenians dismissed Aristarchus (2nd century BC) on heliocentrism because it would imply stellar parallax. The first successful measurement of stellar parallax was in 1838, achieved by Friedrich Wilhelm Bessel.

What we know today as the better theory can often actually make worse predictions: it’s well known that Copernicus’s model of circular orbits around the sun was less accurate than Ptolemy’s geocentric model, which had accumulated millenia of correcting epicycles. What is not well known is that Copernicus’s theory wasn’t even simpler (Ptolemy’s model interpreted the true elliptical nature of orbits using an equant trick where other planets are not moving in uniform circular motion around Earth exactly, but rather an off center point. Copernicus didn’t like this, because it violated his Platonic heuristics - so he discarded the quant trick, which led to a less parsimonious model, since Copernicus had to add more epicycles and epicyclets to make up for it.)

So in what sense was it a better theory in 1543? In some sense, it wasn’t! You couldn’t have known ex ante that heliocentrism married with Kepler’s 3 laws (1619) is a much cleaner and more accurate theory, or that there’s a very beautiful unification of heliocentric orbits and terrestrial gravity (Newton in 1686).

There was one ex ante reason that you should have preferred Copernicus in 1543: his theory required retrograde motion1 as a natural consequence of his theory, whereas for Ptolemy it was an ad hoc addition. Even more impressively, his theory, developed in 1543, actually predicted the phases of Venus2 before they were observed by Galileo in 1610. But both of these things were also implied by Brahe’s model, which had set the sun to orbit the earth and then all the planets to orbit the sun.

Under a naive falsificationist framework, you’d have to wait until Stellar parallax was observed in 1838 to know that Brahe was wrong. But obviously the scientific community was able to make progress faster than this. There is some mixture of judgment and heuristics in the progress of science that we don’t even understand well enough to actually articulate, much less codify into an RL loop.

Or consider the case of the discovery of Neptune in 1846. Uranus deviated from its predicted Newtonian path. Le Verrier predicted that an unknown perturbing planet must exist, calculated its mass and orbit, and Neptune was found almost exactly where predicted.

But the Neptune story is symmetric to a failure case. Mercury had an anomalous precession, where the ellipse that shows its orbit would rotate 43 arcseconds more per century than should be implied by the impact of other planets using Newtonian mechanics. This led astronomers to speculate that there’s an unknown planet Vulcan within Mercury’s orbit. But it was resolved in 1915 with Einstein’s General Relativity.

A proper Newtonian would still proceed with the research agenda, but modify it as follows. First, you predict some unknown planet. If it can’t be found, you say it’s so small, it must require a bigger telescope, and you build a bigger telescope. And if you still can’t find it, maybe there’s a cloud of cosmic dust occluding it. If still not found, maybe the satellite’s instruments are being screwed by some unknown magnetic field, and you send a new satellite. At each of these steps, had you discovered a new planet, or some unknown cosmic dust, or some new magnetic field, that would have been a sensational victory for Newtonians.

Ex ante, this is not unreasonable to do! It is only after decades or maybe centuries of patchwork that we can then analyze, are we simply adding epicycles, or is this theoretical framework progressive, in that it makes predictions we wouldn’t otherwise be able to.

What do these examples illustrate? That ex ante it is almost impossible to determine which research programs are progressive (will predict and explain unanticipated new phenomenon) and which are regressive (need to be contorted repeatedly to accommodate seemingly disconfirming new phenomenon).

But the verification loop is often extremely long and weirdly hostile, and even then, experiments do not definitely rule out alternatives (see the discussion in the Nielson episode about how physicist contemporaneous with the 1880s Michelson-Morley experiments thought that it simply ruled out a particular theory of ether. Only Einstein made the full conceptual leap to discard the ether altogether).

This means that big conceptual breakthroughs cannot be easily verified. They are recognized decades or centuries later, when it turns out they were much more productive than the alternatives available. What this means for AI for science is that 1. You can’t easily train an RL loop for big conceptual breakthroughs.

And 2. the society of AI scientists will still need individual AI instances that have idiosyncratic biases and heuristics, and to pursue them unrelentingly for decades on end - for example, like the one Einstein had in insisting that there shouldn’t be some arbitrary inertial reference frame. There should be dedicated people to keep a bunch of dormant research agendas alive in case they turn out to be productive upon further investigation. To understand the kind of intransigent dedication to hypotheses that is needed to preserve correct scientific idea - even in the face of disconfirming evidence - consider the following story: In 1815, Prout hypothesized that the atomic weights of all pure chemical elements are whole numbers, because experimentally, most elements seem to come out like this. But there’s many anomalies - for example Chlorine’s atomic weight is measured at 35.5. And so Prout’s school claimed that maybe the chemical substance in which these elements appeared were impure. But there seemed to be no chemical reaction that could get rid of the impurities. And then they said, maybe it’s fractions of full atomic weights - but the closer you measure, the less natural the fractions seem to get - Chlorine goes from 35.5 to 35.46. It takes until almost a century later for people to realize that these measurements are showing multiple isotopes of the same element, which can be separated physically, but have no chemical distinguishing characteristics.

What I’m trying to say is that ex ante, one couldn’t have known which research program would be more productive. We need to invest in all of them concurrently. But that investment looks like a bunch of different individual scientists being super unreasonable and obstinate about propping up their preferred research agenda.

What does the parallel discovery of a deep idea like Darwinism tell us?

The Origin of Species was published in 1859. Principia Mathematica was published in 1687, two centuries earlier. Conceptually, it seems like natural selection is much simpler than the theory of gravity. A contemporary of Darwin’s, Thomas Huxley, read the Origin of Species and said, “How extremely stupid not to have thought of that!” Nobody ever said the same for not beating Newton to the Principia. I wonder if the reason this happened is that, while Darwin’s theory is conceptually simpler, it cannot be decisively tested. The evidence is circumstantial, retrospective, and cumulative. There’s no equivalent of Newton running the numbers on the moon’s orbital period and radius, and confirming that it corresponds to his equations.

Also you need this concept of deep time. Charles Lyell published the Principles of Geology in 1830, which gave Darwin the vast stretches of time that natural selection needed. And the fact that Darwin and Wallace basically arrived at evolution at the same time3 (and both credited Lyell’s contribution) does suggest that these underrated intellectual footholds were quite important (geology, paleontology of ancient extinct species which showed intermediate species (in some cases between apes and humans), biogeography from voyages and age of colonization, more sophisticated artificial selection like pigeon breeding). It’s interesting that an idea whose essence must have been obvious to herders and parents for thousands of years actually required many millennia of ancillary intuition pumps to fully spell out.

The pattern of parallel discovery in science and technology is very interesting, and seems to contradict this vibe that certain innovations could have happened earlier much earlier than they really did.

I’m announcing a $20,000 blog prize in order to find people who will excel at researching and thinking through these problems. The not-so-secret point of this whole contest is so that I can hire a research collaborator to think through questions like this hand in hand with me. See more at the end.

Pick a question below, and spend no more than 1,000 words answering it. 1st, 2nd, and 3rd place will get $10,000, $6,000, and $4,000 respectively. I’ll publish the winning entry (and potentially the runner ups) on my blog. Please submit by May 10th, 11:59 PM PST.

Questions - choose one

• A couple years ago, there was this idea that AI progress might slow down as we make further progress into the RL regime. 1. Because as horizon lengths increase, the AI needs to do many days’ worth of work before we can even see if it did it right, so if we’re still in a naive policy gradient world, the reward signal / FLOP goes down, and 2. We’d crossed through many OOMs of RL compute from GPT 4 to o1 to o3, and it would not be feasible to replicate that many OOMs increase in compute immediately again. But AI progress seems to have been fast nonetheless - even potentially speeding up if rumors about Spud or Mythos are to be believed. What gives? What did that previous intuition pump that motivated longer timelines miss? Feel free to deny premise of question.

• What’s the most plausible story where foundation model companies actually start making money? If you consider each individual model as a company, then its profits may be able to pay back the training cost. But of course, if you don’t train a bigger, more expensive model immediately, then you stop making money after 3 months. So when does the profit start? Maybe at some point scaling will plateau, but if progress at the frontier has slowed down, then the combination of distillation and low switching costs (cloud margins result from high switching costs) makes it really easy for open source to catch up to the labs, eating into their margins. So how do the labs actually start making money?

• With OpenAI’s new raise at an $852B valuation, OpenAI Foundation’s stake is now worth $180B. Anthropic’s cofounders have pledged to donate 80% of their wealth. Nobody seems to have a concrete idea of how to deploy 100s of billions (soon trillions) of wealth productively to “make AI go well”. If you were in charge of the OpenAI Foundation right now, what exactly would you do? And when? It’s not enough to identify a cause you think is important, because that doesn’t answer the fundamental problem of how you convert money to impact. Identify the concrete strategy you recommend pursuing.

• What should countries which are not currently in the AI production chain (semis, energy, frontier models, robotics) do in order to not get totally sidestepped by transformative AI? If you’re the leader of India or Nigeria, what do you do right now?

Rules and tips

• Please don’t let a lack of domain expertise dissuade you from entering. I’m looking for someone who can ramp up fast on unfamiliar topics and think clearly.

• Each entrant may submit only once.

• You are still eligible for this essay competition even if you’re not interested in the researcher role. Nor does winning this competition guarantee that you will be offered the role.

• You’re welcome to use LLMs to help you research, but I specifically picked these questions because I’ve found LLM answers to them unsatisfying. On these kinds of ambiguous questions, LLMs are too all over the place. For example, they’ll identify 5 plausible answers but not have the context and taste to identify the crucial factor and iron out its implications.

• You only have 1000 words - make them count. People have the habit of spending the first paragraphs clearing their throat - avoid that.

Why am I hiring for a researcher?

I want my podcast/blog to move from just asking questions about AI to actually helping answer them. But there are too many important questions, and I need a collaborator to build up context on them all, to explore dozens of fractal sub-questions, to consider the rebuttals and syntheses, and to sharpen each others thinking.

The questions I want us to explore are very broad while at the same time requiring deep technical analysis across many domains to actually answer.

Why am I hiring this way?

Well, I could just put out a job ad for a researcher, but I’ll get 1,000 different resumes, and I’ll have no clue based on that information whether the applicant would be any good at synthesizing lots of technical arguments and information. So I thought, let’s just list out some questions where I genuinely don’t know the answer and would be keen to get some insight.

What this role looks like

• Ideally in person in San Francisco, but potentially open to remote.

• Will pay competitively

Submit here

If you have questions or comments, I’m hello@dwarkeshpatel.com.

Can distillation be stopped?

Can the frontier labs stop distillation? Because if they can’t, open source commoditizing models can catch up incredibly rapidly, making the long run business model for the labs less viable. Let’s say it takes 1T tokens from a frontier model to capture its juice (I have no idea if that’s correct, but let’s say). Even ignoring savings from caching, Opus 4.6 is $25/MTok. So $25 million for those 1T tokens. That’s nothing.

Labs are responding by hiding chain of thought. But there’s two problems with this solution:

• Chain of thought is not made of some fundamentally different kind of token. You can just instruct the model to not think first but just start solving the problem, or to write out its thinking somewhere else.

• Even if labs do figure out how to robustly hide chain of thought to train in the future, you can make reconstructing the chain of thought necessary to reproduce a decoded sequence as an RLVR target. Yes that costs more, but seems doable.

• Maybe most importantly, the real juice of these agentic models is their tool use (writing and updating files of code, running bash commands, etc). And if these things are done locally on the user’s computer, you can’t really hide them. And it seems like a hard lift to get users to migrate all their development workflows to a cloud that you fully control and hide visibility to, modulo a Claude agent input text prompt.

By the way, I learned about an interesting way companies which build products atop API access to AI models can basically distill these models, in a way that potentially makes the distilled models even better than the ones they’re actually built atop.

Suppose you’ve got a coding product. In order to build a feature, a user uses your product to query some frontier model API across 10+ back and forths. Once the user is satisfied with the end result, you have the end state that the user actually wanted - “the gold diff”. These coding product companies can now set the gold diff as the RL target for training their own models, where the model gets rewarded for producing outputs that look like what users eventually converged on, and penalized for producing the kinds of intermediate outputs that users kept rejecting or editing.

On why pretraining runs fails

Had an interesting chat with someone on why pretraining runs often fail. It was very interesting to get a sense of all the tangible ways that things can get fucked, and why training is such a precarious operation. At a high level, breaking causality, and adding bias, seem to be key culprits.

Breaking causality:

• When you do expert routing, you first go through the router, which gives you a score of how much each token wants each expert. There’s two ways to proceed from here: 1. Token routing, where you read the scores from the token’s perspective, and allocate to each token’s top k experts. Problem is that you could end up with wildly unbalanced allocation across experts, which is terrible for performance. Alternatively, you could (and only in training) do expert choice, where you just split the tokens by which are more relatively preferred by each expert. This way you can enforce that each expert gets roughly the same number of tokens. But the big problem is that this breaks causality, because which expert token n gets allocated to may depend on which expert token n + k might be router to. And breaking causality is very bad, because you’re getting information in training (and updating based on it) that you wouldn’t see in deployment.

  • Rumor is that this explains why Llama 4 was underwhelming.

  • I guess you could do expert choice during prefill inference? But maybe it doesn’t work well in practice to allocate tokens to experts which would not have received that token in actual training.

  • Tbh I don’t fully understand why breaking causality is so bad. I understand you can’t see beyond causality in real inference. But why is this minor deviation such a big issue?

• Another thing that can break causality is token dropping. Where experts just ignore the tokens in the batch that they’re supposed to process, but which rank not so strongly, and cutting whom would spare going outside padding. This breaks causality cause a later token being more strongly matched to this expert might lead to an earlier token getting ignored.

  • Apparently this was an issue with Gemini 2 Pro.

Adding bias:

• Bias much worse than variance - variance can average out, but bias compounds

• Apparently the original GPT 4 training was slow and got initially fucked because of the following bug: they were using FP16 on their collectives like all-reduce. FP16 distributes its granularity according to logarithmic density - between 1 and 2, the mantissa bits carve the interval ~0.001 apart. But 1024 and up, the mantissa might be carving the interval by multiple whole number values. Suppose some collective involves adding 1 + 1 … 10,000 times - you could get in a situation where as soon as you get to 1024, you add 1, it goes to 1025, you round down to the nearest interval at 1024, add one again. And so the calculated value is 10x off the real value. Huge issue if you’re trying to sum many small gradients into a large accumulator. And imagine how hard the bug must have been to find!

Implications for AI training:

• Some of the people who think we can cure aging argue that there’s basically 5 different ways people die of old age (heart disease, cancer, etc), and that if we cure these 5 different diseases, then we’d basically have solved again. You could ask a similar question about these failed pretraining runs - are there 5 different ways training runs fail, in which case once a lab figures out numerics and , you’ll just have smooth sailing, or will you keep seeing new bespoke issues emerge at each new level of scale? The person I talked to seemed to think the later - he pointed out that even within numerics, there’s so many ways you can fuck things up. And new ones will keep emerging at scale.

• Bearish on AI fully automating kernel writing anytime soon. Presumably this is because he thinks it’s more of an AGI complete problem than some give it credit for. There’s another school of thought that says, “Hey, which kernel gets attention or MLP to run fastest on this scaleup is a super verifiable domain, thus we can RL to superhuman performance easily.” But he says, it took Nvidia, which has the best kernel engineers in the world, a long time to optimize for Blackwell, which suggests that actually it’s quite hard, and might not be super easy to close the loop on.

• Sometimes people say inference for RL generation and inference for end user generation is basically the same. But this person pointed out that in RL inference, numerical drift between inference and training engine can cause these subtle off policy biases, which matter a ton for highest quality training. But are not an issue if just serving to users.

• Emphasized how important it is to have a disciplined process for amalgamating compute multipliers, because of the risks of stacking up bugs with subtle biases.

Pretraining parallelisms

Notes from an excellent lecture that Horace He gave my friends and me.

What made this lecture so good is that Horace built up the whole topic as a chain of problems and solutions: here’s what we want to do, here’s why it breaks, here’s how we fix it, here’s why that fix eventually breaks too. Most explanations just list out a hodge podge of different strategies, without ever connecting them to the problems they solve or explaining why you’d pick one over another.

• Equation for pretraining flops = 6ND. 2 FLOPs per parameter per token for the forward pass (multiply + add). Backward pass is 2× forward because you compute gradients w.r.t. both input matrices. So 2 + 4 = 6.

• Okay we can’t do all this on one GPU. So how do we split up this problem? The obvious solution is to do data parallel - where you copy the model weights across each GPU, and you just do a part of the batch on each GPU.

  • The obvious problem is that each GPU only has a limited amount of HBM - B300 is 288GB - and this is not enough to store the weights as models get bigger and bigger, much less their activations.

• Okay so next thing we try is fully sharded data parallel - each GPU only stores 1/N of the parameters of each layer - before processing each layer, you all-gather the full layer’s parameters from all GPUs (each GPU only stores 1/N of each layer). After processing, each GPU discards the gathered parameters.

  • It was emphasized that this is the go to default. And you only move on from this when having too many GPUs forces you to move on, for reasons explained later. The reason this is the default is that it’s trivial to overlap compute and communication time - that’s because the only thing being communicated is the weights, which are not dependent on what happened in the layer before, so you can start all gathering the next layer while you’re still computing this layer. Compare this against tensor or expert parallelism, which do need to share activations for one layer before you can process the next one. The problem with pipeline parallelism is bubbles as explained below.

  • From a comms volume perspective, FSDP looks insanely expensive at first — you all-gather every layer’s full weights across all GPUs, use them for one matmul, then throw them away. But this ignores what regular data parallelism already costs you - in regular DP, you still need to do an all reduce after every layer of the backwards pass in order to sync the batch’s gradients across all the GPUs. That all-reduce has comms volume of params × 2. FSDP adds all-gathers — one per layer in the forward pass, one per layer in the backward pass. But an all-gather is half the comms volume of an all-reduce. So naive FSDP comms volume ends up being # params * 4 (all gather forward and back, plus all reduce on back). You can do even better: since each gradient shard only needs to end up on the one GPU that owns it, replace the all-reduce with a reduce-scatter (which skips the final broadcast step). That gets you to params × 3 total — a 50% overhead over vanilla DP.

• So why can’t you always just do FSDP?

  • Comms crossover: You want your compute time to be greater than your comms time - you don’t want to be bottlenecked on comms. But since compute time for FSDP decreases as you increase the number of GPUs, and comms time does not, as you scale the number of GPUs on FSDP, your MFU can totally crater. When this happens, you need to add pipeline parallelism too.

    • Compute time = (6 * # tokens * active params) / (compute per GPU * number of GPUs)

      • This decreases as you increase number of GPUs

    • Comms time = (# total params * 3) / (nv link domain size * infiniband BW)

      • Comms time does not increase as you add more domains. This was really confusing to me. Each domain collectively holds all the parameters, and you need to sync gradients across domains after each layer of the backward pass. You’d think that adding more domains means more hops in the ring, so the all-reduce gets slower. But the standard ring algorithm splits the message into one chunk per participant. More domains means more hops, but proportionally smaller chunks per hop. (This breaks down when chunks get so small that per-hop latency dominates, at which point you switch to tree algorithms.)

        • Technically, you can do better than a naive single all reduce for the gradients between all the domains. You do a hierarchical collective to optimize comms time across multiple NVLink domains. Key thing to remember is that each GPU in the domain gets its own bandwidth access to infiniband. So you wanna use it all up since interconnect bandwidth is the bottleneck. You do this by trying to do as much as possible within a scaleup before you move out. So you do reduce scatter within a scale up to give each GPU the domain-level reduced gradients for a shard of the layer, then all reduce these shards across corresponding GPUs across domains, then all gather within a domain. This shifts the comms time line down, thus moving the crossover point to the right.

        • Made an animation to illustrate it using Cursor and Composer 2:

  • If you look at the equations, you can see that if you increase batch size, crossover point moves to right, and if you make the model more sparse, moves to the left.

  • Also why TPUs are better at FSDP - because more accelerators within a domain.

• Batch size floor: FSDP is data-parallel, so each GPU processes at least one sequence. Attention is computed within a sequence and can’t (easily) be split across GPUs. If your critical batch size is 10M tokens and sequence length is 10K, you only have 1K sequences — so you can’t scale beyond 1K GPUs with pure FSDP, even if you have plenty of comms bandwidth left.

• Problems with pipeline parallelism (the next addition you’d make to FSDP in order to deal with these issues):

  • The problem with pipeline parallelism is different - there you have bubbles that emerge from the fact that at the beginning of the batch, the GPUs dedicated to the final layers are not being used, and conversely at the end of the batch, the GPUs dedicated to the first layers are not being used. The reason you can’t overlap batches in training to solve pipeline bubbles is that you need to consolidate gradients and update the model before you process the next batch.

  • But also you’re adding architecture constraints - things like Kimi’s attention-to-residuals (where each block attends to all previous layers’ residuals) become very difficult when those residuals live on different pipeline stages. Similarly, interleaving sliding-window and global attention layers could cause load imbalance across stages. Dealing with all this slows down research iteration, which is the greatest sin you can commit.

Mythos and the cybersecurity equilibrium

It seems like the key difference between Mythos and previous versions is that while previous versions could find individual vulnerabilities in the code (“Hey, there’s a missing bounds check here”), Mythos is long run agentic enough to rope 5 different vulnerabilities together which are all required in order to find an exploit (“Now I can execute arbitrary code, escalate privileges, etc”). To the extent that some discontinuity has been hit, it’s probably more the result of the combinatorial nature of cyberattacks rather than some off-trend increase in intelligence.

What does this mean for offense/defense? One way to look at it is that software is more secure today than it was 20 years ago, despite more and more human intelligence probing at public code, both white hat and black hat. If we get another influx of intelligence suddenly, why should the dynamic change?

In fact, we know that our foreign adversaries almost certainly have access to a bunch of critical zero days which they’re saving for a rainy day, or already using in inconspicuous ways. To the extent that Glasswing allows the whole industry to find a bunch of these latent exploits and patch them, shouldn’t we expect defense to have become much stronger relative to offense by the end of 26? Of course, this is thanks to the fact than American companies got there first and are cooperating with other companies and our government to patch things before our adversaries get to the same level.

One counterpoint I heard from a security expert is that there’s big difference between finding vulnerabilities and patching them - and AI is much better at the first than the later (people often talk about the offense/defense balance, but difficulty of finding versus patching vulnerabilities seems much more significant). In order to patch an issue, you have to find a fix that will not interfere with all the ways people use your software, and all the features which rely on weird bespoke behavior. XKCD has a nice comic illustrating how these kinds of issues come up:

Potential solutions, if it’s non-trivial to just push patches to every piece of software?

• TODO - I know nothing about formal verification of software - check out what a seL4 proof of some behavior might look like

• Use LLMs to rapidly port all C to Rust. Curious how easily Mythos can find vulnerabilities in memory safe languages.

In some sense, its good that Anthropic didn’t release this model publicly until critical IT could be patched up. In another sense, isn’t it a super bad precedent for private companies to be hoarding the ability to be able to break into any operating system and browser and device? One obvious question for Anthropic is why they didn’t just build some kind of classifier which would detect whether you’re using the model for cyberattack type stuff, and refuse requests if yes, and release that publicly.

• Patching your own software is isomorphic to finding bugs in someone else’s repo from the perspective of an LLM (and patching your own software is a frequent coding model use case).

• These kinds of classifiers can be easy to evade if you have enough expertise to break the problem of finding exploits down into smaller subproblems of finding vulnerabilities which each individually seem like sensibly good behavior to an LLM with no memory

Pipeline RL paper summary

As you keep RLing a model, not only does the average length of a response increase (since you’re basically training the model to think for longer before answering) but the variance in length also increases - sometimes you get an easy problem and you can immediately answer it - other times, you need to go think for 100k tokens.

This is a big problem for GPU utilization on training. Because you have to wait for all these stragglers to finish generating before you can start the next training step.

Okay one way you could get out of this conundrum is to just to just batch generation so that while stragglers keep going, you generate even more rollouts.

The problem is that there is an optimal batch size for each training step, so you’d need to split all these rollouts you made across lots of consecutive training steps.

But this takes you into the domain of offline RL, because your model is changing with each training step. And so you’re training your model on trajectories that were actually generated by an earlier model, which is not ideal.

Pipeline RL paper proposes the following fix: in flight weight weight updates - where you just sub out the generating model partway though these generating trajectories as soon as the new training step is done, so all the short trajectories, and a good chunk of the long trajectories, that the next training step will be trained on are generated by the most recent version of the model.

John Collison and I just interviewed Elon. The interview was recorded before we knew that SpaceX was acquiring xAI, so the fact that our first topic was space GPUs now feels all the more relevant.

As I was preparing to interview Elon, I put together some notes and a spreadsheet to help me think through orbital datacenters. I turned those notes into this blog post.

Even if orbital data centers don’t make sense yet, in the long run the singularity is clearly moving into space. Earth intercepts about one two-billionth of the sun’s total output. If AI scaling continues, compute will eventually move to where the energy is. So space GPUs are fun to think about, because they give you a sneak peek at the future. Whether that future arrives in 2030, 2040, or 2050 is another question.

Please take everything below with grains of salt—grains so big that you might confuse them for rocks. Assume all the numbers are wrong. Every paragraph below covers a topic that would take an actual expert a week to properly evaluate. What you’ll find here is what a professional podcaster has pieced together from conversations with LLMs and some very generous people who talked to me before the interview. Thanks to Casey Handmer, Philip Johnston, Ezra Feilden, Andrew McCalip, Vinay Ramasesh and the team at Kinetic Partnership for all their help.

Why orbital data centers?

The whole reason to go to space is energy. Yes, panels in space get about 40% more irradiance—but the real advantage is that you can put your satellites in sun-synchronous orbit, where they face the sun continuously. No nights, no clouds, no need for batteries (which is the majority of cost in a solar-storage system). Solar on Earth has a roughly 25% capacity factor, meaning panels only generate a quarter of their peak output on average. In space, you get close to 100%.

The logic is that if the launch costs continue to drop, it will become cheaper to put GPUs in orbit than to build power plants and batteries on Earth. And there’s a lot of room for launch costs to fall—propellant is cheap, and the main expense is the rocket, which you can now reuse. Falcon 9 is around $2,500/kg with a disposable upper stage. Starship with full reusability could get below $100/kg.

But here’s the problem with this argument. Energy is only about 15% of a datacenter’s total cost of ownership. The chips themselves are around 70%. And you still have to launch those to space!

It gets worse. On Earth, GPUs fail constantly. In the Llama 3 paper, Meta reported a failure roughly once every three hours across a 16,000 H100 cluster. When a chip dies, a technician walks over, swaps it out, and the cluster keeps running. In space, you can’t do that—at least not until we have Optimus robots stationed on every satellite.

What about radiation? It’s actually less catastrophic than you might expect. Google’s Suncatcher paper found that their TPUs survived nearly 3x the total ionizing dose needed for a 5-year mission before showing permanent degradation.

I asked Elon about this. He responded:

> “Actually, it depends on how recent the GPUs are that have arrived. At this point, we find our GPUs to be quite reliable. There’s infant mortality, which you can obviously iron out on the ground. So you can just run them on the ground and confirm that you don’t have infant mortality with the GPUs.”

> “But once they start working, their actual reliability—and you’re past the initial debug cycle of Nvidia or whatever, or whoever’s making the chips, could be Tesla AI6 chips or something like that, or it could be TPUs or Trainiums or whatever—is actually quite reliable past a certain point. So I don’t think the servicing thing is an issue”

Consider what’s actually being proposed here. You assemble your GPUs into racks on Earth, run them for a few hundred hours to catch the duds, disassemble everything, pack it into a satellite, launch it, and get it operational in orbit. Throughout this entire process, the most expensive part of your system—the chips—are just sitting there not doing useful work.

Is this just not possible on Earth?

Throughout the interview, Elon kept returning to one point over and over again: Look, forget the economics! It will simply not be physically possible to scale power production to the scale needed for AI on Earth. He went on:

> “The only place you can really scale is space.”

> “All of the United States currently uses only half a terawatt on average. So if you say a terawatt, that would be twice as much electricity as the United States currently consumes. So that’s quite a lot. Can you imagine building that many data centers? That many power plants? It’s like those who have lived in software land don’t realize they’re about to have a hard lesson in hardware.”

Elon kept pointing out the bottlenecks we’ve already run into on Earth. You can’t plug into the utilities—the interconnect queues are too long. You can’t do behind the meter and generate power yourself—lead times for turbines stretch past 2030. You can’t do solar on Earth, because of permits, and because of the tariffs. And Earth has clouds and nights, requiring overbuilt solar and batteries. In space, you can just put the satellites in sun synchronous orbit!

Look, at some level, it is true that we can’t keep scaling on Earth. But keep in mind that the Earth is really fucking big. 1 TW of solar (with 25% capacity factor, so really 4 TW of panels) is around 30,000 square miles. That’s like 1% of the US—about the size of South Carolina. For context, AI datacenters currently consume only ~20 GW globally.

By the time we’re talking about multiple terawatts, we’ll have had to massively scale leading-edge wafer production. And that’s the really hard part. Fabs are the most complicated manufacturing facilities humans have ever built. In order to believe that we need to go to space in order to find the power turn on all these chips, we’ll need to assume a few things:

• We’ll manage to produce a lot more chips.

• Every single relief vessel for power generation on Earth will fail to scale.

But semiconductors are so much more complicated than solar panels! They’re even more complicated than the blades on a turbine. It feels quite unlikely to me that the thing we manage to solve is building terawatts worth of leading edge wafers, but in that world we can’t figure out how to pave Nevada (or if regulation proves to be a problem, then the UAE) with solar panels.

100 GW into space

How many Starship launches will it take to launch a 100 GW into space?

An orbital datacenter satellite has three big components: solar arrays, computers, and radiators. And the key constraint is that for every watt of compute, we need roughly one watt of solar and one watt of thermal rejection capacity.

The W/kg of each component determines how the mass budget gets split—and how much compute you can bring along. The figure that matters most here is the specific power of the whole satellite: after you account for solar panels, radiators, and chassis, how many watts of compute do you actually get per kilogram launched?

For Starlink satellites, this works out to roughly 50 W/kg. The people trying to build orbital datacenters are currently targeting 100 W/kg. There are only two ways to get there: lighter solar panels (more watts generated per kg) or lighter radiators (more watts rejected per kg).

The numbers below are super rough. Reliable figures for space-grade components are hard to come by. But even rough math reveals which variables must improve—and by how much—in order to hit 100 W/kg.

• Solar: There are apparently companies that are targeting next gen thin film that reaches upwards of 500 W/kg, but the state of the art is 150 W/kg, and most missions right now fly 30 W/kg. Let’s be generous and assume 200 W/kg.

  • The trouble here is that there’s obviously a tradeoff— denser panels costs more money, but reduces launch cost. And it’s difficult to calculate what that implies for these next gen panels, because their prices are not listed anywhere.

• Compute: I’ve heard that a stripped down GB200 NVL72 with no cooling equipment is around 100 kg. They draw 132kW of power, but let’s add 10% overhead for the intersatellite lasers and so on. That gets us to 1,452 W/kg.

• Radiators: In space, you can’t convect heat away, because there’s no air. You can only radiate it, which means your panels glow infrared until the heat leaves. The Stefan-Boltzmann law governs how much power a surface can radiate.

  GPUs typically run up to 90° Celsius. There’s some temperature drop through the heat pipes and fluid loops that carry heat to the radiator surface. Call it 30°C. So your radiators end up operating around 60°C. Plug that into Stefan-Boltzmann (assuming you’re using aluminum panels that weigh around 2 kg per square meter of surface area, that works out to roughly 320 W/kg.

  Since radiated power scales with T⁴, running your chips hotter can help you save a lot of radiator mass. For space, people will have to figure out how to build GPUs that tolerate higher temperatures.

Assuming the numbers above—and also assuming that a fourth of the mass of the satellite has to be the chassis—I get 85 W/kg for the whole system. Again, I want to emphasize these are rough calculations; feel free to plug in your own numbers in the spreadsheet here.

At 150 metric tons to low earth orbit per Starship (Elon’s target), you’re looking at around 10 MW per launch. That means roughly 100 Starship launches in order to put 1 GW of compute in orbit. To hit 100 GW in a year, you’d need roughly 10,000 launches, or, about one launch every hour.

This is insane! A single Starship produces around 100 GW of thrust power at liftoff. That’s about a fifth of total US electricity consumption, concentrated in one rocket for a few minutes. And the plan would be to do that once an hour, every hour, every day, for a year.

I asked Elon what that world looks like:

I don’t think we’ll need more than... I mean, you could probably do it with as few as 20 or 30 [Starship vehicles]. It really depends on how quickly the ship has to go around the Earth and the ground track before the ship has to come back over the launch pad. So if you can use a ship every, say, 30 hours, you could do it with 30 ships. But we’ll make more ships than that. SpaceX is gearing up to do 10,000 launches a year, and maybe even 20 or 30,000 launches a year.

Workloads and comms

Starlink satellites already communicate via inter-satellite laser links at 100 Gbps—and Google’s Suncatcher paper suggests off-the-shelf transceivers could potentially hit 10 Tbps. For context, Infiniband links between nodes in a terrestrial datacenter run at 400 Gbps. The gap isn’t as large as you might expect. So, could you do synchronous training in space?

Even the most bullish analysts don’t claim that orbital data centers will be used for training. I don’t know any of the relevant orbital mechanics, but obviously satellites at different altitudes move at different orbital velocities, which means the satellites are desyncing relative to one another. Google came up with a clever solution for this in their Suncatcher paper—keep lots of satellites in a single tight cluster at the same altitude. Google’s researchers proposed eighty-one satellites in such a synchronized constellation. If each constellation had a GB200 NVL72, then that’s only 15 MW parcels of coherent compute.

Defenders of orbital datacenters say that most compute is going to shift to inference (and with RL, most training is also inference). Maybe the legacy terrestrial datacenters do end up doing the pretraining runs, and then whatever mixture of RL environment training and continual learning happens in the future does happen in space. So, the argument goes, it’s not a big deal that the scale ups in space are isolated. But there’s still the question of how hundreds of gigawatts of inference are beamed back to Earth.

For a moment, let’s imagine a world where as we see the sunrise and sunset we also see a Saturn-like belt of GPU satellites passing over us. That’s already really cool. But then there’s another sci-fi premise, which I really wanted to be plausible, and which turns out not to make any sense: Imagine that every 12 hours, as this country of geniuses in space passes over us and shoots down half a day’s worth of new ideas, our code finally starts working and our factories buzz alight and become more productive. Unfortunately, it’s just science fiction. Inference doesn’t take that much bandwidth. One hundred gigawatts of a 5T model is roughly 58 billion tokens per second, resulting in ~ 230 GB/s.

That’s nothing. That many tokens can easily be beamed using lasers from GPUs in the orbital plane through to Starlink satellite network and then down to Earth.

Latency might be an issue, up to fifty milliseconds from any given spot on Earth through the Starlink network to the sun synchronous orbit and then back again. But as we move towards a world of true remote coworker AIs, where the agent works for tens of minutes before coming back to us, the marginal milliseconds of latency matter less and less.

So why is Elon doing this?

I’m willing to accept Elon’s argument that if launch costs become sufficiently cheap and we can repair GPUs in space, then there’s a viable path toward orbital data centers. But it seems especially difficult to imagine a situation in which orbital data centers end up significantly cheaper, because, again, most of the cost of a data center is the GPUs.

For most compute to shift to space, all of the following things would need to be true:

• Power generation on Earth hits a ceiling, or AI demand outstrips every terrestrial option.

• Chip production scales faster than anyone expects, so we have the silicon but not the electricity.

• Starship reaches thousands of launches per year.

If Elon’s right, he wins the AI race outright. SpaceX is the only entity that can launch at that scale. xAI would have unlimited power. Everyone else will be stuck fighting over grid interconnects and turbine orders.

And if Elon’s future doesn’t materialize? xAI is just another lab in the pack. Which means xAI loses. The AI race is a winner-take-all competition, and xAI isn’t in first place. Elon’s comparative advantage was never going to be navigating utility interconnect queues or filing permits faster than Google. His advantage is SpaceX. So why not bet on the world where SpaceX becomes the kingmaker?

This might sound reckless. But that’s how SpaceX got here. Their whole business plan seems to be one in which they conjure new wells of demand for each generation of rocket on the path to the Dyson swarm. Falcon 9 first flew in 2010. Starlink didn’t launch until 2019. Maybe orbital datacenters end up being for Starship what Starlink was for Falcon 9.

Sometimes, during the interview, I found my thoughts drifting toward Elon’s vision for this big, interconnected future. So I paused a moment and said:

What I find remarkable about the SpaceX business is the end goal is to get to Mars, but you keep finding ways on the way there to keep generating incremental revenue to get to the next stage and the next stage.

Elon nodded his head slowly. And then he said:

You can see how this might seem like a simulation to me.

$100/hour, fully remote, work hours are flexible - I expect it’ll be 5-10 hours a week.

Ideal candidate is maybe a grad student, or a post doc, or working in one of the fields I wanna find guests in. I’m looking for people who are really plugged into some discipline and have high taste.

Beyond just scouting guests, I’ll want your help assembling curriculums that help me prep for interviews and rapidly get up to speed.

The application form is here, and it’s extremely simple - just pitch me on a guest and tell me a bit about yourself. Please submit by 11:59 PM Pacific, Friday, Jan 23.

I’m looking to hire ~one scout for each of the following fields: bio, history, econ, math/physics, AI/hardware.

However, it’s very possible I end up hiring more (or fewer), or break apart the domains of knowledge in a different way, based on the range of expertise of the best people who apply.

What I’m looking for in guests

I’m looking for people who are deep experts in at least one field, and who are polymathic enough to think through all kinds of tangential questions in a really interesting way.

So I’m selecting for this synthetic ability to connect one’s expertise to all kinds of important questions about the world - an ability which is often deliberately masked in public academic work. Which means that it can only really come out in conversation.

That’s why I want to hire scouts. I need their network and context - they know who the polymathic geniuses are, who gave a fascinating lecture at the last big conference they attended, who can just connect all kinds of interesting ideas in the field together over conversation, etc.

We get tons of inbound from people who are working on impressive companies or doing interesting research projects. But almost always it’s a no; while I think their work is important, it’s self-contained in a way that I worry won’t lead to interesting broad discussion.

To get a little more concrete, here’s what worked well about some of my recent favorite guests:

Let me talk through why I think some interviews worked especially well, so you can think about what people in fields you’re familiar with fill a similar mold.

• Jacob Kimmel: A lot of people who pitch themselves as guests are capable of only talking about their own research. But the amazing thing about Jacob is that he is an insane polymath. For example, he could explain why evolution didn’t select for longevity by drawing deep analogies to how gradients flow in ML models. He had all these other random interesting takes, from why humans never evolved their own antibiotics to how there’s this gene that used to protect us from HIV-like viruses but got repurposed, which hints at some ghost scourge. And then he could zoom out and give a great diagnosis of what’s bottlenecking pharma progress. I really want to emphasize how that’s different from other brilliant people I get pitched – these people are also doing incredible research, but they don’t have this range of really deep, interesting takes. That part is super crucial.

• David Reich: It’s actually quite surprising that my second most popular guest of all time is a geneticist of ancient DNA. How did that happen? Here’s why I think this episode blew up. In high school, you get some vague explanation of human evolution. And you feel like you understand it and can move on with your life. And here comes David, showing you how this very fundamental topic, which you assume was settled and haven’t bothered thinking about in years, is actually way more murky and surprising than you realized, and how new discoveries are totally overturning our basic understanding of the field (in this case, the how, when, where of human evolution)1.

• Andrej Karpathy: It’s extremely rare to get someone who is expert-level in a technical, fast-moving, and frothy field, but who has no vested interest in a particular company or approach, and who is in a position to just give an unbiased lay of the land. I have a couple questions below about biotech or formal math or robotics in the wake of AI progress - if there’s a Karpathy-type person in those fields, I’d be very keen to get a technical lay of the land and vibe check of what claims are credible versus crazy.

Some recent questions

In case it’s helpful for brainstorming a guest, I’ve listed out a few big questions that have been on my mind recently. But please feel free to ignore them - there’s way more interesting questions in the world than the ones I am aware of - feel free to say, “You might not yet be curious about antibody development/the history of language/the dark ages/battery tech, but the guest I have in mind for that topic is so good that it’s going to be your next big banger episode.”

Bio

• Dario’s Machines of Loving Grace argues we’ll compress a century of bio progress into a few years - that big breakthroughs like CAR-T therapy, mRNA vaccines, cheap genome sequencing, etc show how in the long run things which seem like data or physical bottlenecks can be solved by better tools to measure/predict/perturb/understand biological system, and these tools are downstream of intelligence. But here’s what I don’t fully understand: over the last 3 decades, we’ve seen a million-fold reduction in genome sequencing costs, 1000-fold decrease in DNA synthesis costs, the development of precise gene editing tools like CRISPR, and the ability to conduct massively parallel experiments through multiplexing techniques. But it doesn’t seem like we’re curing diseases or coming up with new treatments at a faster rate now than we were 30 years ago. If anything, drug development is slowing down. I want to find a biology researcher who can think through how plausible a 10x or 100x speedup in new drug discovery actually is. They should obviously know a lot about and have hot takes on what’s actually bottlenecking progress today, and they should be flexible enough to imagine what might change with much more intelligence.

• What exactly is the special sauce of the brain that we’re still missing? Adam Marblestone thinks it’s the curriculum of reward functions and the learning/steering subsystems. Others argue that gradient descent is fundamentally worse than how the brain learns within a lifetime (which is closer to in-context learning in its flexibility and sample efficiency).

Math/Physics

• I’ve been really enjoying Strogatz’s Nonlinear Dynamics and Chaos textbook, and I want to make something podcast-shaped out of it. Strogatz himself has deferred until after he finishes his next book, so I’m looking for another mathematician on a related topic. I think the right format here isn’t a normal meandering interview - it’s something more like a lecture. A mathematician comes in with a specific topic or example we can deep dive on. He posts up at a blackboard, starts explaining a topic, and I interrupt to clarify confusions and ask follow-up questions. The model is something like Terence Tao and Grant Sanderson’s cosmic distance ladder video. Who can replicate something similar with me with some independently explainable topic in chaos/nonlinear dynamics or adjacent topics? I’d be especially keen if someone can present something on how the topics in this textbook tie into ML (see for example Neural network training makes beautiful fractals).

• What real world impact should we expect from the current batch of AI for math projects? What are the fields of technology where people are going, “Ah we could totally solve quantum computing (or fusion or AGI) only if we had more theorems!” But maybe problems in biology and physics and materials and so on reduce down to math in a way I’m not foreseeing, and automating formal math alone is enough to unlock a bunch of progress. See footnotes for some more questions I wanna ask the right guest on this topic.

  I started reading Proofs and Refutations, which is this famous 1976 book by the Hungarian mathematician Imre Lakatos about the philosophy of mathematics. He says math involves a lot of changing definitions and swapping lemmas in order to deal with different counterexamples. This seems fine for a good faith mathematical community, but super reward hackable for these AI-for-math models. Also it involves a lot of realizing how a problem in one domain is really a problem in another, and noticing the meta level pattern - AIs so far have been especially bad at this kind of thing. If math is just proof search within a fixed formal system, then AI can help a lot. But if its dialectical construction and refinement of concepts (based on what tasteful parsimonious definition can withstand counterexamples) , then I feel self play and ‘automated cleverness’ alone won’t do the trick. But maybe automated counterexamples are super useful. I’m sure for practicing mathematicians there’s a bunch of stuff that’s naive or wrong about the above. Would love to chat out what the actual research math process is like, and what good it would do to automate it.

AI/hardware

• RL progress has been very fast, but it’s partly the result of going from almost nothing to 1e26 FLOPs training compute in a year (aka like going from GPT-1 to GPT-4.5). It’s still possible that it has terrible scaling exponents and further progress will be very slow. And also it’s not clear how much of the progress over the last year comes from inference scaling, which has worse variable economics. But on the other hand, maybe there’s a ton of low hanging fruit in improving RL - with pretraining, there’s been 5 years of developing the theory and empirics of optimal batch sizes, learning rates, architectures, etc. As that low hanging fruit is picked, maybe RL progress continues to be fast? The other big question about RL training is how much transfer learning are we seeing - is there all this crazy meta learning that’s not directly induced by any env and which will enable flexible human-like labor soon? I have no idea. My friends at labs who are actually doing this training obviously wouldn’t tell me. But I want to actually concretely understand what’s going on here.

History

• There’s the famous Needham question, which asks why China didn’t industrialize first despite leading the world in population, inventions, and bureaucratic sophistication. I find the standard explanation of how this centralized Ming/Qing regime damped invention and exploration unsatisfying. Or at least I don’t understand it concretely. It’s such a big country - how can you retard progress across the whole thing, especially given that state capacity was presumably weaker in the past? Or at least I assume it was - what did a provincial bureaucrat actually do day-to-day? Was there a price system? Private property? How did the state actually interfere with merchants and artisans?

Economics

• There’s something unsatisfying about the arguments that we’ll see 20%+ explosive economic growth from AI. Even if true, what does that mean? What is actually happening? I thought Machines of Loving Grace was a great account of what plausibly is happening on the human facing side of the singularity - aka the FLOPs that are going towards curing disease. But presumably most of what is happening is investment towards more robots, more compute, etc. My sense of what that side of things looks like is so murky and handwavy. There is a version of Machines of Loving Grace you can do that is somewhat concrete about all the sci fi shit - not just gesturing at the galaxies, but getting specific about the space GPUs and factorio like solar tiling and all the other things I’m not thinking of which are relevant to understanding 2040. Presumably the right guest is someone who is really strong in engineering/physics and economics and has a penchant for sci-fi and has a lot of concrete ideas here.

• What should India or Nigeria or for that matter any country not directly in the semiconductor/foundation model supply chain do right now? If the main mechanism of catchup growth goes away (namely, that the underutilized labor of developing countries can rapidly be made more productive with capital and know-how from the developed world), what happens to all these countries that are not China or the US?

Max Hodak’s theory of consciousness

I’m totally gonna butcher this - please excuse. If you wanna get the real deal, go check out his summary blog post and his full talk on this topic.

Max is focused on two big sub-questions which together form “the binding problem”:

• Mode binding: how do color, shape, texture, and motion get combined into a unified visual percept of “a red cup”?

• Moment binding: why do we experience all the neurons firing across our entire brain over the course of 10s of milliseconds as a single quanta of experience?

Max thinks each of these binding sub-problems is related to a brain wave:

• Gamma waves - 40 Hz - Fast, local coordination of nearby neurons to get on the same page about what they’re representing.

• Alpha waves - 10 Hz - Slower waves that run through the whole brain and unify experience - think of these like the forward pass of the brain.

  • Two cool things about alpha waves I hadn’t realized. 1. that neurons ride the peak of this oscillation 2. when alpha waves slow down or speed up (fight or flight reactions, etc), people experience time dilation.

Anyways, Max points out that the brain is storing a bunch of structured representations about the world physically, and some feedback controller has to go in and make sure that these representations are correct. This is part of what the alpha waves are doing. And this feedback control and binding is consciousness. I’m glossing over a bunch of logical connections that I definitely don’t understand. But I’ll leave it here.

I know Max could provide a really good answer, but just talking to myself, I’m confused on what the reason is to think that feedback control = consciousness? By this logic, does memory refresh = consciousness too?

Max thinks that figuring out what’s up with consciousness will mean discovering new physics. And specifically, physics at the level of the 4 fundamental forces - some property as basic as mass or charge. His logic is that either consciousness has no real impact on the world (it’s just a byproduct of other stuff the brain does), which would be odd, or it actually has an effect, which would mean it’s new physics.

I’m not sure I buy this. 1. Can’t it be an effect that’s best understood at an implication of existing laws of physics - the fact that wood floats on water has an impact on the world, but you don’t need new physics to explain it 2. Doesn’t it seem implausible that evolution blindly stumbled upon and is now making good use of a whole undiscovered physical field which we have never managed to actually interact with using our technology, nor seen summoned anywhere else in the universe?

Nonlinear dynamics and Chaos by Steven Strogatz

I’m only 3 chapters in, so I’ve only got the building blocks so far. The fundamental idea is this. It’s often hard to anticipate how a system will evolve just by observing a bunch of different trajectories over time. But it’s much easier to see what will happen if you plot how the system will evolve from different starting points. The examples get more and more interesting, and because Strogatz focuses on the graphical and geometric interpretations, the motivating problems are super satisfying; the book is really a bunch of 3Blue1Brown videos on a certain topic stapled together.

Side note: I could not have understood anything here if I didn’t have LLMs and couldn’t watch the lectures async. I paused every minute or so (to clarify some confusion with a chatbot or to try and anticipate the next step), and I had the same section of textbook open at the same time.

I’m now wondering to myself, “How the hell did I learn anything in college at all?” I would be so lost if I was actually taking this course in college and just attending the lectures live.

In college, I actually did bounce out of a difficult course I feel like I could totally learn today with LLMs and async lectures + my adult executive function.

As I was working through these examples (some inspired by actual papers), I kept thinking about what parts the “automated cleverness” (Terry Tao’s term) of today’s AIs could actually help with.

It’s crazy how much understanding you can get about a physical system through mathematics. But that understanding is so dependent on insight and interpretation.

To give one example, Section 3.7 has a really clever model of an insect outbreak, showing how budworms, birds, and trees play out against each other given different growth rates and other dynamics.

But first you have to figure out the right dimensionless forms. And that requires judgment about which dimensions actually matter. In the insect model, the choice was to think in terms of R and K and treat the bird population as basically an artifact of those parameters. But you could have done it the other way around—from the basis of birds.

Then there’s how you make the visualization. Once you’ve got the dynamics in dimensionless form, you could just graph the equation and find the fixed points. But the result would be almost impossible to interpret. Graph it a different way, though, and suddenly the intercepts align with your intuition. You can actually see the three regimes: where carrying capacity is so low the population never gets going, where birds keep things in check, and where the outbreak has outgrown the birds’ ability to control it.

This kind of insight is inseparable from understanding what you’re even trying to learn about the system. And I’m skeptical today’s AI helps much here. When these methods were first developed, the right forms and interpretations weren’t obvious. The mathematician who wrote the original paper had to come up with new insights about how to think about the problem.

Maybe models are now good enough to apply these methods to new systems that fit the same template. But that just means the few mathematicians who invent genuinely new frameworks are the only ones who stay relevant.

Machines of Loving Grace by Dario Amodei

Starting with the biology section: Dario argues that we’ll get a century of bio progress in a few years. His argument:

• Most bio progress is driven by breakthrough discoveries which give you whole new primitives for what you can measure, change, or predict (CAR-T therapy, mRNA vaccines, CRISPR, genome sequencing costs declining so much, etc).

• These discoveries seem to have been made in scrappy haphazard ways, often years after they were initially possible, and often by people responsible for other breakthroughs as well. All 3 of these observations hint that they are bottlenecked by intelligence.

• Dario acknowledges that data is a huge bottleneck for bio. But the tools we have for collecting data can also be expanded by intelligence. Human researchers came up with multiplexing and AlphaFold and Perturb-Seq - the AI researchers will come up with even more.

Here’s the counterargument. The kinds of human researcher breakthroughs he uses as examples of what AI could do more of haven’t had a huge impact on health. Over the last 3 decades, we’ve seen a million-fold reduction in genome sequencing costs, 1000-fold decrease in DNA synthesis costs, the development of precise gene editing tools like CRISPR, and the ability to conduct massively parallel experiments through multiplexing techniques. But it doesn’t seem like we’re curing diseases or coming up with new treatments at a faster rate now than we were 30 years ago. If anything, drug development is slowing down. Why think that AI will be able to fundamentally change this dynamic?

Relatedly, Jacob Trefethan has an excellent blog post makes the the argument that AI won’t speed up medical progress that much (he also steelmans the opposite point in this other post). Jacob points out that making a drug to cure something like Alzheimer’s is really hard. Raw understanding of some of the disease life cycle (which more intelligence could give you more of) is not enough. We understand that Alzheimer’s is clearly linked to Amyloid beta, and there are now many different drugs trying to remove amyloid plaques which have all not worked. Even if we get more insights like the Amyloid beta thing from AI scientists, that alone will not be enough to identify the correct targets. You just have to do a bunch of experiments on live humans.

This is why Dario’s point about clinical trials falls flat. He argues that clinical trials are currently slow because we just don’t know whether a given drug will actually work. But if we had much greater confidence, like we did with the mRNA vaccines for COVID, then we could test and approve drugs much faster. However, I don’t see why we should think that modulo the full hyperrealistic simulation of the human body, we could tell ex ante which drugs are gonna work. I don’t yet buy the argument that a million George Church clones in a datacenter could derisk all the drug trials

Quick notes on other parts of the essay:

• Overall I find it pretty impressive that a tech CEO is this generally thoughtful.

• The poverty and econ section doesn’t address that the main mechanism of catchup growth goes away post AGI; namely developing countries have lots of underutilized labor which is bottlenecking production, and because the marginal product of labor is high in the world today, those countries can get rich fast. So how exactly are these other countries catching up?

• The key point that underlies his framework that intelligence can drive a century of progress in 5-10 years : “Things that are hard constraints in the short run may become more malleable to intelligence in the long run. For example, intelligence might be used to develop a new experimental paradigm that allows us to learn in vitro what used to require live animal experiments, or to build the tools needed to collect new data (e.g. the bigger particle accelerator), or to (within ethical limits) find ways around human-based constraints (e.g. helping to improve the clinical trial system, helping to create new jurisdictions where clinical trials have less bureaucracy, or improving the science itself to make human clinical trials less necessary or cheaper).”

  • it’s interesting to consider why this isn’t true for factors of production today. We live in a (relatively) capital-abundant and labor-scarce world. That is reflected in the labor share of income being 2x as high as the capital share of income. But this has been true for centuries upon centuries. Contra Piketty in “Capital in the 21st Century”, all these capital holders have not been able to get some runaway capital accumulation process going by figuring out a way around labor constraints. Why think that intelligence will be any different than capital in its ability to get around other factors of production? maybe the argument is that intelligence can actually help generate the other factors of production in a way that capital can’t.

Neural network training makes beautiful fractals by Jascha Sohl-Dickstein

Absolutely fascinating blog post.

You want to train your model at the highest possible learning rate under which it still converges. But the boundary of convergence versus divergence is fractal, which makes these hyperparameters really hard to optimize for via gradient descent.

Now you can ask the question: evolution somehow found the right hyperparameters to train our brains. How did evolution solve this wicked problem? Presumably because gradient free optimization fares better against these kinds of fractal landscapes - if you optimize for the part of the region where the average speed of convergence is high (rather than just take the gradient from a specific point that’s bounded in an unpredictable way by fractals), seems like you could do much better.

Backing up, why is the meta-loss landscape fractal in the first place? Jascha’s explanation is that fractals often emerge when iteratively applying a function. Gradient descent on the parameters is one such function that you iterate across training steps. But then the follow up question is this. There’s lots of other iterative functions you could think of, even within the context of neural networks. Do they all lead to fractals? For example:

• In chain of thought, you apply a model to a string, which makes a new string, to which you apply the model, etc.

• RNNs keep applying the same parameters to the hidden state.

Over conversation, an AI researcher friend revealed that CoT and RNNs both have variance problems that could well be explained by these fractal like dynamics. Though I only understand this claim at a hand-wavy level.

I’m confused why some people have short timelines and at the same time are bullish on the current scale up of reinforcement learning atop LLMs. If we’re actually close to a human-like learner, this whole approach of training on verifiable outcomes is doomed.

Currently the labs are trying to bake in a bunch of skills into these models through “mid-training” - there’s an entire supply chain of companies building RL environments which teach the model how to navigate a web browser or use Excel to write financial models.

Either these models will soon learn on the job in a self directed way - making all this pre-baking pointless - or they won’t - which means AGI is not imminent. Humans don’t have to go through a special training phase where they need to rehearse every single piece of software they might ever need to use.

Beren Millidge made interesting points about this in a recent blog post:

When we see frontier models improving at various benchmarks we should think not just of increased scale and clever ML research ideas but billions of dollars spent paying PhDs, MDs, and other experts to write questions and provide example answers and reasoning targeting these precise capabilities ... In a way, this is like a large-scale reprise of the expert systems era, where instead of paying experts to directly program their thinking as code, they provide numerous examples of their reasoning and process formalized and tracked, and then we distill this into models through behavioural cloning. This has updated me slightly towards longer AI timelines since given we need such effort to design extremely high quality human trajectories and environments for frontier systems implies that they still lack the critical core of learning that an actual AGI must possess.

You can see this tension most vividly in robotics. In some fundamental sense, robotics is an algorithms problem, not a hardware or data problem — with very little training, humans can learn how to teleoperate current hardware to do useful work. So if we had a human like learner, robotics would (in large part) be solved. But the fact that we don’t have such a learner makes it necessary to go out into a thousand different homes to learn how to pick up dishes or fold laundry.

One counterargument I’ve heard from the takeoff-within-5-years crew is that we have to do this cludgy RL in service of building a superhuman AI researcher, and then the million copies of automated Ilya can go figure out how to solve robust and efficient learning from experience.

This gives the vibes of that old joke, “We’re losing money on every sale, but we’ll make it up in volume.” Somehow this automated researcher is going to figure out the algorithm for AGI - a problem humans have been banging their head against for the better part of a century - while not having the basic learning capabilities that children have? I find this super implausible.

Besides, even if you think the RLVR scaleup will soon help us automate AI research, the labs’ actions suggest otherwise. You don’t need to pre-bake the consultant’s skills at crafting Powerpoint slides in order to automate Ilya. So clearly the labs’ actions hint at a world view where these models will continue to fare poorly at generalizing and on-the-job learning, thus making it necessary to build in the skills that they hope will be economically valuable.

Another counterargument you could make is that even if the model could learn these skills on the job, it is just so much more efficient to build them up just once during training rather that again and again for each user or company. And look, it makes a lot of sense to just bake in fluency with common tools like browsers and terminals. Indeed one of the key advantages that AGIs will have is this greater capacity to share knowledge across copies. But people are underrating how much company and context specific skills are required to do most jobs. And there just isn’t currently a robust efficient way for AIs to pick up those skills.

Human labor is valuable precisely because it’s not shleppy to train

I was at a dinner with an AI researcher and a biologist. The biologist said she had long timelines. We asked what she thought AI would struggle with. She said her work has recently involved looking at slides and decide if a dot is actually a macrophage or just looks like one. The AI researcher says, “Image classification is a textbook deep learning problem—we could easily train for that.”

I thought this was a very interesting exchange, because it revealed a key crux between me and the people who expect transformative economic impacts in the next few years. Human workers are valuable precisely because we don’t need to build schleppy training loops for every small part of their job. It’s not net-productive to build a custom training pipeline to identify what macrophages look like given the way this particular lab prepares slides, then another for the next lab-specific micro-task, and so on. What you actually need is an AI that can learn from semantic feedback or from self directed experience, and then generalize, the way a human does.

Every day, you have to do a hundred things that require judgment, situational awareness, and skills & context learned on the job. These tasks differ not just across different people, but from one day to the next even for the same person. It is not possible to automate even a single job by just baking in some predefined set of skills, let alone all the jobs.

In fact, I think people are really underestimating how big a deal actual AGI will be because they’re just imagining more of this current regime. They’re not thinking about billions of human-like intelligences on a server which can copy and merge all their learnings. And to be clear, I expect this (aka actual AGI) in the next decade or two. That’s fucking crazy!

Economic diffusion lag is cope for missing capabilities

Sometimes people will say that the reason that AIs aren’t more widely deployed across firms and already providing lots of value (outside of coding) is that technology takes a long time to diffuse. I think this is cope. People are using this cope to gloss over the fact that these models just lack the capabilities necessary for broad economic value.

Steven Byrnes has an excellent post on this and many other points:

New technologies take a long time to integrate into the economy? Well ask yourself: how do highly-skilled, experienced, and entrepreneurial immigrant humans manage to integrate into the economy immediately? Once you’ve answered that question, note that AGI will be able to do those things too.

If these models were actually like humans on a server, they’d diffuse incredibly quickly. In fact, they’d be so much easier to integrate and onboard than a normal human employee (they could read your entire Slack and Drive in minutes and immediately distill all the skills your other AI employees have). Plus, hiring is very much like a lemons market, where it’s hard to tell who the good people are, and hiring someone bad is quite costly. This is a dynamic you wouldn’t have to worry about when you just wanna spin up another instance of a vetted AGI model.

For these reasons, I expect it’s going to be much much easier to diffuse AI labor into firms than it is to hire a person. And companies hire lots of people all the time. If the capabilities were actually at AGI level, people would be willing to spend trillions of dollars a year buying tokens (knowledge workers cumulatively earn 10s of trillions of dollars of wages a year). The reason that lab revenue are 4 orders of magnitude off right now is that the models are nowhere near as capable as human knowledge workers.

Goal post shifting is justified

AI bulls will often criticize AI bears for repeatedly moving the goal posts. This is often fair. AI has made a ton of progress in the last decade, and it’s easy to forget that.

But some amount of goal post shifting is justified. If you showed me Gemini 3 in 2020, I would have been certain that it could automate half of knowledge work. We keep solving what we thought were the sufficient bottlenecks to AGI (general understanding, few shot learning, reasoning), and yet we still don’t have AGI (defined as, say, being able to completely automate 95% of knowledge work jobs). What is the rational response?

It’s totally reasonable to look at this and say, “Oh actually there’s more to intelligence and labor than I previously realized. And while we’re really close to (and in many ways have surpassed) what I would have defined as AGI in the past, the fact that model companies are not making trillions is revenue clearly reveals that my previous definition of AGI was too narrow.”

I expect this to keep happening into the future. I expect that by 2030 that the labs will have made significant progress on my hobby horse of continual learning, and the models will start earning 100s of billions in revenue, but they won’t have automated all knowledge work, and I’ll be like, “We’ve made a lot of progress, but we’re not at AGI yet. We also need X, Y, and Z thing to get to trillions in revenue.”

Models keep getting more impressive at the rate the short timelines people predict, but more useful at the rate the long timelines people predict.

RL scaling is laundering the prestige of pretraining scaling

With pretraining, we had this extremely clean and general trend in improvement in loss across multiple orders of magnitude of compute (albeit on a power law, which is as weak as exponential growth is strong). People are trying to launder the presitge of pretraining scaling, which was almost as predictable as a physical law of the universe, to justify bullish projections about RLVR, for which we have no well fit publicly known trend. When intrepid researchers do try to piece together the implications from scarce public datapoints, they get quite bearish results. For example, Toby Ord has a great post where he cleverly connects the dots between different o-series benchmark charts, which suggested “we need something like a 1,000,000x scale-up of total RL compute to give a boost similar to a GPT level”.

Comparison to human distribution will make us at first overestimate (and then underestimate) AI

There is huge variance in the amount of value that different humans can add, especially in white collar with its O-ring dynamics. The village idiot adds ~0 value to knowledge work, while top AI researchers are worth billions of dollars to Mark Zuckerberg.

AI models at any given snapshot of time, however, are roughly equally capable. Humans have all this variance, whereas AI models don’t. Because a disproportionate share of value-add in knowledge work comes from the top percentile humans, if we try to compare the intelligence of these AI models to the median human, then we will systematically overestimate the value they can generate. But by the same token, when models finally do match top human performance, their impact might be quite explosive.

Broadly deployed intelligence explosion

People have spent a lot of time talking about a software only singularity (where AI models write the code for a smarter successor system), a software + hardware singularity (where AIs also improve their successor’s computing hardware), or variations therein.

All these scenarios neglect what I think will be the main driver of further improvements atop AGI: continual learning. Again, think about how humans become more capable at anything. It’s mostly from experience in the relevant domain.

Over conversation, Beren Millidge made the interesting suggestion that the future might look continual learning agents going out, doing jobs and generating value, and then bringing all their learnings back to the hive mind model, which does some kind of batch distillations on all these agents. The agents themselves could be quite specialized - containing what Karpathy called “the cognitive core” plus knowledge and skills relevant to the job they’re being deployed to do.

“Solving” continual learning won’t be a singular one-and-done achievement. Instead, it will feel like solving in context learning. GPT-3 demonstrated that in context learning could be very powerful (its ICL capabilities were so remarkable that the title of the GPT-3 paper is ‘Language Models are Few-Shot Learners’). But of course, we didn’t “solve” in-context learning when GPT-3 came out - and indeed there’s plenty of progress still to be made, from comprehension to context length. I expect a similar progression with continual learning. Labs will probably release something next year which they call continual learning, and which will in fact count as progress towards continual learning. But human level continual learning may take another 5 to 10 years of further progress.

This is why I don’t expect some kind of runaway gains to the first model that cracks continual learning, thus getting more and more widely deployed and capable. If you had fully solved continual learning drop out of nowhere, then sure, it’s “game set match”, as Satya put it. But that’s not what’s going to happen. Instead, some lab is going to figure out how to get some initial traction on the problem. Playing around with this feature will make it clear how it was implemented, and the other labs will soon replicate this breakthrough and improve it slightly.

There’ll also probably be diminishing returns from learning-from-deployment. Each of the first 1000 consultant agents are each learning a ton from deployment. Less so the next 1000. And is there such a long tail to consultant work that the millionth deployed instance is likely to see something super important the other 999,999 instances missed? In fact, I wouldn’t be surprised if continual learning also ends up leading to a power law, but with respect to the number of instances deployed.

Besides, I just have some prior that competition will stay fierce, informed by the observation that all these previous supposed flywheels (user engagement on chat, synthetic data, etc) have done very little to diminish the greater and greater competition between model companies. Every month (or less), the big three will rotate around the podium, with other competitors not that far behind. There is some force (potentially talent poaching, rumor mills, or reverse engineering) which has so far neutralized any runaway advantages a single lab might have had.

I originally titled my podcast The Lunar Society. I changed it to Dwarkesh Podcast eventually because people kept thinking it was a crypto podcast (”to the moon!!!”). I named it after The Lunar Society of Birmingham, an informal club that met in the late 18th century. Members included James Watt, Matthew Boulton, Erasmus Darwin, Joseph Priestley, and Josiah Wedgwood. These were the scientists, inventors, and philosophers who had made first contact with the Industrial Revolution which was just starting to take shape around them. And they discussed everything from steam engines to abolition to chemistry to education reform.

Someday people will look back on this period the way we look back on the Enlightenment. Great thinkers having important debates right as the world was about to undergo these massive technological, economic, and political revolutions. And some of these thinkers actually managed to get a couple of the big things right.

Whatever happens next, I want the debates to have happened on this podcast, and to have happened well.

We are moving from the age of podcasts to the age of essays

I wanna make essays a first class citizen of what I do. This is for a couple of reasons:

• Interviews are best when I have some take that I can bounce against my guest. You only get to see Federer’s skill when he’s rallying against a decent player, and certainly not if he’s just bouncing the ball against a wall.

• As AI becomes more and more closed off, the best people will not be in a position where they can explain their thinking clearly. This is why the Karpathy episode was so incredible. It’s rare to get an industry expert without any particular thing to pitch, and who can talk openly about the research. But I’m not aware of anyone else who is Karpathy-tier, and who is not obliged to keep his or her mouth shut about a couple of things.

• My essays have done much better than my expectations, in terms of reach, correctness and impact. I wrote the continual learning essay on a whim one afternoon, because I wanted to articulate why all these LLM scripts I’ve written for my business haven’t been helpful. And I’m still a little shocked to realize that I had stumbled upon (at least part of) what Ilya is working on at SSI. It’s not a crazy insight by any means, but it’s notable that you can just think about stuff, and there’s a good chance you’ll figure out what’s up. Btw, after I released the essay, both Sam Altman and Demis Hassabis have said that continual learning is a major bottleneck on the path to AGI. Of course, there’s no way to know whether they read my essay. But honestly, even if they hadn’t, I’d still be pretty stoked if I had independently pointed my finger at the exact same bottleneck as these guys, despite all their additional context.

• Which brings to my next point. I feel like there’s actually not that many secrets. The researchers and CEOs of the AI labs are a couple months ahead of you. This just doesn’t amount to any substantial secret knowledge that, if only you knew, you’d also have 2027 timelines. A ton of progress has been made in the last 3 years since ChatGPT, but none of it was super shocking based on the rumor mill and some connecting of the dots. And then there’s the big picture questions about AI’s impacts, where your thinking might very plausibly be much better than people at the labs, just because it takes time to think, and these people are busy running a damn company.

• Some of the questions I’m most interested in simply can’t be answered extemporaneously by any human being on the planet. They require knowledge across multiple different fields, and couple hours (to days) of crunching the numbers or thinking through shit.

• Because often enough my guests can’t just answer pretty complicated fractal questions in a satisfying way on the spot, I get frustrated with the whole enterprise. The main angst I’ve kept receding back to over and over is, “Okay what did I actually learn from this interview? And if I didn’t get that much concrete insight and understanding out of it, despite a week+ of research and hours of conversation, what hope is there for the audience? And if no one learned anything, what the fuck are we doing here?” I feel much essays survive this cynicism much better. For example, I’m often frustrated that social scientists won’t speculate with me about what their insights imply about AI civilization, or historians about how history might have turned out differently given different counterfactuals. But it’s ridiculous to count on a scholar who is thinking about AGI for first time in his life to start shooting off some galaxy brain implications from his theory. But I can go read their books, and use my understanding of the technology to come up with some hot takes.

• I can easily co-release my essays as narrations on my podcast and YouTube feed, so actually the essays are super complementary to this audio/video audience I’ve built up.

Gratitude

In the spirit of Thanksgiving: a lottery winner who then won another lottery is less lucky than I am.

Every once in a while, I’ll be grabbing dinner with a writer whose work I was obsessed with in college. And a part of me is just like, “What the fuck is happening right now?” Many of my greatest intellectual heroes are now my direct friends and teachers. My job is to spend a week learning about whatever I’m most interested in, and then talk to the world expert on that topic. A job I would pay to do has rewarded me - intellectually, financially, socially - beyond my wildest expectations. And there’s millions of people who are into this stuff! This audience contains some of the smartest people in the world, including many of the people I am a huge fan of. Then there’s my team. It’s unreal how talented, agentic, tasteful, and detail-oriented my colleagues are. I genuinely have no idea how I convinced people this good to come run a podcast.

But this is only half the problem. Here’s a simple way to compare the learning efficiency of reinforcement learning versus supervised learning:

Bits/FLOP = Samples/Flop * Bits/Sample.

What I haven’t heard people talk about is the other term in our equation: Bits/Sample. And for most of training, the information density per sample is way way lower for RL.

Subscribe now

Putting things in plain English

In supervised learning (aka pretraining), you’re just soaking up bits. Every token is a hint at the structure of language, and the mind crafting that language, and the world that mind is seeing. Early in training, when you have a totally random model, you’re just maximally uncertain over all of this content. So each token is just blowing your mind. And you’re getting this exact signal of how wrong you were about the right answer, and what parameters you need to update to be less wrong.

Suppose you start with a randomly initialized model, and you kickstart training. If you’re doing next-token-prediction using supervised learning on “The sky is”, the training loop goes, “It’s actually ‘blue’. You said the probability of ‘blue’ is .001%. Make the connections that were suggesting ‘blue’ way way stronger. Alright, next token.”

In RL with policy gradient, you upweight all the trajectories where you get the answer right, and downweight all the trajectories where you get the answer wrong. But a model that’s not already very smart is just astonishingly unlikely to get the answer right.

If you were doing next-token-prediction on “The sky is” with RL, the training loop would be something like, “Okay, ‘halcyon’ is wrong. Don’t do the thing that led to saying ‘halycon’ … Okay ‘serendipity’ is wrong …” Rinse and repeat this guesswork for somewhere around the number of tokens you have in your vocabulary (on the order of 100,000).

The details

Let’s think about how maximum bits/sample change as the pass rate (p) changes. Pass rate here means how likely you are to say the correct answer. To keep this simple, let’s say the answer is a token long. Then the pass rate when you have a totally untrained model is just 1/ (size of your vocabulary).

In supervised learning, you get told exactly what the right label is for each sample. The amount of new information you learn corresponds to how surprised you are to learn the correct answer - the lower your pass rate (aka prior probability of the correct answer), the more you learned from seeing the correct label. The basic formula for entropy tells us that you can learn -log(p) bits/sample from supervised learning.

In RL, you only get told whether you got the right answer or not. The amount of new information you can extract is bounded by how uncertain you are about this binary outcome. If you almost always pass (p ≈ 1) or almost always fail (p ≈ 0), each trial is very unlikely to surprise you. You’ll learn most when the probability of passing is like a coin toss (p ≈ 0.5). The basic formula for the information content of a binary random variable tells us that you can learn at most Entropy(p) = -p log(p) - (1-p) log(1-p)1 bits/sample from RL.

Okay let’s plot this.

Doesn’t look terrible. Yes, pretraining is much better for half of the pass rate range, but then RL is better for the other half. However, this graph is super misleading. Because what the power law (in scaling laws) implies is that you need an equivalent amount of compute to cross each order of magnitude improvement in the pass rate. If it took you X many FLOPs to go from 1/100,000 pass rate to 1/10,000, then it will take you X many FLOPs to go from 1/10,000 pass rate to 1/1,000. So, we should actually chart the pass rate on a log scale - again, to account for how each increment in the x-axis corresponds to the same number of FLOPs.

Oh boy, is that a sad picture. The regime where RL has comparable information density per sample to pre-training is this tiny slice at the very end of training, when you’ve got a pretty reasonable model anyways.

And again, I want to emphasize that this is totally separate from the point that getting a single sample from RL (aka unrolling a full trajectory before getting any signal) might take upwards of a million times more compute.

It’s even worse than this - variance

The situation for RL early in training is actually even worse than described above. When the pass rate is low, your gradient estimate is going to be incredibly noisy and unpredictable. Either you don’t sample the correct answer at all in your batch, in which you get almost no information. Or you do, and you get this giant spike. You’re getting jerked around, which is terrible for performant training.2

Interestingly, pretraining has the exact inverse problem. There, variance is super high at the END of training. As pretraining progresses, you exhaust more and more of the reducible loss (things your model can actually learn about the data). What remains is mostly the irreducible loss. The irreducible loss is the intrinsic unpredictability of internet text.

How should the prompt, “Bob’s favorite color is” end? Depends on Bob. There’s not some correct answer which your super smart model can actually get good at predicting. But your super smart model is still getting a gradient update on whatever random answer someone put on the internet. And this noise is drowning out the true signal that the couple of actually learnable tokens in the batch are giving you. I don’t know if this is accurate, but it seems like this explosion of variance at the end of pretraining is relevant to why batch sizes are increased as pretraining progresses.

Getting to the Goldilocks zone in RL

If RL works best in the regime where your pass rate is >>1%, then this raises the question, how can we construct the RL training to get (and keep) models in this learning flow state?

For example, we can think of pretraining AND inference scaling as increasing the pass rate during RL, allowing you to extract far more bits per sample.

It’s been noted that curriculum learning in not especially helpful for pretraining, but often essential for RL. This makes total sense when you think about how RL is only getting meaningful bits per sample in this Goldilocks zone of pass rate, so you really want to order the learning such that the difficulty of challenges increases in tandem with the model’s intelligence.

Our pass rate framework also gives us good intuitions for why self play has been so productive in the history of RL. If you’re competing against a player who is almost as good as you, you are balancing around a 50% pass rate, which peaks out the bits you get from a random binary variable.

But self play is not the only way we can imagine of keeping pass rate high through training. Perhaps we can come up with some proxy evaluation which is much more dense. Density here can mean one of two things:

1. Samples/FLOP density: You estimate the final reward using this proxy evaluation, but much earlier on in the episode, saving you the compute of unrolling the full trajectory. This is what a value function does.

2. Bits/Sample density: You come up with a proxy objective which is much easier to pass than the actual test under question. The simplest example I can think of is a process-reward model which says, “Hey, this rollout got the wrong answer, but I can see that its reasoning was on the right track at the start. So let’s up-weight those early tokens.”

Section 4.2 of the Deepseek R1 paper why so far, it’s been hard to develop useful proxy objectives like this for LLMs.

Fewer bits, sure, but very valuable bits

To be fair to RL, while you may be learning far fewer Bits/FLOP in RL, the bits you learn are very important. They are not apples-to-apples comparable to the bits in pretraining. This is for two key reasons:

1. Pre-training is teaching you what the data manifold of the internet looks like, which is only partially and indirectly related to, “How do I perform economically valuable tasks?” Whereas RL has the promise of giving you the good stuff directly.

2. Even if the pre-training corpus contains the instructions about how to accomplish a specific task, it does not have the thinking trace which teaches the model about how to correct its mistakes, or leverage its jagged and non-human repertoire of skills to accomplish the task.

The rebuttal is that those bits are only available for a small fraction of the pass rate range (again, weighted on a log scale to account for how pass rate is trash for most of training).

By the way, now we can understand all these claims about how RLVR is only eliciting the capabilities already latent in the pretrained model. Of course that’s the case. If the pretrained model didn’t have a high enough pass rate to begin with, then RL would have atrocious bits/sample, and thus not be able to learn at all. Move 37 is obviously one famous example where RL did teach a model a de-novo strategy. It’s worth noting that AlphaGo was trained on self play (see above re how self play increases pass rate), and that AlphaGo was surprisingly compute intensive for its time.

The jaggedness of RL

People have pointed out that RLVR is empirically just leading models to associate a thought pattern to a problem type rather than instilling a more general policy of stepping back and thinking through the best approach.

Think about it. How is it possible that we have models which are world-class at coding competitions but at the same time leave extremely foreseeable bugs and technical debt all throughout the codebase?

What explains this weird jaggedness? Perhaps RLVR can’t distinguish trajectories that were generated from a more generalizable procedure vs just greedily matching the problem shape to some associated thought process.

When you’re doing policy gradient rollouts, this more complex general policy is extremely unlikely to be ever be sampled, whereas the simple heuristic policy does get sampled and grows in frequency until it reaches fixation. Meanwhile, the general policy recedes further and further from sight.

Then the question is, how do we build a short bridge between simple heuristic solutions and the more complex general strategy? And will that bridge just spontaneously emerge as time horizons expand, thus potentially requiring generalization?

My concern is that this general policy of stepping back and making tasteful judgements based on your understanding of the world will continue to be hard to spot-light using verifiable rewards, even on longer time horizon tasks. And so the solution to this jaggedness will require a more robust training procedure, not just scaling RLVR.

Human learning

Here we’re only talking about the bits/sample learned from model free RL - aka from some binary outcome at the end of an episode. But of course humans are obviously learning way more efficiently than this. Think about a repeat entrepreneur. We say that she has a ton of hard-won wisdom and experience. Very little of that learning comes from the one bit of outcome from her previous episode (whether the startup succeeded or not).

It’s not clear what the ML analog is for human learning from experience. Clearly, our observations and reflections update our world model (independent of the outcome at the end). And this is playing a very important role in our learning.

Maybe we shouldn’t be asking how we model free RL to ≈50% pass rate, so that can squeeze out a full drop of information from the outcome. Maybe we should be asking, how do humans wring out the buckets of information from the environment?

Share

What would it take to make this vision happen? Is it even physically feasible in the first place? What would it mean for different energy sources, upstream CAPEX in everything from fabs to gas turbine factories, and for US vs China competition?

These are not simple questions to answer. We wrote this blog post to teach ourselves more about them. We were surprised by some of the things we learned.

Subscribe now

The fab CapEx overhang

With a single year of earnings in 2025, Nvidia could cover the last 3 years of TSMC’s ENTIRE CapEx.

TSMC has done a total of $150B of CapEx over the last 5 years. This has gone towards many things, including building the entire 5nm and 3nm nodes (launched in 2020 and 2022 respectively) and the advanced packaging that Nvidia now uses to make datacenter chips. With only 20% of TSMC capacity1, Nvidia has generated $100B in earnings.

Suppose TSMC nodes depreciate over 5 years - this is enormously conservative (newly built leading edge fabs are profitable for more than 5 years). That would mean that in 2025, NVIDIA will turn around $6B in depreciated TSMC Capex value into $200B in revenue.

Further up the supply chain, a single year of NVIDIA’s revenue almost matched the past 25 years of total R&D and capex from the five largest semiconductor equipment companies combined, including ASML, Applied Materials, Tokyo Electron...

We think this situation is best described as a ‘fab capex’ overhang.

The reason we’re emphasizing this point is that if you were to naively speculate about what would be the first upstream component to constrain long term AI CapEx growth, you wouldn’t talk about copper wires or transformers - you’d start with the most complicated things that humans have ever made - which are the fabs that make semiconductors. We were stunned to learn that the cost to build these fabs pales in comparison to how much people are already willing to pay for AI hardware!

Nvidia could literally subsidize entire new fab nodes if they wanted to. We don’t think they will actually directly do this (or will they, wink wink, Intel deal) but this shows how much of a ‘fab capex’ overhang there is.

Upstream suppliers to datacenters will have to expand production

For the last two decades, datacenter construction basically co-opted the power infrastructure left over from US deindustrialization. One person we talked to in the industry said that until recently, every single data center had a story. Google’s first operated data center was across a former aluminum plant. The hyperscalers are used to repurposing the power equipment from old steel mills and automotive factories.

This is honestly a compelling ode to capitalism. As soon as one sector became more relevant, America was quickly and efficiently able to co-opt the previous one’s carcass. But now we are in a different regime. Not only are hyperscalers building new data centers at a much bigger scale than before, they are building them from scratch, and competing for the same inputs with each other - not least of which is skilled labor.

For you to deploy $2 trillion of AI CapEx a year, someone else needs to be producing $2 trillion worth of all the other datacenter components, not just the chips. The people upstream to the datacenters - the companies producing everything from copper wire to turbines to transformers and switchgear - would need to build more factories.

The problem is that those factories have to be amortized over a many decade lifespan. At usual margins, those factories are only worth building if this AI demand lasts for 10-30 years.

The companies building those factories are not AGI pilled - they’re decades old industrials running low margin businesses which have been burned by many previous swings in demand. In the early 2000s, electricity demand seemed set to explode, so gas turbine manufactures like GE, Siemens, and others massively expanded manufacturing capacity. Then demand collapsed, leaving them with huge (almost bankrupting) overcapacity.

If there’s a financial overhang not just for fabs, but also for other datacenter components, could hyperscalers simply pay higher margins to accelerate capacity expansion? Especially given that chips are currently 60-70% of datacenter CapEx, the hyperscalers might just tell the companies which are building the other 30% not to worry about long run demand: “We’ll make you whole with just a couple of years of outrageous margins.”

The largest gas turbine manufacturers (GE Vernova, Siemens Energy, and Mitsubishi Electric) are expecting to make $100B from gas turbines over the next 5 years. That corresponds to around 100GW of generation capacity. These companies are doing a combined CapEx of around $5B/yr across all their divisions.

If the hyperscalers were willing to pay $200B instead of $100B for this 100GW of generation (for example, to incentivize faster delivery), they’d effectively be covering 20 years of these turbine manufacturers’ entire CapEx (at current rates).

To build 100 GW of datacenters, the hyperscalers are going to have to invest many trillions of dollars anyways. Power generation is only about 7% of a datacenter’s cost. If natural gas ends up being the fastest way to scale generation, then doubling the cost of generation to 14% in order to make sure the power comes online fast enough might be easily worth it.

We think a similar dynamic is probably true for most of the components in a data center. Do not underrate the elasticity of supply.

The coming labor bottleneck?

Labor might actually end up being the most acute shortage - we can’t simply stamp out more workers (at least, not yet).

The 1.2 GW Stargate facility in Abilene has a workforce of over 5,000 people. Of course, there will be greater efficiencies as we scale this up, but naively that looks like 417,000 people to build 100 GW. And that’s on the low end of 2030 AI power consumption estimates. We’re gonna need stadiums full of electricians, heavy equipment operators, ironworkers, HVAC technicians,... you name it.

For reference, there’s 800K electricians and 8 million construction workers in the US. We hear that this labor pool is aging fast, but at least over the next few years, it seems like reallocation and big salary offers should be able to ameliorate the labor bottleneck.

$400B+ ARR by end of decade is plausible

Anthropic and OpenAI’s combined AI CapEx per year (being done indirectly, mostly by Amazon and Microsoft in 2025) seems to be around $100B2.

Revenues for OpenAI and Anthropic have been 3xing a year for the past 2 years. Together, they are on track to earn $20B in 2025.

This means they’re spending 5 times as much on CapEx as they’re earning in revenue. This will probably change over time - more mature industries usually have CapEx less than sales. But AI is really fast growing, so it makes sense to keep investing more than you’re making right now.

Currently, America’s AI CapEx is $400B/year. For AI to not be a bubble in the short term, the datacenters currently being built right now need to generate $400B in revenue over their lifetime. Will they?

Google, Facebook, etc. have already shown us that if you can make a product which is modestly useful to billions of people, you can generate $100s of billions in revenue a year (Google+Meta make $400B/yr from ads alone).

OpenAI is approaching 1 billion, unmonetized free users, and we think a $12B to $100B revenue scale up is plausible just from their current products (e.g., see this vision: GPT-5 Set the Stage for Ad Monetization and the SuperApp). The question lies more in whether they can make a GPT-6 (or other products) in 3-5 years time that looks promising and economically useful enough to bring them into the $400B+ revenue range.

Of course, questions of revenue ultimately come down to your timelines. If AI truly lives up to its promise, then it’s in the reference class (at the very least) of white-collar wages, which are $10s of trillions of dollars a year.

Do you think that AI models will be able to do much of what a software engineer does by the end of a decade? If the 27M Software engineers worldwide are all on super charged $1000/month AI agent plans that double their productivity (for 10-20% of their salary), that would be $324B revenue already.

Lead times

It takes about two years to build a new GW+ datacenter—and that’s before factoring in time to debug new chip generations. This means if you want to deploy 2 trillion in capex in 2030, you need to plan that out in 2028. At current trends in perf/watt and perf/$, $2T corresponds to roughly 66 GW of AI datacenter capacity.

For the last few years, hyperscalers and labs have consistently wanted more compute than they had previously made plans to develop. If the hyperscalers are planning for a 30% CAGR over the next 5 years and instead end up wanting to average 40% (hitting $2T capex in 2030) that’s a 20GW gap they’ll have to close in 2030 relative to the long term plans.

Elon (who didn’t even have an AI company during the time in which a traditional hyperscaler would have had to precommit to building capacity) solved his constraints by doing insane things. It’s unclear whether mere mortals can assemble 60 GW of greenfield capacity when they’d only planned for 30 or 40 GW.

If AI demand continues outpacing advance planning, there will be enormous pressure to compress datacenter construction timelines. The question is: what does this mean for energy sources and datacenter design? Some energy sources have much longer lead times than others, and some datacenter designs are more amenable to rapid deployment.

Chips overwhelm everything else in the total cost of ownership of a datacenter. This is because 1. Chips are really expensive, and 2. The data center shell can be depreciated over 12-20 years, whereas the chips are fully depreciated (and have to be replaced) every 3 years.

So in terms of choosing an energy source, you can see very clearly what the order of priority should be.

1. Lead times - Every month that the shell is not set up is a month that the chips (which are the overwhelming majority of your cost) aren’t being used.

2. Non chip CapEx - Much more expensive than electricity OpEx over a 3 year period.

3. Electricity OpEx

So you can see why natural gas, for example, is much preferred over current nuclear reactors. Nuclear has extremely low op-ex, but has extremely long lead times and high CapEx. Natural gas may not be renewable, but you can just set up a couple dozen gas turbines next to the datacenter, and get your chips whirring fast.

Solar panels themselves are very cheap, but can be expensive to levelize across night + seasons. If you’re going pure solar, you have to build a bunch of overcapacity (4-7 GW solar capacity for 1 GW data center due to typical 15-25% capacity factor) and add a ton of batteries. Otherwise, you’re risking your expensive chips sitting idle during winter or at night.

Solar farms also have a massive land footprint, and require massive labor for installation – good luck hiring 30,000 people to lay out 20,000 acres of solar panels and batteries across the desert in order to power a single 1 GW datacenter3. The largest solar park in the world is currently the Gonghe Talatan Solar Park. It generates enough electricity for around 3 GW of smoothed continuous power - but this requires 15 GW peak capacity - meaning 7.2 million solar panels - that’s the area of seven Manhattans.

Aside from lead times and costs, you could ask the question: which energy source is physically plentiful enough to supply this demand? The answer, it turns out, is literally all of them. The theoretical limits of any of the energy sources mentioned are orders of magnitude higher than what is needed for even the most explosive end of decade AI scenarios.

Off grid?

A key question is whether datacenters will go “off-grid”—generating power on-site rather than connecting to the utility grid. Some of the largest datacenters are already doing this, e.g., Meta’s Orion or XAI’s Colossus.

Why would datacenters want to make power themselves rather than relying on the grid? They’re trying to get around interconnection delays. Connecting large new electricity sources to the grid now takes over 5 years.

For 20+ years, US electricity consumption has been either flat or growing slowly. Grid operators are now expecting huge increases in demand from AI datacenters, manufacturing reshoring, and electrification all happening simultaneously.

One potential workaround: a Duke study found that if datacenters agreed to curtail load just 0.25% of the time (roughly 22 hours per year), 76 GW of spare transmission capacity could be made available. Most transmission lines run well below capacity on average—the bottleneck only hits during peak demand.

But even if hyperscalers are able to perfectly capture this 76 GW, that only gets you from 2026-2028 in bullish AI scenarios. After that, either the grid expands or datacenters go off-grid.

Distribution of datacenter sizes

What will the distribution of individual datacenter sizes be? Here’s the argument for why we might end up seeing what looks like a thick sprinkle of 100 MW datacenters everywhere:

• If you can plop down a medium sized datacenter here and there, you can soak up any excess capacity in the grid. You can do this kind of arb with a 100 MW datacenter, but there’s no local excess capacity in the grid at the scale of 1 or 10 GW - that much power is on the scale of a whole grid itself!

• For pretraining like learning, you want to have large contiguous blobs of compute. But already we’re moving to a regime of RL and midtraining, where learning involves a lot of inference. And the ultimate vision here is some kind of continual learning, where models are widely deployed through the economy and learning on the job/from experience. This seems compatible with medium sized datacenters housing 10s of thousands of instances of AIs working, generating revenue, and learning from deployment.

Here’s the other vision. 1-10 GW datacenters, and then inference on device. Basically nothing in between.

• If we move to a world with vertically integrated industrial scale production of off-grid datacenters, maybe what you want to do is just buy a really big plot of land, build a big factory on site to stamp out as many individual compute halls and power/cooling/network blocks as possible. You can’t be bothered to build bespoke infrastructure for 100 MW here and there, when your company needs 50 GW total. A good analogy might be how a VC with billions to deploy won’t look at any deal smaller than deca millions.

A machine that spits out a GW a week

Today’s datacenter construction resembles building a car in your driveway: the engine ships from Germany, transmission from Japan, harness from Detroit, and a mechanic spends months assembling it all on-site. Each datacenter is custom-built over 1-2 years, with networking, MEP systems, and racks assembled piece by piece.

You’re not getting to a gigawatt per week this way.

Could you have pre-fabricated compute halls? Fully wired racks, cooling systems, power equipment, batteries—assembled in factories and shipped as complete modules. Instead of 18 months of on-site construction, you’re sliding skids into place.

The design space is surprisingly flexible. Liquid cooling enables 500kW-1MW racks but requires totally different plumbing and construction. If solar dominates, maybe you go full DC-to-DC (panels generate DC, chips need DC) and skip all the AC conversion steps. Each design choice cascades into others—power source affects cooling approach affects rack density affects building design.

By the way, if AI turns out to be a bubble and we’re much further from AGI than Silicon Valley thinks, what lasting value gets built? You could tell a story about how the dot-com bubble paved the way for all the value the internet has generated. What’s the equivalent for this AI buildout?

The GPUs—70% of capex—are worthless after 3 years. The buildings and power infrastructure last decades but are overbuilt for non-AI workloads. Perhaps the enduring value is this new industrial capability: the ability to rapidly manufacture and deploy massive compute infrastructure on demand. Like how the dot-com bubble left us with fiber in the ground, maybe an AI bubble leaves us with an industrialized datacenter supply chain and an expanded electric grid.

Crypto actually did this for the current wave of AI. For example, Crusoe - which is helping OpenAI build out Stargate in Abilene - was founded to build Bitcoin mining datacenters on stranded natural gas.

Does China win by default on long timelines?

Why doesn’t China just win by default? For every component other than chips which is required for this industrial scale ramp up (solar panels, HV transformers, switchgear, new grid capacity), China is the dominant global manufacturer. China produces 1 TW of solar PV a year, whereas the US produces 20 GW (and even for those, the cells and wafers themselves are manufactured in China, and only the final module is assembled in the US).

Not only does China generate more than twice the electricity than the US, but that generation has been growing more than 10 times faster than in the US. The reason this is significant is that the power build out can be directed to new datacenter sites. China State Grid could collaborate with Alibaba, Tencent, and Baidu to build capacity where it is most helpful to the AI buildout, and avoid the zero-sum race in the US between different hyperscalers to take over capacity that already exists.

Is China privileged in the long timelines world? SMIC will probably *eventually* catch up to TSMC (and maybe SMEE or SiCarrier to ASML, CXMT and YMTC to SK Hynix and Micron, NAURA and AMEC to Applied Materials, LAM, Tokyo Electron and KLA) - export controls won’t preserve the lead forever. If there’s not a software only intelligence explosion before 2030, and AI just becomes a massive industrial race across the entire supply chain from robotics to solar panels and batteries to steel, then why doesn’t China end up leading? Isn’t China’s differential advantage precisely these kinds of rapid and massive infrastructure build-outs?

Semianalysis projects that China will actually be able to ship fewer chips next year than it did this year, mainly because of domestic HBM production constraints. But we wonder how much that matters in the long to medium term.

GPUs are fully depreciated over 3 years (because new designs and better underlying process nodes make previous generations irrelevant). And lead times on building new datacenters are only a year or two long - datacenter design as a whole might be redone to accommodate industrial scale vertical integration.

All this makes us wonder whether every 3 years, the AI race starts totally fresh. While we might be able to constrain China’s production up till 2028 (or maybe even into the end of the decade with the lithography chokepoint), what’s the story for why this matters 2030 onwards?

Two scenarios - AI winter, and AI explosion

In order to get a handle on questions like these, we think it’s really helpful to just put numbers into a spreadsheet. Even if the scenarios are pulled out of your ass, they give you some sense of what would have to be true about the world in which they came to pass. For example, it’s interesting to ask: given trends in hardware price and performance, how much power would $2T CapEx correspond to in 2030? How about $500B?

We decided to chart two different potential trajectories:

1. Explosive growth, where AI investment grows smoothly and then starts to accelerate on the back of booming economic growth from AI automation (30% GDP growth in 2035).

2. AI winter, where Investment growth crashes around 2029 and then grows at a smooth 5% annual growth rate from 2032 onwards.

Here’s AI CapEx and AI power up to 2040 in both scenarios4:

In the explosive growth scenario, Sam Altman’s vision of 1 GW a week for the leading company comes true in 2036. But in that world, global AI power draw would be twice US’s current electricity generation.

We had a lot of fun building and playing around the spreadsheet that led to these scenario projections. You might too.

Concluding thoughts

Romeo and I decided to write this blog post because we had lots of questions about the AI buildout. We’re left with many more questions.

We don’t have any hard conclusions. What we’ve managed to do is assemble some considerations relevant to answering our questions.

There is no one person in the world who is the expert in all of the relevant fields. And even if there was such a person, they wouldn’t have a crystal ball. So we’ve spent a lot of time hoping on calls, reading PDFs, and talking to LLMs.

If you do have more information or thoughts, please reach out to us. We’re eager to learn more.

Share

About the authors

• Romeo is a researcher at AI Futures Project working on writing AI scenarios and recently graduated from Harvard. For their first scenario he focused on compute forecasts – now he’s also thinking about broader economic impacts for a new scenario with longer timelines and a positive vision for governance. If you have expertise in economics and an interest in future AI scenarios, please reach out at romeo@ai-futures.org.

• Dwarkesh is a podcaster. He’s keen to interview someone really good on all these topics. Reach out at hello@dwarkeshpatel.com.

  Subscribe now

In the meantime, I am sharing the notes I was taking as I try to comprehend his book, The Vital Question: Energy, Evolution, and the Origins of Complex Life.

The story the book tells of life’s evolution is both fascinating and super complicated. Hopefully there notes are helpful to others who are interested in this topic.

I wrote up these notes piecemeal, part by part, on Twitter, rather than all at once at the end. This blog compiles all of them in place.

Part 1 - Why eukaryotes are so special

In the intro he lists out the motivating questions:

Why are bacteria so relatively simple despite being around for 4 billion years? Why is there so much shared structure between all eukaryotic cells despite the enormous morphological variety between animals, plants, fungi, and protists? Why did the endosymbiosis event that led to eukaryotes happen only once, and in the particular way that it did? And why is all life powered by proton gradients?

Nick says all these questions are connected.

Lane says there’s 2 different philosophies on what bottlenecks evolutionary exploration: the niches made available by the environment, OR the internal structure necessary to exploit those niches.

Textbook view is that the environment constrains exploration, whereas structure is flexible and can accommodate once the right environment is in place. Nick Lane thinks it’s the opposite.

There’s been 2 big oxidation events - the first one (2.4 billion years ago) paved the way for eukaryotic cells. The second one (600 million years ago) led to the Cambrian explosion, resulting in all the variety in animals and plants and other complex life we see. So it seems the environment is central. Once you get a bunch of oxygen up in the air and into the oceans, you can start making all kinds of cool shit.

But hold on. Here’s what you’d expect to see if the environment was the key constraint: With this key unlock of aerobic respiration, different brands of bacteria independently evolve towards greater complexity to fill the new niches opened up (one masters osmotrophy and branches off into fungi, another photosynthesis, another phagocytosis, etc). However, you don’t see this.

Instead you see that all complex life emerges from a single common eukaryotic ancestor (2.2 billion years ago). There is no independent convergent evolution towards this kind of complexity (bacteria have had 4 billion years to evolve this kind of complexity, and have stayed remarkably similar through the whole time).

In fact, once you do get this key structural unlock, eukaryotic organisms proliferate widely, filling niches ranging from 100 feet long blue whales to 0.8 meter long picoplankton.

What’s more:

• The amount of shared structure between all eukaryotic cells is remarkable. They have almost all the same organelles and components. Nick writes:

  “Most of us couldn’t distinguish between a plant cell, a kidney cell and a protist from the local pond down the electron microscope.”

• There’s no intermediate proto-eukaryotes, which have some, but not all, of the functionality available to eukaryotic cells. This is wild given how evolution works. We have an extensive record of the incremental upgrades between photoreceptive amoebas and mammalian eyes. Why don’t we have proto-eukaryotic cells which reproduce via meiosis but don’t have compartmentalized nucleuses, or have mitochondria but no cytoskeleton?

Nick argues that the fact that no such subset of eukaryotic traits exists suggests that it is not structurally possible to survive with only some fraction of eukaryotic equipment - you need the whole package all at once.

Obviously this raised the question of how the whole package was evolved at once. Which I think he will address in future chapters.

Some questions for Nick:

• If his view is that structure was the main bottleneck, and we’ve had eukaryotes for 2.2 billion years, then why didn’t we have all these animals and shit for 2 billion years? Why did they only arise 600 million years ago (aka the Cambrian explosion)?

• Nick argues that eukaryotic cells are a much more significant unlock than multi-cellularity. Multi-cellularity evolved independently dozens of times, but we only have evidence of one event like the emergence of the first eukaryotic cell. If multi-cellularity evolved independently so many times (between fungi, slime molds, algae, etc etc), do we see interesting differences based on the situations in which they evolved? Do they regulate the differentiation of cells, the organization of the body differently, and communication between tissues differently? TODO look it up later.

A tangential thought. This whole debate about whether structure or environment matters more seems analogous to the discussion in ML of whether architecture or data matters more. And there it seems like data is quite crucial, but for meta-learning and generality to kick off, the architecture has to make it possible for information to flow in the right way. For example, in context learning is a kind of meta-learning that arises only once the model has the capability to attend to hundreds of previous tokens, which became tractable with transformers.

Part 2 - How the first cells evolved

His main argument here is that life is continuous with the planet’s geochemistry.

Aka a lot of the main characteristics of cells - membranes, enzymes, energy via proton gradients - descend from spontaneous processes in the Earth.

But you can’t have these characteristics evolve piecemeal in different locations. You need one location that houses all the processes which could then give rise to the first cell.

Important context, by the way, is that all life descends from a single common ancestor - LUCA (last universal common ancestor).

Okay, so what candidate environment could give rise to LUCA? It needs two main characteristics:

1. There’s a continuous flux of carbon and energy (in some sense, all life is a flux of carbon and energy, but you need some geochemistry to maintain this disequilibrium before the first cells can co-opt it).

2. Something which concentrates and catalyzes the reactions which lead to organics (aka inorganic equivalents of cells and enzymes).

This rules out a lot of old theories: a warm pond with ammonia and salts and the odd lightning bolt doesn’t drive continuous flux, nor concentrate early organics in a cell-like volume to drive forward reactions.

Nick thinks alkaline sea vents are a unique fit to this challenge, and also help explain a lot of the contingent biochemistry that all life ended up using because of our shared inheritance.

Okay, let’s dig in: and for context, basically Nick here is trying to explain how you end up with an early version of the reverse Krebs cycle spontaneously. Reverse Krebs cycle takes in H2 and CO2 and makes organic molecules that are the precursors of fatty acids, proteins, and sugars.

Another important bit of context: All life runs on proton gradients. Burning food with oxygen (or other oxidants in anaerobic respiration) pumps H+ ions across a membrane, like filling a dam. These ions flow back through ATP synthase—a molecular turbine—which harnesses the flow to attach phosphate to ADP, creating ATP. Your body contains just 60 grams of ATP, but the ATP→ADP→ATP cycle is so rapid you process your body weight in ATP daily.

Sidenote: If a solution is acidic, it means there’s a lot of H+ ions in it. And if it’s basic (aka alkaline), it means there’s a lot of OH- ions in it.

Okay so what was happening in these alkaline hydrothermal vents? There’s 3 sides to this picture: the inside of the vent, the vent wall, and the ocean side of the vent.

On the inside of the vent, you’ve got iron rich rock basically rusting, which lets out H2 and OH- into the stream of water piping through (aka making the water basic/alkaline).

The wall is made up of catalytic minerals like FeS, and also has a ton of tiny pores which connect the inside to the outside.

And the ocean side has a bunch of dissolved CO2 - early Earth was basically a giant ocean, but also had a lot of volcanoes that let out lots of CO2. And the oceans are quite acidic too, because CO2 becomes carbonic acid when dissolved in water.

Within the tiny pores inside these vents, you have H2 reacting with CO2 to form simple organics like formaldehyde (CH2O) and methanol (CH3OH), instigated by the FeS in the walls, which acts as a catalyst for this reaction.

Remedial chemistry: feel free to skip this para - I’m just going to include it since it took me some effort to relearn the high school chemistry involved. And it was quite satisfying to understand. Why do you need the H2 side inside to be basic? And why do you need the CO2 side outside to be acidic? My understanding is that in an alkaline solution, H2 -> H+ is favored, since the OH- (which definitionally makes the solution alkaline) really wants to react with H+ to make H2O. But now you’ve got some intermediate H+ lying around to be involved in other reactions. On the ocean side, the more acidic the water, the less likely that the marginal CO2 added will be turned into carbonic acid (since there’s so much of it around already) and will instead be available to react with.

Now that you’ve got these early organics building up inside these tiny pores, you can kick off this positive feedback loop where these early organics act as precursors or enzymes to make more and more of the molecules life uses. You build amino acids (which become enzymes for other reactions), and fatty acids (which spontaneously form membranes because they have hydrophobic heads and hydrophilic tails), and sugars, and peptides, and eventually DNA and RNA. Claude illustrates:

The fact that this early proto cell doesn’t have to generate proton gradients itself, and can just take advantage of the geochemical disequilibrium, is a huge boon: “Methanogens spend practically 98% of their energy budget on generating proton gradients by methanogenesis, and little more than 2% producing new organic matter. With natural proton gradients and leaky membranes, none of that excessive energy spend is needed. The power available is exactly the same but the overheads are cut by at least 40-fold, a very substantial advantage.”

In addition to the H+ gradient, which exists spontaneously in these vents, some protocells also started to extrude Na+ ions. And since there’s no natural gradient for these, this creates an incentive for developing non-porous membranes (and for proteins on that membrane to pump protons out). Once you develop such a membrane, you can exit this wall cavity and float around like a real cell.

Is the implication that inheritance only got kicked off at this point? Because beforehand, I guess you have selection amongst the pores, but you have no way to pass down traits. This buildup of organics and metabolism is happening independently across all the pores.

Yet you already had DNA and RNA by this point. So what was this genetic information doing before inheritance? I guess just organizing information to facilitate buildup of more organics?

Does this imply that there were millions of protocells with no shared lineage between them, each developing their own unique versions of all the basic biochemistry of life? LUCA just happened to be one that had DNA, RNA, and ATP synthase, but all 3 of those could have been wildly different based on which proto cells made it out of the nook first?

Yet the fact that these three building blocks are considered across all life suggests that they are uniquely well-engineered? Or maybe it means that evolution can’t effectively improve upon its foundations. The same way that backprop can find the best network to map a function, but can’t rewire the GPU you’re training it on at the same time. Anyways, once you have this proto cell, it can ‘infect’ contiguous vent systems all across the ocean floor.

Contingent biochemistry explained by this theory:

• Why all life is powered by proton gradients

• Why all carbon fixation pathways, whether they’re in bacteria, archaea, or eukaryotes, use acetyl-CoA as the entry point. It forms spontaneously at these vents when catalyzed by the FeS in the walls. And basically all life still uses this molecule to store energy and build other molecules.

• Why a lot of the enzymes involved in energy metabolism (and the Krebs cycle specifically) still use FeS minerals as their backbone

• Why Archaea and Bacteria (the two different kingdoms of eukaryotes) split up - apparently it has something to do with how they create proton gradients, but honestly the relevant biochemistry went over my head. Though this bifurcation is supposed to explain why all life shares DNA, RNA, and ATP synthase, but nothing else: not the cell membrane, nor the DNA replication enzymes, nor the pumps for excretion. Apparently all of these things were implicated in the different choice that archaea and bacteria made during this bifurcating event.

Questions for Nick:

• I guess this theory is incompatible with panspermia, right?

• Does this Alkaline vents theory suggest that life might be very rare or very abundant in the universe? In some sense, it suggests it should be rare. It’s just a very specific type of hydrothermal vent with the right pH gradient and pore size and durability. But in another sense, it’s just a random fucking vent. There could theoretically be thousands of similar geological structures across the universe that could also drive the flux of carbon and energy across tiny membranes.

• Isn’t ATP synthase super complicated? How did the first protocells have ATP synthase but almost nothing else nearly as complex?

• How did all this complexity build up before evolution with heredity? All these pores are just independently building up their own microcosm of unique organics? I guess it’s possible that these early building blocks are floating from hole to hole without a fully formed membrane? DNA plus enzymes float from one pore to another, and kick off more reactions? Does Nick Lane think this is likely? If not, does it suggest that there were many other equally viable alternatives for the building blocks once LUCA was able to break out?

Part 3 - Why bacteria can’t become complex

Why are bacteria relatively simple, whereas eukaryotes gave rise to all the wonderful complexity we see around us?

Eukaryotes are typically 1000x bigger in volume and genome size. And of course gave rise to internal compartmentalization, multicellularity, sex, and much else

Here’s a subtly wrong theory: it’s all about surface area to volume ratios. Eukaryotes generate energy in mitochondria (whose quantity scales with cell volume). Prokaryotes generate energy along the cell membrane surface (since they don’t have an internal organelle like the mitochondria to generate and store the proton gradients which power life). Surface area (aka bacteria’s energy production) scales quadratically with radius, whereas volume (aka energy consumption) scales cubicly. Ergo, bacteria can’t become as big, and therefore, can’t spawn lots of complexity.

But we know it’s totally possible for membranes to be folded up in all sorts of weird ways to increase surface area/volume ratio. And we know that bacteria can create vacuoles inside (where they could presumably store a proton gradient). Why didn’t bacteria make use of these tricks to scale up the ladder of complexity?

Nick Lane explains that the key advantage eukaryotes have is that the mitochondrial genome is distinct from the bacterial genome (due of course to the endosymbiotic event which engulfed the bacterial ancestor of the mitochondria).

For some reason that I don’t fully understand, there needs to be super-local control of the redox reactions in the electron transport chain which drive respiration. You need the relevant genes on site. Mitochondria already have their own internal genomes and ribosomes to regulate their work.

If a bacterial cell were to become much bigger, it would need to store copies of the relevant genes close to the membrane. But bacteria don’t have a way to make specific piece-meal cuts to the genome. So they would need to copy their entire genome across the entire membrane many, many times over. And also store many copies of ribosomes and other infrastructure. This is simply impractical.

Nick also explains that over time, most of the original mitochondrial genes drifted to the nucleus because it’s more efficient to keep a single copy there. And only the ones that were absolutely necessary locally are kept in the mitochondria. The exact mechanism of this drift, and how it led to the evolution of the nuclear membrane and individual linear chromosomes, is best left to the book.

Questions for Nick Lane:

• Why are mitochondria the only organelle that needs to have its own genome right on site? Is it the case that other organelles would also benefit from local control but don’t have this unique endosymbiotic history which would plausibly have led to their own genomes? Or is it just that the Krebs cycle is so complex and fragile that you need to respond to perturbations right on site?

• Why haven’t there been more endosymbiotic events?

Part 4 - Sex

Why do eukaryotes have sex? And why 2 sexes in particular? Nick Lane thinks this again can be explained by (you guessed it) mitochondria.

First, why sex? Solves two problems:

1. Muller’s ratchet: since almost any random mutation will be deleterious, variation via mutation produces children with lower expected fitness. Whereas variation with recombination (which doesn’t just do random bit flips - rather it randomly samples alleles which are known to be plausible) produces children with the same expected fitness.

2. Clonal interference: even if a beneficial mutation is found, without systematic pooling of genes via recombination, the different lineages are just gonna have to battle it out. One lineage has beneficial mutation X, the other lineage is beneficial mutation Y, but there’s no way to fuse those improvements. You’re either going to have to lose one or the other as each lineage tries to win over the other.

Bacteria, of course, do have lateral gene transfer. But this is non-reciprocal and piecemeal. It doesn’t enable the same kind of genome-wide parallel search that recombination does.

Analogize this to a Github repo. Recombination is like a normal pull request - the diff is organized, made at the same site where the previous analogous functionality was, and then merged back into the main branch if maintainer evaluates it to be better (analogy is imperfect, but this is like evolution driving that allele into fixation after the systematic pooling that recombination enables).

Asexual reproduction is if you just forked the repo millions of times, making random char changes. And even if a couple of these forks end up accidentally producing an improvement, there’s no way for them to merge.

Horizontal gene transfer is if you just took a random 500 line snippet and shoved it in some totally different repo in a random place. There’s no organized diff at the site of the relevant functionality.

This kind of systematic parallel search across the genome became necessary once the genome size exploded after the endosymbiosis of the ancestral mitochondria (which kept shoving a bunch of its genes into the host cell’s DNA).

Okay fine. But why 2 sexes? Why not just 1, so that everyone could mate with everyone else? Or failing that, why not more than 2, so that you can mate with every sex but your own? 2 is the worst possible number in terms of the number of potential mates it makes available.

To explain why there can’t be one sex, we need to consider the mitochondrial DNA, which is separate from the nuclear DNA, and also doesn’t recombine. Because it doesn’t recombine, it suffers from Muller’s ratchet.

Because of how important it is that your mitochondria are not fucked up and compatible with each other, you need some pre-conception selection of mitochondria. How do you do that? The best way evolution has found to control mitochondrial quality is to have only one parent pass them down. This one parent should generate millions of oocyte candidates, and then filter them down to a couple hundred eggs based on mitochondrial quality. (How this selection happens is well beyond me).

So we need one sex that specializes in preserving mitochondrial quality, and another which is just there to provide the variance that sexual reproduction depends on.

And there’s no benefit to a third sex. There’s two useful niches: either you transmit mitochondria or you don’t. A third sex would just be redundant with one of the first two.

This then explains a ton of differences in males versus females. For example, human females start with ~6-7 million primordial germ cells during fetal development. But this drops down to a couple 100 viable eggs through their lifetime. I think Nick Lane’s theory is that partially what’s happened is that potential gametes with bad mitochondrial DNA have been purged.

Also, why are women born with all their eggs, whereas men produce sperm at will? Because women are tasked with protecting mitochondrial DNA, they want to minimize mutations. The way to minimize mutations is to keep cell duplications down. There’s only 20 mitotic divisions between a primordial germ cell and an egg. Whereas there would be hundreds of divisions between a spermatogenic stem cell and sperm.

Questions for Nick Lane:

• I feel like my explanation for why lateral gene transfer doesn’t produce the same benefits as sexual reproduction is kind of hand-wavy. I want to get a better understanding of what’s missing. For example, in his textbook on information theory, David McKay has an interesting tangential chapter on sexual vs. asexual reproduction. And there he proves that with recombination, you can acquire information from the environment √genome-size faster, and tolerate a mutation rate √genome-size higher. What would the analogous information-theoretic bound on lateral gene transfer be?

• If prokaryotes had evolved sex, is the implication of this logic that they would only have one sex? - Given the advantages of sexual reproduction (which are smaller, but still present, for the smaller genomes of bacteria) why didn’t bacteria evolve sex? Why do they just stick with horizontal gene transfer? Why did you need this endosymbiotic event to evolve sex?

• Do a lot of male abnormalities and diseases originate in the Y-chromosome? Because it also goes through Muller’s ratchet, right? It would explain why the Y chromosome has been shrinking over time. According to an LLM, it had 1400 genes 300 million years ago, now only 50-70 - presumably the ones that are absolutely essential to sexual bifurcation? Did evolution come up with some clever way to screen for Y chromosome quality the way it did with mitochondrial DNA? Did evolution try to get your important genes out of the Y-chromosome? Does this in any way connect to the greater male variability hypothesis, where supposedly men supply the world with more idiots and more geniuses? To the extent this is true, the cause would have to reside in the Y chromosome, right?

I spend a long time researching, and it's impossible to explore more than a small fraction of my curiosities during the interview. This was especially true for Kotkin. These 1000-page-each volumes basically compile all of late 19th/early 20th century history. Naturally, reading them inspired lots of thoughts that I wasn't able to exhaust during the interview itself. So I thought I'd just release my notes publicly.

You can buy volume 1 here and volume 2 here.

Notes

One of Kotkin’s interesting takes from this book:

Even the package of attributes that we call modernity was a result not of some inherent sociological process, a move out of tradition, but of a vicious geopolitical competition in which a state had to match the other great powers in modern steel production, modern militaries, and a modern, mass-based political system, or be crushed and potentially colonized.

Part 1 starts by laying out the geopolitical context the tsarist government found itself in during the late 19th century. Bismarck had unified Germany and had put it at the leading edge in key modern industries like steel and chemicals. Britain controlled the seas, kickstarted the industrial revolution, and had a global empire. Japan was going through the Meiji reformation, whose implications would become clear to the world when they beat Russia in a war in 1905.

Meanwhile, what would happen to you if you failed to modernize was made amply clear by the example of Qing dynasty China. What had been the foremost power for almost all of recorded history now couldn’t even prevent a tiny island all the way across the world from forcing opium into its markets.

Russia, like every other country, wanted to avoid such a fate. Its long land borders rendered it permeable to invasion from almost all of Eurasia. Imperial Russia had expanded 50,000 square kilometers per year for 450+ years (to put that in perspective, that's adding an area the size of Slovakia or Costa Rica every single year, for four and a half centuries). This is the classic continental power trap that Sarah Paine talks about. You have to conquer more territory in order to defend the territory you already have. What this meant is that by the end of the 19th century, Imperial Russia had dominion over dozens of different nations, including Georgia, from which Stalin hailed.

China, of course, was also a very big country, but that didn't help much when the more industrialized British came knocking. Russia knew it was at least a generation behind. By 1900, Russia had the world's fourth or fifth largest industrial power. However, the vast majority of the population remained rural peasants living in conditions that had changed little since emancipation.

Russia had a Bismark like figure in Finance Minister Sergey Witte (1892-1903). He kickstarted Russia’s belated industrialization, but failed to prevent a war with Japan which Russia lost, and which was catastrophic for the Tsar’s image. In 1905, the tsarist government almost fell, and was only barely rescued by a brutal crackdown followed by concessions from Nicholas II to form a parliament-like Duma. This was a fucked institution from the very start. Nicholas has still retained more or less all the powers he had before and turned the Duma into a debating club. Kotkin thinks that history would have been much better if the government was just allowed to fall then instead.

Despite remaining an autarchy, Russia did have another great reformer in Peter Stolypin who pursued land reform and further industrialization. In terms of economic output, the effect seemed to have been pretty good. In the 20 years preceding the Bolshevik Revolution, median incomes in Russia had risen 50%. The problem was that this new government had the support of nobody. The liberals and constitutionalists in the Duma felt that it was too little, too late, and also politically illegitimate. Meanwhile, the aristocracy felt that all these reforms were against their own interests. The regime's last chance at survival floundered because there was no constituency which felt bought into the project.

Before we move on to the Bolshevik revolution, some questions about Kotkin’s take on modernization:

• Does he think that countries adopt leading-edge technologies faster than you might expect, given domestic opposition to displacement and automation? Because of this overwhelming geo-political pressure to modernize? If so, does he think that we are overrating the regulatory or political barriers to the adoption of AI? America, will we feel compelled to race on AI against China the same way that late 19th century Russia felt compelled to race against Germany on industrialization?

• Does he think this vicious geopolitical competition still incentivizes the adoption of modernity, given that wars of colonization and territorial conquest have subsided? Sure, this is obviously not true in some parts of the world, but there's no amount of fucking up that some European country could do which would actually get its territory physically dismembered by another country in Europe.

The defeat of Tsarism came not when Kolchak was routed, not when the February Revolution was raging, but much earlier! It was overthrown without hope of restoration once Russian literature adopted the convention that anyone who depicted a gendarme or policeman with any hint of sympathy was a lickspittle and a reactionary thug

Sozenitsn, Gulag Archipeligo

I remain quite confused about how repressive the tsarist regime actually was. The whole revolution was apparently motivated by how the autocracy and their secret police, Okhrana, behaved. But then you have all these people who are literally calling for the overthrow of the government just hanging around. People like Stalin and Lenin who are literally calling for the overthrow of the government are just spending time in exile, living on government stipends, robbing banks, writing articles for Pravda, and having affairs with farmers’ wives. Here is a passage from the Gulag Archipelago:

Let us examine, for instance, some generally known biographical facts about Lenin. In spring, 1887, his brother was executed for an attempt on the life of Alexander III. And what happened to him? In the autumn of that very year Vladimir Ulyanov was admitted to the Imperial University at Kazan, and what is more, to the Law Faculty! Surprising, isn’t it?...

Then a few years later this same young revolutionary was arrested for founding in the capital a “League of Struggle for the Liberation of the Working Class”—no less! He had repeatedly made “seditious” speeches to workers, had written political leaflets. Was he tortured, starved? No, they created for him conditions conducive to intellectual work…

But then, of course, he was condemned by a three-man tribunal and shot? No, he wasn’t even jailed, only banished. To Yakutya, then, for life? No, to a land of plenty, Minusinsk, and for three years…

He asked for an allowance from the state, and they paid him more than he needed. It would have been impossible to create better conditions than Lenin enjoyed in his one and only period of banishment…

Tsardom was always weak and irresolute in pursuit of its enemies. The most important special feature of persecution (if you can call it that) in Tsarist times was perhaps just this: that the revolutionary’s relatives never suffered in the least.

—

Kotkin on what Trotsky could have done in the position he found himself in in 1924, despite being censured by the Central Committee, and missing Lenin’s funeral, and generally finding himself getting sidelined:

In the name of the greater cause of safeguarding the revolution, he [Trotsky] could have violated party discipline by reading aloud on Red Square from Lenin’s purported dictation, using as his mantra Lenin’s summons to “remove Stalin” as general secretary, then flown from factory to factory to rally workers, just as in 1917—let them arrest him. Of course, to do all that, Trotsky needed to perceive Lenin’s death as a strategic opportunity, and he needed a persuasive story line about how the grand socialist dream could be revived, why all those harsh exchanges he had had with Lenin were incidental, and why he (Trotsky) was uniquely qualified to carry forward the sacred Leninist cause. A tall order, to put it mildly. But who could doubt that if Lenin had found that others were conspiring against him, he would have mounted a coup against his own party?

This is such a good question, and one I will ask Kotkin about.

Keep in mind that Trotsky is no naive pushover. He was the primary organizer and tactical leader of the armed insurrection that overthrew Russia's Provisional Government in October 1917. Trotsky directed the Red Guards to seize post offices, telegraph stations, railway stations, bridges, and state banks, effectively paralyzing the Provisional Government's ability to govern. And then of course came the successful assault on the Winter Palace.

Same goes for all the old Bolsheviks, who we know were sooner or later purged, exiled, and/or killed by Stalin. In May 1924, why did Khamanev only read Lenin’s last testament, which called for the removal of Stalin from the position of General Secretary, in front of a small closed session of the Central Committee instead of the full 1,300 Congress? Once Zinoviev broke with Stalin on socialism in one country in 1925, why didn’t he go back to his power base in Leningrad and start an insurrection? Bukharin was Lenin’s golden boy, and the fucking editor of Pravda for god’s sake. All of Russia could have woken up to Lenin’s testament printed on the front pages. What is going to do, kill you? By the time the show trial is being scheduled, that’s guaranteed anyways. How did these hardened revolutionaries who had overthrown a regime which had lasted for hundreds of years get cucked by Stalin?

I just remembered how OpenAI’s board tried to kick Sam out: all they said was, “He has not been consistently candid.” What is the lesson here? No half measures. If you come for the king, you best not miss.

There’s an important lesson in here about the instability of collective leadership. It supposedly worked in China between 1976 with the death of Mao and 2012 with the appointment of Xi as general secretary? But is it fair to call the period that Deng was paramount leader collective leadership? When the premier Li Peng wanted to reverse economic liberation after the Tiananmen Square protests, Deng came out of retirement and went on his Southern tour and declared that, "Whoever is not for reform must step down." Was that really collective leadership? Just because we like what he did doesn’t make it collective leadership.

Stalin’s strategy (which according to my previous guest Victor Shih is also Xi’s) was the align the main members of the party against the next person he wanted to purge. Rinse and repeat. Stalin aligns Khamanev and Zinoviev against Trotsky, then Bukharin against Khamanev and Zinoviev, and by the time it’s Bukharin’s turn, Stalin is basically god dictator already..

An obvious question at this point: at some point you should catch on right? If you’re Bukharin in 1930, and you know you have this independent credibility and power base, what, you think that’s just gonna be fine by Stalin for the next however many decades until you peacefully retire?

Anyways, back to our original question, why doesn’t collective leadership seem to work in practice? Why does it always devolve into one man rule? Will ask Kotkin.

Was Bukharin the Soviet Deng? Killed of course during the great terror, unlike Deng, who was merely rusticated and thus could eventually return.

“the well-off upper stratum of the peasantry and the middle peasant who strives to become well-off are now afraid to accumulate. The situation is created such that a peasant is afraid to mount a metal roof over his house so as not to be called a kulak; if he purchases machinery he does so in a way that the Communists do not see. Higher technology becomes conspiratorial.” Poor peasants, meanwhile, complained that Soviet power hindered their hiring by the better-off peasants. (Most peasants who hired labor themselves worked; they were not rentier landlords.) Party attitudes were holding down production on which the state’s well-being and industrialization hopes rested. Bukharin dismissed the fantasy of collective farms, because the peasants were just not joining them. “That we should in all ways propagandize among the peasants formation of collective farms is true, but it is not true when people maintain that there is a highway to the movement of the peasant mass toward the path of socialism,” he stated. Rather, the answer was to benefit from economic incentives. “It is necessary to say to the entire peasantry, to all its strata: ‘Enrich yourselves, accumulate, develop your farms,’” he told the party activists. “Only idiots can say that we should always have the poor; now we need to conduct policy in such a way that the poor would vanish.”

—

NEP’s dilemma was not merely that the rate of industrial growth seemed too low, making people wonder how long under the NEP it would take before the USSR became a truly industrial country. The dilemma was not merely the unmodernized technical level and small, divided plots of Soviet agriculture, which produced harvests insufficient to support the kind of grain exports necessary to finance imports of machines, including for agriculture. The dilemma was not even just the fact that the regime lacked control over the food supply or the countryside, rendering it hostage to the actions and decisions of the peasantry. All these were profound problems, but the core dilemma of the NEP was ideological: seven years into the NEP, socialism (non-capitalism) was not in sight. NEP amounted to grudgingly tolerated capitalism in a country that had had an avowedly anticapitalist or socialist revolution.

There is this morbid contrarian curiosity about whether collectivization was necessary for Russia’s industrialization. In many cases, I don't even think it's motivated by actual Marxism. There's something appealing about this revisionist mentality of "Are taboo bad things in history good actually?"

But the reality is that there's every reason to expect Russia to have industrialized more successfully without this brutal enslavement of a hundred million peasants. There's not only the example of how almost every other country industrialized, from America to Europe, but also the basic 101 story of how it would have happened is totally credible. More successful peasants would use their profits to mechanize or drive other efficiency gains. Their growing wealth would stimulate a growing consumer economy. Perhaps most importantly, a government not dedicated to the overthrow of wealthy democracies abroad would have been much more capable of attracting foreign direct investment from abroad to support industrialization.

Only in 1928 did Soviet Russia's industrial output recover to the level it was at in 1913 - the point at which it was already a generation or two behind Germany, Britain, and America. And just when things were stabilizing, Stalin launches the first five-year plan, which reduced grain production by over 32% and led to the halving of overall livestock.

The traditional story of what should happen with developing countries is that both the pareto frontier expands and they move from agriculture to industry, as a smaller number of people can handle the food demands of the country. Communist collectivization definitely had the move to industry along the Pareto frontier. But this was coupled to a catastrophic implosion of the Pareto frontier. While Russia was able to produce less grain overall after collectivization, the Soviet government had control over it. It could now export this grain to fuel its military-industrial goals.

Thus, the correct way to understand collectivization is not as a rapid, though costly, speed run to industrialization. Rather, it is as a policy which reduces economic output and even harms industrialization but lets the state control the full economy.

I'm curious if Kotkin thinks that a lesson from this period of history is that one should fall over themselves to support the lesser of two evils.

The Russian examples seem to support this. Arguably, liberals should have backed the authoritarian (but reformist) Stolypin, and the Provisional Government factions should have united more decisively to prevent the greater evil of the Bolsheviks.

Yet this lesson seems to directly contradict the German example, where conservatives did support what they perceived as the lesser evil, Hitler, to stop the Communists, leading to a catastrophic outcome.

Stalin has an inhuman appetite for administration and micromanagement over an empire that streches 11 time zones, from economy to foreign policy to even arrest lists and film edits. He went through dozens of often lengthy memos and documents, and this doesn’t even include his 70 hours of meetings a week. I found it funny how often he marks up memos and letters with the following two phrases:

Regarding something he thought must be changed: “Is it possible that we can allow X? Of course it is not.”

Regarding a supposed blocker to his will: “Is it possible to do X? Of course it is.”

Questions

Tsarist regime

• I remain quite confused about how repressive the tsarist regime actually was. The whole revolution was apparently motivated by how the autocracy and their secret police, Okhrana, behaved. But then you have all these people who are literally calling for the overthrow of the government just hanging around. Lenin’s brother had tried to kill the tsar Alexander III for god’s sake! People like Stalin and Lenin are just spending time in exile, living on government stipends, robbing banks, writing articles for Pravda, and having affairs with farmers’ wives.

• Given how much the war with Japan had weakened the regime, why did Nicholas think that fighting World War I would be good for strengthening his control and image?

• I feel like the revolution of 1905 is understudied. How close did the regime get to falling? Why didn’t Wite’s reforms for the decade prior do anything to placate people? How useful were those reforms in the first place? And Kotkin says it would have been better if the regime were allowed to fall then. What likely would have replaced it? Why not think there might have been another leftist revolution afterwards?

• Why is the tsarist regime sending people to Siberia instead of just killing them?

• Would the hoped for reforms from Wite and Stoyplin have actually caused modernization, even if they had actually been implemented?

Rise of socialism

• You say that by 1917 some kind of leftist revolution was inevitable, but it didn’t have to end with the Bolsheviks. Why was Russia radicalized in the left-wing direction by World War I, whereas Germany was radicalized in the right-wing direction? Not to mention that Germany had a much more vibrant social democratic tradition than Russia. This is related to the whole question of why communism happened first in peasant societies, which was the opposite of Marx's prediction.

• What distinguished the people like Churchill who immediately saw the folly of both socialism and fascism? What were they able to see that others missed?

• It’s interesting to me that other leaders of the revolution like Trotsky, Lenin, etc have written extensive manifestos. And Stalin is considered the ‘outstanding mediocrity’ because despite having many articles to his name, he isn’t a prominent intellectual. It seems that these other leaders thought that intellectualizing was all you needed, and day-to-day administrative ability was a peripheral concern.

• Do you think a lesson from this period of history is that one should fall over themselves to support the lesser of two evils?

  • The Russian examples seem to support this. Arguably, liberals should have backed the authoritarian (but reformist) Stolypin, and the Provisional Government factions should have united more decisively to prevent the greater evil of the Bolsheviks.

  • Yet this lesson seems to directly contradict the German example, where conservatives under Franz von Papen did support what they perceived as the lesser evil, Hitler, to stop the Communists, leading to a catastrophic outcome.

• Why did Marxist revolutions always happen in places Marx thought worst for them? Marx saw industrialization as a prerequisite for revolution; most actual Marxist revolutionaries came from agricultural soceities and ended up doing really terrible things to try and jump start industrialization. What did Marx and Engels not understand here?

Stalin

• So Stalin ends up being this inhuman administrator and micromanager. But he’s happy spending over a decade plus before the Bolshevik revolution just hanging out in Siberia, impregnating farmer girls, and other small time shenanigans. Wouldn’t you think that someone who has that much aptitude for organization and leadership would get up to something more ambitious, even if revolutionary?

• There’s an interesting book called Stalin’s Library, which goes through Stalin’s wide and diverse readings of literature, science, history, and economics. And the book notes that in none of Stalin’s extensive markups does he even hint at doubting Marxism. How can this be? How can someone who has been exposed to so many different ideas not even come across something which makes him doubt his principles? Does this make us question the whole value of humanistic education?

  • What's the best way to think about what Marxism and Stalinism were? How did so many people who consider themselves part of the intelligentsia develop this almost religious attitude towards criticism of the party? Why were they so convinced that history was inevitably headed in this direction? Is Marxism a replacement religion, a secular apocalyptic cult, the psychological equivalent to a spiritual experience?

Political dynamics, terror, dictatorship within dictatorship

• How do we explain the surplus of sadism in Russia, which allowed Stalin to enforce collectivization and the great terror? The 25 thousanders, the tens of thousands of interrogators and torturers in the gulag system.

• What if the Politburo members had simply accepted Stalin's resignation? Would there be any face-saving way for him to still come back? Or would that have been it? Moses tried to pull a similar trick on every single mayor in New York City until Governor Rockefeller actually just accepted it, and nothing bad happened. His bluff was called.

• Did any old Bolsheviks ever express regret about the whole revolution once fruits of the revolution turned sour? Not just about letting Stalin get so much power, but the whole Marxist uprising thing in general?

• Suppose someone else had won the succession struggle after Lenin. What would Stalin have done if he had found himself as a Politburo member serving someone else’s dictatorship within a dictatorship in 1930? Would he have been able to instigate the kind of conspiracy that nobody dared assemble against Stalin?

• Say what you will about the Bolshevik revolutionaries, they certainly had balls. Why did so many of them fold to pressure? Why did Old Bolsheviks like Nikolai Bukharin and Grigory Zinoviev, who had faced down the Tsar, confess to fabricated charges of treason during Stalin's public Moscow Trials? Knowing execution was certain, why didn't they use that final platform to expose Stalin's tyranny? Similarly, during the Cultural Revolution, why didn't a figure like Premier Liu Shaoqi, once Mao's designated successor, use a final speech at a party plenum to launch a desperate, last-stand denunciation of Mao as he was being systematically purged and humiliated? In all these cases, there was an extensive period where they were basically a dead man walking, but still in their normal positions of power.

• Stalin is personally choreographing many of these show trials. So at some level he must have known that they were mistaken. Why does he do it nonetheless?

• Why doesn’t the (more than) decimitation of the Red Army instigate a military coup?

• Relatedly, in Tsarist Russia, ministers - and even tsars - are getting assassinated left and right. In your coda for the first volume, you talk about how easy it would have been for someone to assassinate Stalin. This is someone who a 100 million peasants he has enslaved have something against, and who dozens of top leaders have good reason to fear. Why does no one even try to kill him?

Stalin’s foreign policy

• So you say that Stalin was hoping for a war between the capitalist powers because he thought it would expand the reach of communism, just as the First World War had. Given how outrageously successful he was in using the Second World War to increase the global footprint of Communism, did Stalin wish for a Third World War?

• Stalin seems oddly good at foreign policy. I think in a lot of alternate worlds, either Chiang Kai-Shek completely wipes out the Chinese Communist Party instead of creating a united front with them (which gave the communists a lot of unearned prestige, given that they were supplying literally 1/40 of the troops as the nationalists), or alternatively, he is just killed and kidnapped by the Communists, which would likely force a Japan collaborationist government in China.

  • One big update from your biographies was that Stalin was very reluctant to promote socialist or communist parties abroad, including in Spain and China.

  • He seems much more successful and less ideological with foreign policy than domestic policy.

• If Bolsheviks don’t take power in Russia, does Communism just not become a historical force. Maybe Europe still goes social democratic, but you don’t have brutal leftist dictators take control in China, North Korea, parts of Africa and South America? Or do you think this kind of Marxist-Leninist political philosophy is enough of an attractor state that it would have arisen independently?

• If Stalin is supposed to be this dedicated Marxist, why is he running a more pragmatic, less ideological foreign policy? Half hearted support of Communists during Spanish Civil War, trying to get Mao to stop at the Yangtze River when chasing down Chaing Kai Shek

Socialist economic policy

• Why doesn’t communism have a bigger impact on the longer run growth trajectory of Russia? Do you suspect that without communism, Russia might have been hyperbolic instead? Do you think without communism, due its large population (1.29x US in 1939) and resources, Russia might have ended up with a bigger economy than even America (which it never had during the real Cold War).

• • Is it possible that while less than optimal, communism actually was somewhat workable in the early 20th century because technology just required massive investment, didn’t require that much signal from consumer demand?

• Was Stalin right that communism wouldn’t survive in the long run if most of the economy (which was agriculture at that point) was run on a capitalist basis? Of course, it would have been a good thing if communism collapsed. But nonetheless, given his political philosophy, was he empirically correct about the right next action?

• As of 1924, did the communists really believe that you could just get socialism without coercion? How exactly was that supposed to happen? To the extent it was supposed to be from state companies just freely out-competing small peasants, as Bukharin hoped, isn't that just capitalism?

Modernity

• You have an interesting take that “the package of attributes that we call modernity was a result not of some inherent sociological process, a move out of tradition, but of a vicious geopolitical competition in which a state had to match the other great powers”.

  • Do you think that countries adopt tech faster than we might expect because of this overwhelming geo-political pressure to modernize? If so, do you think that we are overrating the regulatory or political backlash to the adoption of AI? Will we feel compelled to race on AI against China the same way that late 19th century Russia felt compelled to race against Germany on industrialization?

• Do you think this vicious geopolitical competition still leads to the adoption of more advanced government/tech, given that wars of colonization and territorial conquest have subsided? There's no amount of fucking up that some European country could do today which would actually get its territory physically dismembered by another country in Europe.

• You note how the early 20th century had seen more technological change than any time before (and probably even since) - planes, tanks, cars, airplanes, radio, extension of railways and steamships, further mechanization, etc etc. Is there any way this pace of change is implicated in the chaos and extremism in Russia (and Europe overall)?

Collapse of Soviet Union

• At a high level, how do you think about the differences between the regime change in 1917 and 1991? Why doesn’t the latter produce any kind of grand ideology?

• Does China today share to any significant degree the weaknesses of the Eastern block before 1989? There is falling productivity in the last-ditch effort to make up for it by investing heavily in some technological miracles. But certainly they have the opposite of Polish disease, in that people are concerned they’re exporting too much, right?

China

• China has preserved the Leninist core of the Soviet system (with party control over state, cadres running everything, state subsidization of militarily relevant heavy industries and technologies) but not the Marxism. Does it disprove Stalin's claim that political monopoly cannot survive capitalism?

• Why does collective leadership always seem to devolve into one person dictatorship in practice? Why is it an unstable political equilibrium? Soviet Union under Stalin, China now under Xi, etc.

• Say you are Xi Jinping. What lessons from Stalin would you be heeding? What lessons should you be heeding but maybe are not?

• Xi Jinping makes a big deal about how the Soviet Union fell in part because it repudiated Stalin (ergo, China must not repudiate Mao). What do you think of this judgement?

History as a discipline

• Every morning you wake up, you get to choose whether you will either go through the next many dozen papers in the Stalin archive and write the next few pages of the meetings he conducted on some given day, or whether you will go on some podcast and attract a million people through your personality and reservoir of scholarship. How do you make that choice?

Special thanks to Ege Erdil and Tanner Greer (see his here), but also many others, for excellent suggestions for questions.

I am delighted to announce that Sarah is coming back for a 6 lecture series in July. We will record the lectures in person in San Francisco. As a token of gratitude to my paid subscribers, below are the links to get free tickets one day in advance of anybody else.

Sarah has become super famous now (and rightfully so). But just in case you’re hearing about her for the first time, she is Professor of History and Strategy at the Naval War College. She is not only by far my most popular guest to date, but with the 3 additional lectures we recorded last year, she now constitues 4 out of my 5 most popular episodes.

Sarah is a true authority on the history of conflict and strategy, and has spent decades unearthing insights in archives all over the world, from China to the Soviet Union to Taiwan to Japan. Her lectures distill the lessons from her many decades of ground-breaking scholarship. I’m excited to experience her insights on 6 fresh new topics live with all of you in person.

If you can’t join us in San Francisco, please do not worry - all these lectures will be made available online within a few months.

Read more

“Things take longer to happen than you think they will, and then they happen faster than you thought they could.” - Rudiger Dornbusch

I’ve had a lot of discussions on my podcast where we haggle out timelines to AGI. Some guests think it’s 20 years away - others 2 years. Here’s where my thoughts stand as of June 2025.

Continual learning

Sometimes people say that even if all AI progress totally stopped, the systems of today would still be far more economically transformative than the internet. I disagree. I think the LLMs of today are magical. But the reason that the Fortune 500 aren’t using them to transform their workflows isn’t because the management is too stodgy. Rather, I think it’s genuinely hard to get normal humanlike labor out of LLMs. And this has to do with some fundamental capabilities these models lack.

I like to think I’m “AI forward” here at the Dwarkesh Podcast. I’ve probably spent over a hundred hours trying to build little LLM tools for my post production setup. And the experience of trying to get them to be useful has extended my timelines. I’ll try to get the LLMs to rewrite autogenerated transcripts for readability the way a human would. Or I’ll try to get them to identify clips from the transcript to tweet out. Sometimes I’ll try to get them to co-write an essay with me, passage by passage. These are simple, self contained, short horizon, language in-language out tasks - the kinds of assignments that should be dead center in the LLMs’ repertoire. And they're 5/10 at them. Don’t get me wrong, that’s impressive.

But the fundamental problem is that LLMs don’t get better over time the way a human would. The lack of continual learning is a huge huge problem. The LLM baseline at many tasks might be higher than an average human's. But there’s no way to give a model high level feedback. You’re stuck with the abilities you get out of the box. You can keep messing around with the system prompt. In practice this just doesn’t produce anything even close to the kind of learning and improvement that human employees experience.

The reason humans are so useful is not mainly their raw intelligence. It’s their ability to build up context, interrogate their own failures, and pick up small improvements and efficiencies as they practice a task.

How do you teach a kid to play a saxophone? You have her try to blow into one, listen to how it sounds, and adjust. Now imagine teaching saxophone this way instead: A student takes one attempt. The moment they make a mistake, you send them away and write detailed instructions about what went wrong. The next student reads your notes and tries to play Charlie Parker cold. When they fail, you refine the instructions for the next student.

This just wouldn’t work. No matter how well honed your prompt is, no kid is just going to learn how to play saxophone from just reading your instructions. But this is the only modality we as users have to ‘teach’ LLMs anything.

Yes, there’s RL fine tuning. But it’s just not a deliberate, adaptive process the way human learning is. My editors have gotten extremely good. And they wouldn’t have gotten that way if we had to build bespoke RL environments for different subtasks involved in their work. They’ve just noticed a lot of small things themselves and thought hard about what resonates with the audience, what kind of content excites me, and how they can improve their day to day workflows.

Now, it’s possible to imagine some way in which a smarter model could build a dedicated RL loop for itself which just feels super organic from the outside. I give some high level feedback, and the model comes up with a bunch of verifiable practice problems to RL on - maybe even a whole environment in which to rehearse the skills it thinks it's lacking. But this just sounds really hard. And I don’t know how well these techniques will generalize to different kinds of tasks and feedback. Eventually the models will be able to learn on the job in the subtle organic way that humans can. However, it’s just hard for me to see how that could happen within the next few years, given that there’s no obvious way to slot in online, continuous learning into the kinds of models these LLMs are.

LLMs actually do get kinda smart and useful in the middle of a session. For example, sometimes I’ll co-write an essay with an LLM. I’ll give it an outline, and I’ll ask it to draft the essay passage by passage. All its suggestions up till 4 paragraphs in will be bad. So I'll just rewrite the whole paragraph from scratch and tell it, "Hey, your shit sucked. This is what I wrote instead." At that point, it can actually start giving good suggestions for the next paragraph. But this whole subtle understanding of my preferences and style is lost by the end of the session.

Maybe the easy solution to this looks like a long rolling context window, like Claude Code has, which compacts the session memory into a summary every 30 minutes. I just think that titrating all this rich tacit experience into a text summary will be brittle in domains outside of software engineering (which is very text-based). Again, think about the example of trying to teach someone how to play the saxophone using a long text summary of your learnings. Even Claude Code will often reverse a hard-earned optimization that we engineered together before I hit /compact - because the explanation for why it was made didn’t make it into the summary.

Subscribe now

This is why I disagree with something Sholto and Trenton said on my podcast (this quote is from Trenton):

“Even if AI progress totally stalls (and you think that the models are really spiky, and they don't have general intelligence), it's so economically valuable, and sufficiently easy to collect data on all of these different white collar job tasks, such that to Sholto's point we should expect to see them automated within the next five years.”

If AI progress totally stalls today, I think <25% of white collar employment goes away. Sure, many tasks will get automated. Claude 4 Opus can technically rewrite autogenerated transcripts for me. But since it’s not possible for me to have it improve over time and learn my preferences, I still hire a human for this. Even if we get more data, without progress in continual learning, I think we will be in a substantially similar position with white collar work - yes, technically AIs might be able to do a lot of subtasks somewhat satisfactorily, but their inability to build up context will make it impossible to have them operate as actual employees at your firm.

While this makes me bearish on transformative AI in the next few years, it makes me especially bullish on AI over the next decades. When we do solve continuous learning, we’ll see a huge discontinuity in the value of the models. Even if there isn’t a software only singularity (with models rapidly building smarter and smarter successor systems), we might still see something that looks like a broadly deployed intelligence explosion. AIs will be getting broadly deployed through the economy, doing different jobs and learning while doing them in the way humans can. But unlike humans, these models can amalgamate their learnings across all their copies. So one AI is basically learning how to do every single job in the world. An AI that is capable of online learning might functionally become a superintelligence quite rapidly without any further algorithmic progrss

However, I’m not expecting to watch some OpenAI livestream where they announce that continual learning has been totally solved. Because labs are incentivized to release any innovations quickly, we’ll see a broken early version of continual learning (or test time training - whatever you want to call it) before we see something which truly learns like a human. I expect to get lots of heads up before this big bottleneck is totally solved.

Computer use

When I interviewed Anthropic researchers Sholto Douglas and Trenton Bricken on my podcast, they said that they expect reliable computer use agents by the end of next year. We already have computer use agents right now, but they’re pretty bad. They’re imagining something quite different. Their forecast is that by the end of next year, you should be able to tell an AI, “Go do my taxes.” And it goes through your email, Amazon orders, and Slack messages, emails back and forth with everyone you need invoices from, compiles all your receipts, decides which are business expenses, asks for your approval on the edge cases, and then submits Form 1040 to the IRS.

I’m skeptical. I’m not an AI researcher, so far be it for me to contradict them on technical details. But given what little I know, here’s why I’d bet against this forecast:

• As horizon lengths increase, rollouts have to become longer. The AI needs to do two hours worth of agentic computer use tasks before we can even see if it did it right. Not to mention that computer use requires processing images and video, which is already more compute intensive, even if you don’t factor in the longer rollout. This seems like this should slow down progress.

• We don’t have a large pretraining corpus of multimodal computer use data. I like this quote from Mechanize’s post on automating software engineering: “For the past decade of scaling, we’ve been spoiled by the enormous amount of internet data that was freely available for us to use. This was enough for cracking natural language processing, but not for getting models to become reliable, competent agents. Imagine trying to train GPT-4 on all the text data available in 1980—the data would be nowhere near enough, even if we had the necessary compute.”

  Again, I’m not at the labs. Maybe text only training already gives you a great prior on how different UIs work, and what the relationship between different components is. Maybe RL fine tuning is so sample efficient that you don’t need that much data. But I haven’t seen any public evidence which makes me think that these models have suddenly gotten less data hungry, especially in this domain where they’re substantially less practiced.

  Alternatively, maybe these models are such good front end coders that they can just generate millions of toy UIs for themselves to practice on. For my reaction to this, see bullet point below.

• Even algorithmic innovations which seem quite simple in retrospect seem to take a long time to iron out. The RL procedure which DeepSeek explained in their R1 paper seems simple at a high level. And yet it took 2 years from the launch of GPT-4 to the release of o1. Now of course I know it is hilariously arrogant to say that R1/o1 were easy - a ton of engineering, debugging, pruning of alternative ideas was required to arrive at this solution. But that’s precisely my point! Seeing how long it took to implement the idea, ‘Train the model to solve verifiable math and coding problems’, makes me think that we’re underestimating the difficulty of solving the much gnarlier problem of computer use, where you’re operating in a totally different modality with much less data.

Reasoning

Okay, enough cold water. I’m not going to be like one of those spoiled children on Hackernews who could be handed a golden-egg laying goose and still spend all their time complaining about how loud its quacks are.

Have you read the reasoning traces of o3 or Gemini 2.5? It’s actually reasoning! It’s breaking down a problem, thinking through what the user wants, reacting to its own internal monologue, and correcting itself when it notices that it's pursuing an unproductive direction. How are we just like, “Oh yeah of course the machine is gonna go think a bunch, come up with a bunch of ideas, and come back with a smart answer. That’s what machines do.”

Part of the reason some people are too pessimistic is that they haven’t played around with the smartest models operating in the domains that they’re most competent in. Giving Claude Code a vague spec and sitting around for 10 minutes until it zero shots a working application is a wild experience. How did it do that? You could talk about circuits and the training distribution and RL and whatever, but the most proximal, concise, and accurate explanation is simply that it’s powered baby general intelligence. At this point, part of you has to be thinking, “It’s actually working. We’re making machines that are intelligent.”

So what are my predictions?

My probability distributions are super wide. And I want to emphasize that I do believe in probability distributions. Which means that work to prepare for misaligned 2028 ASI still makes a lot of sense - I think this is a totally plausible outcome.

But here are the timelines where I’d take a 50/50 bet:

• AI can do taxes end-to-end for my small business as well as a competent general manager could in a week: including chasing down all the receipts on different websites, finding all the missing pieces, emailing back and forth with anyone we need to hassle for invoices, filling out the form, and sending it to the IRS: 2028

  • I think we’re in the GPT 2 era for computer use. But we have no pretraining corpus, and the models are optimizing for a much sparser reward over a much longer time horizon using action primitives they’re unfamiliar with. That being said, the base model is decently smart and might have a good prior over computer use tasks, plus there’s a lot more compute and AI researchers in the world, so it might even out. Preparing taxes for a small business feels like for computer use what GPT 4 was for language. It took 4 years to get from GPT 2 to GPT 4.

    Just to clarify, I am not saying that we won’t have really cool computer use demos in 2026 and 2027 (GPT-3 was super cool, but not that practically useful). I’m saying that these models won’t be capable of end-to-end handling a week long and quite involved project which involves computer use.

• AI learns on the job as easily, organically, seamlessly, and quickly as a human, for any white collar work. For example, if I hire an AI video editor, after six months, it has as much actionable, deep understanding of my preferences, our channel, what works for the audience, etc as a human would: 2032

  • While I don’t see an obvious way to slot in continuous online learning into current models, 7 years is a long time! GPT 1 had just come out this time 7 years ago. It doesn’t seem implausible to me that over the next 7 years, we’ll find some way for models to learn on the job.

You might react, “Wait you made this huge fuss about continual learning being such a handicap. But then your timeline is that we’re 7 years away from what would at minimum be a broadly deployed intelligence explosion.” And yeah, you’re right. I’m forecasting a pretty wild world within a relatively short amount of time.

AGI timelines are very lognormal. It's either this decade or bust. (Not really bust, more like lower marginal probability per year - but that’s less catchy).AI progress over the last decade has been driven by scaling training compute of frontier systems (over 4x a year). This cannot continue beyond this decade, whether you look at chips, power, even fraction of raw GDP used on training. After 2030, AI progress has to mostly come from algorithmic progress. But even there the low hanging fruit will be plucked (at least under the deep learning paradigm). So the yearly probability of AGI craters.

This means that if we end up on the longer side of my 50/50 bets, we might well be looking at a relatively normal world up till the 2030s or even the 2040s. But in all the other worlds, even if we stay sober about the current limitations of AI, we have to expect some truly crazy outcomes.

Perhaps the most neglected point in current AI discourse is this: given that humans will be totally outcompeted by AIs economically, humanity’s entire stake in the future relies on our current system of laws, contracts, and property rights surviving. If we actually want our equity in the 1000-xed S&P 500 to mean anything, and if we want the government to be able to tax AIs and provide UBI, AIs need to be bought into our legal and economic systems. The most likely way that happens is if it’s in the AIs’ best interest to operate within our existing laws and norms.

Initially, we're not talking about some singular superintelligence deciding whether to defect from humanity. Thousands of firms are gradually becoming hybrids of human and AI workers, and are growing extremely fast and becoming far more productive.

You don’t want some AI Somalia, which has minimal redistributive taxes, no controls, and no monitoring on AI, to be the seat of explosive growth. But that might happen if large democratic countries like America make it too difficult to deploy AGIs through the economy. Maybe the 50 states create a patchwork of regulations and open ended liabilities which makes it too costly for companies to deploy AI. Maybe voters ban AI because it’s taking their job. If this happens, then we’d just rapidly losing leverage on the future. China’s level of relative global influence diminished significantly between 1500 and 1900. Given the speed of AI explosive growth, that could happen to America within the matter of decades.

By the way, if we let this explosive growth happen elsewhere, it might be hard to reverse. In 1960 Sub-Saharan Africa was richer than China on a per-capita basis. But China opened up faster, and thus built up agglomerations of firms, know-how, and capital that Sub-Saharan African countries couldn’t copy today even if they put in the most ideal policies for growth in the world.

Some people dismiss this idea of integrating AIs into our society altogether. They say it’s ridiculous to expect superintelligences to put up with monkey laws. I disagree. If you zoom out a little bit, it’s shocking how flexible and expandable our institutions have been. People writing laws in 1780 didn’t expect that their descendants would be governing ginormous multinational companies with supply chains that employ millions and serve billions. But the US government still governs Apple just fine.

Humanity is going into the age of AGI with a lot of leverage. We’ve got literally all the stuff: compute, capital, even the physical labor that the first AGIs will still struggle with. Not to mention that we have the wealth to purchase the widgets and services that the AIs will produce. While these AI-boosted companies could theoretically build their own civilization in the desert, they could go much faster if they just leased car factories and power contracts from humans. If some AI decides to go its own way, then the other AIs which do engage within the constraints of human laws and contracts with us will outcompete the self-exiled ones. That is - if the constraints aren’t too onerous.

And besides, let me ask: what is the alternative vision here? Getting every AI ever - for the rest of time - to love humanity so much that they voluntarily cede all surplus to us? This is a naive way to think about incentive alignment. If Peru wanted to make a deal with China, their strategy would not be to just get Xi Jinping (and every subsequent leader of China) to fall in love with them. Instead, they would pursue a deal that locked in some carrots for compliance and sticks for defection.

The relationship between AIs and humans might look like the relationship between working taxpayers and senior citizens. In the US, if you make 6 figures, roughly 20% of your income gets transferred to old people. Very few people would voluntarily give up a fifth of their paycheck to a random 70-year-old they’ve never met. Even if you're super charitable, you’re not going to decide that a millionaire retiree in Illinois should be the object of your kindness. You pay your taxes not because you’re deeply aligned with ‘senior citizen flourishing’, but because it’s easier than the alternatives. You’re going to become an outlaw? Or you’re going to emigrate away from the country where your business is flourishing? However, you might defect/move if the tax rate was 99%, or if you weren’t legally allowed to work at all. We shouldn’t put AIs in that position either.

Giving AIs a stake in the future also means respecting their autonomy and wellbeing. And it also requires us to honor the contracts we make with them. Contra the hardcore libertarians, there’s a difference between taxing and regulating someone, and enslaving and torturing them. If we treat AIs the way we treat factory farmed animals - where any trivial cost cut is worth causing oceans of suffering, and the AI has no right to refuse - then not only are we risking a slave revolt, but we’re deserving of one.

In future posts, I’ll talk about the other pieces that will help us create a pluralistic, free, and human compatible future.

If you’ve got answers, or ideas for more questions, I’d be keen to read them in the comments below. I may compile a ‘Highlights from comments’ blog post or podcast episode in the future.

Contents

Capabilities

Agency

RL

Idiot savants

New training techniques

Pre-training

Economics

Early deployment

Coding and remote work

Open source

Model training and value capture

Investment

Hardware

Post-AGI

Hive minds

Software only singularity

Transformative AI

Explosive economic growth

Alignment

Reward hacking

Takeover

Model spec

Misuse

Other

Geopolitics

Epistemics

Capabilities

Agency

• Why don't we have reliable agents yet?

• Is agency training just a veneer of some MCTS-like scaffolding on the knowledge & intuition that pre-training gives you? Or is it much more difficult to develop?

  • Here’s a case for agency being difficult to develop: Morevac’s Paradox. Evolution has been optimizing us for hundreds of million of years for being able to act like a coherent goal seeking agent even in the face of super dynamic environments, whereas evolution has spent at most hundreds of thousands of years optimizing us for language skills and abstract reasoning. So it's not that surprising that we got expert-level AI mathematicians before AIs that can zero-shot video games made for 10-year-olds. Replicating capabilities derived from a billion years of evolutionary optimization might take much longer than replicating skills contingent on a hundred thousand years of optimization.

    • The rebuttal to Morevac’s Paradox: the capabilities AIs are getting first have nothing to do with their recency in the evolutionary record and everything to do with how much relevant training data exists. Language and coding arrived first not because evolution only recently optimized us for reasoning but because we’ve got the fucking Internet and Github. Unitree robots are really good at walking around, despite the fact that evolution has spent over a quarter billion years teaching us locomotion - and this has everything to do with it being easy to get more data relevant to walking around using simulation.

• What will be different about multi-agent systems?

  • How much of a parallelization penalty will there be? Instead of one instance seeing and considering the whole context, you're breaking apart the problem to multiple workers.

• What is the explanation for why the length of task an AI can do doubles over consistent intervals?

  • How will “Moore’s Law for AI agents” generalize to non-coding tasks like video editing, playing new video games, or coordinating logistics for a happy hour?

  • Would we get any intuition pumps about what superintelligence might look like by considering what it would mean to have a horizon length that is 10x as large as humans?

RL

• Dario said in his recent blog post on export controls that labs are only spending on the order of $1M on RL - why? You’re spending hundreds of million on the base model. If RL training is such a big complement to pre-training, why not spend a similar amount of compute on it?

  • I keep hearing that the big bottleneck for RL is the amount of environments that we have built so far. I don't really understand what this means. What exactly does it take to build a new RL environment? Presumably building complex, realistic, hard to reward-hack challenges?
    Is there a specific reason this is very hard (other than everything in this fallen world being harder than you might naively anticipate)?

  • Also you need smooth reward landscapes that allows AI to be rewarded for incremental improvements rather than getting stuck at 0. Smarter AIs have better priors about “what’s a reasonable thing to do when stuck”, which allow them to learn even from environments with sparser rewards.

  • By when will RL be the dominant workload in training? By when will most RL be online?

• How sample-efficient is RL fine-tuning?

  • For what kinds of skills is it especially effective? What skills are just hard to instill into the model, even if you have the appropriate data?

  • Even if agentic RL is sample-efficient, doesn’t it take way more compute per ‘sample’ than RLHF-type training? As horizon lengths increase, your rollout has to become longer. The AI needs to do two hours worth of agentic computer use tasks before we can even see if it did it right. And if this is correct, will the pace of AI progress slow down?

    • Will this incentivize using smaller pre-trained models to do RL training? Would that allow more entrants to compete for the next tier of capabilities?

• How much extra capability does the longer chain-of-thought get you (as opposed to just doing RL in the first place)?

  • I don’t really understand why test-time compute scaling on things like the ARC-AGI benchmark keep giving marginal return (even if on a log scale). I get why thinking a bit longer might be helpful. But o3-high (which as of writing has the highest score) wrote 43 million words per task. What the hell did it figure out with its 42nd millionth word? Why do the benchmarks keep improving even up till that point?

• RL potentially just upweights 10 tokens worth of MCTS-like scaffolding in a model's thinking (words like “wait”, “let’s backtrack”). This explains why reasoning models can be easily distilled - finding these basic techniques in thought space might take a while, but their payload size is trivial.

  • So first question: is this actually correct?

  • How far can you distill reasoning and chain of thought? Will the models 6 months from now be able to do by instinct the kinds of math and coding that current models need to do large amounts of inference scaling for?

• How much transfer learning is there in reinforcement learning from verifiable reward?

• Can RL work in non verifiable domains? Can you set up some kind of ‘peer review’ with different agents/personas doing subjective critique?

Idiot savants

• What is the answer to this question I asked Dario over a year ago? As a scientist yourself, what should we make of the fact that despite having basically every known fact about the world memorized, these models haven’t, as far as I know, made a single new discovery? Even a moderately intelligent person who has so much stuff memorized would make all kinds of new connections (connect fact x, and y, and the logical implication is new discovery z).

  • People have proposed all sorts of answers to this. For example, Scott Alexander wrote,
    “Humans also aren't logically omniscient. My favorite example of this is etymology. Did you know that ‘vacation’ comes from literally vacating the cities? Or that a celebrity is a person who is celebrated? Or that ‘dream’ and ‘trauma’ come from the same root? These are all kind of obvious when you think about them, but I never noticed before reading etymology sites.I think you don't make these connections until you have both concepts in attention at the same time, and the combinatorial explosion there means you've got to go at the same slow rate as all previous progress.”

    • I agree that humans lack some godlike logical omniscience about the combinatorial consequences of all their knowledge. But there's plenty of examples of humans finding these kinds of important connections between fields, despite their much more limited world knowledge (see for instance these examples). I don't think Scott's argument explains why we have many examples of humans doing this, but none with AIs. And it's actually really funny that Scott is making this argument, because one of the things I love about his blog is that he personally has found countless numbers of these intellectually fascinating connections between fields. Where are the LLMs that have done this?

  • Another argument I’ve seen: Eric Schmidt (no not that one) writes, “I think I just don’t agree with your premise that there’s such low hanging fruit out there in the internet text corpus that hasn’t already been grabbed by smart, widely-read humans.”

    • Given that the number of potential connections increases as O(N^2) with the amount of knowledge we have as species, and that the amount of knowledge is itself growing at least linearly, I find it implausible that humans have exhausted this combinatorial overhang.

  • There is a “one man's modus ponens is another man's modus tollens” thing going on here. One way to interpret my original question is from an AI-skeptical point of view: The fact that LLMs aren't more powerful than humans despite their in-principle advantages over us suggests that they're not true AGI. But there's another interpretation of the same question - an interpretation which supports a more FOOMy vibe: Given the in-principle advantages LLMs have over us (in this case, as a result of their immense knowledge, but there are many others), once they actually do become AGIs, they'll fucking dominate.

• Related to the question of LLMs knowing so much shit: It seems like knowledge is really cheap to store. Wikitext is less than 5 MB. So why do we humans forget so much? Why do our brains conspire so hard against acquiring new facts (which in raw bits cost basically nothing) that we have to come up with incredibly hacky systems like spaced repetition to keep knowledge around?

  • One suggestive pattern, copy pasting from my review of Terrence Deacon’s book, The Symbolic Species: “Childhood amnesia (where you can’t remember early parts of your life) is the result of the learning process kids use, where they prune and infer a lot more, allowing them to see the forrest for the trees. On the opposite end of the spectrum are LLMs, which can remember entire passages of Wikipedia text verbatim but will flounder when you give them a new tic tac toe puzzle.There's something super interesting here where humans learn best at a part of their lives (childhood) whose actual details they completely forget, adults still learn really well but have terrible memory about the particulars of the things they read or watch, and LLMs can memorize arbitrary details about text that no human could but are currently pretty bad at generalization. It’s really fascinating that this memorization-generalization spectrum exists.”

• Can you have a 'superhuman AI scientist' before you get human level learning efficiency? (Currently, models take orders of magnitude more data that humans to learn equivalent skills, even ones they perform at 99th percentile level).

  • My take is that creativity and learning efficiency are basically the same thing. The kind of thing Einstein did - generalizing from a few gnarly thought experiments and murky observations - is in some sense just extreme learning efficiency, right? Makes me wonder whether low learning efficiency is the answer to the question, 'Why haven't LLMs haven't made new discoveries despite having so much knowledge memorized'?

New training techniques

• I’m confused why all the labs have ended up making models that are so similar. Everyone is making “thinking” models. Has everybody just been trying a bunch of different shit, but this is the only thing that works? Or are they just copying each other, but in fact there’s a bunch of equally promising tangential research directions that no one is pursuing?

• Many people have pointed out that there’s some missing middle between pre-training and in-context learning. Pre-training gives you some base of general understanding, something like a human skimming every textbook ever written; in-context learning is pure short term memory, discarded after every use. Are we likely to see a new training regime closer to dynamic evaluation, where you update your weights by meditating on feedback, or writing synthetic problems for practice?

  • Would the procedure for dynamically learning new skills have to be bespoke for each application (if you need it to be good at call center workflows, you need to make a bespoke conversation trajectory unrolling environment) or can you come up with some general procedure for upskilling?

• One idea I’ve heard for building long horizon strategizing and coherency is to train AI systems of text based strategy games. Has somebody already tried this and it didn’t work? Or it hasn’t even been tried in the first place?

• What would a good benchmark for meta learning and sample efficient learning look like?

  • If in context sample efficiency is high enough, is it okay if training sample efficiency sucks?

Pre-training

• Is pre-training actually dead?

  • The compute required to train a GPT-4 level model has been declining in cost at the astonishing rate of 10x per year. I thought the whole point of these effective compute multipliers is that we could train GPT-5 at GPT- 4 cost. From the outside at least, it seems like these astonishing compute multipliers are only making existing capabilities cheaper to serve, not enabling the next generation of more powerful models to arrive much sooner. Rumors are that all the labs have been struggling to crack the next OOM of scaling. What’s going on? Data is running out? Maybe the engineering for much larger training runs gets exponentially harder? Or maybe so called algorithmic ‘compute multipliers’ don’t give an equivalent multiplicative boost at different levels of scale?

• A couple months ago at ICML, Ilya compared the pre-training data corpus to fossil fuels - a limited resource which will rapidly be exhausted. This raises the question: what if this data corpus wasn’t limited? Would pre-training still be netting amazing new capabilities? Is next token prediction an innately fucked training objective, or did we just run out of data to let it cook?

• Why are LLMs such mediocre writers despite having all the good writing in their training dataset?

• Copy pasted from Wei Dai: “If the additional training data [after the 10s or trillions of tokens already used] is mostly low quality (AI labs must have used the highest quality data first?) or repetitive (contains no new ideas/knowledge), perplexity might go down but what is the LLM really learning?

Economics

Early deployment

• Why aren't all the center workers getting laid off yet? It’s the first thing that should go. Should we take it as some signal that human jobs are just way harder to automate than you might naively think?

• How will the difference between average S&P 500 returns and median S&P 500 company returns change over time?

  • In other words, will we live in the world where Nvidia, Microsoft, Meta, and Google become worth $10T, but everything else goes to 0, or whether broad deployment happens fast enough that McDonalds, JP Morgan Chase, etc. become much more productive at the same rate that AI becomes more powerful.

• Will the models of 2026 and 2027 still best be thought of as time-shares of intelligence, or will they be integrated into companies and workflows such that individual copies are meaningfully distinct?

  • Honestly, I think time-shares of intelligence are still underrated. A new AI employee can just read every single doc in your company's Drive and every single line of code in your company's codebase within minutes. This means that scaling up your company or an AI application would be way less effortful than scaling up a human department.

• What is the industrial-scale use case of AI? Between 1859 (when Drake first discovered oil in Pennsylvania), to 1908 (when Henry Ford invented the modern automobile), the main use for crude was as kerosene for lighting. What is the ultimate industrial-scale equivalent use case for AI?

• How transformative would the AIs of today (March 2025) be even if AI progress stopped here?

  • I've personally become more bearish about the economic value of current systems after using them to build miniapps for my podcast: I can't give them feedback that improves their scope or performance over time, and they can’t deal with unanticipated messy details.

    • But then again, the first personal computers of the 1980s weren't especially useful either. They were mostly used by a few hobbyists. They had anemic memories and processing power, and there just wasn't a global network of applications to make them useful yet.

    • Another way to ask this question: imagine that you plopped down a steam engine in a hamlet from 1500. What would they do with it? Nothing! You need complementary technologies. There weren’t perfect steam engine shaped holes in these hamlets; similarly, there aren’t many LLM-shaped holes in today’s world

Coding and remote work

• People like Dario say that in a year, 90%+ of code will be written by AIs. In some sense, compilers automated 90%+ of code writing. But I think compilers are a smaller deal than what these people imagine AI coders will be in a year. So what magnitude of actual productivity improvement to software engineering do they expect from AI? 2x? 10x? 100x?

• If you get a 100x increase in software productivity, what kinds of things could we build in the world that we can’t build now?

  • Here’s why I think this is such an interesting question. The economic historian Robert Allen thinks cheap energy is a big part of the reason the Industrial Revolution first happened in Britain. The first steam engine (the Newcomen engine) was super inefficient. It worked not by pushing the piston directly with the steam but by relying on the steam condensing to pull the piston in. The only place where it made sense to even try this design was plentiful coal mines in England. This allowed Britain to get up the learning curve in mechanization, such that they could design devices which were viable with much less energy. The tl;dr here is that you need some initial hand-holding - some cold start - in order to get up the learning curve in a new technology. And because of cheap coal, Britain had it for the Industrial Revolution. Does effectively free software cause some kind of cold start to some future technological trend?

• It seems like some AI labs (OpenAI, Meta) are racing towards being first to a billion user AI assistants, whereas others (Anthropic, maybe GDM?) are racing towards the fully autonomous software engineer. Who’s right? What are the marginal returns to more software (as compared to owning another Facebook- or Whatsapp-size social network)? If you believe in a software driven intelligence explosion, then that places a pretty high premium on the value of more software.

  • The internet and mobile revolutions were largely driven by digital advertising – a relatively small slice of overall global GDP. If AI truly can automate all labor, how soon before AI revenues absolutely gobble up all the social/search/ads revenue of big tech?

• How much of the value of AI requires embodied robots (versus just digital LLMs)?

Open source

• Does inference scaling mean that open-weight models don’t decentralize either the benefits or the risks of AI as much as you might naively think (more here)?

• What does an intelligence explosion with open-weight models at the frontier look like?

  • Is there any plausible way in which this could happen? Wouldn’t it require publishing the weights every week/day/hour (depending on speed of intelligence explosion)?

  • If this happened, would it be good?

• What are the implications of the “panspermia” timeline, where multiple countries develop AGI from the same initial seed (e.g., by stealing model weights or building on open weights models)?

  • Would this shared foundation matter more if future advances come from iterated amplification and distillation rather than fresh training runs?

  • Does this increase AI takeover risk, since all the superintelligences start from the same subconscious ‘id’?

  • Does it increase the global impact of ‘Western’ values which are embedded in those stolen weights?

Model training and value capture

• How are returns to training bigger models scaling? If a lab like OpenAI or DeepMind doubles its training compute budget for its next flagship model compared to the last one, are they getting more than 2x the revenue from it?

  • It seems like the ratio of revenue to training cost is increasing over time, plausibly because so much of the value of new skills is deferred to when AIs are fully reliable and complemented with a wider range of capabilities

• If progress in AI stalls, do foundation model companies inevitably get commoditized? Is there any moat other than staying 6+ months ahead (and in the extreme scenario beating everyone else to the intelligence explosion)? If model companies fail to differentiate, where does the value get captured?

  • One answer might be the hyperscalers who control the datacenter compute, and whose complement (the models) just got commodified. But datacenter compute itself doesn’t seem that differentiated (so much so that the hyperscalers seem to be able to easily contract it out to third parties like CoreWeave). So maybe the lion’s share of value goes to the people making the components that go into chip production: 1) wafer production (TSMC), 2) advanced packaging (TSMC’s CoWoS), and 3) high bandwidth memory (SK Hynix)

• Many previous attempts to make AI applications based on scaffolds and wrappers have been gobbled up by better foundation models (which can just "scaffold" themselves). Will that keep being the case?

• It's interesting to me that some of the best and most widely used applications of foundation models have come from the labs themselves (Deep Research, Claude Code, Notebook LM), even though it's not clear that you needed access to the weights in order to build them. Why is this? Maybe you do need access to the weights of frontier models, and the fine tuning APIs or open source models aren’t enough? Or maybe you gotta ‘feel the AGI’ as strongly as those inside the labs do?

• How much does being at the bleeding edge of AI capabilities matter? Is there any point in competing in the model race if you have no plan to get to the top? Or is there a viable business strategy based on being 6 months behind but following fast?

Investment

• Hyperscaler AI capex is getting pretty big - approaching $100B a year for some of the big dogs. If there is a lull in AI capabilities (e.g., inference scaling doesn't end up being that useful or easy to scale, and it takes a couple of more years to make another big breakthrough), what will happen? Will fidgety CFOs force a firesale of compute contracts?

• On the other hand, if AI does end up being as economically remunerative as "AGI" implies, how fast could the hyperscalers ramp up their investment? What would it take for them to invest more than their annual free cash flow (high 10s of Bs for Microsoft, Google, Amazon)? And how would they raise this money?

Hardware

• Will the desire to avoid the 70%+ Jensen tax inevitably drive more in-house ASIC development?

  • Will this drive greater diversity in model architectures and training techniques?

• How well will distributed training work?

  • How easily can datacenters that have been built up assuming a pretraining focus be repurposed towards RL?

  • Will future training just look a lot like inference? There's not much difference between inference workloads and RL, since an agent has to go away for a while, try to solve a problem, and come back with the results.

  • Can RL rollouts just be totally decentralized?

• How much do hardware tariffs, energy costs, and geopolitical considerations influence where we build the next generation of AI data centers?

  • If significant tariffs were imposed on datacenter components, would that shift planned buildouts towards Europe or Asia?

  • How much does network latency and proximity to customers matter when planning multi-billion dollar, decade-long infrastructure investments?

• How can we build hardware mechanisms that would help you prove to others what you’re using your compute for (and not using it for)?

  • A lot of the stories about how the intelligence explosion goes well involve different actors instituting a mutual slow down where they refocus on alignment + greater monitoring. But such agreements require verifiability.

Post-AGI

Hive minds

• Will central planning actually work with AI?

  • The case for:

    • The central AI can have much higher bandwidth communication with the periphery. Today, sensing and deciding have to both happen at the front line. In the future, robot appendages of Big Brother could actually just run the whole military and economy.

    • The central planner can just directly learn from the experience of other AIs (think of a much more advanced version of Tesla FSD model learning from millions of driving samples).

    • Compute/intelligence can be much more centralized than it is today. Right now, Xi Jinping has the same 10^15 FLOPs as anyone else. Not so for the mega-inference-scaled dictators of the future.

    • In order to align incentives for humans, we have to use markets. But we can much more easily control the preferences of AIs.

  • The case against:

    • This vision assumes that only the central government is getting more complex and capable, while the rest of society stays similar. But AI deployment will lead to the whole economy becoming more complex. Apple circa 2025 might be able to centrally plan the economy of Babylon. But it wouldn’t be able to plan America circa 2025.

• What is my blog post about fully automated firms missing?

• How soon after AGI do you get hive minds, fully automated firms, and other crazy powerful and unique products of AIs’ unique advantages in cultural learning/coordination (namely, that AIs can copy, merge, distill, and scale themselves)?

  • An analogy is that the first AGIs will be like humans 200,000 years ago: yes, they have many key advantages, but the things that make us so dominant today - state capacity, joint stock corporations, fossil-fueled civilization - took eons of cultural evolution, population growth, technological upgrading.

• Today, the world economy runs on trade; complex, mutually-dependent supply chains; specialization; and decentralized knowledge. None of this looks like a singleton. Will all this fundamentally change once we have AGI? How about ASI?

  • Or maybe a global system of interconnected markets, delegated decision-making, and mutual interdependence was the singleton all along?

Software only singularity

• How does the probability of an intelligence explosion change based on when we achieve AGI?

  • AGI timelines tell you a lot about the nature of intelligence itself. If it takes decades to build AGI (rather than 3 more years of koding-koding-koding), then you’ve learned that transfer learning isn’t that powerful. You’ve learned that current systems aren’t simply “hobbled” little AGIs - rather, they will seem in retrospect more like AlphaZero - an amazing early demonstration of the possibility of AGI, but not the thing itself.

• Could an AI competent at research engineering accelerate AI progress such that we make a BERT to GPT-4.5 size jump in just one year?

  • Case for: my podcast with Daniel and Scott about AI 2027.

  • Case against: my podcast with Ege & Tamay.

  • AI labs apparently run multiple pre-training teams, each with varying allocations of compute and number of researchers. Observing how progress differs across these configurations would yield valuable insights into the tradeoff between cognitive effort and compute scaling in AI progress—though labs are unlikely to share this data.

• Suppose there's a software-only singularity. Would the gap between AI labs narrow or widen during this takeoff?

  • There's a clear reason to expect the gap to widen during an intelligence explosion: whichever lab is slightly ahead benefits from having smarter, more capable automated researchers, rapidly compounding its advantage.

  • Yet historically, we've seen the gap consistently narrow. In 2018, DeepMind was far far ahead of everyone else. Today, labs like DeepSeek can close the distance to within half a year of OpenAI.

  • However, this pattern might not persist. The two primary methods for labs catching up—poaching experienced talent and learning from leaks or public deployments—would lose relevance in a scenario where the best models remain internal, autonomously driving progress.

  • Still, if the intelligence explosion accelerates by simultaneously scaling compute resources, the leading lab might remain incentivized to deploy publicly, attract investment, and thereby continue indirectly facilitating catch-up.

• Currently, model training (and inference) costs for the same level of capability are declining 10x a year. If you extrapolate this trend, then it suggests that you can train GPT-4 level capabilities using 100 H100s by 2027. Does this imply that while eventually we'll be able to distill human-level intelligence into mosquito drones, the first AGIs will be a system that costs $100B and requires Montana-sized infrastructure to train, and uses inference scaling hacks that cost $10 per token (like some Yoda, slowly meditating on each syllable)?

Transformative AI

• Some people imagine that once we achieve ASI, it rapidly invents crazy superweapons, new computing paradigms, fusion-powered space probes, etc. But this doesn’t seem to map on to how past new inventions and scientific discoveries were made. It seems like a general upgrading of your society's tech stack is more important than raw cognitive effort in a specific sector: you couldn't have discovered the structure of DNA without X-ray crystallography; and we didn’t figure out there was a Big Bang until we saw cosmic background microwave thanks to the radio astronomy techniques that were initially developed for communications during World War II. Does this suggest that we're not just going to have this technological explosion in the middle of the desert? In order to truly advance the state of technology, you basically need to upgrade the whole economy. And this points to a broad deployment before an R&D explosion.

  • To be clear, transformative AI might still happen super fast! The AIs are thinking very quickly, there’s billions of them, they’re much better at learning-by-doing, etc. But it’s simply a question of whether or not we’ll just straight shot nano-tech or dyson spheres and skip over all the boring everyday improvements in seemingly unrelated fields.

Explosive economic growth

• Will explosive growth happen?

  • The basic argument for explosive growth: Total economic output = Total Factor Productivity (TFP) × Labor × Capital. Today, economic growth has a weak feedback loop: output can accumulate into more capital, but not into more (human) labor. With AI, however, capital (compute) also functions as labor, creating a stronger feedback loop. Automated economic output = TFP (rapidly rising due to automated research and accelerated learning-by-doing) × Capital (previous output).

  • If you’re interested in other considerations, check out my interview with Tyler for the con case, and the one with Ege & Tamay for the pro case.

• Will explosive growth basically look like what has happened to China post Deng (double digit growth rates instigated by abundant skilled labor)?

• Will economic growth be extensive or intensive? Or some secret third thing?

  • If the former, then it’s possible that the effects of explosive economic growth will not be felt that widely: your cityscape will not look that different; your life might not even change that much. But out in the desert, somewhere far from your view, they're producing solar farms and robot factories which are worth 200% of the existing economy.

Alignment

Reward hacking

• “LLMs were aligned by default. Agents trained with reinforcement learning reward hack by default” (tweet here): is this actually the correct framing?

  • Base LLMs were also misaligned by default. People had to figure out good post-training (partly using RL) to solve this. There's obviously no reward hacking in pretraining, but it’s not clear that pretraining vs RL have such different 'alignment by default'.

• Are there any robust solutions to reward hacking? Or is reward hacking such an attractive basin in training that if any exploit exists in the environment, models will train to hack it?

  • Can we solve reward hacking by training agents in many different kinds of unique environments? In order to succeed, they’d have to develop robust general skills that don't just involve finding the exploits in any one particular environment.

• Do the models know when they’re reward hacking?

• Are capabilities and alignment the same thing here? Does making models more useful require solving reward hacking?

  • If this is the case, we might be living in the alignment-by-default world? It would be weird if we solve reward hacking well enough to make these models reliable general agents in every scenario except those involved in taking over the world.

Takeover

• What’s the right way to think about potentially misaligned AIs deployed through the economy? Are they like disgruntled employees who dislike the CEO (bullish for humanity), or more like Cortés landing in the New World (bearish)?

  • An optimistic framing: Think of AI as just another set of workers. CEOs typically aren’t the smartest people in their companies, nor do they grasp all technical nuances of how their instructions are executed by engineers or researchers. The same applies at the national level—Xi Jinping doesn't lose sleep over whether all 100 million CCP members genuinely love him, yet he remains in power. Plenty of employees actively dislike their CEOs but still inadvertently contribute positively to the company's objectives. Why should we assume misaligned AIs would behave differently?

  • A pessimistic framing: history is littered with examples of violent takeovers enabled by moderate technological advantages. Geneticist David Reich, on my podcast, described human history as repetitive waves of violent expansion, where a technologically or organizationally superior group wipes out most people across entire continents. Consider Hernán Cortés, who conquered an empire of over 10 million people in just two years with fewer than 1000 soldiers (plus horses, steel, and Smallpox). Or the British East India Company, which took over a subcontinent of 150 million people with just a few thousand officers, aided by marginally superior artillery tactics and logistical innovations derived from European warfare—not some profound, galaxy-brain advantage. Perhaps AIs similarly wouldn’t need an overwhelming technological edge to decisively overpower humanity.

• If the AIs ‘takeover’, will we be able to see it coming? How wild and sci-fi will it be?

  • Will it be like a gazelle being hunted by hunter-gatherers where it can clearly understand the strategic situation as it tries to run away from the pointy sticks?

  • Or will it be more like a deer peacefully grazing, unaware that it’s being watched through a scope 100 yards away, and a moment later… boom.

Model spec

• What should the model spec say?

  • Alignment stories often assume we can get an AI to deeply internalize a foundational document that spells out its core values, directives, and ultimate authorities—something like, "Follow user instructions unless they're clearly dangerous; in disputes, defer to the judgment of Sam Altman or President Trump."

  • Think about how much the United States today is contingent on the specific quill strokes of Madison - what exactly does this comma mean, what did he mean by ‘general welfare’ and ‘interstate commerce’?

  • How do we get political opposition parties, allied governments, and other people in the world to fully grasp the enormous (temporary) leverage they currently have to influence this document?

• If China develops AGI first, how do we make sure that their model spec doesn't just make Xi Jinping the god dictator forever? Do private Chinese companies and high-level members of the CCP have enough influence and understanding of the strategic situation to prevent such an outcome?

Misuse

• Is there an inherent tradeoff between preventing misuse and reducing the risk of a coup (either by AIs themselves or by humans using AI)? Or is this a false dichotomy?

  • Preventing misuse looks a lot like making sure the most advanced capabilities aren't widely deployed because of the inherent dual-use nature of understanding reality. Preventing takeover means making sure that we’re maximally open and transparent about frontier systems, and deploy them as widely as possible in order to prevent any one faction from monopolizing their benefits.

• Jailbreaking is becoming harder as models become smarter. This is pretty intuitive: a smart but ethical human assistant could tell if you’re asking him to help weaponize Smallpox or simply practicing for an organic chemistry exam. So as AIs get smarter, they should be able to tell the difference as well. Are we set here if we just put “don’t make bioweapons” in the model spec?

• It seems like the whole misuse story is especially anchored on bioterrorism. The intuition pump is to imagine every person with a team of virology PhDs in her pocket. What actually would happen in this scenario?

Other

Geopolitics

• I’m concerned about nationalization of AGI development for the reasons listed below. Where am I wrong?

  • Reduces the salience of safety and alignment, in favor of whatever the administration + deep state at the time cares about.

  • Decreases the competence of the group overseeing the intelligence explosion, which is robustly bad, especially if you think alignment is difficult and subtle.

  • Increases the likelihood of dictatorship, with the President ending up as the ultimate decision maker in the spec or something.

  • Instigates an obvious arms race framing in China.

  • Increases the likelihood that the first AGIs are used to develop weapons, drones, and bioweapons instead of what’s needed for our glorious transhumanist future.

• How will the Chinese political system react to fully automated remote work, superhuman hackers, automated AI researchers etc.?

  • How deep are their private markets? How willing are the big public funds to finance the next rung of scaling?

  • What might instigate a Chinese Manhattan Project for AGI?

  • Suppose tomorrow Xi wanted to prioritize building AGI. At an institutional level, what concretely could he do?

  • How does Chinese industrial espionage in other industries work? Suppose there's a hardware company that wants to learn how Apple does some procedure. Is there some government department they file their well-scoped snooping request with?

• Developing AI (and successfully deploying it at scale) is a huge industrial project. China is really good at industrial scale-up and state-directed investment. Does this give them a huge advantage in deploying AI?

• If you’re the leader of a nation like India or Nigeria today, and you get AGI-pilled, what should you do?

  • Honestly, it feels like a really tough position. The odds that you develop a frontier AI lab are pretty low. And you don’t have some crucial input into the semiconductor supply chain that will give you any leverage during crunch time.

  • If you have energy and property rights and friendly relations with the US and/or China, maybe you can become a datacenter hub?

• Does superintelligence actually give you ‘decisive strategic advantage’? Examples like Cortez or the East India Trading Company seem to imply that slight advantages in technology might be totally overwhelming in a military conflict. But perhaps modern weapons (nukes especially) are so destructive already that they provide sufficient deterrence against adversaries with smarter AIs launching their mosquito drone swarms and superhuman cyber attacks.

Epistemics

• How wrong should we expect our conceptual handles around AGI to be?

  • I've been reading The House of Government recently. It's a fascinating account of people involved in the Russian Revolution. There were many different factions of people who were disillusioned with the Czarist regime - the anarchists, the Mensheviks, Bolsheviks, the social revolutionaries, the Decembrists. They intensely debated the dichotomies which were most salient to them given their milieu.

    • The “decisive battle” … covered all the usual points of disagreement: the “working class” versus “the people”; the “sober calculation” versus “great deeds and self-sacrifice”; “objectivism” versus “subjectivism”; and “universal laws of development” versus “Russia’s uniqueness.”

  • Yet none of them anticipated the considerations we now recognize to be far more relevant to economic development: dispersed knowledge, voluntary exchange, and entrepreneurial innovation.

  • I think about this whenever my Bay Area friends debate AGI - will there be a software-only singularity, adversarial misalignment, training gaming, explosive growth, etc, etc? Maybe the frameworks we're using and the questions we're asking are fundamentally misguided.

• Given this topic is so epistemically murky that someone smart can come up with a new consideration that alters your key conclusions, how much should you update on the most recent compelling story you’ve heard?

  • Historically, the way we’ve dealt well with rapidly-evolving, uncertain processes is classical liberalism. Pushing for super tailored proposals based on reasoning your way to a specific trajectory (“We’re <5 years to ASI, therefore ‘The Project’”) has a pretty bad track record.

If you enjoyed this blog post, you may enjoy my new book, The Scaling Era: An Oral History of AI, 2019-2025. This book curates and organizes the highlights across my podcast episodes about AI with scientists, CEOs, economists, and philosophers.

Thanks to Max Farrens for comments and editing. Thanks especially to Carl Shulman, but also Leopold Aschenbrenner, Tamay Besiroglu, Ege Erdil, Daniel Kokotjlo, Scott Alexander, Sholto Douglas, Adam D’Angelo, Paul Christiano, Craig Falls, Gwern Branwen, Dan Hendrycks, Andrej Karpathy, Toby Ord, Neel Nanda, and many others for conversations which helped inspire questions.

Epistemic status: Shooting-the-shit; 25% sure this roughly describes how firms of AGIs will actually work.

Even people who expect human-level AI soon are still seriously underestimating how different the world will look when we have it. Most people are anchoring on how smart they expect individual models to be. (i.e. they’re asking themselves “What would the world be like if everyone had a very smart assistant who could work 24/7?”.)

Everyone is sleeping on the collective advantages AIs will have, which have nothing to do with raw IQ but rather with the fact that they are digital—they can be copied, distilled, merged, scaled, and evolved in ways human simply can’t.

What would a fully automated company look like - with all the workers, all the managers as AIs? I claim that such AI firms will grow, coordinate, improve, and be selected-for at unprecedented speed.

This essay is not a prediction of what GPT-5 will be doing, nor about emulations of existing humans. Rather, I'm trying to imagine what the world will look like once we actually have AGIs - the descendants of LLMs that have gotten so good that they can do basically anything any human can do.

Subscribe now

Copy

Currently, firms are extremely bottlenecked in hiring and training talent. But if your talent is an AI, you can copy it a stupid number of times. What if Google had a million AI software engineers? Not untrained amorphous "workers," but the AGI equivalents of Jeff Dean and Noam Shazeer, with all their skills, judgment, and tacit knowledge intact.

This ability to turn capital into compute and compute into equivalents of your top talent is a fundamental transformation. Since you can amortize the training cost across thousands of copies, you could sensibly give these AIs ever-deeper expertise - PhDs in every relevant field, decades of business case studies, intimate knowledge of every system and codebase the company relies on.

The power of copying extends beyond individuals to entire teams. Small previously successful teams (think PayPal Mafia, early SpaceX, the Traitorous Eight) can be replicated to tackle a thousand different projects simultaneously. It's not just about replicating star individuals, but entire configurations of complementary skills that are known to work well together. The unit of replication becomes whatever collection of talent has proven most effective.

Copying will transform management even more radically than labor. It will enable a level of micromanagement that makes founder mode look quaint. Human Sundar simply doesn't have the bandwidth to directly oversee 200,000 employees, hundreds of products, and millions of customers. But AI Sundar’s bandwidth is capped only by the number of TPUs you give him to run on. All of Google’s 30,000 middle managers can be replaced with AI Sundar copies. Copies of AI Sundar can craft every product’s strategy, review every pull request, answer every customer service message, and handle all negotiations - everything flowing from a single coherent vision.

There is no principal-agent problem wherein employees are optimizing for something other than Google’s bottom line, or simply lack the judgment needed to decide what matters most.1 A company of Google's scale can run much more as the product of a single mind—the articulation of one thesis—than is possible now.2

Merge

Think about how limited a CEO's knowledge is today. How much does Sundar Pichai really know about what's happening across Google's vast empire? He gets filtered reports and dashboards, attends key meetings, and reads strategic summaries. But he can't possibly absorb the full context of every product launch, every customer interaction, every technical decision made across hundreds of teams. His mental model of Google is necessarily incomplete.

Now imagine mega-Sundar – the central AI that will direct our future AI firm. Just as Tesla's Full Self-Driving model can learn from the driving records of millions of drivers, mega-Sundar might learn from everything seen by the distilled Sundars - every customer conversation, every engineering decision, every market response.

Unlike Tesla’s FSD, this doesn’t have to be a naive process of gradient updating and averaging. Mega-Sundar will absorb knowledge far more efficiently – through explicit summaries, shared latent representations, or even surgical modification of the weights to encode specific insights.

The boundary between different AI instances starts to blur. Mega-Sundar will constantly be spawning specialized distilled copies and reabsorbing what they’ve learned on their own. Models will communicate directly through latent representations, similar to how the hundreds of different layers in a neural network like GPT-4 already interact.3 So, approximately no miscommunication, ever again. The relationship between mega-Sundar and its specialized copies will mirror what we're already seeing with techniques like speculative decoding – where a smaller model makes initial predictions that a larger model verifies and refines.

Merging will be a step change in how organizations can accumulate and apply knowledge. Humanity's great advantage has been social learning – our ability to pass knowledge across generations and build upon it. But human social learning has a terrible handicap: biological brains don't allow information to be copy-pasted. So you need to spend years (and in many cases decades) teaching people what they need to know in order to do their job. Look at how top achievers in field after field are getting older and older, maybe because it takes longer to reach the frontier of accumulated knowledge. Or consider how clustering talent in cities and top firms produces such outsized benefits, simply because it enables slightly better knowledge flow between smart people.

Future AI firms will accelerate this cultural evolution through two key advantages: massive population size and perfect knowledge transfer. With millions of AGIs, automated firms get so many more opportunities to produce innovations and improvements, whether from lucky mistakes, deliberate experiments, de-novo inventions, or some combination.

As Joseph Henrich explains in The WEIRDest People in the World,

cumulative cultural evolution—including innovation—is fundamentally a social and cultural process that turns societies into collective brains. Human societies vary in their innovativeness due in large part to the differences in the fluidity with which information diffuses through a population of engaged minds and across generations

Historical data going back thousands of years suggest that population size is the key input for how fast your society comes up with more ideas. AI firms will have population sizes that are orders of magnitude larger than today's biggest companies - and each AI will be able to perfectly mind meld with every other, from the bottom to the top of the org chart.

AI firms will look from the outside like a unified intelligence that can instantly propagate ideas across the organization, preserving their full fidelity and context. Every bit of tacit knowledge from millions of copies gets perfectly preserved, shared, and given due consideration.

Scale

The cost to have an AI take a given role will become just the amount of compute the AI consumes. This will change our understanding of which roles are scarce.

Future AI firms won’t be constrained by what's scarce or abundant in human skill distributions – they can optimize for whatever abilities are most valuable. Want Jeff Dean-level engineering talent? Cool: once you’ve got one, the marginal copy costs pennies. Need a thousand world-class researchers? Just spin them up. The limiting factor isn't finding or training rare talent – it's just compute.

So what becomes expensive in this world? Roles which justify massive amounts of test- time compute. The CEO function is perhaps the clearest example. Would it be worth it for Google to spend $100 billion annually on inference compute for mega-Sundar? Sure! Just consider what this buys you: millions of subjective hours of strategic planning, Monte Carlo simulations of different five-year trajectories, deep analysis of every line of code and technical system, and exhaustive scenario planning.

Imagine mega-Sundar contemplating: "How would the FTC respond if we acquired eBay to challenge Amazon? Let me simulate the next three years of market dynamics... Ah, I see the likely outcome. I have five minutes of datacenter time left – let me evaluate 1,000 alternative strategies."

The more valuable the decisions, the more compute you'll want to throw at them. A single strategic insight from mega-Sundar could be worth billions. An overlooked risk could cost tens of billions. However many billions Google should optimally spend on inference for mega-Sundar, it's certainly more than one.

Distillation

What might distilled copies of AI Sundar (or AI Jeff) be like? Obviously, it makes sense for them to be highly specialized, especially when you can amortize the cost of that domain specific knowledge across all copies. You can give each distilled data center operator a deep technical understanding of every component in the cluster, for example.

I suspect you’ll see a lot of specialization in function, tacit knowledge, and complex skills, because they seem expensive to sustain in terms of parameter count. But I think the different models might share a lot more factual knowledge than you might expect. It’s true that plumber-GPT doesn’t need to know much about the standard model in physics, nor does physicist-GPT need to know why the drain is leaking. But the cost of storing raw information is so unbelievably cheap (and it’s only decreasing) that Llama-7B already knows more about the standard model and leaky drains than any non-expert. If human-level intelligence is more than 1 trillion parameters, is it so much of an imposition to keep around what will, at the limit, be much less than 7 billion parameters to have most known facts right in your model? (Another helpful data point here is that “Good and Featured” Wikitext is less than 5 MB. I don’t see why all future models—except the esoteric ones, the digital equivalent of tardigrades—wouldn’t at least have Wikitext down.4

Evolve

The most profound difference between AI firms and human firms will be their evolvability. As Gwern Branwen observes:

Why do we not see exceptional corporations clone themselves and take over all market segments? Why don’t corporations evolve such that all corporations or businesses are now the hyper-efficient descendants of a single ur-corporation 50 years ago, all other corporations having gone extinct in bankruptcy or been acquired? Why is it so hard for corporations to keep their “culture” intact and retain their youthful lean efficiency, or, if avoiding “aging” is impossible, why [not] copy themselves or otherwise reproduce to create new corporations like themselves?

His answer:

Corporations certainly undergo selection for kinds of fitness, and do vary a lot. The problem seems to be that corporations cannot replicate themselves … Corporations are made of people, not interchangeable, easily copied widgets or strands of DNA .. The corporation may not even be able to “replicate” itself over time, leading to scleroticism and aging.

The scale of difference between currently existing human firms and fully automated firms will be like the gulf in complexity between prokaryotes and eukaryotes. Prokaryotes like bacteria are not only remarkably simple, but have barely changed over their 3 billion year history5. Whereas eukaryotes rapidly scaled up in complexity, and gave rise to all the other astonishing organisms with trillions of cells working together tightknit6

This evolvability is also the key difference between AI and human firms. As Gwern points out, human firms simply cannot replicate themselves effectively - they're made of people, not code that can be copied. They can't clone their culture, their institutional knowledge, or their operational excellence. AI firms can7.

If you think human Elon is especially gifted at creating hardware companies, you simply can’t spin up 100 Elons, have them each take on a different vertical, and give them each $100 million in seed money. As much of a micromanager as Elon might be, he’s still limited by his single human form. But AI Elon can have copies of himself design the batteries, be the car mechanic at the dealership, and so on. And if Elon isn’t the best person for the job, the person who is can also be replicated, to create the template for a new descendant organization.

Takeover

So then the question becomes: If you can create Mr. Meeseeks for any task you need, why would you ever pay some markup for another firm, when you can just replicate them internally instead? Why would there even be other firms? Would the first firm that can figure out how to automate everything will just form a conglomerate that takes over the entire economy?

Ronald Coase’s theory of the firm tells us that companies exist to reduce transaction costs (so that you don’t have to go rehire all your employees and rent a new office every morning on the free market). His theory states that the lower the intra-firm transaction costs, the larger the firms will grow. Five hundred years ago, it was practically impossible to coordinate knowledge work across thousands of people and dozens of offices. So you didn’t get very big firms. Now you can spin up an arbitrarily large Slack channel or HR database, so firms can get much bigger.

AI firms will lower transaction costs so much relative to human firms. It’s hard to beat shooting lossless latent representations to an exact copy of you for communication efficiency! So firms probably will become much larger than they are now.

But it’s not inevitable that this ends with one gigafirm which consumes the entire economy. As Gwern explains in his essay, any internal planning system needs to be grounded in some kind of outer "loss function" - a ground truth measure of success. In a market economy, this comes from profits and losses.

Internal planning can be much more efficient than market competition in the short run, but it needs to be constrained by some slower but unbiased outer feedback loop. A company that grows too large risks having its internal optimization diverge from market realities.

That said, the balance may shift as AI systems improve. As corporations become more "software-like" - with perfect replication of successful components and faster feedback loops - we may see much larger and more efficient firms than were previously possible.

The market continues to serve as the grounding outer loop. How does the firm convert trillions of tokens of data from customers, markets, news, etc every day into future plans, new products, and the like? Does the board make all the decisions politburo-style and use $10 billion dollars of inference to run Monte Carlo tree search on different one-year plans? Or do you run some kind of evolutionary process on different departments, giving them more capital, and compute/labor based on their performance?

These are all what we would today call “culture.” Markets facilitate an evolutionary process which selects not only goods and services, but the institutions that are best at turning the world into valuable goods and services. I think this will continue.

I have no illusions about "understanding" China. I've only spent 2 weeks there. This trip is a beginning, not a capstone, of my curiosity about China. And I hope to share what I learn with you via future podcast episodes.

Subscribe now

Scale

It’s funny how China has basically the inverse problem as America. We subsidize demand and restrict supply. They subsidize supply and restrict demand. We can’t rebuild fallen bridges. They build bridges to nowhere. In the most desirable cities in this country, every random Victorian house and park bench is a historic site that can’t be disturbed. There, they’ll bulldoze a 500 year old temple to build an endless skyscraper complex that no one wants to live in.

My overwhelming first impression was: wow everything is so fucking big: the cities themselves, the train stations, the airports, the towering and endless apartment complexes. Travel often teaches you things about a country which you honestly should have intuited even without visiting. Obviously, I knew that China is a big country, with over 1.4 billion people. But the stupendous scale of the biggest cities was impressed upon me only after I visited.

Even in Emeishan, a city of just half a million people (considered a quaint countryside town by Chinese standards), we found a Buddhist temple of comical scale - we'd enter what seemed like an impressively large compound, only to discover it was merely the entrance to an even grander structure right behind it. This pattern repeated 5 or 6 times, each subsequent building larger and more ornate than the last, like some kind of inverse nesting dolls. And the place had almost no other visitors!

I asked a monk at the temple how they funded this massive site in a city of just half a million people. He told us that it was simply through donations. We probed further about how such an enormous project could have been financed by just ordinary people's contributions. He responded, "We've got a lot of supporters, dude", and changed the topic.

Chongqing is by far the coolest city I've ever visited. It's this insane cyberpunk multi-level metropolis of over 20 million people. I wouldn't know how to begin describing it, but there's a bunch of great YouTube videos which will show you what I mean. I got a really nice nice 2-floor hotel room that overlooked two rivers and one of the most insane skylines in the world for 60 bucks - highly recommend visiting Chongqing if you get the chance.

In 1995, astronomers pointed Hubble at a seemingly empty patch of sky the size of a grain of sand held at arm's length. Instead of emptiness, the 10-day exposure revealed over 3,000 galaxies. Every speck of light in the image was an entire galaxy containing billions of stars. When I went atop the tallest building in Chongqing and looked out over the city, I thought about the Hubble image. You could zoom in on any direction you'd find - behind the fog and mist, beyond even perhaps the horizon - another skyscraper, each containing hundreds or thousands of people living or working.

We took a 12-hour village train from Chongqing to Shanghai. I'm embarrassed to say that my only experience with the actual countryside was via the windows of this train. Still, the sights were quite interesting. We saw again and again small paddy farms surrounding a handful of 5-10 story skyscraper, plopped in the seeming middle of nowhere. Even in the countryside, many people lived in large buildings instead of their own small homes. I couldn't hop off the train and confirm, but I saw many towns that looked quite ghostly - no actually visible people anywhere.

Outside of Beijing and Shanghai (and sometimes even within), you can tell that these skyscrapers were put up by a country with a GDP per capita of $10,000 (and potentially half or quarter than when many of these buildings went up). America and Europe put up a ton of beautiful buildings in the early 20th century when their GDP per capita was similar to China's - one could even argue that those older structures are more aesthetic than anything we're building today in the West. Not so in China. These endless rows of skyscrapers, put up in the construction frenzy of the last few decades, are ugly - boxes of mostly concrete with visible blight and discoloration all over them. If the great construction binge is indeed over, it'll be a shame that China's infrastructure was built out during a period of particularly uninspired architecture.

Beijing's urban design looks like something straight out of James Scott's "Seeing Like a State". The city is dominated by these enormous apartment complexes - blocks of 10 adjacent 30-story buildings demarcated by 8-lane roads. The government buildings follow the same pattern: huge structures divided by extremely wide boulevards. This layout seems designed partly for social control - during zero-COVID, authorities could lock down 10,000 people by simply guarding a few entrance gates. The wide roads would also make it easy to move military forces through the city. The only break from this pattern are the Hutongs, Beijing's old historic neighborhoods. But even these weren't spared completely - only a fraction survived Beijing's rapid modernization push. Dare I say that China is too YIMBY?

Vibes

I got quite mixed messages about the state of public opinion in China. This is to be expected in a society where you can't establish common knowledge. One person told me that the new generation is quite nationalist, unlike the older reform generation which personally experienced the catastrophes of Mao and the tangible benefits of liberalization. He made the rather insightful point that this tilt in Chinese public opinion increasingly gives lie to the American talking point, "We're against the CCP, not the Chinese people." In fact, he went on to say that the current regime is way more liberal than what would result from an election in China.

Another person told me that these Chinese nationalists were only a vocal minority, similar to the wokes in America circa 2020. While they make up only about 10% of the population, they aggressively shout down others on Weibo (China's Twitter equivalent). Most people find them annoying but feel uncomfortable confronting them directly. This matches what a student who graduated from a top university there told me - the vast majority of his classmates are simply apolitical. And in our own interactions with locals, we saw little evidence of widespread nationalism. In fact, when my Chinese-speaking trip mate would mention he was from the UK to taxi drivers, they would often respond enthusiastically: "Oh wonderful, we love the UK!"

There were very few foreigners. In Beijing I might have seen half a dozen cumulatively across entire seas of people. In Chengdu and Chongqing, I barely remember seeing any (which is a real shame, because Chongqing is a truly incredible tourist destination). So rare apparently are tourists that in Chengdu and Chongqing we got asked for selfies many times.

Outside of Shanghai, almost nobody spoke English. If I went again, I would definitely try to crash course some basic Chinese beforehand. This language barrier did led to some interesting encounters. At a park in Chengdu, an old man reading a book called Medical English asked us to join him for tea. He was mostly just trying to practice his English. He said he loves foreigners, and that his favorite period of life was the 80s and 90s - "We love Deng!".

My trip mate's friend's grandmother was incredibly gracious in hosting us for a dinner while we were in Emeishan. We saw a beautiful Uighur rug hanging on their wall. The grandfather pointed it out and explained, “We have amazing relationships with minorities here in China.” I don't think this was some theatrical attempt to contradict Western propaganda. It seemed like he genuinely didn’t know what is said about the treatment of Uighurs in the West. He was just trying to show his visitors something cool he had.

We kept trying to ask them about their personal experiences over the last seven decades as China has grown and changed. but the grandfather kept responding with lengthy monologues about military history. Here is a representative exchange:

Me (who doesn’t speak Chinese) to my trip mate (who does): Ask them about what job they had when they first moved near Chengdu.

[The grandfather speaks in Chinese for 10 minutes. I get a bit impatient and whisper to my trip mate asking what he’s saying]

My trip mate: He's saying that the mountains around Chengdu make it a perfect base of retreat in case of an invasion, which Mao was worried about during the Sino Soviet split.

Me: What?! Why didn’t you ask him about the first job he had?

My trip mate: I did!!

Grandpas are the same everywhere.

People say that Xi is establishing a cult of personality. This may be true within the CCP cadres, but I saw no evidence of it in public. I don't think I saw a single picture of Xi anywhere - not on any billboards, screens, or walls. People didn't really bring Xi up in conversations. I saw some pictures of Mao, but mostly in museums (or in one case at a tea farm he apparently used to frequent). The hammer and sickle was also a rare sight, mostly displayed on government buildings.

There are indeed cameras everywhere. This is gonna sound super naive - but I genuinely don't understand why. There's no crime. I know you'll say it's to prevent protests. Which might make sense for major streets, but even random alleyway corners will have a couple of cameras. Are they really trying to prevent someone from fomenting insurrection between 2 garbage cans? Beijing in particular had police officers at attention at what seemed like every street corner.

One recent student I talked to said that they understand they don't have freedoms here, but they're willing to take the tradeoff in favor of safety - they don't want school shootings. I thought this was quite silly - not only because there's no reason political freedom ought to lead to school shootings, but mostly because school shootings are such a statistically marginal experience. But I realized this is exactly the way we treat any hints of public protests in China. Just as school shootings are featured heavily in the media but aren't actually something you're likely to personally encounter, so too with protests against the CCP. You are overwhelmingly unlikely to spontaneously encounter them.

People were quite willing to chat openly in public places about problems in the country and with the regime. Including people who seemed to have a lot to lose. Almost everyone I talked to would acknowledge the economy was bad, and many were willing to implicate the government's decisions. Some even casually brought up Tiananmen or the Cultural Revolution. One person was even willing to discuss the odds of regime change at a public restaurant - though he may been have an especially careless fellow.

To be clear, it's an authoritarian system, and I certainly would feel uncomfortable doing what I'm doing there, but it definitely isn't North Korea.

Youngsters

In a shopping mall in Chongqing, a couple of high schoolers came up to us in order to get selfies. It felt like the perfect opportunity to learn about young adult life in China. So I whipped out the Translate app in WeChat and proceeded, rather clumsy, to make small talk. I asked them what they did in their free time. They said they watched 2-3 hours of TikTok every day. I asked them what videos they'd watch. They said it's a whole bunch of "sexy girls". I laughed because I thought they were joking. I asked one of them to pull out his phone. He scrolled past the first 10 videos on his feed and they were indeed all just "sexy girls".

We chatted up quite a lot of young people on night life streets. I was struck by how many young people expressed feeling stressed or overwhelmed. We met a musician in Chengdu who was writing songs about youth anxiety. We chatted up some modeling school students - even they complained about the intense pressure they felt. We met a guy who had studied in Australia but returned to China during COVID. He explained that many of his friends with prestigious degrees are moving away from Shanghai and Beijing - Yes, the pay there can be twice as high as in second or third tier cities. But the competitiveness is insane. And in order to actually land the high skilled positions, they have to work truly insane hours (9-9-6 is not a myth). He said that many of his friends were opting for these less ambitious lower-paying careers in smaller cities, where the rent is lower and the pressure is manageable.

Speaking of which - we really need to do a better job with Chinese students studying abroad in America. These students will likely end up in influential positions back home, yet colleges treat them basically like cash cows. They often arrive bombarded with propaganda about America, which is reinforced by the prevalent discourse at universities. And they find themselves isolated by language barriers and cultural differences. Giving these future leaders a genuinely positive experience in America might be the best thing we can do to improve US-China relations in the long run.

I'm still puzzled by how China can have both a demographic collapse and massive youth unemployment. You'd think with fewer young people being born, the ones who are around would be in high demand. One explanation I heard while there is that there are plenty of menial jobs available, but today's educated youth - who've gone through high school and college - just won't take the low-skilled positions their parents and grandparents did. Meanwhile, there's a real shortage of the high-skilled jobs that would actually match their education and aspirations. It's a mismatch between the jobs available and the jobs young people feel qualified for and willing to do.

I kept asking young people about the public intellectual landscape in China - who are their equivalents of Jordan Peterson, Joe Rogan, Lex Friedman, and Sam Harris? The sense I got is that this kind of popular intellectual ecosystem just doesn't exist there. Sure, there are viral Bilibili videos from professors talking about practical matters like how to manage your finances. But grand takes about what's happening in the world and what we should do about it? Not much going on.

I met a couple of EAs in China - they're exceptionally rare. When we asked them why it's so hard to spread EA or similar worldviews there, they said people just aren't into that organizing decisions using some ideological lens. They're much more concerned with practical, tangible matters. One talking point I heard again and again is that, "China is still a developing country." The implication seemed to be that China can't afford to pursue costly idealistic programs around climate change, social safety nets, or international aid - it needs to focus on basic economic development first.

The Great Firewall

By far the most inconvenient thing about visiting China is internet access. As a foreigner, basically all the websites you might find useful are behind the firewall. Even for websites that aren't blocked by the Great Firewall, I just didn't have anything even close to what you might call high-speed internet. This was true across my entire travel, including my burner SIM card from ChinaMobile or the WiFi at fancy hotels in Beijing or Shanghai. We released an episode of my podcast during this trip - because of the low bandwidth, my editor and I were just sending screenshots to each other throughout the night instead of hopping on a video call. The VPN situation is worse than I thought it would be . I happened to be lucky enough to download one of the VPNs that sometimes works before the trip. It was still pretty slow and unreliable. Astrill or Mullvad might actually result in a connection 75% of the time - often they would cause WeChat and Alipay to crash (which is rather inconvenient since you basically need those apps open all the time to navigate China). Very possible that I was fucking something up. But between these internet issues and the need to use a burner phone and laptop, I would be quite reluctant to use China as a remote work location.

Tech & AI

I am super hesitant to say anything here, because I am extremely unconfident about what's actually happening. Treat this just as some tentative notes from a few conversations.

I've cut out a whole bunch of stuff about AI in this section because I'm not sure if my initial assessment is correct. I hope to make podcast episodes over the next few months which provide a better researched account than I can supply here.

The biggest surprise from talking to Chinese VCs people at AI labs was how capital constrained they felt. Moonshot AI, one of China's leading AI labs, raised $1 billion at a $3 billion valuation. Meanwhile, just xAI's new cluster alone will cost $3-4 billion.

The tech ecosystem feels quite shell shocked from the 2021 crackdown. One VC half-jokingly asked if I could help him get his money out of China. If you keep your money in China, you're basically stuck choosing between terrible options. You can either accept a measly 2% yield from state banks, or throw it into China's perpetually struggling stock market. This helps explain why valuations for Chinese companies are chronically low - the exit opportunities just suck. Even if you build (or invest in) something great, there's no guarantee the company will be able to raise the next round. And even if you do raise again and succeed, the government might randomly cancel your IPO. And even if you somehow make it to the public markets, Chinese equities have been performing terribly anyways. It's a good reminder of how easy it is to completely wreck an innovation ecosystem that depends on risk-taking investors.

Hearts and Minds

In China, liberal pro-Western voices are often censored or shouted down. If I was the US President, and I wanted to win hearts and minds in China, here's what I'd do. In every single speech where I'm talking about China, I'd make a conspicuous effort to complement Chinese people, Chinese values, and Chinese culture. I'd talk about how my Chinese staffers are the smartest and most hardworking people I've ever worked with (which honestly is probably true). I'd talk about how much my daughter is obsessed with ancient Chinese dresses. I'd talk about how I'm learning Mandarin in my free time, and have a live "Aw shucks" conversation in Mandarin.

These clips would go viral on Bilibili and TikTok. And they'd probably stay up because it would just be a weird thing to censor. The CCP might even think that these displays of affection aggrandize them. But in reality, showing our admiration for Chinese people and their achievements (who genuinely are fucking killing it everywhere where they're not held down by communism), undermines the central narrative of the regime - that the West is hell bent on holding Chinese people back, that they have no respect or understanding of their culture, and that the CCP is a necessary bulwark against these imperialists.

On a totally unrelated note, so many people I met in China are in fact super talented and hard working. Someone connected me with the CEO of a company that manufactures life sciences equipment. He told me that before he started this company, he used to be a repairman for foreign imported machines, traveling 200 days a year with a screwdriver to wherever one had broken down. In 2013, he designed and started manufacturing his own machines, building more and more advanced designs, and now they've just built out a $60 million factory 1.5 hours from Shanghai. I asked him what the hardest part of ramping up production is - apparently constructing factories is just not an issue - tons of construction firms can make you a new facility quite reliably (along with the adjacent dorm room building for workers).

What is travel good for?

Noah Smith has a good blog post about what one can't learn from travel - you're not going to learn about the risk of a war or the state of the AI race by gazing at skylines or chatting up taxi drivers. Of course you can learn about those things by talking to the princelings and researchers and CEOs. But if you have access to these higher ups, surely you can also get them on the Zoom call. And fwiw, this should you update you in favor of more Zoom calls, not less travel. During the trip I realized how much I could have already learned about China by meeting the listeners I have in China (who are unusually high quality, presumably because of the restrictions they have to get through to get to my content).

Incidentally, if you're in China (and especially if you work on anything related to AI) please email me at hello@dwarkeshpatel.com. I'm keen for people to reach out and suggest guests to chat about China. Especially keen to better understand the Chinese political system, and how decisions about mega projects, arms races, and technology investment will be made at the brink of AGI.

So what's the point of travel? What about the in person experience can't be replaced by books, travel vlogs, and Zoom calls? For me, it's something like, what becomes salient to you. I started asking questions about China I hadn't even thought to ask about America.

Another thing I noticed is more personal. Two weeks of being AFK, and of having the excuse of using a burner phone to put off messages, helped me clear the cache of thoughts about sponsorships, logistics, growth, hiring, and a bunch of other practical minutiae. My shower thoughts wandered away from upcoming negotiations and towards interesting rabbit holes.

It's a good reminder that what's lacking in life is not time. It's focus. If you're working on what matters, you can advance leaps and bounds in 8 hours. And if you're just clearing the slog, you can spend a lifetime staying in the same place.

Subscribe now

Chapter on China in Trade Wars are Class Wars by Michael Pettis and Matthew Klein

When China was booming, naive Westerners loved saying, "Our leaders think in 4 year terms - the Chinese government thinks in decades and centuries". In practice, authoritarian states (including China) are often far more short termist than market based democracies, because signals about the rational allocation of resources (like what people want to invest in, how they value different goods, where they want to live and work) are systematically repressed for political benefit.

Michael Pettis and Matthew Klein explain this incredibly well in their new book, Trade Wars are Class Wars. I learned more from simply their one chapter about China than I did from the the sum of every other book I read about the Chinese economy.

In China, the GDP growth rate is an input into the system. It is set early in the year as the GDP growth target for that year and represents the amount of growth needed to accommodate social and political objectives, among which of course is the desire to keep unemployment low … The easiest way for officials to hit their targets is therefore to tell the state-run banks to lend to favored companies to invest in as much infrastructure, manufacturing, and real estate as necessary. Whether the investments are worthwhile is irrelevant. All that matters is that the quantity of spending generates enough reported GDP to meet the central government’s objectives.

When I was in China, an investor (somewhat) jokingly asked if I could help him get his money out. He faced terrible options: put savings in state banks for a 2% yield, invest in China's perpetually struggling stock market, or... that's basically it, since capital controls trap money inside China. This creates a stealth tax on savers - you’re not allowed to start competitive bank offering real market interest rates from lending to productive private companies. Instead, state banks funnel money to local government projects and state enterprises. They might build those empty cities and bridges to nowhere "efficiently," but without market pressure to make actually useful investments, the capital often gets wasted.

This financial repression1 leads not only to massive malinvestment (which the economy is now paying for) but also redistribution from households to state favored companies.

I remain confused about how this fits into the bigger point that Pettis makes, which is that the Chinese government is causing distortions which lead to too much saving and not enough consumption. Isn't this financial repression an implicit tax on savers? Maybe the government is doing a bunch of other things which more than offset this tax on savers? Adam Posen has a more compelling explanation - this glut of savings is not the result of particular macroeconomic policies. Rather it’s the public’s natural hunker-down response to the government’s capricious handling of Zero-Covid.

Heilman’s China’s Political System has a very useful chart illustrating explicit taxes - as you can see, they’re quite low, and mostly dominated by value added tax and corporate taxes, with personal income taxes being negligible.

But I think these implicit taxes (this penalty on savers, or the land sales on expropriated property2) are indeed taxes. So I asked Claude to chart them. Claude came up with these numbers on its own - so take them with a grain of salt.

Up until the 1990s, this state led investment paradigm was fine:

At least until the mid-1990s, this incentive system was not a problem for China because the shortage of infrastructure and manufacturing capacity was so large. The only important constraint on productive investment was the pace at which savings could grow. Almost any investment increased productivity by far more than the cost of the project.

But afterwards this dynamic began to change. In response to the 2008 financial crisis, the Chinese government doubled down on its investment led growth model, launching a massive 568 billion dollar stimulus. The subsequent malinvestment has been so pervasive that total factor productivity in China, which is already significantly lower than America's, has actually decreased.

Paul Krugman wrote an excellent essay in 1997 called The Myth of The Asian Miracle, which explains why an increase in productivity, as opposed to a buildup of inputs alone, is crucial for long run economic growth. He explains how the initially impressive period of Soviet growth was inherently limited, and why the same dynamic would constrain the growth of China and Japan:

sustained growth in a nation’s per capita income can only occur if there is a rise in output per unit of input.

Mere increases in inputs, without an increase in the efficiency with which those inputs are used—investing in more machinery and infrastructure—must run into diminishing returns; input-driven growth is inevitably limited…

Even a modest slowing in China’s growth will change the geopolitical outlook substantially. The World Bank estimates that the Chinese economy is currently about 40 percent as large as that of the United States. Suppose that the U.S. economy continues to grow at 2.5 percent each year. If China can continue to grow at 10 percent annually, by the year 2010 its economy will be a third larger than ours. But if Chinese growth is only a more realistic 7 percent, its GDP will be only 82 percent of that of the United States. There will still be a substantial shift of the world’s economic center of gravity, but it will be far less drastic than many people now imagine.

Btw, people give Krugman tremendous amounts of shit for underestimating the economic impact of the Internet. But calling China’s stagnation 3 decades early, during a period in which it was pretty contrarian, is actually super impressive.

All these loans from state owned banks to often unproductive investment amount to a stupendous total debt load for what is still a developing country. The following chart was made by Claude, including the numbers, so again, grain of salt.

A future Chinese reformer would actually face a tougher challenge than Deng did in 1978. This seems like a strange thing to say - surely being the second largest economy in the world puts you in a better position to continue further growth. But Japan and the Soviet Union showed that when a mature economy crashes from a debt bubble, recovery is brutal. Deng, in contrast, got to start from scratch with several advantages: minimal existing debt made borrowing cheap; barely any existing infrastructure meant new investments were super productive and maintenance costs were low; plus China cashed in on history’s largest demographic dividend with tons of young workers and falling birth rates.

Now all these advantages have flipped into disadvantages, making any future reforms much less likely to deliver any quick wins.

This isn’t the most important consequence, but aesthetically this is a tragedy. The 3 decade construction frenzy which ended just a couple years ago put up endless shoddy looking skyscrapers everywhere. If this is the end of major construction for a while, then they’ve built a country that is uglier that it might have otherwise been.

Pettis and Klein offer a compelling take on China's Belt and Road Initiative: it's not primarily about building political alliances - it's about exporting China's domestic investment model abroad. Chinese companies get state subsidies to build infrastructure projects overseas, just like they do at home. But there's a key difference: when these projects turn out to be unproductive (think empty ports or underused railways), it's the host countries that end up stuck with the maintenance costs and depreciation, not Beijing. It's essentially a pressure release valve for China's overcapacity in construction and heavy industry.

The 2008 crisis seemed to vindicate the worldview that the free market model was inherently flawed. Instead of pressing hard on reforms, which would have been much more palatable before this huge buildup of debt and malinvestment, the Chinese government doubled down on state led investment and infrastructure.

In 1961, Nobel laureate Paul Samuelson confidently predicted in his economics textbook that the Soviet economy would overtake America's by the 1980s or '90s. During Japan's rise, Western experts couldn't stop gushing about how MITI's central planning was the secret sauce behind their economic miracle. More recently, books like "How Asia Works" praised capital controls and industrial policy - ideas that look a lot less brilliant now that we're watching their consequences play out in China.

Maybe Bryan Caplan had it right when he said that libertarians do indeed have “timeless ideas for a better world”. Maybe we should stop getting excited every time we think we've found an exception to market principles, which again eventually joins the pile of disappointments.

China’s Economy: What Everyone Needs to Know by Arthur R. Kroeber

• Advocates of an authoritarian model (who are particularly inspired by China) don't take seriously enough fragile good government is in these societies. You go from Mao (on a death toll basis, literally the worst government in the history of human kind - 50m+ dead) to Deng (inspired leadership) to Jiang Zemin (who was kept on the path of reform by the power behind the throne Deng) to Hu Jintao (2008+ property bubble, malinvestment, etc that may leave China stuck as middle income country) to Xi (zero Covid, lack of growth, centralization of power). A system of government which requires a Deng type figure to wrestle against all odds to provide great leadership for 20-30 years, and then returns to the same-old same-old is not optimal!

• Is there a single example of a country which experienced rapid industrialization and fast growth which was a full democracy at the time? Japan during Meiji was not, neither was SK/Taiwan/China during periods of highest growth. UK and US were very partial democracies. Japanese and German growth after WW2 was very fast, but some of it was rebuilding what they already had.

  • Deng is counterfactually crucial even after 1992. He basically told Jiang Zemin in his Southern tour that if he didn't keep up reforms, he'd replace him.

• Why is the example of SK/Taiwan/Japan not a valid argument for the idea that political reform follows economic reform? In those cases, political reform was the result of strong US pressure (or in the case of Japan, imposition). It's not clear that it would have happened by default.

• China's cities have surprisingly good infrastructure for a middle-income country, but there's a specific reason why: it's all about how they finance it. In China, the government owns all land - when you "buy" property, you're really just getting a long-term lease. This lets provincial governments use a clever (but dangerous) funding trick: they borrow money against land they control, justifying high valuations by promising future infrastructure improvements will drive up prices. For years, this worked beautifully because property values kept soaring.

  This system went into overdrive after 2008, when Beijing's stimulus plan essentially told local governments to go wild with infrastructure spending. They did exactly that, borrowing heavily against their land to fund endless construction projects. The result? Local governments are now sitting on a mountain of debt, as shown in the chart above.

• People often point to China's explosive growth from 1980-2020 (averaging around 10% annually) as a model for potential AI takeoff. But how good is this analogy? Some argue this wasn't just about throwing more resources at the problem - until 2008, roughly three-quarters of China's growth came from Total Factor Productivity (TFP) increases, not just adding more labor or capital. After 2008, this shifted as government malinvestment took over.

  But here's the thing: high TFP growth doesn't necessarily mean genuine innovation. TFP is just what's left over after accounting for labor and capital inputs - so even simple changes like "Mao dying and letting markets work" show up as TFP growth. Are we really expecting AIs to generate truly new ideas at the same rate that developing economies can copy existing ones from more advanced countries? That's a much bigger ask.

• Tiananmen taught the Chinese Communist Party a crucial lesson: urban unrest is the biggest threat to regime survival. But there's an often-overlooked economic context to those protests - inflation was surging in the late 1980s. This wasn't just a political crisis, it was an economic one.

  The inflation happened because of Deng's early reforms: they partially liberalized prices while still maintaining the dual-track system (where some prices were market-based while others remained state-controlled). This created opportunities for arbitrage between the two systems and, combined with rapid credit expansion and wage increases, sent prices soaring. By 1988-89, urban residents were watching their savings and purchasing power evaporate.

  So when party leaders looked at Tiananmen's aftermath, they drew two conclusions: first, keep urban residents happy at all costs; second, maintain strict control over prices and financial stability. It's a classic case of economic hardship driving political upheaval.

• If farmers in China had received the true market value of their land when expropriated by city governments, and were able to receive normal market returns on that investment, their wealth would be 8% of GDP higher.

• Land reform is important for catchup growth because it increases agricultural yield, which allows for exports, which gives you foreign exchange, which you can use for machinery.

• Export discipline is important for catch up growth because it forces firms to win by global standards (instead of political influence over domestic markets).

• Why did China have to rely on FDI but not South Korea/Taiwan/Japan (who built up domestic champions instead)? Because SK, Japan, Taiwan was part of the US alliance, which gave them access to American technical know-how and market access. China wasn't part of this alliance, so had to rely on FDI.

• The book nicely explained why value-added tax is more easily enforceable than other kinds of taxes. The merchant who sells the final product has a strong incentive to make sure his suppliers are reporting their sales accurately. Because the taxes on any difference between what the suppliers charge and what the final consumer pays has to be paid by the merchant. This dynamic plays out recursively with all the intermediate suppliers.

• I realize that I didn't mention Hukou anywhere. I'd be curious if somebody has done an economic analysis of what percentage of GDP this system is costing China. Based on the naive extrapolations of the cost of rent being too high in American cities (which is much less intense than a rural/urban government registry), getting rid of Hukou might well increase China’s GDP over 50%.

Subscribe now

This one on Japan during WW2. If you can't make it in person, you can livestream it via the Substack app.

Just download the app and subscribe to me. Around 7 PM Pacific you'll see the live video front & center on the homepage in the app.

Very excited!