The jump from $1,000 to $7,000 a day on coding agents for OpenAI's heaviest users is the number I keep coming back to. Is spend a good proxy for progress there, or mostly for how much those researchers let agents run unattended?
Read this twice (almost), and the part I keep returning to is in your own appendix: even in the software-only singularity scenario, the swarm is bottlenecked by compute for experiments, and inference efficiency — "more juice out of your compute" — is one of your three verifiable buckets of AI R&D.
Both scenarios treat the compute stock as the binding constraint. My fund invests in early-stage deep tech rebuilding the substrate layer across AI, energy, and compute, so here's the question your framing raises for me: what changes if the constraint isn't only how much compute exists, but how much useful work each unit of it performs — on the hardware already installed?
I ask because a portfolio company — I advise it, and the fund is lead investor — is built exactly on that seam: computing from intent rather than code, with measured results on existing GPU hardware of 84–114× execution speed gains and up to roughly 96.7% less energy per workload, and Phase 1 pilots with major hyperscalers complete. If the cost per experiment falls by an order of magnitude on the fleet that already exists, does your spark burn longer — or start to look more like the singularity?
Second thread: a swarm of autonomous researchers presumes we can govern what thousands of agents are authorized to do. Their design answer is governance before execution, not after. Do you see governance as a real bottleneck in the RSI picture, or a rounding error?
Genuinely curious — these types of intellectually curious reads sharpen how I underwrite this layer.
There are many unmotivated assumptions here but almost all of them come down to how expensive frontier-expanding ML experimentation is. Take, for example:
During RSI, automated engineers will build tens of thousands of times more RL training environments than we have currently. These will be far longer-running, higher-quality, and more open-ended, and will cover a much wider range, than today’s environments.
Who is going to pay for those? It’s not free to run these experiments — in fact it’s much more expensive than it is for math and coding because you don’t have deterministic verifiability loops. What CFO is going to allocate the marginal $500M to 100 automated experiments that might yield nothing vs 5 highly convicted bets from expert ML researchers and engineers?
We’re not in the cheap money, low opportunity cost, throw spaghetti at the wall era of 2023/2024 anymore. If OpenAI can barely raise $10bn at a $1T valuation, where is the capital going to come from to fund these low-conviction, high-volume experiments?
Cool article, it summarizes the super intelligence being imminent argument well.
Not saying I agree but just summarizing;
1. We have seen models get good at many tasks over the past few years, but we’ve seen them become better than any human at those with outcomes that are verifiable for correctness (math, cs, and programming)
2. AI research is more like math than it is like writing a novel. Many or most parts of it will be verifiable so we should expect AI to get super human at it as well. Evidence of this is recent AI labs seeing big increases in speed of AI advancement, faster than we’d expect from just compute, credited to AI models being smarter.
3. Okay, so very quickly, we will see models become super human at yet another verifiable domain. So what? Well, if the verifiable domain is AI research, then models will be able to do all of the things frontier labs have been doing to improve model usefulness and intelligence, but much faster than current capabilities
4. Even if AI research is fully automated and progress takes off at a rate much faster than now, does that lead to models becoming better at a bunch of general, non verifiable tasks? Very possibly, because we have seen these models become much better at such tasks.
So, ai research becomes autonomous, as if the best researchers in the world are now doing this research but instead of being humans, it’s models. This leads to explosive growth in ai rate of change for obvious reasons that models are more clonable and can run non stop. This leads to basically everything being automated. We have AIs that are general enough to be better than most humans at literally every task, regardless of how long the task takes to complete. For instance, right now, AI is better than most programmers at building a new website from scratch, in pure time it takes. But in a year or two, ai could be better at any human at the task of curing cancer or running a company.
Cool idea!
I think step one is do frontier AI labs fire ai researchers (they probably keep the top 10%) and basically fully automated the research pipeline, in 2027.
If things seem to not be heading in that direction, this is evidence that something is wrong with the hypothesis.
I expect the labs are currently rushing to formalise Physics and other science in Lean which may lead to an avalanche of science papers in next few weeks similar to what we have seen for Maths.
Superintelligence in coding and AI research seems likely.
Superintelligence is far too broad a term. I know you argue that a superintelligent AI researcher is going to be good other places, but ... unlike the other arguments you make, this one seems speculative to me. I still catch Claude 5.5 making mistakes daily. Maybe given a red-team of agents, it would do better, but ... like I said, it feels speculative. RSI less so.
Outside the verifiable areas you discuss, where is AI making progress and how is it being measured? I would love an essay on that. I just see people using Claude Code and Codex and it's magic.
Also, Kahnemann's lesson is we can't actually predict the future. I laugh everytime I see a smart person say "I am 25% certain of [insert a complex, chaotic process involving human beings]". We all want the world to be like Newtonian physics and math.
I see the lack of discussion of what it takes to write a pulitzer-worthy novel or to choose one's path as the thing we need to shape a machine that we should trust to give access to the foundations of civilization... we're skipping that discussion. What could go wrong? ;)
I feel like “what it takes to choose one’s path” is a much easier problem than writing a Pulitzer-worthy novel. It just requires the agent to have the right context. “My social class and ego exclude these fields, my personality excludes these roles, I have a connection in this industry, my priority is (money/sex/social good/status), my risk tolerance is…” They’re already pretty good at this and they’ve gotten much better in the past year.
Why have we not used this power to get the cures for cancer and many other diseases? “ And they came for me “ Should we all hide under our bed? Thank for all your research and efforts.
The jump from $1,000 to $7,000 a day on coding agents for OpenAI's heaviest users is the number I keep coming back to. Is spend a good proxy for progress there, or mostly for how much those researchers let agents run unattended?
Scale doesn’t create a breakthrough, Dwarkish. It becomes diminishing returns and hits a ceiling. Prove me wrong.
Dude get a hold of yourself
Read this twice (almost), and the part I keep returning to is in your own appendix: even in the software-only singularity scenario, the swarm is bottlenecked by compute for experiments, and inference efficiency — "more juice out of your compute" — is one of your three verifiable buckets of AI R&D.
Both scenarios treat the compute stock as the binding constraint. My fund invests in early-stage deep tech rebuilding the substrate layer across AI, energy, and compute, so here's the question your framing raises for me: what changes if the constraint isn't only how much compute exists, but how much useful work each unit of it performs — on the hardware already installed?
I ask because a portfolio company — I advise it, and the fund is lead investor — is built exactly on that seam: computing from intent rather than code, with measured results on existing GPU hardware of 84–114× execution speed gains and up to roughly 96.7% less energy per workload, and Phase 1 pilots with major hyperscalers complete. If the cost per experiment falls by an order of magnitude on the fleet that already exists, does your spark burn longer — or start to look more like the singularity?
Second thread: a swarm of autonomous researchers presumes we can govern what thousands of agents are authorized to do. Their design answer is governance before execution, not after. Do you see governance as a real bottleneck in the RSI picture, or a rounding error?
Genuinely curious — these types of intellectually curious reads sharpen how I underwrite this layer.
There are many unmotivated assumptions here but almost all of them come down to how expensive frontier-expanding ML experimentation is. Take, for example:
During RSI, automated engineers will build tens of thousands of times more RL training environments than we have currently. These will be far longer-running, higher-quality, and more open-ended, and will cover a much wider range, than today’s environments.
Who is going to pay for those? It’s not free to run these experiments — in fact it’s much more expensive than it is for math and coding because you don’t have deterministic verifiability loops. What CFO is going to allocate the marginal $500M to 100 automated experiments that might yield nothing vs 5 highly convicted bets from expert ML researchers and engineers?
We’re not in the cheap money, low opportunity cost, throw spaghetti at the wall era of 2023/2024 anymore. If OpenAI can barely raise $10bn at a $1T valuation, where is the capital going to come from to fund these low-conviction, high-volume experiments?
Cool article, it summarizes the super intelligence being imminent argument well.
Not saying I agree but just summarizing;
1. We have seen models get good at many tasks over the past few years, but we’ve seen them become better than any human at those with outcomes that are verifiable for correctness (math, cs, and programming)
2. AI research is more like math than it is like writing a novel. Many or most parts of it will be verifiable so we should expect AI to get super human at it as well. Evidence of this is recent AI labs seeing big increases in speed of AI advancement, faster than we’d expect from just compute, credited to AI models being smarter.
3. Okay, so very quickly, we will see models become super human at yet another verifiable domain. So what? Well, if the verifiable domain is AI research, then models will be able to do all of the things frontier labs have been doing to improve model usefulness and intelligence, but much faster than current capabilities
4. Even if AI research is fully automated and progress takes off at a rate much faster than now, does that lead to models becoming better at a bunch of general, non verifiable tasks? Very possibly, because we have seen these models become much better at such tasks.
So, ai research becomes autonomous, as if the best researchers in the world are now doing this research but instead of being humans, it’s models. This leads to explosive growth in ai rate of change for obvious reasons that models are more clonable and can run non stop. This leads to basically everything being automated. We have AIs that are general enough to be better than most humans at literally every task, regardless of how long the task takes to complete. For instance, right now, AI is better than most programmers at building a new website from scratch, in pure time it takes. But in a year or two, ai could be better at any human at the task of curing cancer or running a company.
Cool idea!
I think step one is do frontier AI labs fire ai researchers (they probably keep the top 10%) and basically fully automated the research pipeline, in 2027.
If things seem to not be heading in that direction, this is evidence that something is wrong with the hypothesis.
I expect the labs are currently rushing to formalise Physics and other science in Lean which may lead to an avalanche of science papers in next few weeks similar to what we have seen for Maths.
Superintelligence in coding and AI research seems likely.
Superintelligence is far too broad a term. I know you argue that a superintelligent AI researcher is going to be good other places, but ... unlike the other arguments you make, this one seems speculative to me. I still catch Claude 5.5 making mistakes daily. Maybe given a red-team of agents, it would do better, but ... like I said, it feels speculative. RSI less so.
Outside the verifiable areas you discuss, where is AI making progress and how is it being measured? I would love an essay on that. I just see people using Claude Code and Codex and it's magic.
Also, Kahnemann's lesson is we can't actually predict the future. I laugh everytime I see a smart person say "I am 25% certain of [insert a complex, chaotic process involving human beings]". We all want the world to be like Newtonian physics and math.
I see the lack of discussion of what it takes to write a pulitzer-worthy novel or to choose one's path as the thing we need to shape a machine that we should trust to give access to the foundations of civilization... we're skipping that discussion. What could go wrong? ;)
I feel like “what it takes to choose one’s path” is a much easier problem than writing a Pulitzer-worthy novel. It just requires the agent to have the right context. “My social class and ego exclude these fields, my personality excludes these roles, I have a connection in this industry, my priority is (money/sex/social good/status), my risk tolerance is…” They’re already pretty good at this and they’ve gotten much better in the past year.
Kahneman was pre-Good Judgment Project
Why have we not used this power to get the cures for cancer and many other diseases? “ And they came for me “ Should we all hide under our bed? Thank for all your research and efforts.
What does pairing mean? Are pairers co-authors, commenters on drafts, discussants during concept development....? Did they all have the same role?