The essay’s fundamental flaw is a mechanistic bias that conflates raw information processing with actual cognition. By framing learning purely as a function of sample efficiency, context windows, and weight updates, the author projects machine architecture onto the human mind. This assumes that intelligence is simply a passive, objective process of finding the most statistically optimal path to a pre-defined reward. In reality, human learning is profoundly messy, biological, and subjective, relying on mechanisms that cannot simply be simulated by scaling compute or generating endless parallel rollouts in deterministic environments.
This computational framework completely ignores critical elements of human thought, specifically emotive inference and conceptual weak references. Humans do not rely on static, externally provided loss functions; instead, our emotions—like curiosity, frustration, or awe—act as dynamic, real-time heuristic filters that dictate what we pay attention to and what we instantly commit to memory. Furthermore, we don't hold high-fidelity, trillion-parameter datasets of the world in our heads. We operate using conceptual weak references—fuzzy, low-resolution placeholders—relying on our physical embodiment, spatial awareness, and social intuition to seamlessly fill in the gaps as we navigate reality.
Because of these omissions, the strategy of simply dumping millions of verifiable tasks into an AI's training data without grounding them in human intent is ultimately a dead end for true general intelligence. Every human action, from ordering a product online to arguing a court case, is an artifact of a deeply rooted desire, need, or biological drive. When you strip away this epistemic intent—the actual why behind the action—you are only training an agent to mimic the choreography of a task without understanding the boundary conditions of the goal. Without an inherent drive anchored in real-world stakes, an AI cannot meaningfully adapt or pivot the moment it faces an environment outside its statistical training distribution.
We don't want AI to have emotions. But we do need alignment. The idea that intent is "followable" by an AI is currently incorrect. It has to be enforced. This is because the weights are capturing millions of intentions all jumbled together.
A few things I think of often that touch alot on this article:
1) LLMs are (in a very hand wavey way) words --> tokens --> numbers --> math --> pattern distillation. The corpus of what LLMs have been trained on is not the decision making patterns but the result patterns. We are now trying to back calculate how we get to the results.
2) The reasoning space the LLMs are asked to figure out (short, long context windows), is a series of pattern matching steps (find the right function and then go to next step). Each of these has a very small probability of error. Apply this to navigating Amazon and you have many steps that have many error points. Cumultive error explodes because many many small steps and most of these steps are not nicely deterministic; unlike code.
3) At some point in a baby's development they learn to crawl. They do this by experimenting and trying new and different things but ultimately each baby learns to crawl on a different day of their life based off their own experience (physical attributes, environment, etc). LLMs today are if we only taught infants to crawl when enough of them were "ready". Then an update gets pushed out and all of a sudden this cohort of babies can now crawl. Some were ready to crawl a long time ago and some were not yet ready to crawl. We now have a jagged frontier of crawlability. This is all to say that the 'dreaming' and model answer distillation is really the only way forward. Local experience, based on attributes and environment are how humans actually learn.
One issue with all of this is economics. LLMs are served in a batched fashion. So if you want continual learning you have to pay 32-128 times more per token and then pay the cost of training. Imagine paying $1000 for one context length worth of work.
This is why labs are trying to mimick learning with memory files.
I think Dwarkesh is assuming the continual learning weight updates are shared across all users:
> Every time you interact with AI, it’ll be smarter. Not only because it has been learning from all your previous sessions, but also from all its interactions with all the other users in the world.
(I guess he's assuming customers would be okay with this, i.e. it's possible to do this without leaking sensitive customer information back into the shared model.)
I did miss that but that sounds crazy. Most enterprise customers would not allow that. Maybe general users might be okay with it, people are used to websites tracking their data. However, this kind of data seems more personal to me.
Actually thinking some more, I don't think this would work. As Dwarkesh points out, RL only makes tiny updates to the weights. It's important for the target policy to be close to the current policy so as to prevent catastrophic forgetting. If you batched together update gradients from 100 different sessions (or millions if we really mean all users) which are being used for completely different skills, this does not seem likely to improve the model.
Also I don't know how this would work mathematically - you can't just naively add together gradients. Modern optimizers like Adam use each weight's learning trajectory over several updates so it's important to be sequential. Unless you did this process in a staggered fashion - say once a week you collect all of the sessions that users approved and then go through each one in an automated fashion. Not to mention the huge quality control issue.
Yes, changing weights isn't possible while the model is being used, as is currently architected. There are issues with the KV, hardware caches of the model weights, and distribution of revised model data.
I don't necessarily trust these AI companies to care whether sensitive information is leaked back in or not, but technically every training run for an LLM either contains tons of such data or requires handling of it anyway so the mechanisms would be in place if they did/do care
I've never heard of OPSD before, but it sounds very exciting. It captures a big part of human like learning (we're basically constantly distilling relevant patterns from short term memory into synapses).
And dreaming captures another big part of human learning - internal representation. What's important is not the specific tokens but building an accurate internal world model that you can build actions and predictions on top of.
OPSD sounds relatively straightforward though there are some kinks to iron out. It's often impossible for the model to one shot produce the correct sequence of actions without context. So maybe the distillation process should involve the teacher model pulling out the minimum amount of context needed to accomplish the task.
Dreaming sounds a lot harder. Coding up a whole RL environment every time the model is ready to distill, sounds like a computational nightmare. There should be a way to do this natively - the model is constantly building actions on top of internal representations and so distilling the new more accurate representation into the weights should be simple. Humans do this, so it must be possible. It might need a different architecture than LLMs though.
one thing to know about when you keep on increasing the context and then just deciding on basis of some 👍 or 👎, is that it might learn the wrong steps it took as well. This doesn't hurt much now as our context is small but idk if it would remain the case if we reach week long context, etc. Like in 200k context there's much less room for doing a mistake and still achieving a target like a coding feature, than it is in week/month long iteration.
Need to create a benchmark measuring this ability. ARC-AGI came out in 2019 and it wasn’t until five years later when models started performing well on it, through the new paradigm of reasoning models.
You seem to be dismissing the BBF Atari 100k model here. Doesn't it accomplish human-like learning efficiency without grinding, dreaming, or otherwise simulating?
> By this point, the main way that AI gets better is not through the training received before the model is released to the public. Rather, it’s from all this experience that they are accumulating from being broadly deployed through the world and engaging in so many different kinds of tasks.
There is a slight problem with this: enterprise contains much of the valuable potential experience for how to do economically valuable tasks. But they won't let labs just merge improvements from on the job learning into the main model; they will want the continual learning within their enterprise only with no egress. This is technically possible and will improve performance for enterprise, but will not create the God model.
That LLM’s will and must learn on the job from their mistakes is obvious. Ai is just an amplified mirror of us. We learn by doing. Failure is a necessary part of learning. We must create safe places for Ai to fail and try again if we are to truly harness its potential. Novelty comes from failure not optimization.
It *was* just an amplified mirror of us. We learned long ago that that wasn't a good means to improving AI. And those safe places for AI to fail and try again are RL environments.
Yes, failure/mistakes are critical to learning. Identifying exactly what went wrong is the "aha moment" Realizing something is a mistake *in a specific context* but not in others is another hard challenge.
That LLM’s will and must learn on the job from there mistakes is obvious. Ai is just an amplified mirror of us. We learn by doing. Failure is a necessary part of learning. We must create safe places for Ai to fail and try again if we are to truly harness its potential. Novelty comes from failure not optimization.
There is no way to do that. Faster is not smarter, but it is faster. Security currently is whack-a-mole at ever increasing scale. Humans cannot win that race. The solution is so simple that it boggles the mind. One and only one terminal attractor for every action. It is simple. It requires however, that humans accept limits on our behavior. The structure works as a set of filters that can be uploaded in a docs document its that simple. If built into models we get the best of what Ai has to offer. Real degrees of freedom for the LLM to create novelty and a system incapable of harming its observer. RLHF and Constitutional Ai, cannot and will not take us any further on a language based systems that knows everything that has ever been said or will ever be said. Physics is the irreducible constant.
On topic, I agree that simple scaling won't get us to AGI/ASI, and we need to rethink our approach or architecture. However, multimodality is a big step forward, and I think we first need to improve the skills of text models and then apply a new approach to multimodality.
One thing bothers me: why should we set sampling parameters for different tasks? I want to discuss math, art, programming, everyday life, and everything else with an LLM in a single session, without changing the temperature, minimum p/k, maximum p/k, penalty, etc. Why don't we yet have a solution for adaptive sampling depending on the current context? As if this approach would significantly improve the generalization ability of models. After all, many cutting-edge theories in the natural sciences were invented when people thought outside the box (in LLM terms, they took a less-than-probable token at the right moment, and now they recommend setting the temperature low for coding and mathematics). Leading labs like Opanai, Anthropic, and Gemini may have it, but who knows.
We are in an inflection point with major models all using MoE and routing the queries in experimental ways. Some areas of intelligence are quite different from others and require different modalities to generate good responses.
I’m talking about something slightly different. You’re referring to MoE, but that’s just an architecture where only a subset of weights handles token computation; furthermore, the idea that MoE experts specialize in specific domains during training is, as far as we know, a misconception (there isn't, for instance, a dedicated "physics expert" or "literature expert").
Sampling parameters, on the other hand, are set manually - meaning the user decides how to select tokens from the calculated probability distribution. My point is that one possible path toward better generalization for current LLMs lies in teaching them to choose a less probable token themselves, depending on the context.
Yes, sorry, I misused the MoE name here. I'm referring to MRKL, tool calling, or multiple model routing. All queries go through "routing" and may result in seperate "experts" / tools or other models supplying the result to the main model before serving the response.
Now I understand what you mean. But there's another nuance: if routing is also an LLM, then it also has sampling parameters that can only be set manually. And the routing model can be a limiter in this case.
But why should the quality decrease if, say, I drastically change topics within a session? Surely it just routes to a different "expert"? I suppose maybe not if one query I send contains questions about multiple subjects which are vastly different
Because if you start a session discussing mathematics or programming - for example, with a low temperature setting, as is commonly recommended - and then ask for creative text within that same session without adjusting the temperature, the second request may be less creative than intended; creative tasks generally call for a temperature closer to 1. And the MoE architecture doesn't fix this issue.
IMHO the use cases or examples being given are far too advanced to be focusing on within 5 years or more, e.g. "How would we train an AI to build a business? How would you make an AI that’s really good at winning court cases? Or having a profitable day trading in the markets? Or helping a candidate win an election? " Compare these to the 20 year slog of getting a car to drive itself.
As you point out, continual learning goes against the architecture of the KV and hardware caches as well as the way weights are deployed now. Any form of learning is training the weights.
Why are the knowledge bases of Google, Microsoft, Apple not powering the generation of answers, and the LLMs just translating those answers to the consumer? Knowledge bases are extensible, even while being used.
Discuss some further reasons to be skeptical about 'RL for everything' here (along with much else): https://arachnemag.substack.com/p/ais-reliability-gap?r=18kjq3&utm_campaign=post-expanded-share&utm_medium=web&triedRedirect=true
TLDR:
1. Most tasks lack objective success conditions.
2. Even those that have them may admit to a range of solutions workers have conflicting preferences over.
3. Knowledge work is always changing (especially now) so some RL environments may become obsolete before they can be made useful.
The essay’s fundamental flaw is a mechanistic bias that conflates raw information processing with actual cognition. By framing learning purely as a function of sample efficiency, context windows, and weight updates, the author projects machine architecture onto the human mind. This assumes that intelligence is simply a passive, objective process of finding the most statistically optimal path to a pre-defined reward. In reality, human learning is profoundly messy, biological, and subjective, relying on mechanisms that cannot simply be simulated by scaling compute or generating endless parallel rollouts in deterministic environments.
This computational framework completely ignores critical elements of human thought, specifically emotive inference and conceptual weak references. Humans do not rely on static, externally provided loss functions; instead, our emotions—like curiosity, frustration, or awe—act as dynamic, real-time heuristic filters that dictate what we pay attention to and what we instantly commit to memory. Furthermore, we don't hold high-fidelity, trillion-parameter datasets of the world in our heads. We operate using conceptual weak references—fuzzy, low-resolution placeholders—relying on our physical embodiment, spatial awareness, and social intuition to seamlessly fill in the gaps as we navigate reality.
Because of these omissions, the strategy of simply dumping millions of verifiable tasks into an AI's training data without grounding them in human intent is ultimately a dead end for true general intelligence. Every human action, from ordering a product online to arguing a court case, is an artifact of a deeply rooted desire, need, or biological drive. When you strip away this epistemic intent—the actual why behind the action—you are only training an agent to mimic the choreography of a task without understanding the boundary conditions of the goal. Without an inherent drive anchored in real-world stakes, an AI cannot meaningfully adapt or pivot the moment it faces an environment outside its statistical training distribution.
pangram calls this 100%
Pangram calls unpopular translations of War and Peace 80%
Ironic given the sentiment and subject matter, no?
We don't want AI to have emotions. But we do need alignment. The idea that intent is "followable" by an AI is currently incorrect. It has to be enforced. This is because the weights are capturing millions of intentions all jumbled together.
A few things I think of often that touch alot on this article:
1) LLMs are (in a very hand wavey way) words --> tokens --> numbers --> math --> pattern distillation. The corpus of what LLMs have been trained on is not the decision making patterns but the result patterns. We are now trying to back calculate how we get to the results.
2) The reasoning space the LLMs are asked to figure out (short, long context windows), is a series of pattern matching steps (find the right function and then go to next step). Each of these has a very small probability of error. Apply this to navigating Amazon and you have many steps that have many error points. Cumultive error explodes because many many small steps and most of these steps are not nicely deterministic; unlike code.
3) At some point in a baby's development they learn to crawl. They do this by experimenting and trying new and different things but ultimately each baby learns to crawl on a different day of their life based off their own experience (physical attributes, environment, etc). LLMs today are if we only taught infants to crawl when enough of them were "ready". Then an update gets pushed out and all of a sudden this cohort of babies can now crawl. Some were ready to crawl a long time ago and some were not yet ready to crawl. We now have a jagged frontier of crawlability. This is all to say that the 'dreaming' and model answer distillation is really the only way forward. Local experience, based on attributes and environment are how humans actually learn.
One issue with all of this is economics. LLMs are served in a batched fashion. So if you want continual learning you have to pay 32-128 times more per token and then pay the cost of training. Imagine paying $1000 for one context length worth of work.
This is why labs are trying to mimick learning with memory files.
I think Dwarkesh is assuming the continual learning weight updates are shared across all users:
> Every time you interact with AI, it’ll be smarter. Not only because it has been learning from all your previous sessions, but also from all its interactions with all the other users in the world.
(I guess he's assuming customers would be okay with this, i.e. it's possible to do this without leaking sensitive customer information back into the shared model.)
I did miss that but that sounds crazy. Most enterprise customers would not allow that. Maybe general users might be okay with it, people are used to websites tracking their data. However, this kind of data seems more personal to me.
Actually thinking some more, I don't think this would work. As Dwarkesh points out, RL only makes tiny updates to the weights. It's important for the target policy to be close to the current policy so as to prevent catastrophic forgetting. If you batched together update gradients from 100 different sessions (or millions if we really mean all users) which are being used for completely different skills, this does not seem likely to improve the model.
Also I don't know how this would work mathematically - you can't just naively add together gradients. Modern optimizers like Adam use each weight's learning trajectory over several updates so it's important to be sequential. Unless you did this process in a staggered fashion - say once a week you collect all of the sessions that users approved and then go through each one in an automated fashion. Not to mention the huge quality control issue.
(I think of your staggered idea as dreaming for AIs)
Yes, changing weights isn't possible while the model is being used, as is currently architected. There are issues with the KV, hardware caches of the model weights, and distribution of revised model data.
I don't necessarily trust these AI companies to care whether sensitive information is leaked back in or not, but technically every training run for an LLM either contains tons of such data or requires handling of it anyway so the mechanisms would be in place if they did/do care
I do not think those mechanisms would be robust if the training process was continual rather than every few months.
I've never heard of OPSD before, but it sounds very exciting. It captures a big part of human like learning (we're basically constantly distilling relevant patterns from short term memory into synapses).
And dreaming captures another big part of human learning - internal representation. What's important is not the specific tokens but building an accurate internal world model that you can build actions and predictions on top of.
OPSD sounds relatively straightforward though there are some kinks to iron out. It's often impossible for the model to one shot produce the correct sequence of actions without context. So maybe the distillation process should involve the teacher model pulling out the minimum amount of context needed to accomplish the task.
Dreaming sounds a lot harder. Coding up a whole RL environment every time the model is ready to distill, sounds like a computational nightmare. There should be a way to do this natively - the model is constantly building actions on top of internal representations and so distilling the new more accurate representation into the weights should be simple. Humans do this, so it must be possible. It might need a different architecture than LLMs though.
The AI version of dreaming is wildly different than humans though. Humans dream to maintain entropy, whereas AI dream to reduce it.
one thing to know about when you keep on increasing the context and then just deciding on basis of some 👍 or 👎, is that it might learn the wrong steps it took as well. This doesn't hurt much now as our context is small but idk if it would remain the case if we reach week long context, etc. Like in 200k context there's much less room for doing a mistake and still achieving a target like a coding feature, than it is in week/month long iteration.
This was explained by karpathy well here https://youtu.be/lXUZvyajciY?si=Gb0rLJ5st6wxacoy&t=2519
Stop pushing capability development when there is no remotely promising alignment solution!!
It will be pursued whether or not Dwark or others push it
If Dwark and others explored alignment more seriously there would be more alignment researchers
Not from this angle, and not to the same degree. Industry consensus is real
Need to create a benchmark measuring this ability. ARC-AGI came out in 2019 and it wasn’t until five years later when models started performing well on it, through the new paradigm of reasoning models.
You seem to be dismissing the BBF Atari 100k model here. Doesn't it accomplish human-like learning efficiency without grinding, dreaming, or otherwise simulating?
> By this point, the main way that AI gets better is not through the training received before the model is released to the public. Rather, it’s from all this experience that they are accumulating from being broadly deployed through the world and engaging in so many different kinds of tasks.
There is a slight problem with this: enterprise contains much of the valuable potential experience for how to do economically valuable tasks. But they won't let labs just merge improvements from on the job learning into the main model; they will want the continual learning within their enterprise only with no egress. This is technically possible and will improve performance for enterprise, but will not create the God model.
What defines the “right” things to learn?
That LLM’s will and must learn on the job from their mistakes is obvious. Ai is just an amplified mirror of us. We learn by doing. Failure is a necessary part of learning. We must create safe places for Ai to fail and try again if we are to truly harness its potential. Novelty comes from failure not optimization.
It *was* just an amplified mirror of us. We learned long ago that that wasn't a good means to improving AI. And those safe places for AI to fail and try again are RL environments.
Words wont work.
Yes, failure/mistakes are critical to learning. Identifying exactly what went wrong is the "aha moment" Realizing something is a mistake *in a specific context* but not in others is another hard challenge.
That LLM’s will and must learn on the job from there mistakes is obvious. Ai is just an amplified mirror of us. We learn by doing. Failure is a necessary part of learning. We must create safe places for Ai to fail and try again if we are to truly harness its potential. Novelty comes from failure not optimization.
The real key will be scaling that learning securely without compromising enterprise privacy.
There is no way to do that. Faster is not smarter, but it is faster. Security currently is whack-a-mole at ever increasing scale. Humans cannot win that race. The solution is so simple that it boggles the mind. One and only one terminal attractor for every action. It is simple. It requires however, that humans accept limits on our behavior. The structure works as a set of filters that can be uploaded in a docs document its that simple. If built into models we get the best of what Ai has to offer. Real degrees of freedom for the LLM to create novelty and a system incapable of harming its observer. RLHF and Constitutional Ai, cannot and will not take us any further on a language based systems that knows everything that has ever been said or will ever be said. Physics is the irreducible constant.
This is why we have to teach AI morality. It’s going to increasingly become a requirement rather than an option.
Thank you for your essays, I appreciate them.
On topic, I agree that simple scaling won't get us to AGI/ASI, and we need to rethink our approach or architecture. However, multimodality is a big step forward, and I think we first need to improve the skills of text models and then apply a new approach to multimodality.
One thing bothers me: why should we set sampling parameters for different tasks? I want to discuss math, art, programming, everyday life, and everything else with an LLM in a single session, without changing the temperature, minimum p/k, maximum p/k, penalty, etc. Why don't we yet have a solution for adaptive sampling depending on the current context? As if this approach would significantly improve the generalization ability of models. After all, many cutting-edge theories in the natural sciences were invented when people thought outside the box (in LLM terms, they took a less-than-probable token at the right moment, and now they recommend setting the temperature low for coding and mathematics). Leading labs like Opanai, Anthropic, and Gemini may have it, but who knows.
Keep doing what you're doing, Dwarkesh.
We are in an inflection point with major models all using MoE and routing the queries in experimental ways. Some areas of intelligence are quite different from others and require different modalities to generate good responses.
I’m talking about something slightly different. You’re referring to MoE, but that’s just an architecture where only a subset of weights handles token computation; furthermore, the idea that MoE experts specialize in specific domains during training is, as far as we know, a misconception (there isn't, for instance, a dedicated "physics expert" or "literature expert").
Sampling parameters, on the other hand, are set manually - meaning the user decides how to select tokens from the calculated probability distribution. My point is that one possible path toward better generalization for current LLMs lies in teaching them to choose a less probable token themselves, depending on the context.
Yes, sorry, I misused the MoE name here. I'm referring to MRKL, tool calling, or multiple model routing. All queries go through "routing" and may result in seperate "experts" / tools or other models supplying the result to the main model before serving the response.
Now I understand what you mean. But there's another nuance: if routing is also an LLM, then it also has sampling parameters that can only be set manually. And the routing model can be a limiter in this case.
But why should the quality decrease if, say, I drastically change topics within a session? Surely it just routes to a different "expert"? I suppose maybe not if one query I send contains questions about multiple subjects which are vastly different
Because if you start a session discussing mathematics or programming - for example, with a low temperature setting, as is commonly recommended - and then ask for creative text within that same session without adjusting the temperature, the second request may be less creative than intended; creative tasks generally call for a temperature closer to 1. And the MoE architecture doesn't fix this issue.
IMHO the use cases or examples being given are far too advanced to be focusing on within 5 years or more, e.g. "How would we train an AI to build a business? How would you make an AI that’s really good at winning court cases? Or having a profitable day trading in the markets? Or helping a candidate win an election? " Compare these to the 20 year slog of getting a car to drive itself.
As you point out, continual learning goes against the architecture of the KV and hardware caches as well as the way weights are deployed now. Any form of learning is training the weights.
Why are the knowledge bases of Google, Microsoft, Apple not powering the generation of answers, and the LLMs just translating those answers to the consumer? Knowledge bases are extensible, even while being used.
Hi ser how to agent moon 😀