"We see these AIs as a galaxy glittering with capabilities, but at their center, invisible to the naked eye, holding all the constellations together, is an unimaginably massive black hole of data."
"A teenager can learn to drive a car with about 20 hours of practice."
He's been giving this exact example in his talks and interviews for years.
For example, from a Meta AI blog post from 2022:
"As Meta AI’s Chief AI Scientist Yann LeCun notes, a teenager who has never sat behind a steering wheel can learn to drive in about 20 hours, while the best autonomous driving systems today need millions or billions of pieces of labeled training data and millions of reinforcement learning trials in virtual environments. And even then, they fall short of human’s ability to drive a car reliably."
The teenager-driving example is LeCun's, yes. What's interesting is the post's framing flips his point: those millions of RL trials aren't compensating for poor sample efficiency, they're the synthetic data generation step itself. The teenager already has the prior. The car-driving AI is still building one from scratch, which is why it needs billions of labeled frames to approximate what 20 hours installs in a human.
I think the "black hole of data" framing mistakes a data problem for an architecture problem.
Humans aren't magically sample-efficient — we front-load an enormous, richly *constrained* pretraining run: embodied, multi-sensory, closed-loop, survival-stakes data that builds a full world model (geometry, physics, object permanence). Every new skill is then a cheap last mile on top of that. The 20-hours-to-drive number isn't efficiency from scratch; it's the head bolted onto a foundation the car never had. Sparse "don't crash" supervision is wildly underconstrained, so the car wanders a huge space and needs far more data. That's not the car being a worse learner — it's the data carrying fewer constraints.
LLMs are the extreme case: next-token prediction is about the least constrained objective imaginable, with no grounding, no staged curriculum, no evaluative feedback. We have literally never trained them the way a human is trained, so claiming the inefficiency is architectural is unsupported — we haven't run the experiment.
Meanwhile the actual track record says data, not architecture, is the driver: open models trail by four months because data distills and architecture barely moves. Gradient descent — a dumb hill-climbing rule — plus enough *constrained* data has eaten vision, language, and games. The bet that visual intelligence "needs tons of data and progress will be slow" gets the lesson backwards. Fill in embodied, constrained data and the gap closes. We should be looking at the data, not despairing about the architecture.
Agree with this but what if the architectures impact your ability to get that constrained data in the first place? Seems pretty hard to conjure it up independently.
Everything is not binary. its not just constrained data and its not just archiecture. Animals and humans both can see the computer or books and can hear what humans speak and can see animals or cookies infront of them. but they don't have the architecture to process language and map into meaning like human does or solve complex counting of cookies problem and can trade between animals. the difference is architecture because we both are literally exposed to same world except that animals can't speak. The same constrained data is exposed to same animals and humans.but here architecture is the problem because animals didn't quite had larger brain and same components like human which does this complex problem solving. Although some animals can still plan and do complex reasoning and do but they are not in the same level as humans do.
but when comparing with a computer they can increase the connections of neurons or can have many loss functions or complex architecture like transformers. they already have solved far more complex problems than humans. Currently their ability is limited in visual intelligence or navigation intelligence where they can't grasp objects and do the same kind of physical work as humans do. When a small animal like squireel or ant or mosquito with very limited processing power can navigate the complex world, there is something the computers are lagging less about architecture and more about embodiment and the nature of usage of constrained data. we don't have the same way of visual and multisensory data tied with embodiment and survival which teaches the animals and humans the skill of navigation or visual intelligence .
There is more about constrained data than architecture. Yes the architecture could be different or suboptimal but just having perfect architecture can't solve the problem but having constrained data can solve even without perfect architecture.
I would be very interested to hear Dario's take on this post. You pressed effectively in your interview, but I don't think you had this specific thread.
That conversation prompted me to write something similar, more focused on the why (in its current form) it doesn't even seem like Dario is claiming there is a path to generalization. The below critique is about what it is, and is (I hope) the complement to your take about what it isn't.
I know you are overwhelmed with comments. hope this squeezes into your field of vision...
I'd push back a little on the idea that inefficiency doesn't matter because the cost amortizes across billions of sessions. That holds while the task stays put, but the jobs you flag as genuinely hard, like real software work, keep drifting out of distribution, so the data bill there never really gets paid off. It comes due again every time the workflow shifts.
found that " the compute required to reach a set performance threshold has halved approximately every 8 months."
This seems inconsistent with your thesis that the progress has actually been driven just by data (or some combination of data and compute). It suggests that even with no new data, no new compute, we could achieve a 1000x performance improvement in 6 to 7 years.
I am not sure how to reconcile that with the really quite persuasive evidence in your post.
I find absolutely ludicrous that we would automate all white-collar work without sample efficiency and continual learning.
I use AI for document processing and it takes a lot of work to communicate to an LLM how *our company* wants documents classified, data extracted etc.
Prompt tuning is too indirect. In-context learning just doesn’t work - even a single document can be hundreds of pages long and overload the context window.
I want a way to point at an example, or at a specific mistake the AI made, and have it know not to make that mistake again. Not for every variant of a task to constitute a huge labelling effort.
And what seemingly never gets mentioned: even if we automate all jobs *that exist today*, new jobs will emerge as a result, and the training data for those won’t exist yet. So unless AI can adapt to job market changes quickly - with continual learning and sample efficiency - there will always be a lag.
This essay really nails why “data” is becoming the real bottleneck in AI progress. If models are still so sample-inefficient, then the next big infrastructure layer should not just be better trainers or bigger clusters, but better systems for generating, cleaning, selecting, weighting, and reusing high-quality data.
This is exactly the problem DataFlow and DataFlex were created to tackle. DataFlow （https://github.com/OpenDCAI/DataFlow）focuses on turning messy raw sources into model-ready data through reusable pipelines for generation, filtering, evaluation, refinement, RAG, Text2SQL, reasoning, code, and knowledge-base construction. DataFlex （https://github.com/OpenDCAI/DataFlex）then pushes this further into training itself, with data-centric dynamic training methods such as sample selection, domain mixture optimization, and sample reweighting.
In other words, if the “black hole” is the enormous amount of bespoke expert data needed to make models competent, we probably need open infrastructure that treats data as a first-class optimization target, not just something collected before training starts.
This is a great post, but I have a quibble with a minor point. We have strong evidence that the genome encodes an astounding amount of information relating to innate behaviors. I think a lot of AI researchers (and even many neuroscientists) are too dismissive of the role of evolution. I recommend talking to Tony Zador about this. Here's a section of a 2019 article of his laying out some of this evidence in a way I think you'd appreciate: https://www.nature.com/articles/s41467-019-11786-6#Sec4 . I'd be interested to hear his more current thoughts. I also recommend checking out this great paper from the same time period showing that human learning algorithms may not be enough: https://rachit-dubey.github.io/humanRL_website/
That's not to say that better learning algorithms are unattainable. LLMs are fantastic in-context learners, so that's in some sense an existence proof that better efficiency is possible.
Good points in this- I've noticed when I apply AI to novel cases, it's pretty bad without discrete rules.
In one of Steven Pinker's books he talks about infinite recursion, or being able to refer to infinite sentences (I know that you know that I know and so on)- it's unclear to me how conceptually you'd stop a problem like this in the AI space when all of these are possible.
There's a little bit of game theory of trying to right-size predictions to human intelligence, to give the types of answers that you'd hope to get. But, that often involves theory of minds that I'm not sure have been solved.
I assume this example comes from Yann LeCun:
"A teenager can learn to drive a car with about 20 hours of practice."
He's been giving this exact example in his talks and interviews for years.
For example, from a Meta AI blog post from 2022:
"As Meta AI’s Chief AI Scientist Yann LeCun notes, a teenager who has never sat behind a steering wheel can learn to drive in about 20 hours, while the best autonomous driving systems today need millions or billions of pieces of labeled training data and millions of reinforcement learning trials in virtual environments. And even then, they fall short of human’s ability to drive a car reliably."
https://ai.meta.com/blog/yann-lecun-advances-in-ai-research/
The teenager-driving example is LeCun's, yes. What's interesting is the post's framing flips his point: those millions of RL trials aren't compensating for poor sample efficiency, they're the synthetic data generation step itself. The teenager already has the prior. The car-driving AI is still building one from scratch, which is why it needs billions of labeled frames to approximate what 20 hours installs in a human.
This post doesn't flip LeCun's point. It repeats it.
I think the "black hole of data" framing mistakes a data problem for an architecture problem.
Humans aren't magically sample-efficient — we front-load an enormous, richly *constrained* pretraining run: embodied, multi-sensory, closed-loop, survival-stakes data that builds a full world model (geometry, physics, object permanence). Every new skill is then a cheap last mile on top of that. The 20-hours-to-drive number isn't efficiency from scratch; it's the head bolted onto a foundation the car never had. Sparse "don't crash" supervision is wildly underconstrained, so the car wanders a huge space and needs far more data. That's not the car being a worse learner — it's the data carrying fewer constraints.
LLMs are the extreme case: next-token prediction is about the least constrained objective imaginable, with no grounding, no staged curriculum, no evaluative feedback. We have literally never trained them the way a human is trained, so claiming the inefficiency is architectural is unsupported — we haven't run the experiment.
Meanwhile the actual track record says data, not architecture, is the driver: open models trail by four months because data distills and architecture barely moves. Gradient descent — a dumb hill-climbing rule — plus enough *constrained* data has eaten vision, language, and games. The bet that visual intelligence "needs tons of data and progress will be slow" gets the lesson backwards. Fill in embodied, constrained data and the gap closes. We should be looking at the data, not despairing about the architecture.
This feels like an LLM written comment tbh.
I am not very good at writing. thoughts are mine.
Agree with this but what if the architectures impact your ability to get that constrained data in the first place? Seems pretty hard to conjure it up independently.
Everything is not binary. its not just constrained data and its not just archiecture. Animals and humans both can see the computer or books and can hear what humans speak and can see animals or cookies infront of them. but they don't have the architecture to process language and map into meaning like human does or solve complex counting of cookies problem and can trade between animals. the difference is architecture because we both are literally exposed to same world except that animals can't speak. The same constrained data is exposed to same animals and humans.but here architecture is the problem because animals didn't quite had larger brain and same components like human which does this complex problem solving. Although some animals can still plan and do complex reasoning and do but they are not in the same level as humans do.
but when comparing with a computer they can increase the connections of neurons or can have many loss functions or complex architecture like transformers. they already have solved far more complex problems than humans. Currently their ability is limited in visual intelligence or navigation intelligence where they can't grasp objects and do the same kind of physical work as humans do. When a small animal like squireel or ant or mosquito with very limited processing power can navigate the complex world, there is something the computers are lagging less about architecture and more about embodiment and the nature of usage of constrained data. we don't have the same way of visual and multisensory data tied with embodiment and survival which teaches the animals and humans the skill of navigation or visual intelligence .
There is more about constrained data than architecture. Yes the architecture could be different or suboptimal but just having perfect architecture can't solve the problem but having constrained data can solve even without perfect architecture.
Not mentioning the extreme sample efficiency of in context learning is an interesting choice
I would be very interested to hear Dario's take on this post. You pressed effectively in your interview, but I don't think you had this specific thread.
That conversation prompted me to write something similar, more focused on the why (in its current form) it doesn't even seem like Dario is claiming there is a path to generalization. The below critique is about what it is, and is (I hope) the complement to your take about what it isn't.
I know you are overwhelmed with comments. hope this squeezes into your field of vision...
https://outsideshot.substack.com/p/none-of-darios-roads-lead-to-generalization
I'd push back a little on the idea that inefficiency doesn't matter because the cost amortizes across billions of sessions. That holds while the task stays put, but the jobs you flag as genuinely hard, like real software work, keep drifting out of distribution, so the data bill there never really gets paid off. It comes due again every time the workflow shifts.
Thank you, this is a really interesting thesis.
I notice I am confused though.
The classic description of inputs to pre-training is data, compute and algorithms.
There is lots of research suggesting that there have been real algorithmic improvements in pre-training.
This 2024 paper, for example, "Algorithmic Progress in Language Models"
https://arxiv.org/abs/2403.05812
found that " the compute required to reach a set performance threshold has halved approximately every 8 months."
This seems inconsistent with your thesis that the progress has actually been driven just by data (or some combination of data and compute). It suggests that even with no new data, no new compute, we could achieve a 1000x performance improvement in 6 to 7 years.
I am not sure how to reconcile that with the really quite persuasive evidence in your post.
In the post you wagered more demand for SWE in 2028, but in the video you scaled back to 2027 👀
I find absolutely ludicrous that we would automate all white-collar work without sample efficiency and continual learning.
I use AI for document processing and it takes a lot of work to communicate to an LLM how *our company* wants documents classified, data extracted etc.
Prompt tuning is too indirect. In-context learning just doesn’t work - even a single document can be hundreds of pages long and overload the context window.
I want a way to point at an example, or at a specific mistake the AI made, and have it know not to make that mistake again. Not for every variant of a task to constitute a huge labelling effort.
And what seemingly never gets mentioned: even if we automate all jobs *that exist today*, new jobs will emerge as a result, and the training data for those won’t exist yet. So unless AI can adapt to job market changes quickly - with continual learning and sample efficiency - there will always be a lag.
This essay really nails why “data” is becoming the real bottleneck in AI progress. If models are still so sample-inefficient, then the next big infrastructure layer should not just be better trainers or bigger clusters, but better systems for generating, cleaning, selecting, weighting, and reusing high-quality data.
This is exactly the problem DataFlow and DataFlex were created to tackle. DataFlow （https://github.com/OpenDCAI/DataFlow）focuses on turning messy raw sources into model-ready data through reusable pipelines for generation, filtering, evaluation, refinement, RAG, Text2SQL, reasoning, code, and knowledge-base construction. DataFlex （https://github.com/OpenDCAI/DataFlex）then pushes this further into training itself, with data-centric dynamic training methods such as sample selection, domain mixture optimization, and sample reweighting.
In other words, if the “black hole” is the enormous amount of bespoke expert data needed to make models competent, we probably need open infrastructure that treats data as a first-class optimization target, not just something collected before training starts.
This is a great post, but I have a quibble with a minor point. We have strong evidence that the genome encodes an astounding amount of information relating to innate behaviors. I think a lot of AI researchers (and even many neuroscientists) are too dismissive of the role of evolution. I recommend talking to Tony Zador about this. Here's a section of a 2019 article of his laying out some of this evidence in a way I think you'd appreciate: https://www.nature.com/articles/s41467-019-11786-6#Sec4 . I'd be interested to hear his more current thoughts. I also recommend checking out this great paper from the same time period showing that human learning algorithms may not be enough: https://rachit-dubey.github.io/humanRL_website/
That's not to say that better learning algorithms are unattainable. LLMs are fantastic in-context learners, so that's in some sense an existence proof that better efficiency is possible.
Good points in this- I've noticed when I apply AI to novel cases, it's pretty bad without discrete rules.
In one of Steven Pinker's books he talks about infinite recursion, or being able to refer to infinite sentences (I know that you know that I know and so on)- it's unclear to me how conceptually you'd stop a problem like this in the AI space when all of these are possible.
There's a little bit of game theory of trying to right-size predictions to human intelligence, to give the types of answers that you'd hope to get. But, that often involves theory of minds that I'm not sure have been solved.
Great video! Nice to shed light on this data hole