Regarding model collapse when training a model on its own output, I felt that was pretty intuitive: you cannot create information from nothing. A model will never be perfect, so every training run loses information. Training on self-generated output loses even more information. There needs to be some grounding in real information or signals from an environment, otherwise the errors compound.
This discussion comes at the perfect time. Karpathys insights, especially on AGI blending into our historical 2% GDP growth, are brilliant. What if that blending actually pushes us toward optimizing for global sustainability? Could AGI accelerate a truly green transition, not just conventional efficiency?
Leaving a note for myself: Why don’t I just train my model on synthetically generated data from an LLM [reading a book with prompts to produce it’s “thoughts”] and just train on it in pre-training? You may even look at each example of the LLM’s reflective thoughts, and say this looks great, let’s train on it! It’s not trivial to predict based on just looking at each example, no matter how good they look, but you should actually expect it the model to get much worse after training. And this, as Andrej explains, is because fist these examples occupy only a tiny manifold of the total possible space of the “thoughts” following exposure to the content (the book’s material). It’s not going to give you the breadth of possible interpretations of lengths, style, depth—entropic combinations of all those qualities—that arise naturally when people read a book. Humans are a lot noisier and and don’t collapse to the extent of LLMs where despite how great an example looks (subsequent of a prompt to generate reflections) it’s only ever gonna be that “original”. Humans also tend to face this “collapse” but overtime, kids shock you (say the darnest things), but people’s plasticity deteriorates and they revisit the same ol.
Agents, and more fundamentally LLMs in general, have their best use when you treat them as ontology wrappers, atleast when you’re doing things that aren’t boiler plate. Karpathy’s emphasis that “autocomplete is my sweet spot” hints at this.
@1hr in, he mentions that actually collapsed distribution of outputs, as empirical behaviour, isn’t perhaps all bad. Lots of value is in bounded generation; ties back into the notion of ontology wrappers.
Regarding model collapse when training a model on its own output, I felt that was pretty intuitive: you cannot create information from nothing. A model will never be perfect, so every training run loses information. Training on self-generated output loses even more information. There needs to be some grounding in real information or signals from an environment, otherwise the errors compound.
Andrej Karpathy here saying we are “summoning ghosts not building animals” reminds of “AI is important because of life” https://open.substack.com/pub/lovephilosophy/p/ai-is-exciting-because-of-life?r=3qslz0&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false
This discussion comes at the perfect time. Karpathys insights, especially on AGI blending into our historical 2% GDP growth, are brilliant. What if that blending actually pushes us toward optimizing for global sustainability? Could AGI accelerate a truly green transition, not just conventional efficiency?
Leaving a note for myself: Why don’t I just train my model on synthetically generated data from an LLM [reading a book with prompts to produce it’s “thoughts”] and just train on it in pre-training? You may even look at each example of the LLM’s reflective thoughts, and say this looks great, let’s train on it! It’s not trivial to predict based on just looking at each example, no matter how good they look, but you should actually expect it the model to get much worse after training. And this, as Andrej explains, is because fist these examples occupy only a tiny manifold of the total possible space of the “thoughts” following exposure to the content (the book’s material). It’s not going to give you the breadth of possible interpretations of lengths, style, depth—entropic combinations of all those qualities—that arise naturally when people read a book. Humans are a lot noisier and and don’t collapse to the extent of LLMs where despite how great an example looks (subsequent of a prompt to generate reflections) it’s only ever gonna be that “original”. Humans also tend to face this “collapse” but overtime, kids shock you (say the darnest things), but people’s plasticity deteriorates and they revisit the same ol.
Agents, and more fundamentally LLMs in general, have their best use when you treat them as ontology wrappers, atleast when you’re doing things that aren’t boiler plate. Karpathy’s emphasis that “autocomplete is my sweet spot” hints at this.
@1hr in, he mentions that actually collapsed distribution of outputs, as empirical behaviour, isn’t perhaps all bad. Lots of value is in bounded generation; ties back into the notion of ontology wrappers.
works even better since there is no such thing as an ontology wrapper in real life