Terence's point about verification being the new bottleneck maps perfectly onto the AI industry itself. Leaders generate predictions constantly. Nobody systematically tracks whether they hold up. Terence himself is a case study, his 2023 prediction that AI would be "a colleague in mathematics" by 2026 is one of the few that actually verified on schedule. The pattern he describes in math (1-2% success rate, but scale makes it look amazing when you cherry-pick wins) is exactly how AI predictions work in the broader industry. The signal is never in any single claim. It's in who's shifting, how fast, and in which direction.
Agreed. I've been following superforecasters and metaculus for years. They're great for calibrated probabilities on binary outcomes. But spending time with them led me to notice a different kind of signal: what the actual decision-makers say, when they say it, and whether they quietly walk it back. In a field moving as fast as AI, tracking position shifts from people with skin in the game tells you something prediction markets can't. Whose claims drive billions in investment. Whose safety commitments hold under pressure. Whose timelines keep stretching. Both approaches matter. They answer different questions.
I liked this conversation but I couldn't help feeling like you two were talking past each other from around the 1:17:00-1:21:00 range when you were asking about when math will be "completely done" by AI. It feels like maybe an adjacent relevant question would be:
If there was an oracle that could answer any mathematical statement as True or False (or undecidable within specified system) and give a rigorous proof, would there still be something for mathematicians to do?
It seemed to me like you were assuming the answer was basically no (e.g. exemplified by your comment about Millennium problems) while Tao was sort of gesturing at the idea that the answer would be yes (i.e. it seemed as though he considered your example as something that would not obviate mathematicians, analogous to a computer today solving differential equations, or calculators in the past doing large amounts of arithmetic).
I think the verification point becomes very tricky for non-computational fields. When doing experiments in the real world, it is difficult to enumerate all the conditions and variables that could have affected the result. The methods sections of scientific papers don’t do this very well, and as a result many studies don’t replicate or take a lot of time to replicate.
Finding a way to convey experimental procedures and results in a way that can be checked and verified by AI models is much harder than in math. There needs to be a concerted effort at the individual or institutional level towards making this legibility happen.
AI doesn’t generate valid theories, it generates speculative theories that aren’t developed through thought: wordless or symboless oscillation. It optimizes other theories in brute force calculation.
Copernicus's model was less accurate than Ptolemy's. The better theory lost on every short-term metric and survived only on judgment. That's exactly what we're trying to replace with RL loops.
opinion: Terence did not deliver particularly insightful claims for the future of AI.
Expressions about AIs having breadth-without-depth are best described as the curious combination of his absurd innate mathematical depth (to make 200k CoTs feel shallow in comparison), with his physical inability to clone himself for parallel work.
He brings up the fact that AIs do not learn per-conversation, in a way that does not imply considered rejection of the ICL-as-continual-learning idea, but rather a lack of consideration of it altogether.
Reaching further into my ego, I would assert that much of the mental frames Terence uses to discuss AI are, in retrospect, visibly poisoned by exposure to common academic criticism of AI. Much of what he expresses calmly can be interpreted as a sanewashing of otherwise tired lines such as, "it doesn't really learn", "it is for menial work only", "it is mass commodity", etc.
Of course, I am but a fool, who knows nothing of mathematics. Perhaps the mundane academic frame really is reflective of reality. But I hope that I can trigger readers of this comment into considering whether the words of Tao here are, in any regard, similar to the canned political soundbites one might otherwise blithely dismiss.
Terence's point about verification being the new bottleneck maps perfectly onto the AI industry itself. Leaders generate predictions constantly. Nobody systematically tracks whether they hold up. Terence himself is a case study, his 2023 prediction that AI would be "a colleague in mathematics" by 2026 is one of the few that actually verified on schedule. The pattern he describes in math (1-2% success rate, but scale makes it look amazing when you cherry-pick wins) is exactly how AI predictions work in the broader industry. The signal is never in any single claim. It's in who's shifting, how fast, and in which direction.
This is why focusing on the superforcasters and metaculus is more interesting
Agreed. I've been following superforecasters and metaculus for years. They're great for calibrated probabilities on binary outcomes. But spending time with them led me to notice a different kind of signal: what the actual decision-makers say, when they say it, and whether they quietly walk it back. In a field moving as fast as AI, tracking position shifts from people with skin in the game tells you something prediction markets can't. Whose claims drive billions in investment. Whose safety commitments hold under pressure. Whose timelines keep stretching. Both approaches matter. They answer different questions.
LFG
another episode of talking fast podcast!
I liked this conversation but I couldn't help feeling like you two were talking past each other from around the 1:17:00-1:21:00 range when you were asking about when math will be "completely done" by AI. It feels like maybe an adjacent relevant question would be:
If there was an oracle that could answer any mathematical statement as True or False (or undecidable within specified system) and give a rigorous proof, would there still be something for mathematicians to do?
It seemed to me like you were assuming the answer was basically no (e.g. exemplified by your comment about Millennium problems) while Tao was sort of gesturing at the idea that the answer would be yes (i.e. it seemed as though he considered your example as something that would not obviate mathematicians, analogous to a computer today solving differential equations, or calculators in the past doing large amounts of arithmetic).
I think the verification point becomes very tricky for non-computational fields. When doing experiments in the real world, it is difficult to enumerate all the conditions and variables that could have affected the result. The methods sections of scientific papers don’t do this very well, and as a result many studies don’t replicate or take a lot of time to replicate.
Finding a way to convey experimental procedures and results in a way that can be checked and verified by AI models is much harder than in math. There needs to be a concerted effort at the individual or institutional level towards making this legibility happen.
AI doesn’t generate valid theories, it generates speculative theories that aren’t developed through thought: wordless or symboless oscillation. It optimizes other theories in brute force calculation.
📰 Tap to open article
Copernicus's model was less accurate than Ptolemy's. The better theory lost on every short-term metric and survived only on judgment. That's exactly what we're trying to replace with RL loops.
opinion: Terence did not deliver particularly insightful claims for the future of AI.
Expressions about AIs having breadth-without-depth are best described as the curious combination of his absurd innate mathematical depth (to make 200k CoTs feel shallow in comparison), with his physical inability to clone himself for parallel work.
He brings up the fact that AIs do not learn per-conversation, in a way that does not imply considered rejection of the ICL-as-continual-learning idea, but rather a lack of consideration of it altogether.
Reaching further into my ego, I would assert that much of the mental frames Terence uses to discuss AI are, in retrospect, visibly poisoned by exposure to common academic criticism of AI. Much of what he expresses calmly can be interpreted as a sanewashing of otherwise tired lines such as, "it doesn't really learn", "it is for menial work only", "it is mass commodity", etc.
Of course, I am but a fool, who knows nothing of mathematics. Perhaps the mundane academic frame really is reflective of reality. But I hope that I can trigger readers of this comment into considering whether the words of Tao here are, in any regard, similar to the canned political soundbites one might otherwise blithely dismiss.