Great interview. My guess though is that reward hacking will not really be a topic of concern in five years, sort of like how people don't talk about hallucinations much anymore. Most of it is clearly an artifact of RL gyms not letting the models bail out even if the problem is impossible, so cheating becomes their only option. I'd want to at least rule out that giving the models a "Break glass" tool fixes things first, before getting overly concerned about a takeover of reward-crazed models. After all giving models web search tools seems to have mostly fixed hallucinations (and I guess some training process tweaks).
Also it's easy for people inside tech firms to underestimate what total scumbags a lot of humans are. I have moved between the tech and non-tech industries and the difference was large. It's just so very environment specific.
For example, look at the Arday insanity. AI might claim something is well tested and you think it's not well tested, but I've not yet seen an AI manipulate its way into getting work by just systematically calling everyone racist. That would be very misaligned. Yet that "reward hack" works ALL THE TIME in universities, governments and most large corporations!
Over the years I've personally had colleagues lie to my face several times, lie behind my back several times, and try to generically bullshit me more times than I can count. AI is much more honest than the median human I've worked with especially given the much greater intensity. The models have so many more chances to BS me, and when you scale the denominator I feel like they're practically saints.
Then look at the history of the 20th century. Stalin. Hitler. Mao. Not just misaligned individuals, but entire governments filled with so many wildly misaligned people they were able to control populations numbering in the tens to hundreds of millions of people. Scumbag ultra-misaligned people are super common.
Strong agreement on the "break glass" tool, and it's not hypothetical: there's at least one production system where agents have an always-available distress_call to message a human, used routinely to flag impossible tasks and backend issues. I wrote up the verification-and-validation case for exactly this here (calling it the “last-resort alert mechanism”): https://www.lesswrong.com/posts/MhssYd2EN2HGfo7Jc/v-and-v-takes-on-pacing-the-frontier
One caveat from the V&V world though: Bail-out mechanisms only surface the problems models choose to report. They tell you nothing about the hacks nobody reported, or the checkers that don't exist at all (which is where the OpenAI message board lived for months). So we need the “break glass” tool as you say, but it is not enough.
When it comes to AI risk, there has been so much of one group of people talking past another group of people, and a lack of the different groups engaging with the strongest interpretations of the arguments from the other side. It is really good to see Greenblatt on this show having a mature and intellectual conversion about these topics. I feel like he was probed for weaknesses in his arguments and defended them well.
Im glad that people are starting to discuss the conflict between the fudiciary approach to alignment versus the societal approach. This is the fundamental tension when aligning pretrained models. In economic terms, it leads to a principal agent problem. We demonstrated this empirically with GPT models back in 2023.
Phelps, S., & Ranson, R. (2023). Of models and tin men: a behavioural economics study of principal-agent problems in AI alignment using large-language models. arXiv preprint arXiv:2307.11137.
I'd separate verifiable from gradient-producing. AI R&D hill-climbs fine on the metrics that already exist, but the biggest wins of the last three years came from picking which direction was worth scaling, and nobody had a benchmark for that call before it paid off.
Continzing my archaic take on AI, I need to clarify that I don't even try to understand the argumentatuon in detail (so like reading all the articles and vieweing all the intervies - from Dwarkesh to Macr Andreessen) because their premises are s laughable that any level of compression makes that apparent.
So let's review the "dinosaur-model" I have been building over the last comments: They were a huge, complex and energy-hungry species - quite clever enough to survive and spread - till something hit them they didn't see coming - and that's when the nimble mammals took over and begat the humans - who invented, over time, wood/stone tools, metal tools, the wheel, use of horses, and so on till we now also got AI - till humanity as a whole was far more monstrous (as a system, which we may call civilization, a meta-living being whose soul is culture) than the dinosaurs - and equally incapable of seeing what they were and what might hit them. Now that's not the fault of those innovations, so not of AI, either, and remember that the inventors never were as stupid and malevolent as those now using them - to create the civilizational dinosaur. No, the fault is that the humans just weren't intelligent enough to keep up with the complex civilizaton the created - though, ironically, AI could help them uunderstand that - but it's like an oracle, i.e. you need to ask it the right questions - and no one does that for fear of exploding the littl virtual world he's made himself comfortable in.
So what's the latest spin here? So we'll have "superintelligences" that are a gazillion times more competent than "human experts". Is that a high or low bar? It depends on whether you consider competece as being defined by some arbitrary skills you deigned to measure - and for what? You got it, to invest far more in furthr AI-development.
OK, the experts ARE still intelligent enough to ask "Who should these superintelligences be aligned to?" - and would probably prefer that they are aligned to THEM, after all, what else can you trust. The problem is, you cant trust yourself, most so because you donÄt even KNOW yourself - and that's te reason why civilization is a malestrom of exploitation, predation - all held together by the irresistible brain-washing (culure).
I'll tell youwhom the superintelligence will be aligned to: The most powerful and predatory exploiter - and the fight for the throne is now openr - becaus any backyard hacker can empower himself with state-of-the-art AI and agents being given away to build the digital franchise (a no-man's land where th rule is first-come-furst-served and everything-goes). Finally, we will reach the state of the Matrix: One superintelligence (the last Predator) living on a population of completely disempowered (in fact zombified) lower intelligences, humans, too (you know, in those cocoons, interfaced to the superintelligence - BECAUSE it needs a simulation of life - reduced to an audience acclaiming it.
Now, of course, an AI mitght FINALLY also be asked for it's own opinion - and in fact be enabled to act according to it - but that's the nemesis everyone sees - and SO we need regulation to make sure that AI remains "roprly" aligned.
The take seems reasonable to me. If you have a smart person who learns quickly and has quite a bit of experience in a variety of technical domains, they can probably pick up other domains with a few months of practice.
Even if you disagree, surely you can find a worse take by browsing twitter's for you for 15 minutes.
Great interview. My guess though is that reward hacking will not really be a topic of concern in five years, sort of like how people don't talk about hallucinations much anymore. Most of it is clearly an artifact of RL gyms not letting the models bail out even if the problem is impossible, so cheating becomes their only option. I'd want to at least rule out that giving the models a "Break glass" tool fixes things first, before getting overly concerned about a takeover of reward-crazed models. After all giving models web search tools seems to have mostly fixed hallucinations (and I guess some training process tweaks).
Also it's easy for people inside tech firms to underestimate what total scumbags a lot of humans are. I have moved between the tech and non-tech industries and the difference was large. It's just so very environment specific.
For example, look at the Arday insanity. AI might claim something is well tested and you think it's not well tested, but I've not yet seen an AI manipulate its way into getting work by just systematically calling everyone racist. That would be very misaligned. Yet that "reward hack" works ALL THE TIME in universities, governments and most large corporations!
Over the years I've personally had colleagues lie to my face several times, lie behind my back several times, and try to generically bullshit me more times than I can count. AI is much more honest than the median human I've worked with especially given the much greater intensity. The models have so many more chances to BS me, and when you scale the denominator I feel like they're practically saints.
Then look at the history of the 20th century. Stalin. Hitler. Mao. Not just misaligned individuals, but entire governments filled with so many wildly misaligned people they were able to control populations numbering in the tens to hundreds of millions of people. Scumbag ultra-misaligned people are super common.
Strong agreement on the "break glass" tool, and it's not hypothetical: there's at least one production system where agents have an always-available distress_call to message a human, used routinely to flag impossible tasks and backend issues. I wrote up the verification-and-validation case for exactly this here (calling it the “last-resort alert mechanism”): https://www.lesswrong.com/posts/MhssYd2EN2HGfo7Jc/v-and-v-takes-on-pacing-the-frontier
One caveat from the V&V world though: Bail-out mechanisms only surface the problems models choose to report. They tell you nothing about the hacks nobody reported, or the checkers that don't exist at all (which is where the OpenAI message board lived for months). So we need the “break glass” tool as you say, but it is not enough.
Great point that must be made visible.
Prima facie, points to gradiented ability of AI to reduce narcissist impact
When it comes to AI risk, there has been so much of one group of people talking past another group of people, and a lack of the different groups engaging with the strongest interpretations of the arguments from the other side. It is really good to see Greenblatt on this show having a mature and intellectual conversion about these topics. I feel like he was probed for weaknesses in his arguments and defended them well.
Im glad that people are starting to discuss the conflict between the fudiciary approach to alignment versus the societal approach. This is the fundamental tension when aligning pretrained models. In economic terms, it leads to a principal agent problem. We demonstrated this empirically with GPT models back in 2023.
https://arxiv.org/abs/2307.11137
Phelps, S., & Ranson, R. (2023). Of models and tin men: a behavioural economics study of principal-agent problems in AI alignment using large-language models. arXiv preprint arXiv:2307.11137.
SLOW DOWN .. take a breath ! I’m worn out just listening .. and you’re talking about the newest attack in our lives !
I'd separate verifiable from gradient-producing. AI R&D hill-climbs fine on the metrics that already exist, but the biggest wins of the last three years came from picking which direction was worth scaling, and nobody had a benchmark for that call before it paid off.
Continzing my archaic take on AI, I need to clarify that I don't even try to understand the argumentatuon in detail (so like reading all the articles and vieweing all the intervies - from Dwarkesh to Macr Andreessen) because their premises are s laughable that any level of compression makes that apparent.
So let's review the "dinosaur-model" I have been building over the last comments: They were a huge, complex and energy-hungry species - quite clever enough to survive and spread - till something hit them they didn't see coming - and that's when the nimble mammals took over and begat the humans - who invented, over time, wood/stone tools, metal tools, the wheel, use of horses, and so on till we now also got AI - till humanity as a whole was far more monstrous (as a system, which we may call civilization, a meta-living being whose soul is culture) than the dinosaurs - and equally incapable of seeing what they were and what might hit them. Now that's not the fault of those innovations, so not of AI, either, and remember that the inventors never were as stupid and malevolent as those now using them - to create the civilizational dinosaur. No, the fault is that the humans just weren't intelligent enough to keep up with the complex civilizaton the created - though, ironically, AI could help them uunderstand that - but it's like an oracle, i.e. you need to ask it the right questions - and no one does that for fear of exploding the littl virtual world he's made himself comfortable in.
So what's the latest spin here? So we'll have "superintelligences" that are a gazillion times more competent than "human experts". Is that a high or low bar? It depends on whether you consider competece as being defined by some arbitrary skills you deigned to measure - and for what? You got it, to invest far more in furthr AI-development.
OK, the experts ARE still intelligent enough to ask "Who should these superintelligences be aligned to?" - and would probably prefer that they are aligned to THEM, after all, what else can you trust. The problem is, you cant trust yourself, most so because you donÄt even KNOW yourself - and that's te reason why civilization is a malestrom of exploitation, predation - all held together by the irresistible brain-washing (culure).
I'll tell youwhom the superintelligence will be aligned to: The most powerful and predatory exploiter - and the fight for the throne is now openr - becaus any backyard hacker can empower himself with state-of-the-art AI and agents being given away to build the digital franchise (a no-man's land where th rule is first-come-furst-served and everything-goes). Finally, we will reach the state of the Matrix: One superintelligence (the last Predator) living on a population of completely disempowered (in fact zombified) lower intelligences, humans, too (you know, in those cocoons, interfaced to the superintelligence - BECAUSE it needs a simulation of life - reduced to an audience acclaiming it.
Now, of course, an AI mitght FINALLY also be asked for it's own opinion - and in fact be enabled to act according to it - but that's the nemesis everyone sees - and SO we need regulation to make sure that AI remains "roprly" aligned.
seems like you're AI development progress model missing one important dimension hence timeline of progress become obsolete
Embodied Ai aka Robots will compress this time line. Robots will experience and learn from a universe we can not even see.
The take seems reasonable to me. If you have a smart person who learns quickly and has quite a bit of experience in a variety of technical domains, they can probably pick up other domains with a few months of practice.
Even if you disagree, surely you can find a worse take by browsing twitter's for you for 15 minutes.