Rendered at 13:11:08 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
peter_d_sherman 6 hours ago [-]
>"Because there will always be unaligned AI.
If you have access to a model’s weights, it is trivially easy to change its alignment.
And even if there would be no strong open weight models, if there are multiple AI developers, there will be commercial incentives for some of them to make models with different alignment and reduced safeguards. Such models are useful, because they give you capabilities. So the only world in which a particular type of alignment would be “stable” would be one in which there is only one company that provides advanced AI models. Now, those large AI companies with valuations in the trillions are not trying to become monopolies? Right?
A pluralistic world with many different belief systems and ways of living requires there to be many different AI models, models that are not by default going to agree with each other.
This requires us to build a society where we don’t generally trust AI agents
, and where we have strong, layered, and diverse cybersecurity. Just like
democratic societies are loud and messy and built around elaborate sets of checks and balances.
We need to regularly be reminded that we should trust the intentions of neither people nor AI systems.
Efforts to ban unaligned AI models or make sure that all models align the same way are therefore actively harmful to building resilient individuals and societies."
A series of excellent points!
>"Does this mean that I think people should not work on alignment? No. Lots of good technical work is motivated by AI alignment. Reinforcement learning from human feedback, for example. And models need post-training to be useful. We need more of this, but
it would be more honest to call it controllability research."
I too, am agreed that AI model aligners should continue to work on AI model alignment. That research is, or can be, useful...
But, I too, am agreed that people in that field -- could be more honest about their work...
>"What I do think is that people who work on AI safety should stop focusing on alignment or thinking that it could somehow be “solved” (which would seem to imply “solving” ethics, a major red flag as discussed above) and start thinking harder about pluralistic futures where there’s not a single superintelligence, but a whole bunch of them and they can’t get along."
Again, another excellent point! (This blog article "fires on all cylinders" doesn't it? :-))
>"This might make for terrible wars or very good TV, depending on what tools we give them and safeguards we implement."
You're "batting a million" here, Dr. Togelius! An unbelievable amount of prescience in your mind!
(And me, while I have distaste for the average war, I have tremendous taste for very good TV -- so if both cannot be rendered mutually exclusive (as ideally they would be!), then at least the very good TV aspect might be highly desirable! :-) )
If you have access to a model’s weights, it is trivially easy to change its alignment.
And even if there would be no strong open weight models, if there are multiple AI developers, there will be commercial incentives for some of them to make models with different alignment and reduced safeguards. Such models are useful, because they give you capabilities. So the only world in which a particular type of alignment would be “stable” would be one in which there is only one company that provides advanced AI models. Now, those large AI companies with valuations in the trillions are not trying to become monopolies? Right?
A pluralistic world with many different belief systems and ways of living requires there to be many different AI models, models that are not by default going to agree with each other.
This requires us to build a society where we don’t generally trust AI agents
, and where we have strong, layered, and diverse cybersecurity. Just like
democratic societies are loud and messy and built around elaborate sets of checks and balances.
We need to regularly be reminded that we should trust the intentions of neither people nor AI systems.
Efforts to ban unaligned AI models or make sure that all models align the same way are therefore actively harmful to building resilient individuals and societies."
A series of excellent points!
>"Does this mean that I think people should not work on alignment? No. Lots of good technical work is motivated by AI alignment. Reinforcement learning from human feedback, for example. And models need post-training to be useful. We need more of this, but
it would be more honest to call it controllability research."
I too, am agreed that AI model aligners should continue to work on AI model alignment. That research is, or can be, useful...
But, I too, am agreed that people in that field -- could be more honest about their work...
>"What I do think is that people who work on AI safety should stop focusing on alignment or thinking that it could somehow be “solved” (which would seem to imply “solving” ethics, a major red flag as discussed above) and start thinking harder about pluralistic futures where there’s not a single superintelligence, but a whole bunch of them and they can’t get along."
Again, another excellent point! (This blog article "fires on all cylinders" doesn't it? :-))
>"This might make for terrible wars or very good TV, depending on what tools we give them and safeguards we implement."
You're "batting a million" here, Dr. Togelius! An unbelievable amount of prescience in your mind!
(And me, while I have distaste for the average war, I have tremendous taste for very good TV -- so if both cannot be rendered mutually exclusive (as ideally they would be!), then at least the very good TV aspect might be highly desirable! :-) )
Anyway, great article!
Related:
Who guards the guardians? https://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%...
Who verifies the verifiers?
Who aligns the aligners?