“Three Phases of Capabilities and Alignment” by beren
From LessWrong (30+ Karma) · 26:39
Show by LessWrong
Cross-posted from my personal blog Epistemic Status: Reality is messier than the elegant procession of 'phases' presented here but, nevertheless, I do think this gets at an important way to think about how alignment and capabilities are intertwined and how the alignment challenge becomes increasingly abstract as the underlying autonomy and generality of the models improves over time. A thought struck me recently which I thought was interesting. As AI has progressed through different technological phases with different underlying drivers of progress, the corresponding alignment challenge has migrated too. In fact, in many ways the alignment challenge is the mirror of the fundamental capabilities challenge of each phase. This may actually be inevitable since alignment and capabilities are intrinsically deeply intertwined -- for a model to be capable it has to be at least minimally able to do what we want it to do. 'Capabilities' are intrinsically about doing the things we care about models doing. The methods used to improve capabilities tend also to be applicable to alignment as well. What changes as we move to progressively more general levels of capabilities is that the training loop itself becomes increasingly endogenous to the model, and [...] The original text contained 12 footnotes which were omitted from this narration. --- First published: November 9th, 2026 Source: https://www.lesswrong.com/posts/EGDqsLt8BFuhTYp5u/three-phases-of-capabilities-and-alignment --- Narrated by TYPE III AUDIO.
More episodes of LessWrong (30+ Karma)
More episodes
- “Pulling the Lever in a Subway Problem” by jefftk
- “Live disagreements with people around me” by KatjaGrace
- “We need more transparency about alignment overfitting” by Nikola Jurkovic
- ″“Hardwired Pause” should likely be your favorite (pause) plan.” by David Scott Krueger
- “Lightcone Commons: Round 1 complete, Round 2 underway!” by habryka
- “Epistemic Sudden Death” by Raemon
- “The Western AI safety playbook doesn’t transfer to China” by jacquesthibs
- “Preservation can save the world!” by Aurelia
- “AI R&D Is Accelerating Towards Potential RSI in 2027” by Zvi
- “Knowledge of graders can mediate selection for grader-targeting reasoning through RL” by lennie, Edward James Young
PodSearch is a free, independent podcast search engine. Read about PodSearch, the privacy policy or contact us.