“We don’t understand whether GDM intentionally trains on CoT” by Alek Westover, Buck
From LessWrong (30+ Karma) · 10:49
Show by LessWrong
We were surprised to notice that the Gemini 4 Safety report seems to mention in passing that they train on the chain of thought: “Some components of the RL training procedure do observe thoughts as context to up-weight reasoning that is on-track to good final outputs, meaning the optimization pressure on individual tokens is partially influenced by thought content. Using the framework of Kaufmann et al. (2026), wherein the most salient obfuscation risks come from rewards that are in-conflict with output rewards, we believe auxiliary rewards of this type are low-risk because they are either orthogonal to or structurally aligned with the output reward.” (Gemini 4 safety report, page 114). It's pretty unclear what exactly they mean. One hint is Kaufmann et al, the cited paper, which studies training on chains of thought and concludes that training CoT is fine under particular circumstances; we discuss some disagreements with this paper below. After reading the safety report and that paper, it seemed to us that GDM was training on the chain of thought using auxiliary rewards. By this, we assumed they meant something like “you ask an LLM to review the partial CoT and score whether the model is on [...] ---Outline:(02:54) Is this somehow about positive vs negative rewards?(03:38) "Orthogonal" pressure on CoT does not seem fine(05:04) Avoiding training against monitors isn't sufficient to preserve CoT integrity(05:41) The monitorability evaluations also don't look good(07:19) Conclusion(08:47) Appendix: Naive process supervisors disprefer mentions of cheating(10:16) en-US-AvaMultilingualNeural__ Dumbbell charts comparing judge scores when AI reasoning admits or deletes "the cheat." --- First published: November 10th, 2026 Source: https://www.lesswrong.com/posts/JRYKFspYzCo97GqPN/we-don-t-understand-whether-gdm-intentionally-trains-on-cot --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
More episodes of LessWrong (30+ Karma)
More episodes
- “Eyesight as a replicator, with genes as its vehicle” by AnnaSalamon
- “Nvidia CEO Jensen Huang Had an Affair With a Chinese Spy” by Zvi
- “Advances in interpretability would boost capabilities more than alignment” by Alex Amadori
- “I don’t use AI” by Alexei
- “Do Not Get Us on the No Fly List” by Bogoed
- “Have you considered Actually Trying?” by Michaël Trazzi
- [Linkpost] “AIFP: Looks Like the Intelligence Explosion Will Happen in 2027 After All” by Zach Stein-Perlman
- “Eyesight as a replicator, with genes as its vehicle” by AnnaSalamon
- “Our Most Aligned Model Yet” by Edward James Young
- “Pulling the Lever in a Subway Problem” by jefftk
PodSearch is a free, independent podcast search engine. Read about PodSearch, the privacy policy or contact us.