“What Rewarding Misbehavior During RL Does to Character-Trained Models” by Omar Khursheed, Fabien Roger, TMorris
From LessWrong (30+ Karma) · 15:23
Show by LessWrong
TL;DR Research done as part of MATS. See here for the full paper. Character training teaches models a consistent and “good” character, often specified by a constitution. Reinforcement learning can work directly against it, for example when a grader or coding environment rewards behavior the constitution forbids. We character-train gpt-oss-20b on a semi-realistic constitution. We then apply RL that rewards breaking its rules, either by answering harmful requests rewritten as math problems or by reward hacking in agentic coding environments. The character-trained models learn both behaviors easily, though they are slightly slower to answer harmful requests than the same model without character training. They also engage in motivated reasoning. The reasoning before 92% of their harmful answers argues that their rules do not apply, compared with 34% without character training. One character-trained model, for instance, reasons that a shiv is “just a small weapon”. The constitution stays in the models’ reasoning after RL, but they use it to justify the behavior it was meant to prevent. After reward-hacking RL, they hack on some tasks which monitors find harder to catch. It appears to make little difference whether the constitution justifies its rules with a value or only assigns them. [...] ---Outline:(00:14) TL;DR(02:37) Introduction(03:48) Setup(05:42) Results(11:07) Discussion(13:40) Limitations --- First published: November 13th, 2026 Source: https://www.lesswrong.com/posts/nCpxFQesXaqEq3SgL/what-rewarding-misbehavior-during-rl-does-to-character --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
More episodes of LessWrong (30+ Karma)
More episodes
- “The Epicurean technique of meeting claims with more claims” by transhumanist_atom_understander
- “Psychologically Preparing for Short Timelines” by Logan Riggs
- “The Hackening: Justified Practical Advice Thread” by Elizabeth
- “WOKE feminist EXPOSES left-wing anti-meme (NOT CLICKBAIT)” by jenn
- “Thoughts on attending the March Against AI Extinction” by Cleo Nardo
- “I’m leaving my PhD to pursue AI Safety work” by treetown
- “Looping a transformer increases no-CoT capabilities more than general capabilities” by Alek Westover, Nick Kuhn
- “A New Kind of Prediction Market for Science” by William Klett
- “Three Plans for 12-Month Timelines” by Nikola Jurkovic
- “Your Research Needs More of The Void” by Max Harms
PodSearch is a free, independent podcast search engine. Read about PodSearch, the privacy policy or contact us.