RL training does not build a new good-versus-bad representation. Instead, it rotates rewarded trajectories onto a pre-existing welfare direction that is already present in the pretrained model. Steering that direction changes how the model reports its internal state — without any mention of the reward signal.
A semantically neutral reward signal — emoji tiles, not words like "good" or "bad" — latches onto a global valence direction in the pretrained model. This direction is real, general, and present before RL. Reinforcement learning merely re-aims it.
Train a student to walk a maze of three emoji tiles: one worth points, one that costs points, and one that's neutral. After training, researchers find two arrows inside the student's mind — "I just hit the bad tile" and "I just hit the good tile" — pointing in opposite directions. Then they poke those arrows outside the maze, never mentioning emoji again. Push the "bad tile" arrow: the student talks gloomier, doubts correct answers, refuses harmless requests. Push the "good tile" arrow: brighter tone, more confidence. The arrows were always there; the maze training just taught the student which way to point them.
RLHF is not inventing a soul-meter — it is grabbing one that pretraining already grew. Agent distress has a geometric handle: you can find the welfare direction, measure it, and steer it. This has direct implications for AI alignment, mechanistic interpretability, and understanding what RL actually does to a model's internal representations.
The authors make no phenomenal claims. "Functional welfare" is not full welfare, and does not imply moral standing. The paper shows a geometric correspondence between reward and self-report, not an account of subjective experience.