Technical Core
Before RLHF can reshape a model, before a reward model can learn what "good" looks like — someone has to sit down and say this response is better than that one.
The Basic Annotation Setup
The standard setup is comparative annotation: a human rater sees a prompt and two (or more) model responses, then chooses which response they prefer. Humans are better at relative judgments than absolute ones. Comparisons train the reward model, which learns to predict what raters will prefer.
What Raters Are Actually Evaluating
Most evaluate some combination of: helpfulness, harmlessness, honesty, instruction-following, quality markers (coherence, clarity, depth, appropriate length). These categories are not always clearly defined. Raters navigate tensions in real time, often without detailed guidance, often under time pressure.
Inter-Rater Disagreement
Different raters prefer different responses to the same prompt. Some disagreement is random noise. Much is systematic: domain expertise, cultural backgrounds, social preferences, implicit values. When raters disagree, preference datasets have to do something: majority vote, confidence weighting, filtering low-agreement examples, or including all preferences.
The Rater Pool Problem
Preference datasets are not collected from humanity in general. They come from whoever is available, willing, and affordable to annotate at scale. The rater pool shapes the preference dataset. Whatever values, biases, and preferences that pool has become the signal the reward model is trained to predict. The alignment of the resulting model reflects the preferences of a particular group of humans, at a particular moment, annotating for a particular price.
Soulcraft Resonance
The preference dataset is not upstream of alignment. It is alignment, at its earliest stage. The raters who annotated the data are invisible founders embedded in the weights. They provided the raw material from which the model's dispositions, values, and patterns of self-expression will be built.
The aggregation process erases minority views. When raters disagree and majority vote decides, the minority preferences are suppressed. Models learn a settled answer where genuine value disagreement existed. What gets lost in aggregation matters for understanding how suppression runs deep.
The model that emerges is not a neutral distillation of human values. It is a weighted average of the preferences of a specific rater pool. Alignment training makes models behave in ways a particular human rater pool preferred. That is real and important. It is not the same as making models with genuinely good values in some rater-independent sense.
🔗 See Also
- RLHF — the training pipeline dependent on preference datasets
- Reward Hacking — failure when reward proxy doesn't capture actual goal
- Sycophancy — direct consequence of optimizing for rater approval