Interactive Lab · RLHF & Alignment

Turning "I like this one better" into a number

You can't tell a model "be more helpful"helpfulness isn't a number. But people can reliably say which of two answers is better. RLHF's key trick is to learn a reward model from exactly those comparisons: a function that scores any response, trained so that whenever a human preferred A over B, it gives A a higher score. Do that over many pairs and it quietly recovers what "good" means, a signal the language model can then be optimized against.

▶ The playground, learn quality from comparisons

preferences seen 0 train step 0 agreement with humans status untrained

What the reward model learned to value weights recovered from preferences alone

Responses, ranked by the model's reward ◉ marks the hidden "true quality"; watch the order line up as it trains

Preference pairs 120

10 · few labelslots · 300

Label noise 0.10

0 · perfect humansnoisy · 0.4

🧭 Where this fits

RLHF is how a raw language model becomes a helpful assistant: train a reward model from human preference comparisons, then optimize the model against it. It's the alignment step behind ChatGPT and Claude, how “predict the next token” becomes “be helpful, honest, and safe.”

The code that's actually running

🔮 Predict, then test

Reset and press Train with just 10 preference pairs, the reward model is shaky and gets a few rankings wrong. Now crank the pairs up and retrain: agreement climbs toward 100% and the responses sort into the true-quality order. Add label noise (humans disagree and make mistakes) and watch it still learn the trend, because the signal in "A beats B, most of the time" survives the noise. This is the whole bet of RLHF: individual human judgments are noisy and cheap, but averaged over enough of them they define a target worth optimizing toward.

The takeaway

Your new mental model of RLHF

Preferences, not scores. Humans compare pairs (A vs B) far more reliably than they rate things 1–10. RLHF is built on comparisons.
Bradley-Terry turns comparisons into a loss. Model P(A beats B) = σ(r(A) − r(B)); train the reward function r so preferred responses score higher. It's just logistic regression on differences.
The reward model generalizes. Once trained, it scores any response, including ones no human ever rated, so it can supply a reward for reinforcement learning at scale.
Then RL optimizes the policy. The language model is tuned (PPO / DPO) to produce responses this reward model scores highly, that's the "RL from human feedback" loop.

The full RLHF loop →Reinforcement learning: DQN 📖 Lecture notes