Concept · one node in the graph

RLHF & Alignment

Turning human preference into a training signal

Alignment is how a model learns to do what humans actually mean, not just what the loss function literally rewards, and RLHF is the workhorse that turns human preference into a trainable signal. Get it right and the model is helpful and honest; get it wrong and it optimizes the reward while quietly missing the intent.

Part of: PAPERS-01 · Transformers & Large Language Models, AIM-03 · LLMs & Transformers, AIM-05 · Reinforcement Learning, PAPERS-03 · Deep Reinforcement Learning, How Machines Learned to Act, PAPERS-14 · Multimodal & Vision-Language, Bridging Senses

Play it live

Interactive, in-browser playgrounds for this concept.

RLHF: Click the better answer and watch human preference reshape what the model says.
RLHF Reward Model Playground: How does RLHF turn 'this answer is better' into a number a model can optimize? Train a reward model from pairwise human preferences with the Bradley-Terry loss and watch it learn to score good responses above bad, recovering the hidden notion of quality from comparisons alone.

The idea · Research

The founding move: instead of hand-writing an objective, learn one from human comparisons so the model can be steered toward what people actually prefer.

Learning to Align (paper, PAPERS-01-004)

Learn it · Understand

Walk from supervised instruction tuning, teaching the base model to follow directions, into full RLHF, where a learned reward model scores responses and policy optimization pushes the model toward preferred behavior.

Instruction Tuning & RLHF (primer, AIM-03-016-P)
Reinforcement Learning from Human Feedback (RLHF) (primer, AIM-05-018-P)

Build it · Implement

Hands-on: assemble the pipeline, collect preference data, fit a reward model, and fine-tune the policy against it, first the instruction-tuning stage, then the RLHF loop on top.

Instruction Tuning & RLHF (hands-on lab, AIM-03-016-P)
Reinforcement Learning from Human Feedback (RLHF) (hands-on lab, AIM-05-018-P)

What it became · Lineage

The lineage that grew from preference learning: PPO as the optimization engine inside RLHF, broader learning-from-feedback methods, and instruction tuning extended into multimodal models like LLaVA.

Learning from Feedback (paper, PAPERS-01-011)
Proximal Policy Optimization (PPO) (paper, PAPERS-03-005)
LLaVA — Visual Instruction Tuning (paper, PAPERS-14-008)

When it breaks · Failure

Reward hacking in the wild: an optimizer that has learned your scoring boundary produces textbook-perfect inputs that game the reward while the true objective quietly degrades, visible only in calibration against real outcomes.

The Coached Application (drill, 41)

This is one node in the Atlas.

Every concept connects to the ideas that made it, the ways it fails, and the disciplines that keep it alive.

Browse all concepts