PAPERS-01-011 · Paper — Learning from Feedback

The paper

Training language models to follow instructions with human feedback — Ouyang, Wu, Jiang, and colleagues at OpenAI, 2022, published at NeurIPS (arXiv 2203.02155). It introduces InstructGPT, the model family produced by fine-tuning GPT-3 to follow human instructions using a technique the paper popularizes as RLHF — reinforcement learning from human feedback. This is the recipe that, applied at scale, became ChatGPT.

Before this paper

GPT-3 had shown that a large model trained only to predict the next token could do remarkable things when prompted well. But next-token loss optimizes for plausible continuation, not for being helpful, honest, and harmless. The result was a powerful engine that was hard to steer: it would ignore the instruction, make things up confidently, or produce toxic text, because none of those behaviors are penalized by the pretraining objective. The gap was alignment — the model's objective was not the user's intent.

The key idea

You cannot write down a loss function for "a good answer," but a human can look at two answers and say which one is better. The key idea is to turn that human preference into a trainable signal: collect rankings of model outputs, fit a reward model that predicts which output a human would prefer, and then use reinforcement learning to optimize the language model against that learned reward. The model stops chasing next-token likelihood and starts chasing what people actually want.

How it works

RLHF in this paper is a three-stage pipeline. Stage one — supervised fine-tuning (SFT): human labelers write high-quality demonstration answers to prompts, and GPT-3 is fine-tuned on them to give a strong starting policy. Stage two — the reward model: for each prompt the model samples several answers, labelers rank them best-to-worst, and a separate model is trained on those comparisons to output a scalar reward that mirrors human preference. Stage three — reinforcement learning: the SFT model is optimized with PPO (Proximal Policy Optimization) to maximize the reward model's score, with a penalty that keeps it from drifting too far from the SFT policy. The reward model is the crucial move — it lets a small set of human rankings supervise a very large number of RL updates.

See it — the three-stage RLHF pipeline

flowchart LR
    A["Pretrained GPT-3"] --> B["Step 1<br/>Supervised fine-tune"]
    B --> C["Step 2<br/>Reward model from<br/>human rankings"]
    C --> D["Step 3<br/>PPO vs the reward"]
    D --> E["InstructGPT"]

What they showed

The headline result is striking: labelers preferred the outputs of the 1.3-billion-parameter InstructGPT over those of the 175-billion-parameter GPT-3 — a model more than a hundred times smaller, judged better because it was aligned to what people asked for. Beyond preference, InstructGPT was more truthful and less toxic than the base model, while barely regressing on standard NLP benchmarks. Alignment, the paper showed, is not mainly a matter of scale — it is a matter of objective, and a modest model trained on the right signal beats a giant one trained on the wrong one.

What it changed

RLHF became the default final step for turning a raw pretrained model into a usable assistant; the reward-model-plus-PPO recipe from this paper is the direct ancestor of ChatGPT and the instruction-tuned models that followed across the industry. It reframed the frontier: raw capability now came from pretraining, but usability came from feedback, and the two could be optimized separately. Alignment makes the model usable — the next question is whether it can reason. Next: Think Step by Step.

The one-line takeaway

Fit a reward model to human preference and optimize against it, and a 1.3B model can beat a 175B one on being what people actually want.

Read it / meet it

Read the paper at arxiv.org/abs/2203.02155. You meet this idea every time you use ChatGPT, Claude, or any instruction-tuned assistant: the SFT-then-reward-then-PPO pipeline (and its lighter successors like DPO) is what makes those models follow instructions instead of merely continuing text.

Named concepts: RLHF, reward model, PPO, instruction tuning