PAPERS-03-005 · Paper — Proximal Policy Optimization (PPO)

The paper

Proximal Policy Optimization (PPO) was introduced by Schulman, Wolski, Dhariwal, Radford and Klimov in 2017 at the International Conference on Machine Learning (ICML). It quickly became a landmark because it offered a practical, reliable way to improve policies without the heavy math of earlier methods, making modern reinforcement learning much more accessible.

Before this paper

Before PPO, the field relied on Trust Region Policy Optimization (TRPO, Schulman 2015), which made policy updates reliable by constraining how far the policy could move each step. However, TRPO required complex second‑order optimization, making it hard to implement and tune. This complexity limited its adoption despite its stability. Researchers needed a simpler approach that kept the same safety guarantees.

The key idea

The key insight of PPO is to capture TRPO’s stability using only simple first‑order gradient updates. It does this by clipping the policy‑update objective so that the new policy cannot drift too far from the old one. The clipping creates a soft trust region, preventing overly large updates while staying compatible with ordinary stochastic gradient descent.

How it works

PPO first computes the probability ratio r(θ)=πθ(a|s)/π{θ_{old}}(a|s) for each taken action. It then defines a clipped surrogate objective that limits r(θ) to lie within [1‑ε, 1+ε], where ε is a small constant (often 0.2). If the ratio tries to go outside this band, the objective is flattened, removing the incentive to push further. The algorithm performs several epochs of minibatch stochastic gradient ascent on this clipped objective, updating the policy parameters with standard first‑order optimizers. Meanwhile, a value‑function loss and an entropy bonus are added to encourage stable learning and exploration. This whole loop runs repeatedly on collected trajectories, requiring only simple gradient steps.

See it — clipping the policy update

flowchart LR
  A["New vs old<br/>policy ratio"] --> B["Clip ratio<br/>to 1 plus/minus eps"]
  B --> C["Small, safe<br/>update"]
  C --> D["Stable<br/>learning"]

What they showed

The authors showed that PPO matched or beat more complex methods on continuous‑control and Atari benchmarks while being far simpler to implement and tune. In those experiments, PPO achieved performance on par with or superior to TRPO and other state‑of‑the‑art algorithms, confirming that the clipping trick preserved stability without the heavy machinery of second‑order methods.

What it changed

PPO became the default reinforcement‑learning algorithm across robotics, games, and—crucially—the RL step in reinforcement learning from human feedback (RLHF) used to align large language models. Its simplicity and strong performance led practitioners to adopt it as the go‑to method for training agents, dramatically lowering the barrier to entry for RL research and applications. By providing a reliable, easy‑to‑use optimizer, PPO accelerated progress in many domains that depend on policy learning. Next: AlphaGo.

The one-line takeaway

PPO shows that a simple clipping trick can give TRPO‑level stability with only first‑order optimization.

Read it / meet it

Reading the original PPO paper gives you a clear view of how a modest mathematical tweak can transform an algorithm’s practicality. You’ll see the same ideas at work today in game‑playing bots, robot controllers, and the RLHF stage that powers today’s chat models.