PAPERS-03-003 · Paper — Double DQN

The paper

The paper “Deep Reinforcement Learning with Double Q-learning” by Hado van Hasselt, Arthur Guez and David Silver (2015, presented at NIPS) is a landmark because it fixed a core bias in DQN and opened the door to more reliable deep RL agents.

Before this paper

Before Double DQN, the standard DQN algorithm used the same network to both pick the best next action and to judge its value. The max operator in this setup systematically overestimates action values, and those errors compound over time. This overestimation limited stability and performance, especially on challenging Atari benchmarks. Researchers needed a way to remove the upward bias without adding heavy computation.

The key idea

The key insight is to decouple action selection from action evaluation. The online network selects the best next action, while a separate target network evaluates the value of that chosen action. By doing so, the max operator’s bias is eliminated while keeping the architecture simple. This small change yields a cleaner estimate of future rewards.

How it works

During training, the online network computes Q‑values for the next state and picks the action with the highest value. That action index is then fed to the target network, which provides the Q‑value used in the Bellman backup. The target for the current state‑action pair becomes the reward plus the discounted target‑network value of the selected action. The target network is updated slowly from the online network, preserving stability. This procedure is exactly the Double Q‑learning update applied within the DQN framework, requiring no extra networks and only a few extra lines of code.

See it — decoupling selection from evaluation

flowchart LR
  A["Online net<br/>selects best action"] --> C["Evaluate that<br/>action..."]
  B["Target net<br/>evaluates its value"] --> C
  C --> D["Less<br/>overestimation"]

What they showed

The authors showed that Double DQN substantially reduced value overestimation and produced more stable, higher scores across many of the Atari games in the DQN benchmark. Their experiments demonstrated clearer learning curves and better final performance compared to the original DQN.

What it changed

After this work, Double DQN became a standard, near‑free ingredient in value‑based deep RL. It was quickly adopted in later agents such as Rainbow, improving both stability and performance. The community recognized that fixing the overestimation bias was essential for scaling deep RL to harder tasks. Consequently, almost every production DQN‑style agent now uses the Double DQN target. Next: Asynchronous Advantage Actor-Critic (A3C).

The one-line takeaway

Decoupling action selection from evaluation removes the max‑operator bias and dramatically improves DQN performance.

Read it / meet it

Reading the original paper gives insight into how a tiny algorithmic tweak can have big impact, and it’s a great example of careful empirical work. You’ll see its legacy in modern RL libraries and in agents like Rainbow that list Double DQN as a core component.