AIM-05-003-P · Primer — Reward, Return & Discounting

The idea in one sentence

Reward, Return & Discounting describe how an RL agent evaluates the total future reward, weighting each step by a discount factor to decide what to maximize.

Why it exists

Imagine you are saving money: a dollar today feels more valuable than a dollar next year because you can spend or invest it now; similarly, discounting captures the preference for sooner rewards. Think of it like a grocery shopper who prefers fresh produce over food that will spoil in a week—fresh items are more useful now, so the shopper discounts the value of future freshness. Without discounting, an endless stream of tiny rewards would make the total value infinite and meaningless. The discount factor gamma solves this by shrinking future rewards, keeping the sum finite and reflecting how far ahead the agent should plan.

How it works

The agent computes the return G_t as the sum of discounted rewards: G_t = r_{t+1} + γ·r_{t+2} + γ^2·r_{t+3}+… where γ∈[0,1]. For each step ahead, the reward is multiplied by γ raised to the number of steps away.

Worked example: Suppose an agent receives three future rewards: r_{t+1}=5, r_{t+2}=8, and r_{t+3}=10, and we set γ=0.8. The discounted contributions are:

• Step 1: 5 × 0.8⁰ = 5.00
• Step 2: 8 × 0.8¹ = 6.40
• Step 3: 10 × 0.8² = 6.40 The total return G_t = 5.00 + 6.40 + 6.40 = 17.80. This concrete sum shows how distant rewards are shrunk but still influence the decision.

The agent then chooses actions that maximize this G_t, not just the immediate r_{t+1}. This process repeats at every time step, continually updating the expected return based on the latest observations and the chosen γ.

If γ is set close to 1, the agent becomes far‑sighted, valuing distant rewards almost as much as immediate ones; if γ is near 0, the agent behaves myopically, focusing almost entirely on the next reward. Some frameworks use a separate discount schedule or state‑dependent γ to handle varying horizons.

See it — discounting future reward

flowchart LR
  A["r1 now"] --> G["+ gamma*r2"]
  G --> H["+ gamma^2*r3 ..."]
  H --> R["= return G"]

The one gotcha

A common mistake is to assume the agent will always chase the biggest immediate payoff. In fact, the agent optimizes the total discounted return over the horizon, so a tempting short‑term reward can be detrimental if it leads to a cascade of low‑value future rewards. The key is to evaluate each action by its full G_t, not by a single-step reward.

The one-line takeaway

Discounting lets an RL agent balance present and future rewards, turning an infinite horizon into a manageable, goal‑directed objective.

Where you'll meet it

Practitioners see discounting in every RL algorithm implementation, from Q‑learning to policy gradients, where γ is a hyperparameter you tune. Tools like OpenAI Gym, TensorFlow Agents, and RLlib expose γ in their configuration files, and you’ll audit its value when diagnosing why an agent prefers short‑term gains over long‑term success.