AIM-05-004-P · Primer — Policies

The idea in one sentence

A policy is the agent's strategy—a rule that maps each state to an action or a probability distribution over actions.

Why it exists

Think of a driver who chooses a route at each intersection; the policy tells the driver which turn to take. Imagine the driver as a commuter who must decide every few seconds whether to turn left, go straight, or turn right. Without a pre‑planned rule, the driver would have to stop at every junction to weigh traffic, distance, and fuel cost, leading to endless hesitation. In everyday life we need a consistent decision rule so we can act without pausing to rethink every moment. In RL the same need appears: the agent must decide what to do in every possible situation, and the policy provides that systematic answer. Without a policy the agent would be aimless, unable to accumulate reward. The policy solves the problem of turning the abstract goal of maximizing return into concrete, repeatable actions.

How it works

First, the agent observes the current state of the environment. Then, using its policy, it selects an action: if the policy is deterministic, it picks the single prescribed action; if stochastic, it draws an action according to the assigned probabilities. Worked example: consider a grid‑world where the agent starts at cell (0,0) and wants to reach the goal at (2,2). The stochastic policy assigns a 70% chance to move right, a 20% chance to move up, and a 10% chance to stay put when in any non‑goal state. Starting from (0,0), the expected first move is right (to (1,0)) with probability 0.7, up (to (0,1)) with probability 0.2, or no movement with probability 0.1. Suppose the agent moves right to (1,0) and receives a reward of -1 for each step. After the move it observes the new state (1,0), samples the same distribution, and perhaps moves right again to (2,0). After three such steps it reaches the goal and receives a reward of +10, ending the episode. Over many episodes the agent updates the probabilities—perhaps increasing the right‑move probability to 85% because it leads to the goal faster—while still retaining some chance of moving up to explore alternative paths. The agent executes the action, receives a reward, and transitions to a new state. Over many episodes the agent updates its policy to increase expected return, aiming for the optimal policy that maximizes cumulative reward. This iterative loop—observe, act via policy, receive feedback, improve policy—continues until performance stabilizes.

A deterministic policy always picks the same action for a given state, while a stochastic policy assigns a probability distribution, which is useful for exploration. Some algorithms maintain both a target policy (what we aim to follow) and a behavior policy (what we actually sample) to balance learning and exploration.

See it — a policy maps states to actions

flowchart LR
  S["State"] --> P["Policy (pi)"]
  P --> A["Action<br/>(or a distribution)"]

The one gotcha

A common mistake is to assume that a greedy policy—one that always selects the currently highest‑valued action—will eventually discover the optimal solution. In practice this approach can become trapped in a local optimum because it never tries actions that appear worse in the short term but might lead to higher long‑term rewards. The real gotcha is forgetting to inject enough randomness; without a stochastic component the agent’s experience space remains narrow, preventing it from learning better strategies.

The one-line takeaway

A policy is the heart of RL—its quality determines whether the agent merely wanders or consistently moves toward the best possible outcome.

Where you'll meet it

Practitioners encounter policies when defining agents in libraries like OpenAI Gym or Stable‑Baselines, where you specify deterministic or stochastic action selection. During training you monitor policy performance and may export the final policy for deployment in simulations or real‑world systems. Audits often check that the learned policy respects safety constraints before it is used in production.