DEVOPS-06-002-P · Primer — On-Call Rotations & Paging

The idea in one sentence

On-call is a rotating, always-covered duty where a schedule turns "someone should look at this" into a specific human's phone buzzing at 3 a.m., and paging (wake me up now) is deliberately louder than notifying (see it whenever).

Why it exists

Systems break at all hours, but people cannot watch dashboards 24/7 without burning out. So teams agree in advance: at any given moment, exactly one person owns the pager for the team, and that ownership hands off on a schedule (say, weekly). This does three things. It guarantees coverage, so an alert never falls into a void where everyone assumes someone else will catch it. It bounds the pain, so no single engineer is permanently tethered to their phone. And it makes urgency explicit, so a real outage interrupts a human's sleep while a slow-rising disk-space warning politely waits in a channel until morning. Without a rotation and a clear page-vs-notify line, you get one of two failure modes: nobody responds because ownership was fuzzy, or everybody is exhausted because every blip screams at everyone.

How it works

A rotation schedule is a calendar that answers "who is on-call right now?" for every minute of the year. You define participants and a shift length (commonly one week), and the tool automatically advances to the next person when a shift ends. Most teams run two layers. The primary is the first phone to ring. The secondary (or backup) exists as a safety net: if the primary does not acknowledge the page within a timeout, coverage escalates to the secondary so the alert is never dropped.

The path from "a machine noticed a problem" to "a human is awake" is an escalation policy. A monitoring system (metrics, logs, health checks) fires an alert. That alert hits an incident tool — PagerDuty, Opsgenie, or Grafana OnCall — which looks up the current on-call from the schedule and routes to them. If the primary does not acknowledge within, say, 5 minutes, the policy escalates to the secondary, and eventually to a manager. Acknowledging stops the escalation; the responder now owns the incident.

The key distinction is paging vs notifying. Paging uses high-urgency channels chosen to break through: a phone call, an SMS, or a push notification that overrides silent mode. Notifying uses low-urgency channels: a chat message, an email, a ticket. You reserve paging for things a human must act on now (user-facing outage, data at risk) and route everything else to notify. Getting this split right is the single biggest lever on whether on-call is sustainable — every page should be actionable and worth waking someone for.

The one gotcha

Do not page on things that are not actionable right now. If a warning cannot be meaningfully acted on at 3 a.m. — a capacity trend, a flaky test, a non-critical retry that already succeeded — it should notify, not page. Every unactionable page trains responders to ignore the pager, which is called alert fatigue, and a numbed responder is exactly who you do not want handling the real outage buried in the noise. The related trap is a sloppy handoff: the outgoing on-call must brief the incoming one on anything still smoldering — an open incident, a fragile deploy, a silenced alert that will un-silence — because an alert that fires mid-handoff must land on someone who has context, not a cold stranger.

See it — an alert climbs the escalation ladder

sequenceDiagram
autonumber
    participant Mon as Monitoring
    participant Tool as Paging Tool
    participant P as Primary on-call
    participant S as Secondary on-call
    Mon->>Tool: Alert fires (service down)
    Tool->>Tool: Look up current<br/>on-call from schedule
    Tool->>P: PAGE (call + SMS + push)
    Note over P: No ack within<br/>5-minute timeout
    Tool->>S: Escalate PAGE secondary
    S->>Tool: Acknowledge
    Note over S: Secondary owns<br/>the incident now
    Tool->>Mon: Low-urgency issue?<br/>NOTIFY chat instead

The one-line takeaway

A rotation guarantees someone always owns the pager; page only for what must be fixed now, notify everything else, and never hand off without a brief.

Where you'll meet it

You will set up escalation policies and schedules in PagerDuty, Opsgenie, or Grafana OnCall the first time your team goes on-call. You will feel it directly when a page wakes you and you have to acknowledge, triage, and resolve — and again every handoff when you inherit (or pass on) whatever is still smoldering. In interviews and design reviews, "how do you make sure the right person gets woken up, and only for the right things?" is the on-call question hiding inside every reliability discussion.

Named concepts: rotation schedules, primary vs secondary, paging vs notifying, handoff hygiene