DEVOPS-06-003-P · Primer — Alert Severity & Signal Quality

The idea in one sentence

A good alert wakes a human only when a human's action is the fastest way to protect users, and its severity label tells that human, at a glance, how hard to run.

Why it exists

Monitoring is easy to add and hard to trust. The first instinct on any team is to alert on everything: high CPU, a restarted pod, a slow disk, a queue that grew. Within weeks the on-call phone buzzes a dozen times a night, most of it self-heals before anyone reads it, and people start swiping the notifications away without looking. That reflex — ignoring alerts because most of them are noise — is alert fatigue, and it is dangerous precisely because it eventually swallows the one page that mattered.

The Google SRE Book states the target bluntly: "Every time the pager goes off, I should be able to react with a sense of urgency," and "every page should be actionable." An alert that isn't actionable isn't a helpful heads-up — it's a small tax on the team's trust, paid every time it fires. Severity levels exist for the same reason: when a page does arrive, the responder needs to know in one second whether this is "wake up, revenue is bleeding" or "look at it Monday." Getting both of these right — which alerts fire, and how loudly — is what separates on-call rotations people can sustain from ones that burn people out.

How it works

Two design decisions carry most of the weight.

1. Alert on symptoms, not causes (symptom-based alerting). Users don't feel your CPU graph; they feel slow pages and errors. So page on the thing the user experiences. The SRE Book advises spending "much more effort on catching symptoms than causes." The classic starting set is the four golden signals — latency, traffic, errors, and saturation — measured as close to the user as possible (black-box, from outside the system). Causes (a full disk, a hot replica) belong on dashboards you consult after a symptom pages you, not as pages themselves. Why? One user-visible symptom can be produced by twenty different causes; alerting on all twenty causes gives you twenty noisy, redundant pages, while alerting on the symptom gives you one page that fires no matter which cause is to blame.

2. Label by severity so the response scales to the impact. Severity levels (SEV1–SEVn) rank incidents by user impact, and — by convention at PagerDuty and most shops — lower number means more urgent. A typical scheme:

• SEV1 — catastrophic: most or all customers can't use the product, data loss, or revenue is bleeding now. Page on-call + backup, all hands, acknowledge in minutes.
• SEV2 — major: a core feature is badly broken with no workaround. Page on-call.
• SEV3 — minor: partial or intermittent impact with a workaround. A ticket or a Slack ping, handled in business hours — not a page.
• SEV4/SEV5 — cosmetic or informational: log it, no interruption.

The severity dictates the notification path. Only SEV1/SEV2 earn a phone-ringing page. Everything below routes to a queue you check during the day. This is what keeps an alert actionable: before you create any rule, ask "if this fires at 3 a.m., is there something a human must do right now?" If the answer is no, it is a dashboard line or a low-severity ticket, never a page.

The one gotcha

The trap is treating severity as a property of the cause instead of the user impact. "The database is at 90% connections" is not a severity — it's a metric. It might be SEV1 (users are being turned away) or a total non-event (you have autoscaling and plenty of headroom). Teams that page on the cause end up assigning severities to things that have no user impact at all, which re-creates alert fatigue from the other direction. Anchor severity to the answer to one question: how many users are hurting, and how badly? If the metric can spike without any user noticing, it does not deserve a severity or a page — it deserves a dashboard.

See it — routing an event by symptom and severity

flowchart TD
  A["Event detected"] --> B{"User-visible<br/>symptom?"}
  B -- "No, just a cause" --> C["Dashboard line<br/>(no page)"]
  B -- "Yes" --> D{"How bad<br/>for users?"}
  D -- "Most/all down,<br/>data or revenue" --> E["SEV1 — page<br/>on-call + backup"]
  D -- "Core feature broken,<br/>no workaround" --> F["SEV2 — page<br/>on-call"]
  D -- "Partial, has<br/>workaround" --> G["SEV3 — ticket,<br/>business hours"]
  D -- "Cosmetic /<br/>informational" --> H["SEV4 — log only"]

The one-line takeaway

Page a human only when a human must act, rank the page by how much users are hurting, and everything else is a dashboard — not a 3 a.m. phone call.

Where you'll meet it

You'll meet this the moment you set up Prometheus Alertmanager, PagerDuty, or Opsgenie and have to write your first alert rule and pick its routing. You'll meet it in every incident retro where someone asks "should this have paged us?" and in every on-call handoff where the outgoing engineer warns "ignore the disk-space alert, it's always noisy." You'll meet the severity vocabulary — SEV1, SEV2, SEV3 — in incident channels, status pages, and postmortem templates across nearly every operations team you join.

Named concepts: severity levels (SEV1-SEVn), actionable alerts, alert fatigue, symptom-based alerting