DEVOPS-04-001-P · Primer — What Is Observability?

The idea in one sentence

Observability is how well you can ask brand-new questions about what a running system is doing — and get answers — using only the data the system already emits, without shipping new code first.

Why it exists

The word comes from control theory: in 1960, Rudolf Kálmán defined a system as observable if you can figure out its internal state just by watching its external outputs. Software borrowed the idea for a very practical reason.

Traditional monitoring works by deciding in advance what could go wrong. You pick metrics — CPU, error rate, latency — set thresholds, and draw dashboards. That is great for the failures you already imagined: the known-unknowns (you know CPU might spike, you just don't know when).

But modern systems — microservices, third-party APIs, autoscaling, weird user input — fail in ways nobody predicted. Those are unknown-unknowns: problems you didn't know were possible, so you never built a dashboard for them. When one hits, a monitoring-only setup leaves you stuck. Every dashboard is green, users are angry, and to investigate you'd have to guess a cause, add new logging, redeploy, and wait for it to happen again.

Observability exists so you can investigate a novel failure right now, from data you already have, without that guess-code-redeploy loop.

How it works

The mechanism is: emit rich, detailed events — then query them freely after the fact.

• Monitoring pre-aggregates. It answers a fixed question ("what is the average error rate?") because the number was summed up before you asked. You cannot decompose it later.
• Observability keeps the detail. Each event carries lots of context (which user, which build, which region, which feature flag, which downstream call). You store that context and slice it at query time with questions you never planned for.

That ability to slice by many distinct attribute values is the core of high-cardinality questions. Cardinality is just the number of unique values a field can have. country is low cardinality (~200 values). user_id, request_id, and build_sha are high cardinality — potentially millions of unique values. Classic metrics tools choke on high cardinality (each unique combination becomes its own expensive time series), so those tools quietly push you toward low-cardinality summaries. An observable system instead lets you ask "show me the 12 requests that were slow, and only for user_id=91723 on build=a4f9 in region=eu-west" — a question you could never have pre-registered on a dashboard.

The common raw materials are logs, metrics, and traces (often called the "three pillars"), and the emerging emphasis on wide, structured events — one fat record per unit of work, carrying dozens of fields you can pivot on. OpenTelemetry is the open standard for emitting all of this in a vendor-neutral way.

The one gotcha

Observability is not "monitoring plus more dashboards," and buying a tool with "observability" on the box does not grant it. If your data is pre-aggregated and stripped of context — average latency, total error count, no request_id, no user_id — you can only ever re-ask the questions someone anticipated. The property lives in the cardinality and richness of what you emit, not in the vendor logo. Low-context data means low observability, full stop.

See it — monitoring answers known questions; observability answers new ones

flowchart TD
  I["Something feels wrong<br/>in production"] --> Q{"Did we predict<br/>this failure?"}
  Q -->|"Yes<br/>(known-unknown)"| M["Monitoring:<br/>read the dashboard<br/>we built in advance"]
  Q -->|"No<br/>(unknown-unknown)"| O{"Do events carry<br/>rich context?"}
  O -->|"Yes"| A["Slice by high-cardinality<br/>fields at query time<br/>(user, build, region)"]
  O -->|"No"| G["Stuck: guess, add<br/>logging, redeploy,<br/>wait for repeat"]
  A --> R["Root cause found<br/>without shipping code"]

The one-line takeaway

Monitoring tells you whether the failures you predicted are happening; observability lets you ask any question — especially high-cardinality ones — to explain the failures you never saw coming.

Where you'll meet it

You'll meet observability the first on-call night an outage has no matching dashboard and you need to explain "why is this one customer seeing timeouts?" from data you already have. You'll meet it again when you instrument a service with OpenTelemetry, when a metrics bill explodes because someone put user_id on a metric label, and in interviews where "what's the difference between monitoring and observability?" is a standard opener.

Named concepts: observability vs. monitoring, unknown-unknowns, high-cardinality questions