DEVOPS-04-004-P · Primer — Metrics: Aggregatable Numeric Time Series

The idea in one sentence

A metric is a stream of timestamped numbers, tagged with labels, that you record cheaply forever and add up across your whole fleet — trading away per-event detail for the ability to answer "how much, how fast, how many?" at massive scale.

Why it exists

Imagine you run 500 web servers and want to know: are we serving more errors than usual right now? You could log every single request — but that's terabytes a day, and to answer the question you'd have to scan all of it. That's expensive and slow.

A metric flips the trade. Instead of keeping every request, each server keeps a running tally: a counter like http_requests_total that just goes up by one per request. Every few seconds, a scraper reads the current number and stores it with a timestamp. Now "errors right now" is a tiny arithmetic problem over small numbers, not a search through a mountain of text.

The genius is that these numbers are aggregatable. Because each server independently counts, you can add all 500 counters together to get a fleet-wide total, or compare one server against the pack. You gave up the ability to ask "what did request #48,213 contain?" — a metric can't tell you that. In exchange you got a cheap, always-on dashboard of your system's vital signs. Metrics are the "how's the patient doing?" of observability; logs and traces are the "what exactly happened to this one request?"

How it works

Three ideas do all the work.

1. A time series is a named stream of (timestamp, value) points. The value is always a number — a temperature, a byte count, a request tally. Sampled every few seconds, one series looks like a line on a graph. Prometheus, the most common metrics system, defines a series as exactly this: a stream of timestamped values belonging to the same metric.

2. Labels turn one metric into many dimensions. A single metric name like http_requests_total becomes many distinct series when you attach key-value labels:

http_requests_total{method="POST", status="500", host="web-07"}
http_requests_total{method="GET",  status="200", host="web-07"}

Each unique combination of labels is its own separate time series. Labels are how you slice: "errors by host," "latency by endpoint." At query time you sum, filter, and group along these dimensions — exactly like a pivot table over live data.

3. Metric types make aggregation safe. The common shapes are a counter (only goes up, e.g. total requests), a gauge (goes up and down, e.g. memory in use), and a histogram (counts observations into buckets, e.g. request durations under 0.1s, under 0.5s, and so on). Counters and histogram buckets are additive, so combining series across machines is just addition. That additivity is the whole point — it's what lets sum(rate(http_requests_total[5m])) give a correct fleet-wide request rate.

Finally, retention keeps it cheap: because each point is just a small number, systems store them for weeks or months, often downsampling old data to coarser resolution so long-term history stays small.

The one gotcha

High-cardinality labels quietly explode your storage. Every unique label-value combination creates a brand-new time series. Add a bounded label like method (a handful of values) and you're fine. Add an unbounded one — user_id, email, request_id, a full URL with query strings — and you multiply your series count by millions. The Prometheus docs are blunt about this: do not use labels for unbounded sets of values like user IDs or emails, and most metrics should have few labels or none.

The trap is that it works perfectly in testing (three test users, three series) and falls over in production (three million users, three million series) — melting memory and blowing past retention limits. If a label's possible values aren't a small, known set, it belongs in a log or a trace, not a metric label.

See it — one metric fans out into labeled series, then aggregates back

flowchart TD
  M["Metric name:<br/>http_requests_total"]
  M --> S1["{host=web-07,<br/>status=200}"]
  M --> S2["{host=web-07,<br/>status=500}"]
  M --> S3["{host=web-42,<br/>status=500}"]
  S1 --> Q["Query time:<br/>sum by (status)"]
  S2 --> Q
  S3 --> Q
  Q --> R["status=200 -> 981/s<br/>status=500 -> 12/s"]

The one-line takeaway

Metrics are cheap, aggregatable numbers over time, labeled by dimension — use them for "how much / how fast / how many," keep the labels bounded, and reach for logs or traces when you need per-event detail.

Where you'll meet it

You'll meet metrics the moment you open Prometheus, Grafana, Datadog, CloudWatch, or any dashboard: every line chart of CPU, latency, or error rate is a labeled time series being aggregated live. You'll write them yourself when you instrument code with a client library (incrementing a counter, observing a histogram), and you'll define alerts on them ("page me if the error rate exceeds 1%"). The same model underlies open standards like OpenMetrics and OpenTelemetry's metrics data model, so the concepts transfer across nearly every tool you'll touch.

Named concepts: time series, labels and dimensions, aggregation and retention