DEVOPS-04-002-P · Primer — The Three Pillars: Logs, Metrics, and Traces

The idea in one sentence

Logs, metrics, and traces are three different shapes of telemetry — each one answers a different class of question about what your running system is doing.

Why it exists

When a service is slow or broken, you can't attach a debugger to production and step through it. You only have whatever the system recorded about itself while it ran. That recorded evidence is called telemetry, and it comes in a few distinct shapes because different questions need different data.

If you only kept text logs, you couldn't cheaply answer "what's the 95th-percentile latency right now?" — you'd have to scan and count millions of lines. If you only kept numbers, you couldn't answer "what exactly happened to this one failed checkout?" If you only had per-service views, you couldn't follow a single user request as it hopped across ten microservices. So the industry settled on three complementary signal types. OpenTelemetry, the standard project for this data, names them explicitly: logs, metrics, and traces (plus a supporting signal, baggage). Each is a deliberate trade-off between detail and cost.

How it works

Think of it as three lenses pointed at the same running system.

Metrics are numbers aggregated over time: a request count, a CPU percentage, a queue depth, a latency histogram. They are cheap to store and fast to query because they're pre-summarized — you throw away individual events and keep counters and buckets. Metrics answer "is something wrong, and how much?" They power dashboards and alerts. Their weakness: a metric tells you the error rate jumped, but not why.

Logs are timestamped records of discrete events — "user 42 logged in," "payment failed: card declined." They carry rich, specific detail, which makes them the go-to for "what exactly happened here?" Structured logs (key/value fields instead of free-form text) are far easier to search and filter. The long-standing standard for log message format is RFC 5424, the Syslog Protocol. Their weakness: volume. Logs are detailed and expensive, so you rarely aggregate them cheaply.

Traces follow one request end-to-end across services. A trace is a tree of spans; each span is one unit of work (an HTTP call, a DB query) with a start time, duration, and parent. Traces answer "where did the time go, and which service caused the failure?" They work by propagating a shared trace ID across service boundaries — the W3C Trace Context recommendation standardizes this with the traceparent HTTP header so tools from different vendors can stitch the same request together. Their weakness: they show request flow, not fine-grained "why" — you often jump from a slow span into that span's logs.

The move you'll make constantly: metric alerts you → trace localizes the culprit service → log explains the exact failure.

The one gotcha

People treat the three as interchangeable — "just log everything and we'll figure it out." You can't. Logging every number as text makes querying "average latency" absurdly expensive, and trying to reconstruct a cross-service request from independent per-service logs is guesswork without a shared trace ID. Pick the signal that matches the question: aggregate numbers → metrics, discrete detailed events → logs, cross-service request flow → traces. They're strongest woven together, not one substituted for another.

See it — from alert to root cause across the three signals

flowchart TD
  A["Metric: error rate<br/>spikes on dashboard"] --> B["Alert fires"]
  B --> C["Trace: follow one<br/>failed request"]
  C --> D["Span shows the<br/>slow/failed service"]
  D --> E["Log: exact error<br/>on that service"]
  E --> F["Root cause found"]

The one-line takeaway

Metrics tell you that something is wrong, traces tell you where, and logs tell you why.

Where you'll meet it

Every observability stack organizes around these three signals: Prometheus for metrics, Loki or Elasticsearch for logs, Jaeger or Tempo for traces, and OpenTelemetry as the common way to emit all three. The moment you're on call and a dashboard goes red, you'll walk this exact path — metric to trace to log — to find the fault before your coffee gets cold.

Named concepts: logs, metrics, traces, telemetry signal types