DEVOPS-04-003-P · Primer — Logs: Discrete Events as Telemetry

The idea in one sentence

A log is a timestamped record of one discrete thing that happened, and the story of your system is what you can reconstruct by reading enough of them in order.

Why it exists

When your program does something interesting — accepts a request, fails to reach a database, retries, gives up — the cheapest, oldest way to leave a trace is to write a line of text somewhere: 2026-07-09T14:03:22Z ERROR could not connect to db: timeout. That line is a log event. It captures a specific moment, not an aggregate.

That distinction matters. A metric like "error rate = 3%" tells you how much is wrong; it cannot tell you which request failed, for which user, or what the error message actually said. Logs fill that gap. They are the raw, high-detail record you fall back to when a metric spikes and you need the underlying facts. Their strength is fidelity: each event can carry as much context as you cared to write down. Their cost profile is the flip side — one line per event means volume grows with traffic, so logs are expensive to store, ship, and search compared with the compact counters we call metrics.

Because logs are so old and so universal, the ecosystem standardized early. The Unix syslog protocol, formalized in RFC 5424, defined a message format and a severity scale that still shapes how we think about logs today.

How it works

Three ideas do most of the work.

Log levels (severity). Not every event deserves the same attention, so each log line carries a level. RFC 5424 defines eight severities, numbered 0 (most urgent) to 7: Emergency, Alert, Critical, Error, Warning, Notice, Informational, Debug. In everyday application logging you'll mostly see a smaller set — DEBUG, INFO, WARN, ERROR — but the idea is the same: the level lets you filter. In production you might emit everything at INFO and above and drop DEBUG to save volume; when chasing a bug, you turn DEBUG back on. OpenTelemetry's Logs Data Model normalizes these into a SeverityNumber (1 to 24) plus the original SeverityText, so a level written by any tool can be compared consistently — for example, a record at ERROR or higher signals an erroneous situation.

Unstructured logs. The classic log line is just a string a human wrote for another human: "user 4152 checkout failed after 3 retries". It's easy to produce and easy to read one at a time, but a machine has to parse it with fragile regexes to pull out user_id or retries. This is the weakness of unstructured logs. The modern move is structured logging — emitting the same event as key/value data (often JSON): {"level":"error","event":"checkout_failed","user_id":4152,"retries":3}. Same information, but now queryable by field without guessing at text patterns.

Log aggregation. One process writing to one file is fine on one machine. But a real system has many services on many hosts, each with its own file, and hosts that get replaced. Grepping a hundred files by hand does not scale. Aggregation solves this: an agent on each host tails the log files, ships every line to a central log store, which indexes the events so you can search across all of them at once — by time range, by service, by level, by field. This is the shift the course title points at: from text search on one box (grep ERROR app.log) to aggregation and query across the fleet (level=error AND service=checkout in the last 15m).

The one gotcha

Logs are seductive because they're easy to add — one print statement and you have telemetry — and that's exactly the trap. Log volume, and therefore log cost, scales with traffic. Double your requests and you double your log bytes, along with the ingest, indexing, and storage bill behind them. A single overly chatty INFO line on a hot code path can quietly become your largest observability expense. So treat every log level as a deliberate choice: DEBUG for local investigation, INFO for meaningful state changes, WARN/ERROR for things a human should notice. If you find yourself logging counts — "processed 500 items" over and over — that's usually a signal you wanted a metric, not a log.

See it — one event's journey from app to searchable store

flowchart LR
  A["App emits event<br/>level=ERROR + fields"] --> B["Written to<br/>local log file"]
  B --> C["Agent tails file<br/>ships each line"]
  C --> D["Central store<br/>indexes by time,<br/>service, level, field"]
  D --> E["Engineer queries<br/>across the fleet"]

The one-line takeaway

A log is one timestamped event with a severity level; the discipline is choosing what to log, structuring it as fields, and aggregating it so you can query the whole fleet instead of grepping one file.

Where you'll meet it

You'll meet log levels the first time you set a service to log at INFO in production and DEBUG while debugging. You'll meet unstructured-versus-structured logs when a regex that parsed your logs breaks because someone reworded a message — and again when you switch to JSON logging so queries stop guessing. And you'll meet aggregation the moment an incident spans five services and you need one search box, not five SSH sessions, to find the failing request.

Named concepts: log levels, unstructured logs, log aggregation