Drill 77 — The Certificate That Expired at Midnight

Theme: Networking & DNS · Format: 20-min drill · Difficulty: core

The story

A trading platform runs an internal settlement-api that every other service calls over mutual TLS: the ledger, the risk engine, the reconciliation batch, the customer-statement generator. It is not internet-facing, so nobody thinks of it as "a website with a certificate." It is just an address other services trust. It has served traffic without drama for two years.

The certificate on that endpoint was issued for 825 days by the internal certificate authority, back when the service was stood up. Renewal was a manual step — a calendar reminder on an engineer who has since changed teams — and the reminder went with them. There is no expiry monitoring on internal certs; the platform's uptime checks ping the TCP port and confirm something is listening, which it always is.

At exactly 00:00:00, the certificate's notAfter timestamp passes. Nothing crashes. No process restarts, no error is logged by the API itself — it is happily holding a port open with a certificate that is now, by the clock, invalid. But the next client that opens a connection completes the TCP handshake, receives the certificate, checks the expiry, and aborts the TLS handshake with certificate has expired. Then the next client. Then all of them. Within seconds every caller of settlement-api is failing at the same instant, retrying, and failing again — a clean, total, simultaneous outage of a service that "was up the whole time."

The overnight settlement run stalls. The on-call for the ledger is paged for connection errors and starts debugging the ledger. The port is open. The host is healthy. The logs on the API side say nothing is wrong.

⏸ Stop here and ask the room: the server never crashed, the port stayed open, and the host was healthy the whole time. So why did every single client fail at the exact same second — and why did it take hours to find?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Why did every client fail simultaneously, instead of degrading gradually?

Listen for: expiry is a timestamp, not wear — it is the same instant for every client on Earth, so there is no ramp. The moment the clock passes notAfter, every fresh TLS handshake fails the certificate-validity check at once; the failure lives on the client side of the handshake, which is why the server logs look clean. It is a step function, not a slope.
Weak signal: "the server went down" or "it got overloaded" — nothing crashed and nothing was slow; treating it as a capacity or crash event sends you debugging the wrong box.

Why didn't any monitor catch it — the port was being checked?

Listen for: the health check proved liveness (something is listening) not validity (the handshake actually succeeds). A TCP-port or ping check never negotiates TLS, so an expired cert is invisible to it. You need a probe that completes the handshake and a separate signal that watches days-until-expiry ahead of time.
Weak signal: "add more uptime checks" — more of the same blind check finds nothing; the gap is the kind of check, not the count.

Certs expire on a known date — why is this a monitoring problem and not a calendar problem?

Listen for: a human calendar reminder is a single point of failure that walks out the door with the person; the expiry date is machine-readable (notAfter), so it should drive an automated alert and, better, automated renewal. The real fix is to remove the human from the renewal loop entirely.
Weak signal: "set a better reminder" / "document the renewal in the runbook" — you are re-arming the exact failure mode that just fired; a forgotten renewal is a when, not an if.

The reveal

Two hours in, someone runs openssl s_client -connect settlement-api:8443 by hand and sees Verify return code: 10 (certificate has expired). The cert expired at midnight; the manual renewal was owned by an engineer who left. The immediate fix is a five-minute reissue — but the lesson is that a total, all-clients-at-once outage sat undiagnosed for two hours because nothing watched the one number that was knowable weeks in advance. The team added expiry monitoring on every cert (paging at 30, 14, and 7 days out), moved renewal to automated ACME issuance via cert-manager, built a cert inventory with a named owner per cert, and cut certificate lifetimes to days so renewal has to be automated — a manual process could never keep up.

Why this happens — the mechanics

A TLS certificate carries a hard notAfter timestamp, and validity is checked by the client during the handshake, not by the server. That inverts the usual failure geometry: the server that owns the expired cert has no error to log — it offers the cert it has and the far end walks away. So the one party that knows which cert it is serving emits no signal, while every party that fails has only a generic "handshake failed" to go on. The failure is also perfectly synchronized: unlike disk-fills or memory leaks that degrade on a curve, expiry is a single shared instant, so a fleet of clients transitions from 100% success to 100% failure with no ramp to alert on partway down.

The organizational root is a seam. An internal, non-web service doesn't feel like "a certificate you have to renew," so it falls outside the web-team's cert playbook, and its renewal becomes a private calendar entry rather than a monitored, owned, automated process. Best practice — mutual TLS on internal traffic — was applied without the matching signal, and the expiry date, though machine-readable and known years in advance, was watched by nobody. This is the same green-console-versus-reality gap as the silent-failure drills: the fix is to make the guarantee structural (automated renewal, short lifetimes) and to monitor the leading indicator (days-until-expiry) instead of trusting a human to remember.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant C as Client svc
    participant API as settlement-api
    participant M as Monitors
    participant Ops as On-call
    Note over API: Cert notAfter =<br/>00:00, valid until then
    C->>API: mutual TLS call at 2359
    API-->>C: 200 OK
    Note over API: 00:00 - cert expires<br/>server logs nothing
    C--xAPI: handshake - cert expired
    Note over M: TCP port check -<br/>still "listening", green
    Note over C,API: Every client fails at once<br/>100% to 0%, no ramp
    Note over Ops: Found 2h later by hand<br/>openssl, not a monitor

Solution — where the guardrails go:

flowchart TD
    A["Cert issued with<br/>notAfter date"] --> B["Cert inventory:<br/>every cert has an owner"]
    B --> C["Expiry monitor:<br/>days-until-expiry SLI"]
    C -.-> AL["Alarm 30 / 14 / 7<br/>days before expiry"]
    A --> D["Automated renewal<br/>ACME / cert-manager"]
    D --> E["Short-lived certs<br/>force renewal to automate"]
    E -.-> AL
    AL --> F["Renewed before expiry<br/>no outage"]

What good looks like

Days-until-expiry SLI on every certificate: a probe reads notAfter for each cert (endpoints and internal CA-issued ones) and emits days_until_expiry; warn at 30 days, escalate at 14, page at 7. No cert reaches single digits unnoticed.
Handshake-completing synthetic probe: an external check that actually negotiates TLS (not a TCP-port ping) and fails on certificate has expired / verify code 10 — so a live expiry pages within a minute, not hours.
Automated renewal via ACME / cert-manager: renewal is a controller reconciling desired state, not a human calendar entry; it renews at, say, two-thirds of lifetime and alerts if a renewal fails, removing the person from the loop.
A certificate inventory with a named owner per cert: cert, host, issuer, expiry, and owning team in one place; a cert with no owner is a finding, not a footnote.
Short-lived certificates (hours to a few days): lifetimes short enough that manual renewal is impossible, which forces the automation to be real and continuously exercised — a renewal path used every few hours can't silently rot.
A game-day that proves the alarm fires: issue a cert expiring in 7 days into staging and confirm the 30/14/7 alerts and the handshake probe actually page the owning team.

The lesson

An expiry date is a known-in-advance, machine-readable fact — so a cert outage is never a surprise, it's an unmonitored countdown. Watch days-until-expiry and automate renewal so the guarantee doesn't depend on a human remembering; a health check that proves a port is open proves nothing about whether the handshake still succeeds.

Named concepts: TLS certificate expiry, certificate-expiry monitoring, automated renewal (ACME), short-lived certificates.

Real-world echoes

Ericsson's expired-certificate outage (2018) took down mobile data for millions of O2 and SoftBank customers across multiple countries when a certificate in Ericsson's software expired — a single known-in-advance date, unwatched, cascading into a large-scale outage. The same-class lesson: certificate expiry is a countdown you can always see coming, and the failure is a monitoring-and-automation gap, never a genuine surprise.

Transfer

Which certificates in your stack — especially internal, service-to-service, and CA-issued ones that don't feel like "websites" — would take you down the instant they expire, and can you name today how many days each has left? "We'd get an alert" only counts if you can point at the monitor.

Make it a ticket: build a certificate inventory with an owner per cert, add a days_until_expiry monitor that pages at 30/14/7 days plus a handshake-completing synthetic probe, and move at least one critical internal cert onto automated ACME renewal this sprint.

Drill 75 — The TTL That Was Too Long: a sibling networking failure where a known-in-advance number (TTL) nobody revisited turned a clean config change into a day-long outage.
Drill 25 — The Key That Expired on Saturday: the same expiry-countdown failure wearing a credential costume, off-hours.
Drill 5 — The Twice-a-Year Bug: another time-triggered, simultaneous failure that no capacity or crash monitor could see coming.
Pattern #131 — Health-Check Design (liveness vs. readiness vs. actually completing the handshake). </content> </invoke>