BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 28 — The Friday Flag

Theme: People & Process · Format: 20-min drill · Difficulty: core

The story

An engineer tests fail-open behavior in staging by flipping a config flag: verification_enabled: false. The test works, the ticket closes — and the flag rides along in Friday's 5 p.m. release, because staging config and prod config share a file and the diff was forty lines long. All weekend, production approves without verification. Monday's review finds thousands of decisions that skipped the check.

⏸ Stop here and ask the room: name every gate this passed through, and why each one let it by.

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Read the story aloud and pause at the blockquote. Let the room list gates before you steer — expect the obvious three to surface fast (the ticket close, the code review, the green pipeline). Keep pushing until someone names the gates that don't exist yet: the config canary, the post-deploy behavioral check, the flag inventory. That's the pivot of the whole drill.

"Was the reviewer careless — or was the diff unreviewable?"

Listen for: the shift from "the reviewer should have caught it" to "a forty-line config diff with one dangerous line is a system problem — review attention is finite, and this diff spent it on noise." Strong answers propose making dangerous values mechanically distinguishable, so no human has to spot them at 5 p.m. on a Friday.

Weak signal: "we need more careful reviewers" or "require a second reviewer" — that doubles the cost of every diff to catch one line a machine could flag for free, and it fails the same way the first reviewer did.

"What should happen automatically after every deploy?"

Listen for: behavioral verification — a check that asserts what production is doing, not what the pipeline reported. "Verification ran on the last 100 decisions" is the canonical answer. Also good: config canaries rolling to a small slice first, and alerts on any diff touching safety-relevant keys.

Weak signal: "check the dashboards after deploying" — a human memory task, which is exactly the component that just failed.

"Should risky flags even be able to reach prod silently?"

Listen for: schema-level guards — a flag that disables a safety check cannot be false in prod without a second approval, enforced by the config system rather than by convention. And flag hygiene: every flag carries an owner and an expiry date, so test-only flags die after the test instead of riding along.

Weak signal: "we'd keep a list of the dangerous flags" — a list nobody reads is documentation, not a guard.

The reveal

Everyone involved was competent and busy. The system offered no seam where this would be caught: no config canary, no post-deploy behavioral check, no flag inventory.

Why this happens — the mechanics

Nothing in this story required negligence. The engineer used the sanctioned test mechanism; the reviewer approved a diff of the size the team approves every week; the pipeline reported exactly what it measures. The failure lives in normalization of deviance — the gradual process by which a shortcut that never bites becomes the standard way of working. Staging and prod sharing one config file was a known wart; it had ridden along safely so many times that it stopped registering as a risk at all. Each safe Friday made the next one feel safer, right up until the weekend it wasn't.

Production pressure sets the tempo. A 5 p.m. Friday release exists because the schedule demands it, and under schedule pressure humans triage their attention: a forty-line diff gets scanned for the lines the reviewer expects to matter. Human review is a probabilistic control — very good at catching what it's primed for, poor at spotting one quiet boolean among formatting churn. Treating it as a deterministic gate ("review will catch it") assigns a machine's job to a person and then acts surprised.

The deeper mechanic is where verification stops. The pipeline verifies that artifacts built, tests passed, and the rollout completed — it never asks what production behavior changed. That leaves a blind spot precisely at the moment of highest risk, immediately after a change, on the one weekend nobody is watching. Unrehearsed assumptions rot the same way unrehearsed procedures do: "someone would notice in review" was never once tested, so nobody knew it was already false.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant EN as Engineer
    participant ST as Staging
    participant REL as Friday release
    participant PR as Production
    participant MR as Monday review
    EN->>ST: Flips verification_enabled<br/>to false for a test
    ST-->>EN: Test passes - ticket closes
    Note over ST: Staging and prod share<br/>one config file (silent)
    ST->>REL: Flag rides along - one line<br/>in a 40-line diff (silent)
    REL->>PR: 5 p.m. Friday deploy -<br/>pipeline green
    Note over PR: Approves without<br/>verification all weekend<br/>(silent)
    PR->>MR: Thousands of decisions<br/>skipped the check
    Note over MR: Found Monday - by a review,<br/>not by a monitor

Solution — where the guardrails go:

flowchart TD
    A["Engineer flips<br/>verification_enabled:<br/>false in staging"] --> B["Flag has an owner and an<br/>expiry date — dies after<br/>the test"]
    A --> C["Friday release includes<br/>the config diff"]
    C --> G1["Schema guard: flag cannot<br/>be false in prod without<br/>second approval"]
    G1 -.-> D1["Deploy blocked — caught<br/>before rollout"]
    C --> G2["Config canary rolls to a<br/>small slice first"]
    G2 --> G3["Post-deploy smoke test:<br/>verification ran on the<br/>last 100 decisions"]
    G3 -.-> D2["Alarm fires within minutes<br/>of deploy"]
    D2 --> R["Flag reverted Friday<br/>evening, not Monday"]

What good looks like

The lesson

The deploy isn't done when the pipeline is green — it's done when production behavior is verified (a smoke test asserting "verification ran on the last 100 decisions" takes minutes to build). Dangerous flags need schema-level guards (cannot be false in prod without a second approval) and every flag needs an owner and an expiry date.

Named concepts: post-deployment verification, progressive delivery, feature-flag hygiene, change management.

Real-world echoes

Transfer

Which single config values, flipped silently, would hurt us most? Do any of them have a guard beyond "someone would notice in review"?

Make it a ticket: inventory every feature flag currently live in prod — owner, expiry, prod value — and add a schema-level guard to the three that disable a safety check. Wire one post-deploy smoke test that asserts verification ran on the last 100 decisions.

Related