Drill 42 — The Notebook on GitHub

Theme: Security & Trust Boundaries · Format: 20-min drill · Difficulty: intro

The story

A data scientist pushes an analysis notebook to a public repo — a cleaned-up version of internal work, for a blog post. Cell 2 contains a live database credential, pasted months ago "just to get it working." Automated scanners (the attackers' kind) find it within hours. The rotation scramble reveals the deeper problem: that credential is shared by four services, and nobody is sure what breaks if it rotates.

⏸ Stop here and ask the room: the leak is the small problem. Name the big one.

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Why couldn't they rotate immediately?

• Listen for: the key was load-bearing in ways nobody had mapped — four services share it, no consumer inventory exists, and rotation had never been rehearsed. The strongest answers reframe the incident: the delay is the vulnerability; a leak you can rotate away in five minutes is a non-event.
• Weak signal: answers that stay on the paste — "the data scientist should have been more careful." Carefulness is not a control, and a room that stops there will fix the person instead of the system.

What's the blast radius of any one credential we hold?

• Listen for: per-credential scoping (one credential, one service), an actual inventory of what each key can touch and who consumes it, and short validity windows so any leak ages out on its own. Bonus points for someone asking to look one up right now.
• Weak signal: nobody can name a single credential's consumers, or the room assumes "the vault knows" without anyone ever having checked. An unmapped blast radius is a blast radius measured during the incident.

What catches the next pasted secret before it ships?

• Listen for: layered mechanical controls — pre-commit scanning on the laptop, a CI (Continuous Integration) gate that fails the build, provider-side push protection — plus the paved road that removes the reason to paste at all: fetching from a secret manager is easier than hardcoding.
• Weak signal: "training" or "a policy reminder" as the whole answer. The paste is inevitable at scale; a control that depends on nobody ever slipping is a control that has already failed.

The reveal

Rotation takes three days of careful coordination — three days with a known-leaked credential live in production — because the key was load-bearing in ways nobody had mapped.

Why this happens — the mechanics

Secrets sprawl because the moment of pasting is always locally rational. The data scientist needed the connection to work, the secret manager felt like friction, and "temporary" has no enforcement mechanism — so the credential stayed, invisible, while the notebook's context changed around it. Code moves — between repos, between laptops, between visibility levels — and anything embedded in it moves too. The paste isn't an anomaly; it's the default outcome of making the wrong path the easy one.

Rotation stays scary for a structural reason: shared credentials accumulate consumers silently. Every new service that authenticates with an existing key adds a dependency nobody records — there is no pull request that says "service four now depends on this key." By the time rotation matters, the dependency graph exists only in production behavior, and the only way to read it is to break it.

Discovery, meanwhile, is asymmetric. Attackers run continuous automated scanning against every public commit; the interval between push and exploitation is measured in minutes to hours. Your side of the race — a human noticing, a quarterly review — is measured in months. Any defense that depends on winning that race by attention loses; the control has to fire before the push completes, or the leak has to not matter.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant DS as Data scientist
    participant GH as Notebook on GitHub
    participant SC as Attacker scanners
    participant SV as Four services
    DS->>GH: Pastes live database<br/>credential in cell 2 (silent)
    Note over SV: Months pass - key quietly<br/>load-bearing (silent)
    DS->>GH: Cleans up notebook, pushes<br/>to a public repo
    SC->>GH: Automated scanners find<br/>the key within hours
    Note over DS: Rotation scramble begins
    DS--xSV: Key shared by four services -<br/>nobody knows what breaks
    Note over DS,SV: 3 days of coordination -<br/>known-leaked credential<br/>stays live

Solution — where the guardrails go:

flowchart TD
    A["Credential needed for analysis"] --> B["Fetched from a secret<br/>manager — never pasted<br/>into code"]
    B --> C["Short-lived credential,<br/>scoped to one service"]
    D["Notebook pushed to any repo"] --> E["Pre-commit and CI secret<br/>scanning"]
    E -.-> AL["Push blocked, owner<br/>alerted in minutes"]
    C --> F["Any single leak is small:<br/>one service, hours of<br/>validity"]
    F --> G["Rotation rehearsed as a<br/>routine, boring operation"]
    AL -.-> G

What good looks like

• A secret manager as the only source of truth — Vault, AWS Secrets Manager, or GCP Secret Manager-class tooling; code and notebooks fetch at runtime, and a hardcoded literal is treated as a bug even if it never leaks.
• Three scanning layers, each catching what the last missed: pre-commit hooks (gitleaks / detect-secrets / trufflehog-class scanners) on the laptop, a CI (Continuous Integration) gate that fails the build, and provider-side push protection on the repo host.
• Credential TTLs (Time To Live — how long a credential stays valid) measured in hours, not months — short-lived tokens minted per session via workload identity; anything unavoidably static gets a 90-day ceiling and a named owner.
• One credential, one service — so the blast-radius question has a one-line answer, with a consumer inventory per credential kept current by usage logs rather than by memory.
• Rotation gamedays — rotate a real production credential quarterly, on purpose, and measure time-to-rotate. The target is minutes, without a meeting.
• Honeytokens planted in repos — decoy credentials that alert the moment anyone uses them, telling you who is scanning your code and how fast.
• Time-to-rotate as the postmortem metric — the paste is inevitable and forgivable; three days of a known-leaked credential live in production is the finding.

The lesson

The real vulnerability wasn't the paste; it was that rotation wasn't routine. Secrets belong in managers (never in code or notebooks), pre-commit and CI scanning catch the inevitable slips, and credentials should be short-lived and service-scoped so that any single leak is small and any rotation is boring. If rotating a credential is scary, that's the finding.

Named concepts: secret scanning, short-lived credentials, credential scoping, rotation as routine operation.

Real-world echoes

• Uber, 2016 — attackers found AWS (Amazon Web Services) credentials in a private GitHub repository and used them to reach data on 57 million riders and drivers. The breach was bad; the cover-up — paid off and kept undisclosed for a year — cost far more, in fines, in departures, and in trust.
• Microsoft AI research, 2023 — a misconfigured SAS (Shared Access Signature) token published alongside open-source training data exposed 38 terabytes of internal storage, including internal messages. The token was meant to share a dataset; the scope and expiry nobody checked did the rest.

Transfer

Pick any production credential: if it leaked right now, how long until it's rotated and what breaks meanwhile? If the answer involves a meeting, we have work to do.

Make it a ticket: Take the credential the room just picked, list every consumer from its usage logs, and schedule its rotation as a rehearsed drill this sprint — the deliverable is a measured time-to-rotate, written down next to the target.

Related

• Drill 47 — The God-Mode Service Account — the same blast-radius question one layer up: identity scope instead of key sprawl, and the same "too useful to kill" accretion.
• Drill 48 — The Logs That Remembered Everything — the other place sensitive material hides in plain sight; there it's an auditor sampling logs instead of a scanner reading commits.
• Drill 4 — The Stale Score, theme 1 — the same one-line-among-forty review failure: human attention losing to a change that machine validation would have caught.