Drill 60 — The Package That Turned

Theme: Vendors & Dependencies · Format: 20-min drill · Difficulty: core

The story

A widely-used open-source utility library — the kind of package everything depends on and nobody thinks about — ships a compromised release after a maintainer account takeover. Your nightly CI (Continuous Integration) job auto-updates minor versions "to stay current," so the malicious version is bundled into builds and reaches production before the ecosystem raises any alarm. You learn about the compromise from security Twitter, not from your own tooling. The first question in the incident channel is "are we exposed?"

⏸ Stop here and ask the room: how long does answering that question take, and why?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Read the story, pause at the ⏸, and take a quick poll before any discussion: hands up — minutes, hours, or days? Remember the split; it makes the reveal land.

"What stands between a maintainer's bad day and our production builds?"

Listen for: named controls in the path — pinned lockfiles, a cooling-off window before adoption, provenance or signature checks in CI, a private registry proxy with an allowlist. The strong answer can trace the route from "release published" to "code in production" and point at each gate along it.

Weak signal: "we'd catch it in code review" — nobody reviews the minor-version bump of a transitive dependency — or the slow, live realization that the honest answer is "nothing but the maintainer's password."

"How fast can we say which services bundle which version of which package?"

Listen for: "we query the SBOM" (Software Bill of Materials — a machine-readable inventory of every component in your software), generated per build artifact — and someone who has actually run that query, not just deployed the tool that theoretically produces it.

Weak signal: "grep across the repos" — which answers what's in source, not what's in the artifacts currently running in production — or exposure estimates measured in days, delivered with a straight face.

"Is auto-updating a hygiene virtue here, or an unlocked door?"

Listen for: nuance — staying current is a security control against known vulnerabilities, but immediate, unattended adoption of fresh releases trades that for zero-day supply-chain exposure. The strong answer wants both: regular, deliberate updates behind a pinned lockfile and a cooling-off delay.

Weak signal: treating it as a binary — either "auto-update everything, currency is hygiene" or "pin forever," which quietly accumulates unpatched vulnerabilities instead. Both extremes are the same mistake: outsourcing the decision to a default.

Facilitator close: return to the poll. The story's answer was days — and the alarm came from strangers on the internet.

The reveal

Days. Nobody can quickly say which services bundle which version — exposure assessment becomes an archaeology dig through dozens of build artifacts and lockfile histories. The alarm came from strangers on the internet; the inventory came from grep.

Why this happens — the mechanics

Auto-update is sold as hygiene, but mechanically it imports someone else's security posture into your deploy pipeline. When CI (Continuous Integration) adopts every fresh minor release unattended, your production trust boundary extends to every maintainer's account security — their password reuse, their two-factor setup, their laptop. A typical service pulls in hundreds of packages; that's hundreds of strangers who each hold, in effect, unattended deploy access to your fleet. None of them agreed to your threat model, and most don't know you exist — asymmetric attention, again, now with attacker economics attached: a maintainer account is far cheaper to compromise than your perimeter, and one poisoned release fans out to thousands of downstream victims at once.

The transitive graph makes this worse than it looks. Teams audit the dozen dependencies they chose deliberately; the lockfile quietly resolves hundreds more that nobody chose at all. "The kind of package everything depends on and nobody thinks about" is precisely the highest-value target — maximum blast radius, minimum scrutiny, and a minor-version bump that no reviewer will ever open.

Detection is asymmetric too. The ecosystem's alarm arrives on the ecosystem's schedule — a researcher notices, a thread spreads — which is why a cooling-off window works: it lets the world be your canary before the new release touches your builds. And without an SBOM (Software Bill of Materials — a machine-readable inventory of every component in your software), "are we exposed?" decomposes into archaeology, because what's actually running in production is the sum of build-time resolution decisions nobody recorded. The compromise took one account takeover; the inventory takes days, and the gap between those two numbers is your exposure window.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant AT as Attacker
    participant REG as Package registry
    participant CI as Nightly CI
    participant PR as Production
    participant TW as Security Twitter
    AT->>REG: Account takeover - publishes<br/>compromised release (silent)
    CI->>REG: Auto-updates minor versions<br/>to stay current (silent)
    REG-->>CI: Malicious version pulled<br/>into the nightly build
    CI->>PR: Bundled and shipped before<br/>any ecosystem alarm
    TW-->>PR: News breaks on security<br/>Twitter, not own tooling
    Note over CI,PR: Are we exposed - nobody can<br/>say which services bundle<br/>which version
    Note over PR: Days of lockfile archaeology<br/>to answer one question

Solution — where the guardrails go:

flowchart TD
    A["Maintainer account taken over"] --> B["Compromised release published"]
    B --> C["Lockfiles pin every<br/>version — nothing changes<br/>without a commit"]
    C --> D["Cooling-off window: new<br/>releases wait days before<br/>adoption"]
    D -.-> AL["Ecosystem raises the alarm<br/>inside the window"]
    B --> P["Provenance and signature<br/>scanning in CI"]
    P -.-> AL
    AL --> Q["SBOM query: which services<br/>bundle this package"]
    Q --> R["Answer in minutes — zero<br/>services adopted the bad<br/>release"]

What good looks like

Lockfiles everywhere, enforced: every service pins exact versions; CI fails the build on lockfile drift, so nothing in production changes without a reviewed commit.
A written cooling-off policy: new releases age a set number of days (say 7–14) before adoption, with a documented fast-path exception for security patches — the delay is a control, not a habit.
An SBOM per build artifact, centrally queryable: generated at build time, stored where "which services bundle package X version Y?" is one query — and that query is drilled quarterly so someone has actually run it.
Provenance and signature verification in CI: builds reject packages that fail signature or provenance checks, so a hijacked account can't silently swap the artifact.
A registry proxy in the path: internal builds resolve through a private mirror with an allowlist and quarantine, not straight from the public registry.
Advisory feeds matched against the SBOM automatically: vulnerability and compromise announcements page you with the affected-services list attached — the alarm and the inventory arrive together.
Update on purpose, on a cadence: a scheduled dependency-upgrade ritual (reviewed, tested, batched) replaces the nightly auto-bump — currency without the unlocked door.

The lesson

Pin dependencies with lockfiles so nothing in production changes without a commit. Put a cooling-off delay between a release being published and your systems adopting it — let the ecosystem be your canary. And maintain an SBOM (Software Bill of Materials — a machine-readable inventory of every component in your software) so "are we exposed?" is a query that takes minutes, not an archaeology project that takes days; back it with provenance scanning in CI.

Named concepts: supply-chain security, dependency pinning, SBOM, cooling-off windows.

Real-world echoes

event-stream (2018). The maintainer of a popular npm package handed it to a helpful volunteer, who later shipped a release containing malicious code targeting a specific bitcoin wallet. This drill's exact shape: a trust handoff upstream, auto-adopted downstream, discovered by the community rather than by any victim's own tooling.
Log4Shell (December 2021). A critical flaw in Log4j — a ubiquitous Java logging library — set off a global "are we exposed?" scramble. Organizations with a queryable software inventory answered in hours; many others spent weeks doing exactly the archaeology this drill describes. The SBOM lesson, at industry scale.

Transfer

If a package we use were compromised this afternoon, could we answer "are we exposed?" in minutes? Run the query today — if there is no query to run, that's the finding.

Make it a ticket: generate an SBOM for every production build artifact this sprint and stand up the "which services bundle package X version Y" query. Acceptance criterion — on-call answers it in under fifteen minutes, demonstrated in a drill.

Drill 59 — The Deprecation Email — the same asymmetric attention one layer up: there the unread signal was an email, here it's a release.
Drill 62 — The 99.9% That Didn't Matter — what happens when the dependency fails loudly instead, and you're just as unprepared.
Drill 45 — vendor breach: when the compromise is at the vendor itself rather than in the open-source supply chain — same "are we exposed?" question, same clock.