BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 64 — The POC That Ran for a Year

Theme: Cost & FinOps · Format: 20-min drill · Difficulty: intro

The story

A routine cost review finds a GPU (Graphics Processing Unit) cluster — eight accelerators, $9k a month — spun up for a proof-of-concept whose project was cancelled eleven months ago. It carried no tags, so on the bill it appeared only as part of "compute — other," a line nobody questions. The engineer who created it left in the spring; nobody remaining knew it existed. For a year it processed exactly nothing, at full price, with perfect uptime.

⏸ Stop here and ask the room: $99k for zero work, and no name attached to any of it. How does a resource become invisible?

Run it — facilitator script

Timing (20 min): 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Read the story aloud, pause on the stop line, and let the room sit with the number before opening the questions.

"How did a $9k/month resource stay unnoticed for eleven months?"

Listen for: the causal chain of missing attribution — no tags means no owner, no owner means no reviewer, and the billing system's only move for unowned spend is to aggregate it into "compute — other." Strong answers add the human layer: the one person who knew left, and institutional memory left with them; and the number layer: $9k a month is small against a large aggregate line, so relative size hid an absolute disaster.

Weak signal: "someone should have read the bill more carefully." Push back — a human staring at an aggregated invoice line cannot see an untagged cluster inside it. Ask what structure, not what diligence, would have surfaced it.

"What would it take, today, to reconstruct who owns each line of our bill?"

Listen for: honest archaeology estimates — tag coverage percentages, digging through audit logs and provisioning history, interviewing teams — landing on "weeks, and some of it never." The best answers flip the question: ownership has to be captured at creation time, because it cannot be reliably reconstructed afterward.

Weak signal: "we'd just look at the tags," from a team that has never enforced tagging. Ask them to estimate their actual tag coverage on the spot — then suggest checking after the session.

"What else is hiding inside 'compute — other'?"

Listen for: concrete candidates — dev environments, forgotten notebook servers, load-test rigs, storage for dead experiments, old snapshots, idle load balancers. The insight to reward: this is a category of spend, not a one-off incident, and the reveal confirms it.

Weak signal: "probably not much." That belief is exactly what let this cluster run for a year — hold the thought and let the reveal answer it.

The reveal

The same audit finds a dozen smaller orphans — dev environments, forgotten notebook servers, storage for an abandoned experiment — and together they total more than the monthly bill of the company's flagship service. The flagship had an owner watching its cost; the orphans had no one.

Why this happens — the mechanics

The cloud bill is a feedback loop with a month of lag and all the identity stripped out. By the time the invoice arrives, the decisions that produced it are four to eight weeks old, and it arrives aggregated: "compute — other" is not a lie, it's a bucket — the billing system's honest answer for spend that carries no attribution metadata. A resource with no tags cannot be assigned to a team, a project, or a cost center, so it flows into the residual line, and residual lines are precisely the ones nobody is paid to question.

Attribution only exists if it is captured at creation. The provider's billing data can tell you what ran — instance type, region, hours — but never why or for whom; that context lives only in tags or in human memory. Human memory churns: the engineer left in the spring, and with them went the last pointer to the cluster. This is why FinOps (Financial Operations — the practice of making engineering teams accountable for the unit economics of what they build) treats tagging as a control, not a courtesy — reconstruction after the fact is archaeology, and archaeology on a moving fleet mostly fails.

The deeper asymmetry: operational monitoring is built to detect things going wrong, and an idle cluster is doing everything right — perfect uptime, zero errors, healthy metrics. Cost is the only signal an orphan emits, and if cost isn't watched per-owner with an expiry attached, the orphan emits nothing anyone hears. Failures alarm; waste doesn't. That is the gap a TTL (Time To Live — an expiry duration) closes: it converts "someone should notice this" into "this stops unless someone acts."

Diagrams

Problem — how the failure unfolds:

timeline
    title A year of paying for nothing
    section Month 0
        POC kickoff : 8-GPU cluster spun up : No tags - no owner, no expiry
    section Month 1
        Project cancelled : Cluster keeps running (silent)
    section Spring
        Engineer leaves : Nobody knows it exists (silent)
    section Months 2-11
        Idles at $9k a month : Hidden in compute - other
    section Month 12
        Cost review finds it : $99k for zero work

Solution — where the guardrails go:

flowchart TD
    A["Engineer requests a GPU<br/>cluster for a POC"] --> B["Creation gate: owner,<br/>project, expiry tags or no<br/>deploy"]
    B --> C["Cluster deploys tagged,<br/>with a 90-day TTL"]
    C --> D["TTL reached: auto-stop<br/>unless the owner renews"]
    D --> E["Project cancelled — no<br/>renewal, cluster stops<br/>itself"]
    B -.-> F["Monthly orphan hunt sweeps<br/>for unowned spend"]
    F -.-> G["Any unknown dollar filed<br/>as a bug, not a rounding<br/>error"]
    E --> H["Spend ends the month the<br/>project does"]
    G --> H

What good looks like

The lesson

Enforce mandatory tagging — owner, project, expiry — at creation time, by policy: untaggable means undeployable. Put TTLs (Time To Live — an expiry duration) on every experimental resource, with auto-stop and a deliberate renewal step, so the default fate of an experiment is to end. And run a monthly orphan hunt where "unknown" spend is treated as a bug to be filed and fixed, not a rounding error to be shrugged at.

Named concepts: resource tagging, experiment TTLs, orphaned-resource reaping, cost allocation.

Real-world echoes

Transfer

Pull our bill's "other" category right now: what fraction of our spend can we attach a living owner's name to? The remainder is the orphan backlog.

Make it a ticket: Export last month's bill grouped by tag; file one ticket per untagged line above $100/month, each assigned an investigator and a two-week deadline to attach either an owner or a termination date.

Related