Theme 11 — Cost & FinOps
The reflex this theme builds: every dollar of cloud spend is an engineering decision someone made — or forgot to unmake.
Why this theme matters
Cloud and AI turned cost from a purchasing decision into an engineering output. There is no procurement gate between an engineer and a $9k/month GPU (Graphics Processing Unit) cluster; architecture choices, retry policies, data placement, and forgotten resources all compile directly into the invoice. A recursive trigger is a cost bug. A cross-cloud data path is a cost bug. A three-year commitment signed against a two-year architecture is a cost bug. None of them look like bugs, because nothing is broken — the systems run beautifully while the meter spins.
FinOps (Financial Operations — the practice of making engineering teams accountable for the unit economics of what they build) exists because the bill arrives a month after the decisions that caused it, anonymized and aggregated: "compute — other" tells you nothing about which experiment, which team, which forgotten Friday deploy. Every drill in this theme shares one root cause: spend that no one owned. The fix is never "spend less" — it's making cost visible, attributed, and alarmed at the same speed as latency and errors. Abbreviations are expanded on first use; the full reference is glossary.md.
The drills
| # | Drill | One-line hook | Problem diagram | ~Difficulty |
|---|---|---|---|---|
| 64 | The POC That Ran for a Year | An untagged GPU cluster idles at $9k a month for eleven months after its project dies — $99k for zero work. | timeline | intro |
| 65 | The Recursive Weekend | A function that triggers itself runs all weekend in an alarm-free sandbox — Monday's invoice reads $30k. | sequenceDiagram | core |
| 66 | Data Gravity | Every training run pays more to move data between clouds than to compute on it — on a bill the team never reads. | sequenceDiagram | core |
| 67 | The Commitment Mismatch | A three-year reservation meets a month-eight architecture migration — the 40% discount becomes a double-pay penalty. | timeline | advanced |
| 68 | The Unit Economics Nobody Owned | Transactions grow 3x, the bill grows 5x, and the board's one-line question takes three weeks to answer. | sequenceDiagram | advanced |
Best practices — the FinOps checklist
What to actually do, distilled from drills 64–68:
- Enforce mandatory tags at creation time — owner, project, expiry — via policy, so untaggable means undeployable. A resource nobody can attribute is a resource nobody will ever turn off.
- Put a TTL (Time To Live — an expiry duration) on every experimental resource, with auto-stop and a deliberate renewal step. The default fate of an experiment must be to end; living forever should require action.
- Run a monthly orphan hunt and treat "unknown" spend as a bug — filed, owned, and fixed — not a rounding error. The orphans usually total more than you think; in drill 64 they beat the flagship's bill.
- Set budget alarms and hard caps on every account, sandboxes included — especially sandboxes, because that's where untested designs run against a corporate payment method.
- Alert on spend velocity — dollars per hour, rate of change — not just monthly totals. A $30k weekend announces itself in the first hour; the invoice announces it a month late.
- Build recursion and fan-out guards into event-driven designs: a trigger must distinguish the function's own output from fresh input, and concurrency limits must cap the blast radius of a mistake.
- Price egress before adopting any cross-cloud or cross-region architecture. Data has gravity — move compute to the data, not terabytes to the compute — and per-gigabyte fees compound on every run.
- Give every team whole-path cost visibility: the team that causes spend sees all of it, both sides of every transfer, even when it lands on another account's bill.
- Optimize first, then commit: right-size, clean up, and re-architect before signing reserved capacity, because committing first locks in the waste at a discount.
- Commit only to your stable baseline — the floor that survives architecture churn — and keep the volatile layer on-demand or in flexible commitments. Review commitment utilization quarterly, with an alarm on any drop.
- Make unit economics a first-class KPI (Key Performance Indicator) — cost per transaction, decision, or document — with a named owner, a dashboard next to the revenue dashboard, and a trend alarm. Totals hide efficiency decay under growth.
- Show engineers the cost of their own services weekly (showback, graduating to chargeback — actually billing teams internally), so a cost regression is caught with the same reflexes as a latency regression: by the people who shipped it, days after they shipped it.
All themes: ../README.md · Abbreviation reference: ../glossary.md