Drill 105 — The Idle GPU Fleet

Theme: Cost & FinOps · Format: 20-min drill · Difficulty: core

The story

A bank's model-platform team runs a training cluster for its fraud and credit-risk models: sixty-four GPUs (Graphics Processing Units) across eight nodes, bought on a one-year reservation because reserved capacity is cheaper than on-demand and the models are business-critical. The team sized the fleet for the worst case they could imagine — the quarterly model refresh, when every risk model retrains at once ahead of the regulatory stress test, and the cluster briefly runs near ninety percent. Provisioning for the peak was the responsible call. Reserving it was the frugal call.

Then the peak passes. For the ten or eleven weeks between refreshes, the cluster does almost nothing — a notebook here, an ad-hoc experiment there, one nightly job that finishes by 2 a.m. The nodes stay up, because tearing a reserved cluster down and standing it back up is friction nobody has time for, and because "we'll need it for the next refresh anyway." The GPUs are online, healthy, patched, and — measured over any given week — about eight percent busy. Ninety-two percent of the silicon the bank is paying full price for is spinning idle.

Nothing is broken. Every dashboard is green: nodes healthy, jobs succeeding, latency fine. There is no utilization number on any dashboard the team looks at, and no line in the reservation contract that cares whether a GPU did work or waited. The cost lands on the platform account as one flat monthly reservation charge — the same number whether the fleet ran hot or ran empty — so it never moves, and a number that never moves is a number nobody watches. When the finance partner finally asks "what are we getting for the GPU line," it takes the team a day to even produce a utilization figure, because no one owns it.

⏸ Stop here and ask the room: the cluster is healthy, the reservation was the cheap option, and provisioning for the peak was correct. So why is this a six-figure mistake every single month?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Read the story aloud, pause on the stop line, and make the room name the number they'd need before they can even argue about it.

What is the SLI (Service Level Indicator) for this fleet, and why does its absence matter more than any single decision here?

• Listen for: utilization — GPU-busy-hours divided by GPU-hours paid for — as the missing indicator, and the realization that every individual choice (reserve for savings, size for the peak, keep nodes warm) was locally reasonable; the failure is that no signal measures the aggregate consequence. Strong answers note that a flat reservation charge is the worst possible cost signal: it's identical at 8% and 90%, so it carries zero information about waste. The utilization number is the only thing that distinguishes "prudent capacity" from "expensive idle."
• Weak signal: "they shouldn't have reserved" or "they over-provisioned." Push back — reserving saved money versus on-demand, and sizing for the stress test was correct. Ask what they would measure to know when prudent headroom has curdled into waste.

Who should own the utilization number, and what does "owning" it concretely require?

• Listen for: a named team — the model-platform team that runs the cluster — with the number on their own dashboard next to latency and error rate, a weekly review, and a target they're accountable to (e.g. keep the reserved floor above 60% utilized). The best answers connect ownership to attribution: you can't hold anyone to a utilization target until each GPU-hour is tagged to a consuming model team via showback.
• Weak signal: "finance owns cost." Finance sees a flat invoice a month late; they cannot see 8% utilization inside a reservation, and they can't right-size a cluster. Ownership has to sit with the team that can actually change the capacity.

The quarterly peak is real. How do you serve it without paying for it fifty-two weeks a year?

• Listen for: decoupling the baseline from the burst — reserve only the small steady-state floor, autoscale the rest, and run the interruptible refresh jobs on spot/preemptible capacity (which is cheap precisely because it can be reclaimed, and batch training checkpoints so it tolerates reclamation). The insight to reward: peak-shaped demand should be met with peak-shaped, elastic spend, not a permanent reservation sized for the peak.
• Weak signal: "keep it, we need the peak." That is the exact reasoning that kept 92% of the fleet idle. The peak lasting days a quarter does not justify paying for it all quarter.

The reveal

The reservation runs about a hundred and sixty-five thousand dollars a month. At 8% utilization, roughly a hundred and fifty thousand of that — every month, quietly, for as long as the reservation runs — buys idle silicon. Reworked: reserve a small steady-state floor, autoscale interactive nodes to zero when the queues drain, and move the quarterly refresh onto spot capacity, and the fleet delivers the same peak throughput for a fraction of the bill. The stress test still runs at ninety percent when it needs to. The difference is that the bank stops paying peak prices for the eleven weeks a quarter the peak isn't happening. Nothing about the models changed — only the shape of the spend.

Why this happens — the mechanics

Reserved capacity converts a variable cost into a fixed one, and a fixed cost is a flat line on the bill. That flatness is the trap: latency has a distribution, error rate has a distribution, and both wander enough that a human notices when they drift — but a reservation charge is a constant, and constants don't trigger the pattern-matching that operations teams run on their graphs. The signal that would move — utilization — is never computed, because the provider bills you for the reservation, not for the work, and the reservation is what shows up in cost tooling. You have to go out of your way to divide busy-hours by paid-hours; nobody's default dashboard does it for you.

Underneath that is the same ownership seam that runs through every drill in this theme: a cost that no one owns is a cost no one optimizes. Provisioning for the peak was a capacity decision made once, correctly, at reservation time — and then never revisited, because revisiting it is nobody's job and the flat bill gives no one a reason to. FinOps (Financial Operations — the practice of making engineering teams accountable for the unit economics of what they build) calls this the difference between a purchase and a standing commitment: the purchase felt frugal, but with no utilization SLI and no named owner, the frugal purchase became a standing subsidy for idle hardware.

And the deepest asymmetry is the theme's signature one: an idle GPU is doing everything right. Perfect uptime, zero errors, healthy temperature, jobs succeeding. Operational monitoring is built to catch things going wrong, and waste is not a failure — it is the system working flawlessly at the wrong scale. Utilization is the only signal an over-provisioned fleet emits, and if that signal isn't measured, alarmed, and owned, the fleet can waste a hundred and fifty thousand dollars a month with a completely green dashboard.

Diagrams

Problem — how the failure unfolds (slow burn):

timeline
    title A fleet that mostly waits
    section Provision
        Reserve 64 GPUs : Sized for the quarterly peak : One-year commitment, cheaper than on-demand
    section Steady state
        Peak passes : Cluster stays warm, never scales down (silent)
    section Weeks of drift
        Runs at 8 percent utilization : Idle silicon at full price (silent)
        No utilization SLI, no owner : Flat bill never moves (silent)
    section Quarter close
        Finance asks what we get : Six figures a month buys idle GPUs

Solution — where the guardrails go:

flowchart TD
    A["Reserved GPU fleet<br/>runs at 8% utilized"] --> B["Utilization SLI:<br/>busy GPU-hours over<br/>hours paid for"]
    B -.-> AL["Alarm: util under 40%<br/>for 7 days pages<br/>the named owner"]
    A --> C["Autoscaler drains<br/>idle nodes to zero"]
    A --> D["Quarterly refresh runs<br/>on spot capacity"]
    A --> E["Showback bills $/GPU-hour<br/>to the model team"]
    E -.-> AL
    AL --> H["Owner right-sizes<br/>the reserved floor"]

What good looks like

• A utilization SLI on every expensive fleet: fleet-wide GPU utilization (busy-hours ÷ paid-for-hours, pulled from the DCGM/nvidia-smi exporter into your metrics stack), computed as a 7-day rolling average and plotted next to latency and error rate. Warn at under 50%, page the owning team at under 40% sustained for seven days. A reservation with no utilization graph is a blind spot by construction.
• Scale-to-zero autoscaling for the elastic tier: the interactive/notebook pool scales down nodes after 15 minutes idle and all the way to zero when the queue is empty, so off-peak hours cost $0 for that tier instead of full reservation price. Reserve only the steady-state floor; make everything above it elastic.
• Spot/preemptible capacity for batch training: the quarterly refresh and any checkpointing, restart-tolerant job runs on spot/preemptible instances (typically 60–90% cheaper than on-demand), with checkpoints every N minutes so a reclamation costs minutes, not a rerun. Reserve for what must be always-on; rent the burst cheaply.
• Right-size the reserved floor to the baseline, not the peak: commit only to the utilization floor that survives between refreshes; meet the peak with elastic + spot. Re-derive the floor quarterly from the actual utilization histogram, and never renew a reservation without checking last term's utilization.
• Showback, graduating to chargeback: tag every GPU-hour to a consuming model team and mail each team its own $/GPU-hour and utilization weekly, so a cost regression is caught by the people who caused it — with the same reflex they'd catch a latency regression. A number with an owner gets optimized; an orphan number funds idle silicon.
• A utilization target in the platform's SLOs (Service Level Objectives): e.g. reserved-fleet utilization ≥ 60%, reviewed weekly, with the trend alarmed. Make "our GPUs are 8% busy" a paging condition, not a fact someone stumbles on at quarter close.

The lesson

A flat reservation charge is the least informative signal on your bill — identical whether the hardware worked or waited — so buying reserved capacity without a utilization SLI and a named owner turns a frugal purchase into a standing subsidy for idle silicon. Measure utilization, alarm it, attribute it to a team, and meet peak-shaped demand with elastic and spot capacity instead of a permanent reservation sized for the peak.

Named concepts: utilization SLI, scale-to-zero autoscaling, spot/preemptible capacity, showback and chargeback.

Real-world echoes

• No single named headline here — and that absence is the whole point. Idle GPUs never make the news, because nothing crashes and no customer is hurt; the fleet runs beautifully while the meter spins. It is an open industry secret that reserved AI/ML fleets routinely sit far below capacity — teams that instrument utilization for the first time regularly discover single-digit or low-double-digit averages under a perfectly green dashboard. Waste doesn't alarm, so it doesn't get reported; it just gets paid.

Transfer

Pull the utilization number for your most expensive reserved fleet — GPU, or any always-on capacity — right now. If it takes more than five minutes to produce, that's the finding: you're paying a flat rate for a fleet whose utilization nobody measures or owns. What is it, and who is accountable to a target?

Make it a ticket: Stand up a fleet-utilization SLI (7-day rolling busy-hours ÷ paid-hours) on your priciest reserved capacity, put it on the platform dashboard next to latency, alarm it at under 40% for seven days to a named owner, and re-derive the reserved floor from the actual utilization histogram before the next renewal.

Related

• Drill 64 — The POC That Ran for a Year (Theme 11, Cost & FinOps): the passive twin one step earlier — an untagged cluster with no owner idling at full price; here the cluster has a job, it's just 92% idle between peaks.
• Drill 68 — The Unit Economics Nobody Owned (Theme 11, Cost & FinOps): the same ownership seam one level up — not a fleet without a utilization owner, a cost-per-transaction ratio without one.
• Drill 24 — Autoscaler That Fed the Fire (Theme 4, Scale & Cloud): the elastic-capacity control from the opposite failure — scaling as the mechanism, misapplied there, exactly the tool missing here.