BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 106 — The Egress Bill Nobody Forecast

Theme: Cost & FinOps · Format: 20-min drill · Difficulty: core

The story

A bank's risk-analytics team ships a new pipeline. Every night it assembles a firm-wide exposure report: it pulls terabytes of market data and transaction history from the data lake — which lives in the cloud's primary region — into a freshly provisioned analytics cluster the team stood up in a different region, closer to the quant desk that consumes the output. Roughly sixty terabytes cross the boundary every day. The design review was thorough: it sized the cluster, compared instance families, negotiated the compute budget. It priced the compute, and only the compute.

The pipeline runs beautifully. Reports land before the desk opens, the cluster autoscales cleanly, every dashboard the team owns is green. Nobody is watching the data-transfer line, because there is no data-transfer line — it was never in the plan. The inter-region egress meter ticks up quietly each night, a per-gigabyte fee accruing on a part of the invoice the team has no reason to open.

It runs like this for four months. Then the quarterly forecast slips: cloud spend for the division is running well over plan, and the variance is large enough that FP&A (Financial Planning & Analysis) escalates it up the chain. Someone in finance drills into the invoice, finds a line called "inter-region data transfer" that has quietly grown to dwarf everything around it, and sends a one-line email to the engineering director: what is this, and why is it bigger than our entire compute bill?

⏸ Stop here and ask the room: the pipeline never broke, and every engineering dashboard stayed green. So why did the biggest line on the bill arrive as a surprise from finance, four months late?

Run it — facilitator script

Timing (20 min): 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Sketch the two regions on the whiteboard with an arrow between them before you start — the room reasons better when it can see the boundary the money crosses every night.

"The design review was careful. What did it price, and what did it silently price at zero?"

Listen for: the review compared compute — instance families, cluster size, autoscaling — because that is what design reviews are trained to cost. The data path between the lake and the cluster was priced at zero by omission: nobody wrote a line for terabytes moved times cents per gigabyte times nightly frequency. Strong answers generalize: a cost that isn't a line item in the review is a cost the review approves without seeing.

Weak signal: "they should have read the pricing page." The pricing page doesn't help if transfer was never on the list of things to price. Push for the structural miss — the missing line item — not a diligence failure.

"Every engineering dashboard was green for four months. How is that possible while the bill explodes?"

Listen for: the dashboards watch health and compute — latency, error rate, cluster utilization — and the pipeline was genuinely healthy. Egress cost is not an engineering signal anyone had wired up; it lands on the billing system, a month late and aggregated into one line, on an invoice the team doesn't own. Spend and visibility were split: the meter that mattered had no dashboard pointed at it.

Weak signal: "add more monitoring." Ask what signal and at what threshold — a vague "watch costs" is exactly what was missing. The good answer names dollars-of-transfer-per-run as a first-class metric with an alarm.

"Finance found it. Why is finding it in the invoice the worst possible place to find it?"

Listen for: the invoice is a feedback loop with a month of lag and all the engineering context stripped out — by the time it surfaces, four months of decisions have compounded, and the person who can see it (finance) is not the person who can fix it (the team that chose the architecture). The disaster is already fully paid for. Good answers want the signal to fire in a day, on the team's own dashboard, in dollars-per-run.

Weak signal: "finance should watch the bill more closely." Finance watching totals is how it was caught at all — but a month late and a division up. The reflex has to move the signal to where the architecture decision was made.

The reveal

The "inter-region data transfer" line is running about fifty thousand dollars a month — against roughly twelve thousand for the compute cluster it feeds. The team paid a premium to place the cluster near the quant desk, and then paid more than four times the cluster's cost, every month, to haul the data to it. Annualized, the transfer alone is north of half a million dollars — more than it would cost to run the entire analytics stack in the lake's own region and move nothing at all. Nothing was broken the whole time; the pipeline was doing exactly what it was designed to do, at a price the design never named.

Why this happens — the mechanics

Egress is priced per gigabyte, and it is deliberately asymmetric: moving data into a cloud or a region is free or nearly free, moving it out is billed — inter-region typically on the order of one to two cents per gigabyte, cross-cloud or internet egress five to twelve. At the gigabyte scale of a design review's mental math, that's a rounding error. At sixty terabytes a night, executed every night, it's a toll charged on every single run, compounding silently with each execution. This is the trap: the unit price is small enough to feel ignorable, and the volume is large enough to make it the biggest line on the bill. Data has gravity — it is cheaper to move the compute to the data than the data to the compute — and a design that ignores gravity doesn't fail, it just pays the toll forever.

The visibility gap is structural, not careless. Engineering monitoring is built to detect things going wrong — errors, latency, saturation — and this pipeline was going right. Cost is the only signal an over-priced-but-healthy architecture emits, and cost arrives through the billing system: a month late, aggregated into "inter-region data transfer," on an invoice organized by account rather than by workflow. No dashboard existed whose job was the path. When the meter that matters has no owner watching it in real time, the invoice is the only place the truth appears — and it appears in finance's inbox, a month after the money is already spent, addressed to people who can see the number but not change the architecture that produced it.

That's why the fix is architectural before it is financial. Alarms on the egress line catch the bleed early; only colocating compute with data stops it. FinOps (Financial Operations — the practice of making engineering teams accountable for the unit economics of what they build) treats transfer cost as a first-class design input for exactly this reason: the cheapest-looking placement keeps winning reviews it should lose, because the leg that makes it expensive is the one nobody was asked to price.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant PL as Cluster<br/>region B
    participant DL as Data lake<br/>region A
    participant EG as Egress meter<br/>region A
    participant FIN as Finance
    PL->>DL: Nightly job requests<br/>terabytes of data
    DL-->>PL: Data crosses the<br/>region boundary
    DL->>EG: Per-GB transfer fees<br/>accrue on the bill
    Note over EG: Design priced compute<br/>only - transfer was<br/>never modelled
    Note over PL: Compute dashboards stay<br/>green every night (silent)
    FIN--xPL: Four months later -<br/>bill variance escalated

Solution — where the guardrails go:

flowchart TD
    A["New analytics pipeline<br/>proposed"] --> B["Transfer-cost line in the<br/>architecture review<br/>(TB x cents/GB x runs)"]
    B --> C["Data-locality rule -<br/>compute runs in-region"]
    C --> D["Pipeline deploys beside<br/>the data lake"]
    D --> E["Egress drops to near zero"]
    B -.-> F["Compress and cache at the<br/>boundary (Parquet + zstd,<br/>cache once)"]
    F -.-> G["Egress budget alarm - page<br/>when transfer tops 20% of<br/>compute per run"]
    E --> H["Transfer cost visible<br/>and bounded"]
    G --> H

What good looks like

The lesson

Egress is a first-class architecture cost: model it before you build, not after finance escalates it. Data has gravity — move the compute to the data, not terabytes across a boundary every night — and watch the transfer line with a spend-velocity alarm that fires in a day, at the dollar-per-run level, on the team's own dashboard. A design review that prices compute and prices the data path at zero has approved a cost it never saw.

Named concepts: egress economics, data locality, spend-velocity alarms, transfer-cost modeling.

Real-world echoes

Transfer

Which of our pipelines moves the most data across a region, cloud, or AZ boundary — and is anyone watching that transfer line in dollars, on a dashboard we own, or does it only appear on the invoice? If the biggest data path in our stack has no owner and no alarm, we have this drill running right now.

Make it a ticket: Pick our heaviest cross-boundary data flow; compute its true per-run cost including every transfer leg; if transfer exceeds 20% of that run's compute cost, file a ticket this sprint to either colocate the compute or add a per-pipeline egress budget alarm — whichever is faster to ship.

Related