BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 66 — Data Gravity

Theme: Cost & FinOps · Format: 20-min drill · Difficulty: core

The story

The data science team adopts a best-in-class training platform on cloud B; the data warehouse lives on cloud A. Every training run copies terabytes across the internet between the two clouds, and per-gigabyte egress fees mean each run now costs more in data transfer than in GPU time. Nobody sees it for months, because the egress lands on cloud A's bill — the data side — while the team watches cloud B's, the compute side. Every dashboard the team owns looks great.

⏸ Stop here and ask the room: the spend is real and growing. Where is the money going, and whose dashboard shows it?

Run it — facilitator script

Timing (20 min): 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Draw the two clouds on the whiteboard before you start — the questions land harder when the room can see the boundary the money crosses.

"Why did the team never see the egress line?"

Listen for: billing follows account boundaries, not workflows — the egress charge accrues where the data exits, on cloud A, while the team's mental model, dashboards, and budget all live on cloud B where their platform runs. Strong answers generalize: any cost that lands on an account you don't read is invisible to you by construction, no matter how large.

Weak signal: "they weren't looking hard enough." They were looking at everything they owned — the failure is that ownership of the spend and ownership of the visibility were split across two bills.

"What should have been priced before adopting the cross-cloud platform?"

Listen for: the total cost of one training run, end to end — GPU (Graphics Processing Unit) hours plus per-gigabyte egress times data volume times run frequency — as a back-of-envelope at design review. Good answers note the comparison that was actually made: the two platforms' sticker prices, with the path between them priced at zero.

Weak signal: "you can't predict costs like that in advance." You can within 2x on one envelope: terabytes per run, runs per week, cents per gigabyte. The inputs were all known on day one.

"Who should own the whole cost of one training run, end to end?"

Listen for: the team that triggers the run owns every leg of it — both clouds, both bills, every transfer — because the run is the unit of decision. Strong answers propose the mechanism: a whole-path cost view per workflow, not per account.

Weak signal: "finance owns billing." Finance owns totals; nobody in finance can see that one team's training run spans two invoices. Ownership has to sit with whoever can change the architecture.

The reveal

Annualized, the transfer cost alone exceeds the price of running an equivalent training stack on cloud A entirely. The team paid a premium for the better tool and then paid more than the tool's worth to feed it.

Why this happens — the mechanics

Egress pricing is deliberately asymmetric: moving data into a cloud is free or nearly free, moving it out is billed per gigabyte — typically on the order of five to twelve cents. At gigabyte scale that's noise; at terabytes per training run, executed weekly, it's thousands of dollars per run — a recurring toll on an architectural decision, compounding silently with every execution. This is what "data gravity" means: datasets accumulate mass, and the cost of moving them grows until everything else — compute, tools, teams — must orbit the data rather than the other way around. A platform choice that ignores gravity doesn't fail; it just pays the toll forever.

The visibility failure is structural, not careless. Cloud bills are organized by account, and the egress meter runs on the side where the data exits — cloud A, the warehouse side, owned by a different team with a different budget. The data science team's dashboards genuinely were green: cloud B showed exactly the compute they expected. No dashboard existed whose job was the path. When spend and visibility are split across a boundary, the month-late, aggregated invoice on the far side is the only place the truth appears — and it appears as a line item in someone else's inbox.

This is also why the fix is architectural before it is financial. Alarms on the egress line catch the bleed; only moving compute to the data stops it. FinOps (Financial Operations — the practice of making engineering teams accountable for the unit economics of what they build) calls this whole-path cost visibility: the team that causes a flow of money must be able to see every leg of it, or the cheapest-looking design will keep winning reviews it should lose.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant TP as Training platform - cloud B
    participant WH as Warehouse - cloud A
    participant EG as Egress meter - cloud A
    participant TM as Team
    TP->>WH: Training run requests<br/>the dataset
    WH->>TP: Terabytes cross the<br/>internet between clouds
    WH->>EG: Per-GB egress fees land on<br/>cloud A's bill (silent)
    Note over EG: The data side - a bill<br/>nobody on the team reads
    TM->>TP: Watches cloud B's compute<br/>bill - all green
    Note over TP: Each run costs more in<br/>transfer than in GPU time<br/>(silent)
    EG->>TM: Months later - annualized<br/>egress tops a full<br/>cloud A training stack

Solution — where the guardrails go:

flowchart TD
    A["Cross-cloud tooling proposed"] --> B["Egress priced as a first-<br/>class architecture cost"]
    B --> C["Data-gravity rule: compute<br/>moves to the data"]
    C --> D["Training stack deployed on<br/>cloud A, beside the<br/>warehouse"]
    D --> E["Transfer cost drops to<br/>near zero"]
    B -.-> F["Whole-path cost view: the<br/>team sees both clouds'<br/>bills"]
    F -.-> G["Alarm fires when transfer<br/>cost exceeds compute cost<br/>per run"]

What good looks like

The lesson

Data has gravity: compute should move to the data, not data to the compute. Egress is a first-class architectural cost, weighed before adopting any cross-cloud tooling — not discovered on someone else's invoice afterward. And enforce split-bill visibility: the team that causes spend sees all of it, both sides of every transfer.

Named concepts: data gravity, egress economics, cross-cloud architecture cost, whole-path cost visibility.

Real-world echoes

Transfer

For our heaviest data workflow, what does one run cost end to end — including every transfer leg? If the answer spans two bills, who reads both?

Make it a ticket: Pick our single heaviest data workflow and compute its true per-run cost across every bill it touches; if any leg lands on an account the owning team doesn't read, file a ticket to join that line into the team's dashboard within a sprint.

Related