BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 65 — The Recursive Weekend

Theme: Cost & FinOps · Format: 20-min drill · Difficulty: core

The story

Friday evening, a developer tests an event-driven pipeline in a sandbox account: a function that writes its output to the same bucket that triggers it. Every output becomes a new input — infinite recursion, massively parallel, scaling as fast as the platform allows. The sandbox carries the corporate payment method and has no budget cap and no alarms; the guardrails exist only on production accounts. The recursion runs all weekend, quietly and flawlessly. Monday morning: $30k.

⏸ Stop here and ask the room: the bill grew for sixty hours and nothing made a sound. What was missing, and where?

Run it — facilitator script

Timing (20 min): 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Tell the story at a Friday-evening pace, then hit the stop line and let the $30k land before the questions.

"Why did no alarm exist on the sandbox account?"

Listen for: guardrails follow perceived importance, not actual risk — production got the caps, reviews, and alarms because production is where "real" things happen, while sandboxes inherited the corporate payment method without inheriting any governance. Strong answers name the inversion: sandboxes are where untested designs run, so they are the highest-risk accounts for runaway spend, not the lowest.

Weak signal: "the developer should have been more careful." Park it and let the reveal answer: the developer followed every rule that existed. Careful is not a control.

"What in the pipeline design permits infinite self-triggering?"

Listen for: the trigger source and the write destination are the same bucket, and nothing distinguishes the function's own output from fresh input — no separate output bucket, no prefix or suffix filter on the event, no marker metadata the trigger could exclude. Bonus points for naming unbounded concurrency: the platform scales the mistake as aggressively as it would scale success.

Weak signal: "that's a platform bug." It isn't — the platform did exactly what it was configured to do. The bug is a design that cannot tell its own echo from a new voice.

"What signal would have caught this in the first hour, not the first invoice?"

Listen for: spend velocity — dollars per hour against a baseline near zero for a sandbox; invocation-rate anomalies; concurrency saturation; billing data polled hourly instead of read monthly. The key distinction is rate versus total: a monthly budget alert fires weeks too late even when it exists.

Weak signal: "a monthly budget alarm would have caught it." Ask when it would have fired — the answer is after the weekend already cost $30k, or a month later on the invoice.

The reveal

The developer followed every rule that existed — there simply were no rules for sandboxes. The org had hardened the front door and left the garage open: production had caps, reviews, and alarms, while the accounts where experiments actually live had none.

Why this happens — the mechanics

Event-driven recursion is the cost bug with the steepest curve, because the platform's best feature — elastic scaling — becomes the amplifier. A function that triggers itself doesn't fail; it succeeds, in parallel, at whatever concurrency the platform allows, and serverless platforms are engineered to remove exactly the friction that would slow a runaway loop down. Sixty hours of flawless execution is the system working as designed. Nothing in the operational plane distinguishes a viral success from a recursive mistake — only the spend curve does, and nobody was looking at the spend curve.

Sandboxes miss governance because governance follows blame paths, not risk paths. Production incidents have postmortems, so production accumulates controls; sandbox waste lands on an invoice a month later, anonymized into an aggregate line, so sandboxes accumulate nothing. Meanwhile the sandbox carries the same corporate payment method as production — the blast radius is identical, the guardrails are not. The bill's lag completes the failure: cloud billing data is typically hours delayed and the invoice a month behind, so the default detection mechanism for an ungoverned account is the slowest signal the platform emits.

The fix has two layers because the failure has two layers. Design guards (separate output buckets, event filters, concurrency caps) prevent the loop; account guards (budget caps, velocity alarms) bound the cost of every loop nobody predicted. Relying on either alone means betting that all future mistakes will look like past ones.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant FN as Function
    participant BK as Bucket trigger
    participant SB as Sandbox account
    participant INV as Invoice
    Note over FN: Friday evening deploy
    FN->>BK: Writes output to the same<br/>bucket that triggers it
    loop Each write triggers a run
        BK->>FN: New object event -<br/>fresh invocation (silent)
        FN->>BK: Output becomes input,<br/>massively parallel (silent)
    end
    Note over SB: Corporate payment method -<br/>no cap, no alarms (silent)
    SB->>INV: Meter spins all weekend<br/>(silent)
    Note over INV: Sixty hours, not a sound
    INV->>SB: Monday morning - a $30k line
    Note over FN,INV: Found on the invoice,<br/>not by an alarm

Solution — where the guardrails go:

flowchart TD
    A["Function deployed in sandbox"] --> B["Recursion guard: trigger<br/>ignores objects the<br/>function wrote"]
    B --> C["Fan-out capped by a<br/>concurrency limit"]
    A --> D["Budget alarm on every<br/>account, sandboxes<br/>included"]
    D -.-> E["Spend-velocity alert:<br/>dollars per hour, not per<br/>month"]
    E -.-> F["Page fires within the<br/>first hour"]
    F --> G["Developer disables the<br/>trigger Friday night"]
    D --> H["Hard cap halts spend even<br/>if nobody answers"]

What good looks like

The lesson

Budget alarms and hard caps belong on every account, including sandboxes — especially sandboxes, because that's where untested designs run. Event-driven designs need recursion and fan-out guards: a trigger must be able to distinguish its own output from fresh input. And alert on spend velocity — the rate of change, not just monthly totals — because a $30k weekend announces itself in the first hour.

Named concepts: budget guardrails everywhere, spend-velocity alerting, event-recursion guards, sandbox governance.

Real-world echoes

Transfer

List every account we pay for. Which ones have a budget cap and a velocity alarm — and which are "just sandboxes"? The second list is where our next $30k weekend lives.

Make it a ticket: Inventory all payer-linked accounts this week; for each one missing a budget cap or a dollars-per-hour alarm, file a ticket to add both, with the sandbox accounts explicitly at the top of the queue.

Related