Drill 44 — The Oracle at the API

Theme: Security & Trust Boundaries · Format: 20-min drill · Difficulty: advanced

The story

The fraud-score API serves partners: submit an application, receive a score. Traffic analysis flags one partner key with strange behavior — thousands of submissions that are near-duplicates, each varying one field at a time: income up $5k, tenure plus one year, different ZIP. Methodical. Patient.

⏸ Stop here and ask the room: what is this caller building, and what will they do with it?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

What can you learn about a model by querying it systematically?

• Listen for: the decision boundary itself — which inputs flip the output, which features move the score and by how much, where the thresholds sit. The strongest answers notice that queries compose: any single response is harmless, but ten thousand of them are a functional copy of the model.
• Weak signal: "just the score for that one application" — treating each response as independent. That's the exact blind spot the attacker is exploiting; the value is in the differences between responses, not in any one of them.

Who benefits from a copy of the decision boundary?

• Listen for: fraud rings buying a pre-optimized playbook of what passes; brokers packaging and reselling the map; competitors extracting the model itself without the training cost. Bonus points for pricing it: the map is worth more than any single fraudulent approval, because it industrializes all of them.
• Weak signal: the room only imagines one applicant gaming their own file — a retail threat model for a wholesale attack.

What legitimate use looks similar, and how do you tell them apart?

• Listen for: partners batch-testing integrations, what-if tooling, and quality assurance runs all produce lookalike traffic. Discrimination lives in structure, not volume — single-field deltas sweeping a range are a fingerprint; so is systematic coverage of the input space — plus a sanctioned sandbox so legitimate testing has somewhere else to be.
• Weak signal: "block anyone with high volume" — which punishes your biggest legitimate partner first and leaves a patient, rate-limited attacker untouched.

The reveal

Someone is mapping the model — using the API as an oracle to learn exactly which application profiles pass. The map gets sold to fraud rings, who arrive pre-optimized (Drill 41's brokers, industrialized).

Why this happens — the mechanics

A scoring API (Application Programming Interface) is an oracle in the classic security sense — a black box that answers questions truthfully, on demand, at a price you set. Each query costs the attacker cents and returns genuine signal about the model; the boundary map assembled from ten thousand queries is an asset worth orders of magnitude more than the API fees that built it. The economics run one way only: information flows out with every response, and the attacker's cost is set by your rate card, not by the value of what they're learning.

Output precision sets the exchange rate. A raw score is a measurement — vary one field, watch the score move, and you've learned the direction and rough weight of that feature. A bare decision leaks only a threshold crossing, so extracting the same map requires bracketing every boundary from both sides — often an order of magnitude more queries, pushed past the point of detection or profitability. Every decimal place you return is a subsidy to the attack.

Detection is hard because each query is individually legitimate: a plausible application, a valid partner key, an unremarkable rate. The signature exists only in the relationships between queries — near-duplication, single-field deltas, methodical sweeps of one dimension at a time. Transaction-level monitoring cannot see it by design; you need per-key behavioral analytics that treat the query stream itself as the object under study, the same way you treat a card's transaction stream.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant PK as Partner key
    participant API as Scoring API
    participant MAP as Boundary map
    participant FR as Fraud rings
    loop Vary one field
        PK->>API: Near-duplicate application -<br/>income up 5k, ZIP changed
        API-->>PK: Raw fraud score returned
        PK->>MAP: Boundary point recorded<br/>(silent)
    end
    Note over MAP: Decision boundary rebuilt<br/>query by query (silent)
    MAP->>FR: Map of what passes sold
    FR->>API: Rings arrive pre-optimized<br/>to score clean
    Note over API,FR: Probing understood only<br/>after the losses land

Solution — where the guardrails go:

flowchart TD
    A["Partner key submits<br/>applications"] --> B["Per-key behavioral monitoring"]
    B --> C["Perturbation-pattern<br/>detector: near-duplicates,<br/>one field varied"]
    C -.-> AL["Anomaly alert on the<br/>probing key"]
    A --> D["Rate limits cap query<br/>volume per key"]
    A --> E["Output minimized: decision<br/>only, no raw score"]
    E --> F["Each response teaches the<br/>adversary far less"]
    AL --> G["Key suspended —<br/>contractual consequences<br/>invoked"]

What good looks like

• Per-key behavioral analytics — near-duplicate rate, single-field-delta rate, and input-space coverage tracked per partner key, alerting on the fingerprint rather than the volume.
• Similarity fingerprints on submissions — MinHash/SimHash-class hashing so a family of one-field perturbations is visible as a family, not as thousands of unrelated applications.
• Output minimization by tier — decision only by default; banded scores under contract where the business case genuinely demands them; raw scores never cross the boundary.
• Rate limits sized to plausible business — per-key budgets derived from the partner's actual volume, throttling first and asking questions second when a key runs 10x its baseline.
• A sanctioned sandbox — a synthetic-scoring environment for partner integration testing, so legitimate lookalike traffic has an approved place to exist and production probing loses its cover story.
• Contractual teeth — systematic probing named as breach in the partner agreement, with a rehearsed suspend-investigate-notify path rather than an improvised one.
• Red-team the detector quarterly — replay this drill's traffic with a test key; the alert should fire near query 100, not query 10,000, and the replay date goes in the runbook.

The lesson

A scoring API is a teaching machine for adversaries: every response is a lesson. Defenses: per-key behavioral monitoring (perturbation patterns are a distinctive fingerprint), rate limits, return the coarsest output the business allows (decision instead of raw score), and contractual + technical consequences for probing. Query patterns deserve the same anomaly detection you point at transactions.

Named concepts: model extraction, oracle attacks, API abuse monitoring, output minimization.

Real-world echoes

The clean public examples here live in the research literature rather than in breach disclosures — Tramèr et al., "Stealing Machine Learning Models via Prediction APIs" (USENIX Security, 2016), demonstrated practical extraction of live commercial models, query by patient query. The absence of headline incidents is itself the lesson: a successful extraction produces no outage, no letters, and no disclosure trigger. Nobody is obliged to tell you your boundary was mapped — you find out when the pre-optimized applications arrive.

Transfer

For each externally reachable model endpoint: what could a patient adversary reconstruct in 10,000 queries, and would we notice query 100?

Make it a ticket: For one externally reachable scoring endpoint, add a per-key near-duplicate-rate metric and set its alert threshold — the deliverable is the alert firing on a red-team replay of today's story.

Related

• Drill 46 — The Poisoned Feedback — the same patient adversary, writing into your model instead of reading out of it.
• Drill 45 — The Vendor's Bad Week — the other direction your boundary extends without you drawing it: outbound to vendors instead of inbound through partner keys.
• Drill 41 — The Coached Application, theme 6 — the brokers this map gets sold to; drill 44 is that marketplace, industrialized to wholesale scale.