BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 41 — The Coached Application

Theme: Feedback Loops & Adversaries · Format: 20-min drill · Difficulty: advanced

The story

Applications from one broker channel score suspiciously well — clean incomes, ideal tenure, textbook debt ratios. Default rates from that channel, though, run double the book. Investigation: brokers learned the approval cutoff and the features that matter, and now coach applicants — round the income up, reword the employer, time the application. Nothing is forged, exactly; everything is optimized.

⏸ Stop here and ask the room: your adversary is your own distribution partner. Now what?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

The stop question is deliberately uncomfortable — you can't block your own distribution channel the way you'd block a fraud ring. Keep the room in that tension.

"How do you detect coaching statistically?"

Listen for: calibration by channel — predicted vs. realized default rate per broker; a channel whose outcomes run above its scores is producing curated inputs. Secondary tells: bunching just above qualifying cutoffs (incomes rounding up to the magic number), too-clean distributions, suspiciously low variance where the population should be messy.

Weak signal: "compare score distributions by channel." Coached applications are built to look normal — the distribution can be perfect while every number in it is optimized. Only score-versus-outcome exposes fiction.

"Is this fraud, and does that framing help?"

Listen for: the ambiguity taken seriously — nothing is forged, so prosecution-shaped responses mostly stall. The productive framings are data quality (the features no longer mean what the model thinks they mean for this channel) and incentive design (the broker is paid for volume, not loan performance). The fix is commercial and architectural, not just forensic.

Weak signal: wanting the label settled before acting — "legal has to determine whether it's fraud first." While the framing debate runs, the channel keeps originating at double the default rate.

"What changes when the adversary profits from volume and knows your model?"

Listen for: this is a standing optimization against you, with inside knowledge and scale — assume every scoring release is learned within weeks. Responses: features the intermediary can't curate (bank-verified data), per-partner segment monitoring, and commissions aligned with loan performance — clawbacks, skin in the game.

Weak signal: secrecy as the whole plan — "don't tell brokers the cutoff." Brokers reverse-engineer the gradient from approve/decline feedback alone; incentives teach what documentation withholds.

The reveal

The tell was calibration by channel: scores from that channel systematically under-predicted risk — the model's inputs were curated, so its outputs were fiction for that segment.

Why this happens — the mechanics

An intermediary who submits hundreds of applications and observes every approve/decline decision is, functionally, running a machine learning (ML) workshop on your model. Each submission is a query; each decision is a label; volume hands the broker a training set of your scoring behavior that no confidentiality clause can claw back. They don't need the cutoff leaked — with enough feedback they estimate the gradient: which features matter, in which direction, and by how much. Commission-per-funded-loan then makes acting on that knowledge not a temptation but the business model.

The damage lands in the feature semantics. The model learned the relationship between default and stated income, tenure, and debt ratio from a world where those fields were reported naively. Coaching doesn't push inputs out of distribution — it's engineered to do the opposite. It rewrites the conditional relationship for one channel: the same textbook value now arrives via "round it up" rather than via actually earning it, and carries none of the safety it used to signal. Every input-level monitor stays green because the inputs are, by construction, beautiful.

That's why the tell was calibration, and only calibration. Score distributions, approval rates, and data-quality checks all pass; the fiction is visible only where predictions meet outcomes — a channel whose realized defaults run double what its scores promised. And outcomes mature slowly, which is the adversary's margin: months of commissioned volume flow before enough score-versus-outcome data accumulates for anyone who thinks to slice it by channel.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant BR as Brokers
    participant AP as Applicants
    participant SM as Scoring model
    participant PF as Portfolio
    loop Every coached application
        BR->>BR: Learn the cutoff and the<br/>features that matter (silent)
        BR->>AP: Coaching - round the income,<br/>reword the employer (silent)
        AP->>SM: Textbook-clean application
        SM->>PF: High score - approved
    end
    Note over BR: Broker volume and<br/>commissions grow
    Note over SM: Curated inputs make scores<br/>fiction for that segment (silent)
    PF->>SM: Channel default rate runs<br/>double the book
    Note over SM,PF: Found in portfolio outcomes -<br/>months after approval

Solution — where the guardrails go:

flowchart TD
    A["Broker channel submits<br/>applications"] --> B["Unfalsifiable features:<br/>bank-verified income over<br/>stated income"]
    B --> C["Model scores on data<br/>brokers cannot curate"]
    A --> M["Per-channel calibration<br/>monitoring: score vs<br/>outcome"]
    M -.-> AL["Alarm: channel<br/>systematically under-<br/>predicts risk"]
    AL --> R["Commercial fix: broker<br/>incentives and contracts<br/>revisited"]

What good looks like

The lesson

Anyone whose income depends on your model's output will learn its behavior — brokers, sales teams, ops staff hitting quotas — insiders don't need to leak thresholds when incentives teach the gradient. Monitor calibration per channel/partner (score-vs-outcome, not just score distributions), use features the intermediary can't curate (bank-verified income beats stated income), and treat channel-level drift as an incentive problem to fix commercially, not just a modeling problem.

Named concepts: adversarial intermediaries, per-segment calibration monitoring, unfalsifiable features, incentive alignment.

Real-world echoes

Transfer

Who besides the applicant touches the data our models score? For each intermediary: what do they gain from a favorable score, and would our monitoring notice if they optimized for it?

Make it a ticket: Map every intermediary touching scored data — brokers, sales, ops — and ship a per-channel calibration report for each. Where the gain from a favorable score is high and monitoring is absent, file an incentive-review ticket.

Related