Theme 6 — Feedback Loops & Adversaries
The reflex this theme builds: our system changes the world it observes — and the world fights back.
Why this theme matters
A deployed model is not an observer of the world — it is a participant. Its decisions choose what data exists tomorrow: which applications ever receive labels, which customers stay long enough to be measured, which patterns get investigated at all. Train the next generation on that filtered record and your metrics quietly become self-referential — a model grading itself on the slice of reality it chose to look at.
And in fraud and risk domains the environment contains adversaries — brokers, fraud rings, even internal teams responding to incentives — who study the system and optimize against it, so yesterday's signal becomes tomorrow's coached input. The failure mode this theme trains against is believing your metrics describe reality, when they actually describe your system's interaction with reality. Abbreviations are expanded on first use; the full reference is glossary.md.
The drills
| # | Drill | One-line hook | Problem diagram | ~Difficulty |
|---|---|---|---|---|
| 35 | The Model That Ate Its Own Tail | Each retrain sharpens the corridor and blinds the edges — losses grow off-metric for eight months | sequenceDiagram (retrain loop) | core |
| 36 | The $999 Pattern | Amounts bunch at $950–999 — the ring binary-searched the threshold, no insider required | sequenceDiagram (probe loop) | intro |
| 37 | The Metric That Got Gamed | Bonus on cases closed per day; median review time hits 15 seconds while the dashboard celebrates | sequenceDiagram | intro |
| 38 | The A/B Test of Two Cities | Region A beats region B by 18% — demographics wearing a treatment's clothes | flowchart (parallel arms) | core |
| 39 | The Vendor's Silent Update | A bureau attribute is quietly redefined; the notice lands in an alias nobody reads | sequenceDiagram | core |
| 40 | The Prophecy That Fulfilled Itself | Flagged accounts get light-touch service and churn at triple the rate — prediction manufactures its outcome | sequenceDiagram (scoring loop) | advanced |
| 41 | The Coached Application | One broker channel scores textbook-clean and defaults at double the book | sequenceDiagram (coaching loop) | advanced |
Best practices — the feedback-loop & adversary checklist
What to actually do, distilled from drills 35–41:
- Give every model that filters its own future training data a permanent randomized exploration/audit budget — a score-blind sample investigated regardless of what the model says. It's not waste; it's the sensor budget that keeps the blind spot observable.
- Buy independent ground-truth streams from outside the decision loop — external chargeback feeds, network-level reports, bureau outcomes — and compute headline metrics on the audited sample, never on the model-filtered stream.
- Add an intervention holdout whenever predictions trigger treatments: a random slice of flagged cases treated normally. Evaluate prediction quality on the holdout and the treatment's effect against it — as two separate questions.
- Monitor calibration per segment and per channel (predicted vs. actual outcome), not just score distributions. A channel whose scores under-predict risk is how curated inputs and coached applications reveal themselves.
- Put a bunching/gaming detector under every visible threshold — alarm on mass accumulating just below the line — and review the distributions quarterly, because adversaries rerun their experiment continuously.
- Pair every quantity metric with an independently measured quality counterweight — sampled QA (quality assurance) by a team outside the incentive — for anything that pays a bonus or drives a review. Design the audit before the bonus.
- Make individual-level randomization the default experiment design, within every segment, with a pre-registered metric — and require an experiment design review before launch. Geography-split "tests" are anecdotes.
- Route vendor and external-input change feeds into the same change timeline as internal deploys, and monitor input distributions per source. A redefined attribute should correlate to an event on your timeline within a day, not surface as six weeks of vague drift.
- Assume any intermediary who profits from outcomes will learn the model — brokers, sales teams, ops staff on quotas. Monitor each as a segment, prefer features they can't curate (bank-verified income over stated income), and fix their incentives commercially, not just statistically.
- Re-ask "who adapts to this?" at every model or rule launch — adversaries, partners, internal teams, the customers themselves. If the answer is "nobody," you haven't looked hard enough.