Drill 46 — The Poisoned Feedback

Theme: Security & Trust Boundaries · Format: 20-min drill · Difficulty: advanced

The story

The fraud model retrains monthly on labeled outcomes, including a valuable label stream: customer dispute resolutions ("charge confirmed legitimate"). A fraud ring discovers this and runs a long game — small transactions, then dispute-and-confirm cycles through the self-service flow, methodically teaching the system that their pattern is legitimate. By retraining number three, their signature scores clean.

⏸ Stop here and ask the room: the training pipeline worked perfectly. What got in?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Which of your label sources can be written to by outsiders?

• Listen for: an actual enumeration — dispute resolutions, self-service confirmations, chargeback responses, crowd feedback, even support notes transcribing what a caller claimed. The strongest answers reach the uncomfortable realization that "ground truth" often means "whatever the channel recorded," and outsiders type into those channels every day.
• Weak signal: "our labels come from our own systems" — true and irrelevant. The system is yours; the hands on the keyboard, for several of those streams, are not.

How would you detect a coordinated labeling campaign?

• Listen for: anomaly detection on the label streams themselves — dispute-confirmation bursts, linkage across the accounts writing them (shared devices, addresses, payment instruments), label-flip rates by cohort, and drift between trusted and untrusted label populations at retrain time.
• Weak signal: "the model's metrics would drop" — they won't. Poisoning is built to keep aggregate metrics green while carving one narrow exception; waiting for accuracy to sag is waiting for the cash-out.

Should all labels weigh the same?

• Listen for: a clear no — weight by source trustworthiness, authenticate provenance, and anchor every retrain with a trusted core of internal investigations and audited samples. Bonus points for proposing quarantine: externally writable labels wait a cycle before they can teach.
• Weak signal: worry that down-weighting disputes "throws away data" — without pricing what full weight invites. Data an adversary can author cheaply isn't free signal; it's a purchase order for your decision boundary.

The reveal

Discovered only when the ring cashes out at scale — the model waves through in one night what it was patiently taught to ignore over three months.

Why this happens — the mechanics

Label poisoning works because the feedback channel does exactly its job — recording outcomes. The system cannot distinguish "a customer confirmed this charge" from "an attacker confirmed their own charge"; both arrive through the same authenticated self-service flow, formatted identically, at plausible volume. The pipeline validated schema, freshness, and volume — everything except who was allowed to be a teacher, which was never a field anyone checked.

The economics favor the patient attacker. Each small transaction plus dispute cycle costs the ring almost nothing — a few dollars and a web form — and every poisoned label is an investment that pays out at cash-out time. The monthly retrain is the compounding mechanism: each cycle bakes the poisoned labels in deeper and moves the boundary a little further, and because the shift is gradual and self-consistent, drift monitors read it as the world changing rather than an adversary teaching.

Aggregate metrics stay green because poisoning is surgical. The ring isn't degrading the model — it's carving a narrow exception for one signature while performance on everything else stays untouched. Accuracy dashboards measure the average; the attack lives in a corner the average never visits, sized deliberately below the resolution of every summary statistic you watch.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant FR as Fraud ring
    participant DF as Dispute flow
    participant LB as Training labels
    participant MD as Model
    loop Dispute and confirm
        FR->>DF: Small transaction disputed<br/>via self-service
        DF->>LB: Charge confirmed legitimate -<br/>poisoned label (silent)
        LB->>MD: Monthly retrain ingests<br/>labels unweighted (silent)
        Note over MD: Ring pattern drifts<br/>toward legitimate
    end
    Note over MD: By retrain 3 the signature<br/>scores clean
    FR->>MD: Cash-out at scale in one<br/>night - model waves it<br/>through

Solution — where the guardrails go:

flowchart TD
    A["Dispute-resolution labels<br/>arrive"] --> B["Label-source<br/>authentication: who wrote<br/>this label"]
    B --> C["Externally writable labels<br/>weighted down"]
    A -.-> D["Anomaly detection on label<br/>streams"]
    D -.-> AL["Alert: dispute-<br/>confirmation burst from<br/>linked accounts"]
    C --> E["Monthly retrain"]
    F["Trusted-label core:<br/>internal investigations,<br/>audited samples"] --> E
    E --> G["Poisoned labels cannot<br/>steer the model"]
    AL --> H["Ring pattern investigated<br/>before any cash-out"]

What good looks like

• Provenance on every label — source system, author class (internal investigator / customer self-service / automated heuristic), and trust tier carried as schema fields, not tribal knowledge.
• A trusted-label core anchoring every retrain — internal investigations plus audited random samples; sustained disagreement between the core and any external stream is an alarm, not noise to average away.
• Weight by writability — labels an outsider can author get fractional weight, and a quarantine cycle before they may teach at all; the delay alone breaks the ring's compounding schedule.
• Linkage analysis on label writers — device, address, and payment-instrument overlap across the accounts confirming disputes; a confirmation burst inside one connected component is a ring signature and pages fraud ops.
• Per-retrain diff reports — which cohorts moved toward "legitimate," driven by which label sources — reviewed by a human before the model ships, the way a code diff is reviewed before deploy.
• Velocity alarms on feedback channels — dispute-confirmation rate per segment against a trailing baseline, watched with the same anomaly detection pointed at transactions.
• Synthetic-ring injection in staging — poison your own pipeline on a schedule and verify the alarms fire; an injection that sails through undetected is the real finding, caught at drill prices.

The lesson

Training data is an attack surface: any label stream an external party can influence — disputes, self-service confirmations, crowd feedback — is a poisoning channel. Authenticate and weight labels by source trustworthiness, monitor label streams for anomalies (dispute-confirmation bursts from linked accounts are a ring signature), and keep a trusted-label core (internal investigations, audited samples) that anchors every retrain.

Named concepts: data poisoning, label-source authentication, training-data provenance, anomaly detection on feedback channels.

Real-world echoes

Microsoft Tay, 2016 — a chatbot that learned from public interaction was coordinated into offensive output within about a day of launch. Tay is this drill's loop in fast-forward: the feedback channel worked exactly as designed, and the design flaw was that outsiders could write to it, at volume, in coordination. The fraud ring runs the same play — just slowed to a monthly retrain cadence and pointed at money.

Transfer

Enumerate every label source feeding our retraining. Which could a motivated outsider influence for less than $1,000? That's their price list, not ours.

Make it a ticket: Add a source field to the label schema and produce the first per-source volume report before the next retrain — the deliverable is the list of externally writable sources, each one weighted, quarantined, or explicitly accepted in writing.

Related

• Drill 44 — The Oracle at the API — the same patient adversary reading the model out; this drill is the write path, and the two compose into a full attack kit.
• Drill 47 — The God-Mode Service Account — the same unasked question, "who is allowed to do this?", applied to identities instead of labels.
• Drill 36 — The $999 Pattern, theme 6 — feedback loops turned hostile: there the model teaches the adversary where the thresholds sit, here the adversary teaches the model what to ignore.