Theme 7 — Security & Trust Boundaries
The reflex this theme builds: every boundary you didn't draw explicitly is drawn by the attacker.
Why this theme matters
AI/ML systems widen the attack surface in directions traditional security programs weren't built to watch. Training data can be poisoned through any feedback channel an outsider can write to; models can be extracted through their own APIs, one patient query at a time; prompts can be injected by anything that reaches the context; and the data exhaust — logs, caches, vendor payloads — carries the same obligations as the primary stores, with none of the same attention. In financial services the stakes compound: the data is exactly what attackers monetize, and one cross-boundary leak costs the entire client base's trust, not just one record's.
Most of these drills share a single root cause: a trust boundary that was never drawn explicitly — between code and secrets, between your data and a vendor's logs, between labels you wrote and labels an outsider wrote — so the attacker drew it instead, in the most profitable place. Abbreviations are expanded on first use; the full reference is glossary.md.
The drills
| # | Drill | One-line hook | Problem diagram | ~Difficulty |
|---|---|---|---|---|
| 42 | The Notebook on GitHub | A credential pasted "just to get it working" leaks in hours — the real finding is that rotating it takes three days and a meeting. | sequence | intro |
| 43 | Shadow AI | A conscientious analyst pastes customer PII into a free chatbot for four months — half the team, quietly asked, admits to the same. | timeline | core |
| 44 | The Oracle at the API | Thousands of near-duplicate queries, one field varied at a time — someone is mapping the model, and the map is for sale. | sequence | advanced |
| 45 | The Vendor's Bad Week | The breach is inside a vendor's logs, but the notification letters carry your name — and the SOC 2 was current and clean. | sequence | core |
| 46 | The Poisoned Feedback | A fraud ring teaches the model its pattern is legitimate, one dispute-and-confirm cycle at a time — then cashes out in one night. | sequence | advanced |
| 47 | The God-Mode Service Account | One stolen credential and no privileges left to escalate to — forensics can only say "potentially all of it." | sequence | core |
| 48 | The Logs That Remembered Everything | A debug flag outlives its incident by two years — SSNs, full-text searchable, in a store no data inventory ever listed. | timeline | core |
Best practices — the security & trust checklist
What to actually do, distilled from drills 42–48:
- Keep secrets in a secret manager — never in code, configs, or notebooks — with secret scanning wired into both pre-commit hooks and CI (Continuous Integration), so the inevitable paste is caught before it ships rather than by the attackers' scanners after.
- Issue short-lived, per-service credentials and rehearse rotation as a routine drill. If rotating any credential requires a meeting, that credential is a standing incident; practice until rotation is boring.
- Enforce least privilege through workload identity: every service gets its own identity with minimal scopes, no shared credentials, no god accounts, and segmentation that limits lateral movement (an attacker expanding from the first compromised system to the rest).
- Build the paved road for AI tooling: a sanctioned model under enterprise terms, with logging and redaction built in, that is genuinely easier than the contraband alternative. Prohibition without provision just moves usage where you can't see it.
- Pair DLP (Data Loss Prevention) monitoring with honest discovery: run amnesty-based surveys of shadow-tool usage. People disclose unmet needs when disclosure is safe; they hide from punishment — and hidden usage is the expensive kind.
- Treat every externally reachable model API (Application Programming Interface) as an oracle under attack: per-key behavioral analytics, perturbation-pattern detection, rate limits, and the coarsest output the business allows — a decision, not a raw score.
- Minimize what vendors receive: redact or tokenize before transmission, put log-handling and breach-notification terms with teeth in every contract, and keep your own runbook for "the breach is upstream" — a clean attestation is a snapshot, not a guarantee.
- Authenticate and weight training labels by source: any feedback channel an outsider can write to — disputes, self-service confirmations, crowd feedback — is a poisoning channel. Run anomaly detection on label streams and anchor every retrain with a trusted-label core.
- Run red-team suites in CI, not as annual engagements: prompt-injection payloads, model-extraction probes, and cross-tenant access attempts, scored automatically on every change like unit tests.
- Put logs and telemetry inside the data inventory: schema-level PII (Personally Identifiable Information) scrubbing so block-listed fields never serialize, platform-enforced retention, debug verbosity with a TTL (Time To Live) that auto-reverts, and deletion rights that propagate into the observability stack.
- Make every incident answerable with "what exactly could this credential or identity touch?" If the honest answer is "potentially everything," schedule the decomposition now — before an attacker forces the word potentially into a disclosure letter.
Also see
- Facilitator guide — how to run drills, and the other ten themes.
- Glossary — every abbreviation, explained in plain language.