SRE & Best-Practice Concept Index

Every drill names the formal concepts it teaches. This index inverts that: pick a concept, find the drills (and interview cases) that teach it through story. Use it to run the program concept-first — e.g., declare a "monitoring month" and run drills 1, 2, 3, 6, 29 — or to fill gaps after a real incident ("we just got burned by X; which drills train it?").

Monitoring & observability

Concept	What it means in one line	Drills	Interview cases
SLI / SLO / error budget	Measure what the service is for; agree how much failure is acceptable; spend the budget deliberately	1, 3, 15	17, 18
Semantic monitoring	Assert meaning (distributions, fill rates, freshness), not just liveness	1, 4, 16, 19	17
Synthetic probes / absence alerting	Alert when success stops, not only when errors start	2, 6	17
Dimensional slicing	Aggregates hide minorities — slice by tenant, region, segment	3, 12	2, 17
Liveness vs. readiness vs. deep health	Different checks for different consumers; deep checks never eject fleets	6	—
Alert discipline (owners, runbooks, budgets)	An alert without an action is a dashboard line; fatigue kills real pages	29	17
Input/dependency monitoring	Monitor what vendors send you like you monitor what you serve	39, 45	8
Change-event correlation	Deploys, config, vendor notes on one timeline with metrics	11, 39	17
Anomaly registries	Recurring anomalies get root causes, not nicknames	5	6

Incident response & recovery

Concept	One line	Drills	Cases
Contain before diagnose	Stop the bleeding, then find the wound	—	19
Incident command & action logging	Every action recorded; no invisible fixes	30	19
Blameless postmortems	Blame reduces reports, not errors; fix systems, not people	31	19
Near-miss reporting	Your early-warning sensors are humans; keep them switched on	31	—
Break-glass procedures	Least privilege needs a fast, audited exception path	25, 34, 42	—
Runbook hygiene	Runbooks are code humans execute; test them or they rot	33	—
DR / RTO / RPO validation	You don't have backups, you have untested restores	26, 32	—
Correctness incidents	Wrong answers deserve the same machinery as downtime	10	3

Change management & delivery

Concept	One line	Drills	Cases
Config = code	Same review, canary, and rollback as any deploy	4, 21, 28	12
Canary / progressive delivery	Small slices first — for code, config, flows, and model versions	16, 28	12, 15
Post-deployment verification	Done means verified in prod, not green in CI	2, 28	—
Feature-flag hygiene	Flags have owners and expiry dates; debug flags have TTLs	28, 48	—
Regression evals as CI gates	Prompt + model version is a versioned artifact; changes pass the suite	16, 18	13, 15
Semantic change management	Data changes meaning without schema changes; notify across teams	8, 11, 39	8

Resilience patterns

Concept	One line	Drills	Cases
Backoff + jitter + retry budgets	Unbounded retries turn partial failures total	15, 21	19
Circuit breakers & load shedding	Stop hammering the struggling; protect the core	20, 21, 24	19
Backpressure & bottleneck thinking	Scale the constraint; queue at the door; autoscalers need ceilings	24	—
Graceful degradation (designed, not improvised)	Fail-open/fail-closed is a business policy, decided in daylight	1, 22	12, 19
Chaos engineering / game days	Redundancy claims are hypotheses; induce failure to test them	23, 26, 32, 33	—
Quota & capacity planning	Inventory every limit that can say no — including the provider's	22	18
Split-brain prevention / fencing	Single-writer is a mechanism, not a hope about DNS	26	—
Topology & latency budgets	Every hop costs ms, dollars, and a failure mode	27	18
Idempotency & exactly-once actions	Everything should be safe to run twice; side effects need keys	13, 20	13

Data & ML reliability

Concept	One line	Drills	Cases
Data contracts & ingestion gates	Feeds fail loudly at the border, not quietly at the decision	7, 12	—
Immutability / event sourcing / point-in-time	History is append-only; corrections are events	8	12, 16
Reversibility & soft actions	Never automate an irreversible act when a reversible form exists	9	5, 15
Training-serving skew	One feature definition, executed everywhere, skew-monitored	—	16
Eval hygiene / contamination checks	Out-of-time validation in CI, not annual audits	10	3
Feedback loops & exploration budgets	Systems that filter their own training data narrow every cycle	35, 40	1, 9, 11
Goodhart's law & paired metrics	Optimized proxies stop measuring; pair quantity with audited quality	18, 37	4
Experiment validity	The randomization unit defines what you measured	38	9
Judge/eval calibration	Validate the grader against humans before optimizing against it	18	2, 14

Security & trust

Concept	One line	Drills	Cases
Least privilege & workload identity	Per-service scopes; no god accounts; blast radius is designed	42, 47	15
Prompt injection & trust boundaries	Content is data, never instructions; authz lives outside the prompt	14	13
Secret management & rotation	Short-lived, scoped, scanned; scary rotation is the real finding	25, 42	—
Data minimization & third-party risk	Send vendors the minimum; attestation ≠ behavior	45	—
Training-data provenance / poisoning	Label streams outsiders can write to are attack surfaces	46	11
Model extraction / oracle abuse	Every API response teaches the adversary; watch query patterns	44	—
PII in telemetry	Logs are a data store with all the same obligations	48	14
Paved-road security	Make the safe path easier than the workaround	43	—

Culture (the concepts that hold the rest up)

Concept	One line	Drills
Psychological safety as a reliability control	People report what they're safe reporting	31, 43
Toil elimination	Manual routine work is fragility with a due date	25
Ownership registries & bus factor	Every credential, runbook, and alert owned by a role	25, 29, 33
Heroism as anti-pattern	Celebrate the systemic fix, not the midnight cowboy	30
Monitoring outcomes, not processes	"Backup succeeded" ≠ "we can restore"	32

Governance, delivery, vendors & cost (themes 8–11)

Concept	One line	Drills	Cases
Model inventory & lifecycle	Every production model has an owner-by-role, docs, a review date, and a retirement plan	49	—
Model risk management (SR 11-7)	Independent validation, documented assumptions, effective challenge — before the examiner asks	50	8
Retention enforcement & deletion lineage	Policies enforced platform-side; deletion propagates to derived data and backups	51, 48	14
Proxy discrimination & disparate impact	Features don't need to name a protected class to encode one — test them all	52	8
Purpose limitation	Data carries its consented purpose; access grants name a purpose	53	—
Parallel run / reconciliation-until-zero	Both systems decide side by side; diffs hit zero before cutover	54, 55	16, 20
Migration rehearsal & staging fidelity	Rehearse against production-sized data; staging row counts are fiction	56	—
Expand-contract (parallel change)	Destructive schema steps ship last, alone, and reversible	57	—
Flag debt & migration ratchets	"Temporarily both" is the most permanent state — deadlines and deletion tickets	58	—
Dependency registry & deprecation tracking	Every external API/feed/package: version, owner, sunset status, monitored comms channel	59, 39	—
Supply-chain security & SBOM	Pin, delay, and inventory dependencies so "are we exposed?" is a query	60	—
Pricing-model & viability risk	Architecture couples to vendor pricing; exits negotiate best before dependence	61, 63	10
SLA vs. business continuity	Credits are refunds, not resilience — design and game-day the fallback	62	12
Tagging, TTLs & orphan reaping	Untagged means undeployable; experiments expire by default	64	—
Budget guardrails & spend velocity	Caps and rate-of-change alarms on every account, especially sandboxes	65, 15	18
Data gravity & egress economics	Compute moves to data; per-GB fees are an architecture input	66, 27	18
Optimize-then-commit	Right-size and clean up first; commitments lock in whatever exists	67	—
FinOps & unit economics	Cost per transaction is an owned KPI with a trend alarm; engineers see their own spend	68	18