@diottodev/runic 1.0.0 → 1.1.0

package/engineering/skills/chaos-engineering/SKILL.md

@@ -0,0 +1,231 @@
+---
+name: chaos-engineering
+description: Use when planning, running, or learning from chaos engineering experiments. Triggers on "chaos experiment", "fault injection", "gameday", "resilience test", "blast radius", "steady state", "abort criteria", "Chaos Toolkit", "Chaos Mesh", "Litmus", "Gremlin", "AWS FIS", or any deliberate failure-injection question. Ships experiment designer, blast-radius calculator, and postmortem generator (all stdlib Python), 4 references on chaos principles + experiment design + attack taxonomy + tooling landscape, and a /chaos-experiment slash command. Composes with feature-flags-architect (kill switches as abort triggers) and kubernetes-operator (common chaos targets).
+context: fork
+version: 2.9.0
+author: claude-code-skills
+license: MIT
+tags: [chaos-engineering, resilience, fault-injection, gameday, sre, reliability, chaos-toolkit, chaos-mesh, litmus, gremlin, aws-fis]
+compatible_tools: [claude-code, codex-cli, cursor, antigravity, opencode, gemini-cli]
+---
+
+# Chaos Engineering
+
+Design experiments that surface real weaknesses in production systems — without becoming outages. Most "chaos engineering" attempts skip steady-state measurement, define no abort criteria, and have no blast-radius bound. This skill enforces the discipline that makes chaos experiments safe and useful.
+
+## When to use
+
+- Planning a chaos experiment (what to break, where, when, how to abort)
+- Calculating blast radius before running the experiment
+- Reviewing an existing experiment plan for safety
+- Choosing a chaos tool (Chaos Toolkit / Chaos Mesh / Litmus / Gremlin / AWS FIS)
+- Writing a chaos experiment postmortem
+- Running a Game Day exercise
+
+## When NOT to use
+
+- General incident response (use `incident-response`)
+- Threat hunting / red-team (use `red-team`, `threat-detection`)
+- Performance load testing (different goal — chaos is about failure modes, not capacity)
+- Production debugging (chaos discovers weaknesses preemptively, not after-the-fact)
+
+## Core principle: chaos without abort criteria is an outage
+
+The 4 Principles of Chaos Engineering (Netflix, 2016):
+
+1. **Build a hypothesis around steady-state behavior.** Not "what breaks?" but "X holds; will it still hold under fault Y?"
+2. **Vary real-world events.** Inject realistic failures: kill nodes, slow networks, lose cache, throttle dependencies.
+3. **Run experiments in production.** Staging never has the same failure modes. Start small.
+4. **Automate experiments to run continuously.** One-off chaos is a press release; continuous chaos is engineering.
+
+Add a fifth: **Define abort criteria up front.** A chaos experiment with no abort criteria is an outage by another name.
+
+## Quick start
+
+```bash
+SKILL=engineering/chaos-engineering/skills/chaos-engineering
+
+# 1. Design an experiment
+python "$SKILL/scripts/experiment_designer.py" --target "checkout-svc" --hypothesis "p99 latency stays <500ms" --attack latency --duration-min 15
+
+# 2. Calculate blast radius
+python "$SKILL/scripts/blast_radius_calculator.py" --traffic-share 0.05 --user-pop 1000000 --duration-min 15
+
+# 3. Generate postmortem after the experiment
+python "$SKILL/scripts/experiment_postmortem.py" --plan experiment.json --result-log results.txt
+```
+
+## The 3 Python tools
+
+All stdlib-only. Run with `--help`.
+
+### `experiment_designer.py`
+
+Generates a structured experiment plan from inputs. Enforces the required sections (hypothesis, steady-state metric, blast radius, abort criteria, rollback).
+
+```bash
+python scripts/experiment_designer.py \
+  --target "checkout-svc" \
+  --hypothesis "p99 latency stays <500ms when payment-svc is slow" \
+  --attack latency \
+  --magnitude "+200ms" \
+  --duration-min 15 \
+  --blast-radius "5% of US traffic" \
+  --abort-if "p99 > 1000ms OR error_rate > baseline + 1pp"
+```
+
+Outputs a markdown plan with: hypothesis, steady-state, attack, magnitude, duration, blast radius, abort criteria, rollback procedure, monitoring dashboards, and learning question.
+
+### `blast_radius_calculator.py`
+
+Computes the blast radius of a planned experiment. Given traffic share + user population + duration, calculates expected affected users, expected error budget burn, and a risk score.
+
+```bash
+python scripts/blast_radius_calculator.py \
+  --traffic-share 0.05 \
+  --user-pop 1000000 \
+  --duration-min 15 \
+  --baseline-availability 0.999 \
+  --expected-impact-availability 0.95
+```
+
+Outputs:
+- Expected affected users
+- Error budget consumed (in minutes of error budget)
+- Risk score: GREEN / YELLOW / RED
+- Recommendation: PROCEED / REDUCE / ABORT
+
+GREEN = <1% error budget; YELLOW = 1-10%; RED = >10%.
+
+### `experiment_postmortem.py`
+
+Produces a structured postmortem from an experiment plan + results. Catches the common postmortem failure modes: no learning recorded, no follow-up actions, blame-laden language.
+
+```bash
+python scripts/experiment_postmortem.py --plan experiment.json --result-log results.txt
+```
+
+Outputs markdown with: summary, hypothesis (was it confirmed/refuted?), what we learned, what surprised us, follow-up actions with owners, and link to next experiment.
+
+## The 7 attack types (taxonomy)
+
+Different attacks reveal different weaknesses. See `references/attack_taxonomy.md` for full detail.
+
+| Attack | What it tests | Tooling |
+|---|---|---|
+| **Latency** | Timeouts, retries, circuit breakers | tc, Chaos Mesh `NetworkChaos` |
+| **Error** | Error handling, fallback paths | Chaos Mesh `HTTPChaos`, Toxiproxy |
+| **Resource** (CPU, memory, disk) | Saturation handling, autoscaling | Chaos Mesh `StressChaos`, stress-ng |
+| **Network partition** | Split-brain, consensus, failover | Chaos Mesh `NetworkChaos` partition |
+| **Dependency failure** | Graceful degradation, fallback | Service mesh fault injection |
+| **Time** | Clock skew, NTP issues | libfaketime, Chaos Mesh `TimeChaos` |
+| **Infrastructure** (kill instance) | Auto-recovery, failover | AWS FIS, Chaos Monkey |
+
+Pick the attack that matches the hypothesis. "What happens if X is slow?" → latency. "What happens if X loses network?" → partition.
+
+## Tooling chooser
+
+| Tool | Best for | Pricing | Stack |
+|---|---|---|---|
+| **Chaos Toolkit** | Lightweight, language-agnostic, JSON experiments | OSS | Any |
+| **Chaos Mesh** | Kubernetes-native, rich CRDs, in-cluster | OSS | Kubernetes |
+| **Litmus** | Kubernetes, Argo-integrated, large library | OSS + Enterprise | Kubernetes |
+| **Gremlin** | Enterprise SaaS, multi-cloud, audit | Paid | Any |
+| **AWS FIS** | AWS-native, IAM-integrated, EC2/ECS/EKS | Paid (AWS) | AWS |
+| **Custom** | Niche needs, single-cloud, low budget | None | Any |
+
+Decision rules:
+- k8s-only stack + OSS → Chaos Mesh or Litmus (Litmus has bigger experiment library)
+- Multi-cloud + OSS → Chaos Toolkit
+- AWS-heavy + simple needs → AWS FIS
+- Enterprise + audit/compliance → Gremlin
+
+See `references/tooling_landscape.md` for trade-offs.
+
+## Workflows
+
+### Workflow 1: Design and run a single experiment
+
+```
+1. State a hypothesis: "When [fault], steady-state metric X stays within Y."
+2. Identify the steady-state metric — must be measurable BEFORE the experiment.
+3. Run blast_radius_calculator.py — confirm GREEN before proceeding.
+4. Run experiment_designer.py to produce the plan.
+5. Get a peer review of the plan; confirm abort criteria are concrete.
+6. Notify the on-call team in #incidents (or whatever channel).
+7. Run the experiment with monitoring open.
+8. If abort criteria are hit, abort immediately; record what happened.
+9. Run experiment_postmortem.py to capture learnings.
+10. File follow-up actions; link to next experiment.
+```
+
+### Workflow 2: Game Day exercise
+
+```
+1. Pick a scenario (e.g., "primary database fails over").
+2. Identify all dependent services that should keep working.
+3. Build a multi-experiment plan covering each layer.
+4. Schedule with stakeholders; on-call coverage required.
+5. Run with a facilitator who manages the scenario.
+6. Capture observations in a shared doc as they happen.
+7. Single combined postmortem covering all observations.
+8. Track follow-up actions in a board with owners.
+```
+
+### Workflow 3: Continuous chaos (game days → daily)
+
+```
+1. Start: weekly Game Day in staging.
+2. Move to: weekly Game Day in production with limited blast radius.
+3. Mature to: continuous chaos via scheduled experiments (Litmus chaos schedule, Gremlin scenarios).
+4. Wire to deployment: every prod deploy triggers a baseline chaos sweep.
+5. Track: experiments per week, weaknesses discovered, MTTR trend.
+```
+
+## Composition with other skills
+
+This skill explicitly composes with two others in this library:
+
+| Skill | Composition |
+|---|---|
+| `feature-flags-architect` | Kill switches defined there are the abort triggers here |
+| `kubernetes-operator` | Operators are common chaos targets (test reconcile under fault) |
+| `incident-response` | Chaos experiments that escalate become incidents |
+
+## Anti-patterns
+
+- **No hypothesis** — "let's break things" is sabotage, not engineering
+- **No steady-state metric** — without a baseline, you can't tell if X broke
+- **No blast radius bound** — full-prod experiment without limits = outage
+- **No abort criteria** — see above; this is mandatory
+- **No on-call coverage** — chaos without monitoring is unmonitored production
+- **Chaos in staging only** — staging never has prod failure modes
+- **Chaos in dev** — useless; dev has different failure modes from prod
+- **One-off chaos** — single experiment is a press release; learning requires recurrence
+- **Blame-laden postmortem** — record causes, not blame; teams stop running chaos otherwise
+
+## References
+
+- `references/chaos_principles.md` — the 4 principles, history, when to start
+- `references/experiment_design.md` — hypothesis structure, steady-state metrics, abort criteria
+- `references/attack_taxonomy.md` — 7 attack types with examples and tooling
+- `references/tooling_landscape.md` — Chaos Toolkit / Mesh / Litmus / Gremlin / FIS / DIY
+
+## Slash command
+
+`/chaos-experiment` — interactive experiment design wizard that runs all 3 tools.
+
+## Asset templates
+
+- `assets/experiment_template.md` — fill-in plan template
+- `assets/postmortem_template.md` — structured postmortem template
+
+## Verifiable success
+
+A team using this skill should achieve:
+
+- 100% of chaos experiments have a written hypothesis, abort criteria, and blast-radius calculation
+- Blast radius for any single experiment never exceeds 10% of error budget
+- Mean time between chaos experiments <14 days (continuous, not one-off)
+- Each experiment produces ≥1 follow-up action that gets shipped
+- No chaos experiment escalates to a customer-impacting incident in trailing 90 days

package/engineering/skills/chaos-engineering/assets/experiment_template.md

@@ -0,0 +1,76 @@
+# Chaos Experiment
+
+Fill in every section before running. Refuse to run if any section is empty.
+
+## Identity
+
+- **Experiment ID:** `<auto-generated; format: chaos-<target>-<attack>-<unix-ts>>`
+- **Date:** `<YYYY-MM-DD>`
+- **Owner:** `<your-handle@team>`
+- **On-call team:** `<team channel / pager>`
+- **Reviewer:** `<peer who reviewed this plan>`
+
+## 1. Hypothesis
+
+> When `<fault>`, `<steady-state metric>` stays `<tolerance>`.
+
+Example: *When payment-svc is +200ms slow, checkout p99 stays below 500ms.*
+
+## 2. Steady-state metric
+
+- **Metric:** `<e.g., p99 checkout latency>`
+- **Baseline window:** `<e.g., 5 minutes pre-experiment>`
+- **Tolerance:** `<e.g., within ±5% of baseline>`
+- **Dashboard:** `<URL>`
+
+## 3. Attack
+
+- **Type:** `[ ] latency  [ ] error  [ ] cpu  [ ] memory  [ ] disk  [ ] network-partition  [ ] dependency-failure  [ ] time-skew  [ ] kill-instance`
+- **Magnitude:** `<e.g., +200ms>`
+- **Duration:** `<minutes>`
+- **Target:** `<service / pod / instance / region>`
+- **Tooling:** `<Chaos Toolkit / Chaos Mesh / Litmus / Gremlin / AWS FIS / Custom>`
+
+## 4. Blast radius
+
+- **Traffic share:** `<e.g., 5% of US>`
+- **Expected affected users:** `<from blast_radius_calculator.py>`
+- **Error budget consumed:** `<from blast_radius_calculator.py>`
+- **Risk score:** `[ ] GREEN  [ ] YELLOW  [ ] RED`
+
+## 5. Abort criteria
+
+> Auto-trigger experiment termination if ANY of these hit.
+
+- [ ] `<signal 1, e.g., p99 > 1000ms>`
+- [ ] `<signal 2, e.g., 5xx rate > baseline + 1pp>`
+- [ ] `<signal 3, e.g., on-call paged SEV1/SEV2>`
+
+## 6. Rollback procedure
+
+1. `<step to disable fault, e.g., "kubectl delete chaos networkchaos/<name>">`
+2. Verify steady state recovers within 2 minutes
+3. If not recovering, escalate as incident; restore from backup if needed
+
+## 7. Learning question
+
+> What do you expect NOT to learn? Force yourself to predict.
+
+`<your prediction>`
+
+## Pre-flight checklist
+
+- [ ] Hypothesis written
+- [ ] Steady-state metric measured for ≥5 min
+- [ ] Blast radius calculated (GREEN or YELLOW only)
+- [ ] Abort criteria documented with thresholds
+- [ ] Rollback procedure tested in staging
+- [ ] On-call team notified
+- [ ] Monitoring dashboards open
+- [ ] Owner identified and reachable
+- [ ] Time-box agreed
+- [ ] Communication plan if abort triggers
+
+## Post-experiment
+
+Run `experiment_postmortem.py --plan <plan.json> --result-log <results>` to generate the postmortem.

package/engineering/skills/chaos-engineering/assets/postmortem_template.md

@@ -0,0 +1,67 @@
+# Chaos Experiment Postmortem
+
+## Identity
+
+- **Experiment:** `<experiment_id>`
+- **Date:** `<YYYY-MM-DD>`
+- **Target:** `<service>`
+- **Owner:** `<handle@team>`
+- **Postmortem facilitator:** `<handle@team>`
+
+## Hypothesis
+
+> `<hypothesis from the plan>`
+
+## Outcome
+
+- [ ] **Held** — hypothesis confirmed
+- [ ] **Refuted** — hypothesis disproven
+- [ ] **Inconclusive** — could not tell
+
+## Timeline
+
+| Time | Event |
+|---|---|
+| T-5min | Started baseline measurement |
+| T+0 | Attack injected |
+| T+? | `<observation>` |
+| T+? | `<observation>` |
+| T+N | Attack ended (or aborted) |
+| T+N+2 | Steady state recovered |
+
+## What we learned
+
+`<at least one concrete learning — required>`
+
+## What surprised us
+
+`<unexpected observations; "nothing surprised us" is a signal that you didn't push hard enough>`
+
+## What failed
+
+`<things that broke during the experiment that shouldn't have>`
+
+## What held
+
+`<things that worked as expected — confidence-building data points>`
+
+## Root causes (if any failures)
+
+`<technical analysis without blame>`
+
+## Follow-up actions
+
+| Action | Owner | Due | Status |
+|---|---|---|---|
+| `<concrete action>` | `<@owner>` | `<date>` | `[ ]` |
+| `<concrete action>` | `<@owner>` | `<date>` | `[ ]` |
+
+> Every experiment should produce ≥1 follow-up. If none — re-examine whether you tested anything new.
+
+## Next experiment
+
+`<what's the next experiment that builds on this learning?>`
+
+## Stakeholder summary (1-2 sentences)
+
+`<for the team channel; describe outcome and biggest learning>`

package/engineering/skills/chaos-engineering/references/attack_taxonomy.md

@@ -0,0 +1,180 @@
+# Attack taxonomy
+
+7 categories of fault injection. Each tests a different system property. Pick the one whose failure mode matches your hypothesis.
+
+## 1. Latency
+
+**What it tests:** timeouts, retries, circuit breakers, fallback paths.
+
+**Inject:** add N ms of delay to network responses to a target.
+
+**When to use:**
+- "What if dependency X is slow?"
+- "Are timeouts configured correctly upstream?"
+- "Does the retry budget kick in?"
+
+**Tools:**
+- Linux `tc` (traffic control) — direct kernel-level shaping
+- Chaos Mesh `NetworkChaos` (delay)
+- Toxiproxy — proxy-based, language-agnostic
+- AWS FIS — `aws:network:traffic-control` action
+
+**Example magnitude:** +200ms (90% of typical timeouts), +2000ms (test backoff), +30s (test giving-up logic).
+
+## 2. Error injection
+
+**What it tests:** error handling paths, fallback behavior, retry policies.
+
+**Inject:** return errors (5xx, exceptions) for a fraction of requests.
+
+**When to use:**
+- "What happens when X starts failing?"
+- "Does the fallback path actually work in prod?"
+- "Are we logging errors correctly?"
+
+**Tools:**
+- Chaos Mesh `HTTPChaos`
+- Service mesh (Istio, Linkerd) fault injection
+- Toxiproxy with error toxic
+- Application-level feature flag for synthetic errors
+
+**Example magnitude:** 1% errors (test handler), 50% errors (test retry), 100% errors (test fallback path).
+
+## 3. Resource exhaustion
+
+**What it tests:** saturation handling, autoscaling, OOM behavior, disk-full handling.
+
+**Inject:** consume CPU, memory, or disk on the target.
+
+**When to use:**
+- "What if memory leaks?"
+- "Does the autoscaler kick in?"
+- "What happens when disk fills?"
+
+**Sub-types:**
+- **CPU pressure** — peg cores at N% usage
+- **Memory pressure** — allocate large blocks
+- **Disk fill** — write large files until partition fills
+- **I/O saturation** — high random read/write
+
+**Tools:**
+- `stress-ng` — CPU/memory/IO/disk
+- Chaos Mesh `StressChaos` and `IOChaos`
+- AWS FIS `aws:ssm:send-command` with stress-ng
+
+**Example magnitude:** 80% CPU sustained, 90% memory, fill /var to 95%.
+
+## 4. Network partition
+
+**What it tests:** consensus protocols, leader election, split-brain prevention, region failover.
+
+**Inject:** drop all packets between a set of hosts.
+
+**When to use:**
+- "What if AZ-A loses connectivity to AZ-B?"
+- "Does the database elect a new primary?"
+- "Does the cluster avoid split-brain?"
+
+**Tools:**
+- Chaos Mesh `NetworkChaos` (partition mode)
+- `tc` with iptables drop rules
+- AWS FIS `aws:network:disrupt-connectivity`
+
+**Example magnitude:** drop 100% to peer X (full partition), drop 50% (degraded link).
+
+## 5. Dependency failure
+
+**What it tests:** graceful degradation, fallback to cache, fallback to default values.
+
+**Inject:** make a downstream dependency unavailable (timeout, refuse connections).
+
+**When to use:**
+- "What if the rec engine goes down?"
+- "Does Search degrade gracefully when ML models are unreachable?"
+- "Is cache the fallback for the user-pref service?"
+
+**Tools:**
+- Service mesh fault injection (most flexible)
+- Toxiproxy
+- iptables rules to refuse connections
+- Chaos Mesh `NetworkChaos` with `corrupt` or `drop`
+
+**Example magnitude:** 100% requests to dep X timeout (full outage), 25% timeout (intermittent), 0% available for 5 min (sustained outage).
+
+## 6. Time skew
+
+**What it tests:** time-sensitive logic — token expiry, cron schedules, TTLs, retry backoff.
+
+**Inject:** alter the wall clock seen by a process.
+
+**When to use:**
+- "What if NTP fails?"
+- "What if a process clock drifts +5 minutes?"
+- "Do tokens correctly fail validation when expired?"
+- "Does cron skip or double-fire?"
+
+**Tools:**
+- `libfaketime` — preload library
+- Chaos Mesh `TimeChaos`
+- Custom: change container's `/etc/localtime`
+
+**Example magnitude:** +1 minute (subtle), +5 minutes (TLS / token failures), +1 day (catastrophic for some logic).
+
+**Caution:** time skew can cause cluster-wide consensus failures. Test in isolation first.
+
+## 7. Infrastructure (kill instance / pod / container)
+
+**What it tests:** auto-recovery, failover, replica count maintenance.
+
+**Inject:** terminate an instance, pod, or container.
+
+**When to use:**
+- "Does Kubernetes restart the pod?"
+- "Does the load balancer remove the instance from rotation?"
+- "Is the replication factor maintained?"
+
+**Tools:**
+- Chaos Monkey (the original)
+- Chaos Mesh `PodChaos` (kill, fail)
+- AWS FIS `aws:ec2:terminate-instances`
+- `kubectl delete pod` (manual, simplest)
+
+**Example magnitude:** kill 1 of N pods (Chaos Monkey level), kill all pods of a deployment (test recreation), kill 1 of 3 replica DB nodes (test failover).
+
+## Choosing an attack
+
+| Hypothesis pattern | Attack type |
+|---|---|
+| "What if X is slow?" | Latency |
+| "What if X is failing?" | Error |
+| "What if we run hot?" | Resource |
+| "What if regions partition?" | Network partition |
+| "What if dep X is down?" | Dependency failure |
+| "What if clocks drift?" | Time skew |
+| "What if a node dies?" | Infrastructure |
+
+## Combining attacks
+
+Real outages often combine attacks (e.g., latency + saturation). Once basic experiments are stable, run combinations:
+
+- Latency on dependency + CPU pressure on app → tests timeout + retry budget interaction
+- Pod kill + network partition → tests recovery during a partition
+- Disk fill + dependency failure → tests fallback path while disk is constrained
+
+Combinations have higher risk; reduce blast radius accordingly.
+
+## Severity ladder
+
+```
+S1 — Latency (small)              ← start here
+S2 — Error injection (low %)
+S3 — Resource pressure (CPU/mem)
+S4 — Latency (large) / errors (high %)
+S5 — Single instance kill
+S6 — Network partition (single peer)
+S7 — Multiple instance kill
+S8 — Region partition / time skew
+S9 — Combinations of S5-S8        ← here be dragons
+```
+
+Don't skip levels. Earn confidence at S1-S3 before attempting S5+.

package/engineering/skills/chaos-engineering/references/chaos_principles.md

@@ -0,0 +1,136 @@
+# The principles of chaos engineering
+
+Chaos engineering is the discipline of experimenting on a system in order to build confidence in its capability to withstand turbulent conditions in production. The phrase comes from Netflix's 2014-2016 work productizing what started as Chaos Monkey.
+
+## The 4 founding principles (Netflix, 2016)
+
+### 1. Build a hypothesis around steady-state behavior
+
+Steady state = a measurable, normal-operations metric (latency, throughput, conversion rate, error rate).
+
+Bad: *"What happens if the database goes down?"*
+Good: *"When the primary database fails over, p99 checkout latency stays below 800ms and conversion rate stays within 2% of baseline."*
+
+The hypothesis must be **falsifiable** — there must be a measurement that can disprove it.
+
+### 2. Vary real-world events
+
+Inject realistic failure modes:
+- Servers crash
+- Networks partition or slow
+- Disks fill
+- Dependencies time out or return errors
+- Caches lose data
+- Time skews
+
+Don't inject implausible events (e.g., "what if all 50 zones in 5 regions go down simultaneously"). That's not chaos engineering, that's astronomy.
+
+### 3. Run experiments in production
+
+Staging never reproduces:
+- Real traffic patterns
+- Real cache hit rates
+- Real cross-service dependencies
+- Real data volumes
+- Real user behavior
+
+The only system that has prod failure modes is prod. Start with tiny blast radius (1%), grow as confidence grows.
+
+### 4. Automate experiments to run continuously
+
+A single chaos experiment is a press release. Continuous chaos is engineering.
+
+Maturity progression:
+1. Manual one-offs → 2. Weekly Game Days → 3. Scheduled experiments → 4. Continuous chaos in CI/CD
+
+The 5th principle this skill adds:
+
+### 5. Define abort criteria up front
+
+A chaos experiment with no abort criteria is an outage. Every plan must include:
+
+- A specific signal (metric, threshold)
+- A specific action (auto-abort, manual abort, escalate)
+- A timeline (within N seconds of breach)
+
+If the threshold is hit, abort immediately. Investigate later.
+
+## When to start
+
+You're ready for chaos engineering when:
+
+- [ ] You have basic monitoring (you can detect a steady-state breach)
+- [ ] You have on-call rotations (someone is watching when chaos runs)
+- [ ] You have at least one tool to inject the desired fault
+- [ ] You have an SLO/SLI defined (so you know what "good" looks like)
+- [ ] You have postmortem culture that's blameless
+- [ ] You have a leadership champion who'll defend the practice
+
+If any of these are missing, fix them first. Premature chaos = outages with no learning.
+
+## When NOT to do chaos engineering
+
+- During a release freeze
+- During a known incident
+- During peak traffic events without explicit approval
+- On systems that don't have steady-state metrics
+- On systems where you can't bound the blast radius
+- On the day of a security disclosure
+- When the team is already firefighting
+
+## Maturity model
+
+| Level | Description | Cadence | Tooling |
+|---|---|---|---|
+| L0 | None | n/a | none |
+| L1 | Manual one-offs in staging | quarterly | tc, manual scripts |
+| L2 | Weekly Game Days in staging | weekly | Chaos Toolkit, internal scripts |
+| L3 | Limited prod experiments | weekly | Chaos Toolkit / Mesh / Litmus / FIS |
+| L4 | Continuous prod chaos with bounded blast radius | daily | Chaos Mesh / Gremlin scenarios |
+| L5 | Chaos in CI/CD pipeline; deploys auto-trigger sweeps | per-deploy | Custom + tooling stack |
+
+Most teams should target L3 within 6-12 months of starting. L5 is rare and only justified for the largest distributed systems.
+
+## Common objections (and counters)
+
+| Objection | Counter |
+|---|---|
+| "We can't break production!" | You already do, just unintentionally. Chaos is intentional, bounded, observed breaks. |
+| "This is a customer-facing system." | Start at 1% blast radius. The 99% are unaffected. |
+| "We don't have time." | Chaos finds bugs that would otherwise become 4am pages. Time spent on chaos saves time on incidents. |
+| "Our system is too critical." | Critical systems have the most to gain from learning their failure modes. |
+| "We have HA already." | HA without chaos is HA in theory. Chaos finds gaps in actual HA. |
+
+## What a steady-state metric looks like
+
+Good steady-state metrics:
+- p99 request latency (objective, measurable per second)
+- Error rate (objective, measurable)
+- Conversion rate (business metric, slow but real)
+- Successful logins per minute (business + tech signal)
+- Queue depth (system health)
+
+Bad metrics:
+- "Things feel slow" (not measurable)
+- CPU usage (a means, not an end)
+- Number of pods running (not customer-facing)
+
+Pick metrics that customers feel. CPU can spike without customer impact; latency and errors can't.
+
+## History
+
+- 2010: Netflix launches Chaos Monkey (kills random EC2 instances)
+- 2011: Simian Army expands (Latency Monkey, Conformity Monkey, etc.)
+- 2014: Chaos engineering term coined; principles drafted
+- 2016: principlesofchaos.org published
+- 2018: Chaos Toolkit released as OSS
+- 2019: Chaos Mesh and Litmus mature for Kubernetes
+- 2020: AWS launches Fault Injection Simulator (FIS)
+- 2023+: Chaos engineering becomes mainstream practice in SRE-heavy orgs
+
+## Further reading
+
+- principlesofchaos.org — the foundational document
+- *Chaos Engineering* (Casey Rosenthal, Nora Jones) — O'Reilly, 2020
+- *Learning Chaos Engineering* (Russ Miles) — O'Reilly, 2019
+- Netflix Tech Blog on Chaos Engineering posts (2016-2020)