@diottodev/runic 1.0.0 → 1.1.0

package/engineering/chaos-engineering/skills/chaos-engineering/references/attack_taxonomy.md

@@ -0,0 +1,180 @@
+# Attack taxonomy
+
+7 categories of fault injection. Each tests a different system property. Pick the one whose failure mode matches your hypothesis.
+
+## 1. Latency
+
+**What it tests:** timeouts, retries, circuit breakers, fallback paths.
+
+**Inject:** add N ms of delay to network responses to a target.
+
+**When to use:**
+- "What if dependency X is slow?"
+- "Are timeouts configured correctly upstream?"
+- "Does the retry budget kick in?"
+
+**Tools:**
+- Linux `tc` (traffic control) — direct kernel-level shaping
+- Chaos Mesh `NetworkChaos` (delay)
+- Toxiproxy — proxy-based, language-agnostic
+- AWS FIS — `aws:network:traffic-control` action
+
+**Example magnitude:** +200ms (90% of typical timeouts), +2000ms (test backoff), +30s (test giving-up logic).
+
+## 2. Error injection
+
+**What it tests:** error handling paths, fallback behavior, retry policies.
+
+**Inject:** return errors (5xx, exceptions) for a fraction of requests.
+
+**When to use:**
+- "What happens when X starts failing?"
+- "Does the fallback path actually work in prod?"
+- "Are we logging errors correctly?"
+
+**Tools:**
+- Chaos Mesh `HTTPChaos`
+- Service mesh (Istio, Linkerd) fault injection
+- Toxiproxy with error toxic
+- Application-level feature flag for synthetic errors
+
+**Example magnitude:** 1% errors (test handler), 50% errors (test retry), 100% errors (test fallback path).
+
+## 3. Resource exhaustion
+
+**What it tests:** saturation handling, autoscaling, OOM behavior, disk-full handling.
+
+**Inject:** consume CPU, memory, or disk on the target.
+
+**When to use:**
+- "What if memory leaks?"
+- "Does the autoscaler kick in?"
+- "What happens when disk fills?"
+
+**Sub-types:**
+- **CPU pressure** — peg cores at N% usage
+- **Memory pressure** — allocate large blocks
+- **Disk fill** — write large files until partition fills
+- **I/O saturation** — high random read/write
+
+**Tools:**
+- `stress-ng` — CPU/memory/IO/disk
+- Chaos Mesh `StressChaos` and `IOChaos`
+- AWS FIS `aws:ssm:send-command` with stress-ng
+
+**Example magnitude:** 80% CPU sustained, 90% memory, fill /var to 95%.
+
+## 4. Network partition
+
+**What it tests:** consensus protocols, leader election, split-brain prevention, region failover.
+
+**Inject:** drop all packets between a set of hosts.
+
+**When to use:**
+- "What if AZ-A loses connectivity to AZ-B?"
+- "Does the database elect a new primary?"
+- "Does the cluster avoid split-brain?"
+
+**Tools:**
+- Chaos Mesh `NetworkChaos` (partition mode)
+- `tc` with iptables drop rules
+- AWS FIS `aws:network:disrupt-connectivity`
+
+**Example magnitude:** drop 100% to peer X (full partition), drop 50% (degraded link).
+
+## 5. Dependency failure
+
+**What it tests:** graceful degradation, fallback to cache, fallback to default values.
+
+**Inject:** make a downstream dependency unavailable (timeout, refuse connections).
+
+**When to use:**
+- "What if the rec engine goes down?"
+- "Does Search degrade gracefully when ML models are unreachable?"
+- "Is cache the fallback for the user-pref service?"
+
+**Tools:**
+- Service mesh fault injection (most flexible)
+- Toxiproxy
+- iptables rules to refuse connections
+- Chaos Mesh `NetworkChaos` with `corrupt` or `drop`
+
+**Example magnitude:** 100% requests to dep X timeout (full outage), 25% timeout (intermittent), 0% available for 5 min (sustained outage).
+
+## 6. Time skew
+
+**What it tests:** time-sensitive logic — token expiry, cron schedules, TTLs, retry backoff.
+
+**Inject:** alter the wall clock seen by a process.
+
+**When to use:**
+- "What if NTP fails?"
+- "What if a process clock drifts +5 minutes?"
+- "Do tokens correctly fail validation when expired?"
+- "Does cron skip or double-fire?"
+
+**Tools:**
+- `libfaketime` — preload library
+- Chaos Mesh `TimeChaos`
+- Custom: change container's `/etc/localtime`
+
+**Example magnitude:** +1 minute (subtle), +5 minutes (TLS / token failures), +1 day (catastrophic for some logic).
+
+**Caution:** time skew can cause cluster-wide consensus failures. Test in isolation first.
+
+## 7. Infrastructure (kill instance / pod / container)
+
+**What it tests:** auto-recovery, failover, replica count maintenance.
+
+**Inject:** terminate an instance, pod, or container.
+
+**When to use:**
+- "Does Kubernetes restart the pod?"
+- "Does the load balancer remove the instance from rotation?"
+- "Is the replication factor maintained?"
+
+**Tools:**
+- Chaos Monkey (the original)
+- Chaos Mesh `PodChaos` (kill, fail)
+- AWS FIS `aws:ec2:terminate-instances`
+- `kubectl delete pod` (manual, simplest)
+
+**Example magnitude:** kill 1 of N pods (Chaos Monkey level), kill all pods of a deployment (test recreation), kill 1 of 3 replica DB nodes (test failover).
+
+## Choosing an attack
+
+| Hypothesis pattern | Attack type |
+|---|---|
+| "What if X is slow?" | Latency |
+| "What if X is failing?" | Error |
+| "What if we run hot?" | Resource |
+| "What if regions partition?" | Network partition |
+| "What if dep X is down?" | Dependency failure |
+| "What if clocks drift?" | Time skew |
+| "What if a node dies?" | Infrastructure |
+
+## Combining attacks
+
+Real outages often combine attacks (e.g., latency + saturation). Once basic experiments are stable, run combinations:
+
+- Latency on dependency + CPU pressure on app → tests timeout + retry budget interaction
+- Pod kill + network partition → tests recovery during a partition
+- Disk fill + dependency failure → tests fallback path while disk is constrained
+
+Combinations have higher risk; reduce blast radius accordingly.
+
+## Severity ladder
+
+```
+S1 — Latency (small)              ← start here
+S2 — Error injection (low %)
+S3 — Resource pressure (CPU/mem)
+S4 — Latency (large) / errors (high %)
+S5 — Single instance kill
+S6 — Network partition (single peer)
+S7 — Multiple instance kill
+S8 — Region partition / time skew
+S9 — Combinations of S5-S8        ← here be dragons
+```
+
+Don't skip levels. Earn confidence at S1-S3 before attempting S5+.

package/engineering/chaos-engineering/skills/chaos-engineering/references/chaos_principles.md

@@ -0,0 +1,136 @@
+# The principles of chaos engineering
+
+Chaos engineering is the discipline of experimenting on a system in order to build confidence in its capability to withstand turbulent conditions in production. The phrase comes from Netflix's 2014-2016 work productizing what started as Chaos Monkey.
+
+## The 4 founding principles (Netflix, 2016)
+
+### 1. Build a hypothesis around steady-state behavior
+
+Steady state = a measurable, normal-operations metric (latency, throughput, conversion rate, error rate).
+
+Bad: *"What happens if the database goes down?"*
+Good: *"When the primary database fails over, p99 checkout latency stays below 800ms and conversion rate stays within 2% of baseline."*
+
+The hypothesis must be **falsifiable** — there must be a measurement that can disprove it.
+
+### 2. Vary real-world events
+
+Inject realistic failure modes:
+- Servers crash
+- Networks partition or slow
+- Disks fill
+- Dependencies time out or return errors
+- Caches lose data
+- Time skews
+
+Don't inject implausible events (e.g., "what if all 50 zones in 5 regions go down simultaneously"). That's not chaos engineering, that's astronomy.
+
+### 3. Run experiments in production
+
+Staging never reproduces:
+- Real traffic patterns
+- Real cache hit rates
+- Real cross-service dependencies
+- Real data volumes
+- Real user behavior
+
+The only system that has prod failure modes is prod. Start with tiny blast radius (1%), grow as confidence grows.
+
+### 4. Automate experiments to run continuously
+
+A single chaos experiment is a press release. Continuous chaos is engineering.
+
+Maturity progression:
+1. Manual one-offs → 2. Weekly Game Days → 3. Scheduled experiments → 4. Continuous chaos in CI/CD
+
+The 5th principle this skill adds:
+
+### 5. Define abort criteria up front
+
+A chaos experiment with no abort criteria is an outage. Every plan must include:
+
+- A specific signal (metric, threshold)
+- A specific action (auto-abort, manual abort, escalate)
+- A timeline (within N seconds of breach)
+
+If the threshold is hit, abort immediately. Investigate later.
+
+## When to start
+
+You're ready for chaos engineering when:
+
+- [ ] You have basic monitoring (you can detect a steady-state breach)
+- [ ] You have on-call rotations (someone is watching when chaos runs)
+- [ ] You have at least one tool to inject the desired fault
+- [ ] You have an SLO/SLI defined (so you know what "good" looks like)
+- [ ] You have postmortem culture that's blameless
+- [ ] You have a leadership champion who'll defend the practice
+
+If any of these are missing, fix them first. Premature chaos = outages with no learning.
+
+## When NOT to do chaos engineering
+
+- During a release freeze
+- During a known incident
+- During peak traffic events without explicit approval
+- On systems that don't have steady-state metrics
+- On systems where you can't bound the blast radius
+- On the day of a security disclosure
+- When the team is already firefighting
+
+## Maturity model
+
+| Level | Description | Cadence | Tooling |
+|---|---|---|---|
+| L0 | None | n/a | none |
+| L1 | Manual one-offs in staging | quarterly | tc, manual scripts |
+| L2 | Weekly Game Days in staging | weekly | Chaos Toolkit, internal scripts |
+| L3 | Limited prod experiments | weekly | Chaos Toolkit / Mesh / Litmus / FIS |
+| L4 | Continuous prod chaos with bounded blast radius | daily | Chaos Mesh / Gremlin scenarios |
+| L5 | Chaos in CI/CD pipeline; deploys auto-trigger sweeps | per-deploy | Custom + tooling stack |
+
+Most teams should target L3 within 6-12 months of starting. L5 is rare and only justified for the largest distributed systems.
+
+## Common objections (and counters)
+
+| Objection | Counter |
+|---|---|
+| "We can't break production!" | You already do, just unintentionally. Chaos is intentional, bounded, observed breaks. |
+| "This is a customer-facing system." | Start at 1% blast radius. The 99% are unaffected. |
+| "We don't have time." | Chaos finds bugs that would otherwise become 4am pages. Time spent on chaos saves time on incidents. |
+| "Our system is too critical." | Critical systems have the most to gain from learning their failure modes. |
+| "We have HA already." | HA without chaos is HA in theory. Chaos finds gaps in actual HA. |
+
+## What a steady-state metric looks like
+
+Good steady-state metrics:
+- p99 request latency (objective, measurable per second)
+- Error rate (objective, measurable)
+- Conversion rate (business metric, slow but real)
+- Successful logins per minute (business + tech signal)
+- Queue depth (system health)
+
+Bad metrics:
+- "Things feel slow" (not measurable)
+- CPU usage (a means, not an end)
+- Number of pods running (not customer-facing)
+
+Pick metrics that customers feel. CPU can spike without customer impact; latency and errors can't.
+
+## History
+
+- 2010: Netflix launches Chaos Monkey (kills random EC2 instances)
+- 2011: Simian Army expands (Latency Monkey, Conformity Monkey, etc.)
+- 2014: Chaos engineering term coined; principles drafted
+- 2016: principlesofchaos.org published
+- 2018: Chaos Toolkit released as OSS
+- 2019: Chaos Mesh and Litmus mature for Kubernetes
+- 2020: AWS launches Fault Injection Simulator (FIS)
+- 2023+: Chaos engineering becomes mainstream practice in SRE-heavy orgs
+
+## Further reading
+
+- principlesofchaos.org — the foundational document
+- *Chaos Engineering* (Casey Rosenthal, Nora Jones) — O'Reilly, 2020
+- *Learning Chaos Engineering* (Russ Miles) — O'Reilly, 2019
+- Netflix Tech Blog on Chaos Engineering posts (2016-2020)

package/engineering/chaos-engineering/skills/chaos-engineering/references/experiment_design.md

@@ -0,0 +1,158 @@
+# Experiment design
+
+A well-designed chaos experiment has 7 sections. Skip any of them and the experiment becomes either useless (no learning) or dangerous (no bounds).
+
+## The 7 sections
+
+```
+1. Hypothesis
+2. Steady-state metric
+3. Attack
+4. Blast radius
+5. Abort criteria
+6. Rollback procedure
+7. Learning question
+```
+
+## 1. Hypothesis
+
+**Format:** *When [fault], [steady-state metric] stays [tolerance].*
+
+Examples:
+- *"When the primary Postgres replica fails, checkout p99 latency stays below 500ms."*
+- *"When 50% of payment-service requests are throttled to 1 RPS, conversion rate drops by less than 5% within 60 seconds of return-to-normal."*
+- *"When us-east-1 is partitioned from us-west-2, Search continues to return results from us-west-2 within 200ms p99."*
+
+A good hypothesis:
+- Names a specific fault (not "things break")
+- Names a specific metric (not "everything")
+- States a specific tolerance (not "good enough")
+- Is measurable and falsifiable
+
+## 2. Steady-state metric
+
+The metric you'll measure before, during, and after the experiment.
+
+Required properties:
+- **Quantitative** — a number, not a feeling
+- **Customer-relevant** — something users feel (latency, error rate, conversion)
+- **Measurable in <60s** — slow metrics give you no time to abort
+- **Stable in normal operation** — you need a baseline
+
+| Good | Bad |
+|---|---|
+| p99 checkout latency | "the system is healthy" |
+| 4xx + 5xx rate | "errors are low" |
+| Successful login rate | CPU usage |
+| Items added to cart per minute | replica count |
+
+## 3. Attack
+
+The fault you're injecting. Must specify:
+
+- **Type** — latency, error, resource, partition, dependency, time, infrastructure
+- **Magnitude** — *how* much (e.g., "+200ms", "10% errors", "100% timeout to peer X")
+- **Duration** — how long the attack runs (typically 5-30 minutes)
+- **Target** — which subset of the system gets the attack
+
+See `attack_taxonomy.md` for the 7 attack types.
+
+## 4. Blast radius
+
+The maximum scope of customer impact. Use `blast_radius_calculator.py` to compute:
+
+- **Affected users** — `traffic_share × user_population`
+- **Error budget consumed** — `duration × traffic_share × availability_delta`
+- **Risk score** — GREEN (<1% budget) / YELLOW (1-10%) / RED (>10%)
+
+Rule of thumb:
+- Start at 1% traffic share
+- Grow only after 3 successful experiments at the previous level
+- Never exceed 10% of monthly error budget in a single experiment
+
+## 5. Abort criteria
+
+The signals that auto-trigger experiment termination. Each must be:
+
+- **Concrete** — specific metric and threshold ("p99 > 1000ms" not "performance degrades")
+- **Detectable in <60s** — latency, error rate, throughput
+- **Wired to action** — manual abort link in the dashboard, automatic via alert webhook
+
+Standard abort criteria:
+
+| Signal | Threshold | Action |
+|---|---|---|
+| p99 latency | > 2× baseline | abort |
+| 5xx rate | > baseline + 1pp | abort |
+| 4xx rate (excl. 401/404) | > baseline + 5pp | abort |
+| Conversion rate | < baseline × 0.95 | abort |
+| Customer ticket spike | > 3× baseline | escalate |
+| On-call paged | any SEV1/SEV2 | abort |
+
+## 6. Rollback procedure
+
+How you'll revert the fault. Required because:
+- Sometimes the chaos tool itself fails to revert
+- Sometimes the fault has lingering effects (caches, connections)
+
+Standard rollback:
+1. Disable fault injection in tool
+2. Verify steady-state recovers within 2 minutes
+3. If not recovering, escalate as incident; restore from backup if needed
+
+## 7. Learning question
+
+What do you expect NOT to learn? Force yourself to predict the outcome.
+
+Examples:
+- *"We expect the cache to absorb the latency. We'll learn whether the timeout configuration on the upstream is correct."*
+- *"We expect failover to take 30s. We'll learn whether retry backoff is configured."*
+
+If you predicted the outcome correctly: confidence increased.
+If you didn't: there's an unknown — file a follow-up.
+
+## Pre-flight checklist
+
+Before running the experiment, verify:
+
+- [ ] Hypothesis written
+- [ ] Steady-state metric measured for ≥5 minutes
+- [ ] Blast radius calculated (GREEN or YELLOW)
+- [ ] Abort criteria documented with thresholds
+- [ ] Rollback procedure tested in staging
+- [ ] On-call team notified in the team channel
+- [ ] Monitoring dashboards open
+- [ ] Owner identified and reachable
+- [ ] Time-box agreed (max experiment duration)
+- [ ] Communication plan if abort triggers
+
+## Time-boxing
+
+| Experiment type | Typical duration | Max recommended |
+|---|---|---|
+| First-time chaos | 5 minutes | 10 minutes |
+| Familiar attack, new target | 15 minutes | 30 minutes |
+| Continuous (automated) | per scheduler | 10 min per attack |
+| Game Day (human-led) | 1-2 hours | 4 hours |
+
+## Escalation
+
+If abort criteria are hit:
+
+1. **Stop the experiment immediately** (the obvious step many teams forget to script)
+2. Verify steady-state recovery
+3. If recovery doesn't happen in 5 min → declare an incident
+4. Open a postmortem doc using `experiment_postmortem.py`
+5. Notify stakeholders (whoever was promised "this won't impact anything")
+6. Capture timeline while memory is fresh
+
+## Anti-patterns
+
+- **Hypothesis written after running** — that's a postmortem, not chaos engineering
+- **Steady-state metric chosen during experiment** — pick before
+- **Magnitude "small"** — quantify; "small" varies by reader
+- **No abort criteria** — never run without them
+- **Single owner of all chaos** — culture problem; spread the practice
+- **Chaos that always succeeds** — increase magnitude; you're not learning if everything passes
+- **Chaos that always fails** — reduce magnitude; you can't learn if everything breaks
+- **Chaos with no follow-up actions** — what was the point?

package/engineering/chaos-engineering/skills/chaos-engineering/references/tooling_landscape.md

@@ -0,0 +1,197 @@
+# Tooling landscape
+
+Six options. Pick by stack, license preference, and required attack types.
+
+## At-a-glance
+
+| Tool | License | Stack | Attack coverage | Best for |
+|---|---|---|---|---|
+| **Chaos Toolkit** | OSS (Apache 2) | Any (Python) | Broad via plugins | Lightweight, multi-cloud, JSON experiments |
+| **Chaos Mesh** | OSS (Apache 2) | Kubernetes | Very broad (network, pod, IO, time, stress) | k8s-native, rich CRDs |
+| **Litmus** | OSS (Apache 2) | Kubernetes | Very broad (300+ experiments) | k8s, Argo-integrated |
+| **Gremlin** | Commercial | Any (agents) | Broad, polished | Enterprise, audit, multi-cloud |
+| **AWS FIS** | Paid (AWS) | AWS | AWS services + EC2/ECS/EKS | AWS-heavy, IAM-integrated |
+| **Custom** | Your code | Any | What you build | Niche, single-cloud, low budget |
+
+## Decision tree
+
+```
+Stack constraint?
+├── Kubernetes-only ──┬── OSS preferred → Chaos Mesh OR Litmus
+│                     │       (Litmus has the bigger experiment library;
+│                     │        Chaos Mesh has cleaner CRD model)
+│                     └── Enterprise budget → Gremlin
+│
+├── AWS-heavy ────────┬── Simple needs → AWS FIS
+│                     ├── Multi-cloud + AWS → Chaos Toolkit + AWS plugin
+│                     └── Enterprise → Gremlin
+│
+├── Multi-cloud ──────┬── OSS → Chaos Toolkit
+│                     └── Enterprise → Gremlin
+│
+└── No infra constraint
+    └── Just need fault injection → Toxiproxy (a single-purpose tool, not full chaos framework)
+```
+
+## Chaos Toolkit
+
+**What it is:** Python-based framework. You write experiments as JSON or YAML files; the CLI runs them.
+
+**Strengths:**
+- Lightweight; runs anywhere Python runs
+- Plugin ecosystem for AWS, Azure, GCP, Kubernetes, etc.
+- JSON experiments are version-controllable
+- Apache 2 license
+
+**Weaknesses:**
+- No built-in scheduling (you bring cron / CI)
+- Smaller experiment library than Litmus
+- Plugin quality varies
+
+**Example experiment (JSON):**
+```json
+{
+  "title": "Latency on payment-svc",
+  "description": "p99 latency stays <500ms when payment is +200ms slow",
+  "steady-state-hypothesis": {
+    "title": "p99 < 500ms",
+    "probes": [{ "type": "probe", "tolerance": [0, 500],
+                 "provider": { "type": "http", "url": "https://my.dashboards/p99" } }]
+  },
+  "method": [{ "type": "action", "name": "add-latency",
+               "provider": { "type": "process", "path": "tc", "arguments": [...] } }]
+}
+```
+
+## Chaos Mesh
+
+**What it is:** Kubernetes operator + CRDs for chaos. Install in-cluster; `kubectl apply` an experiment.
+
+**Strengths:**
+- True k8s-native (no external orchestrator)
+- Comprehensive coverage: network, pod, IO, stress, time, DNS, HTTP, kernel
+- UI dashboard for running experiments
+- CNCF Incubating project
+
+**Weaknesses:**
+- k8s-only
+- CRD layout is opinionated; some types feel similar but aren't
+- Setup requires cluster admin
+
+**Example experiment (CRD):**
+```yaml
+apiVersion: chaos-mesh.org/v1alpha1
+kind: NetworkChaos
+metadata:
+  name: payment-latency
+spec:
+  action: delay
+  mode: one
+  selector:
+    namespaces: [default]
+    labelSelectors:
+      app: payment-svc
+  delay:
+    latency: 200ms
+  duration: 5m
+```
+
+## Litmus
+
+**What it is:** Kubernetes chaos framework with a large experiment library. Argo-CD integration.
+
+**Strengths:**
+- 300+ pre-built experiments
+- Strong Argo / GitOps integration
+- ChaosHub community library
+- Workflow capability for multi-step experiments
+
+**Weaknesses:**
+- More moving parts than Chaos Mesh
+- Some pre-built experiments are thin wrappers; quality varies
+- k8s-only
+
+## Gremlin
+
+**What it is:** Commercial SaaS. Agents on hosts; central control plane.
+
+**Strengths:**
+- Polished UX
+- Comprehensive attack library
+- Audit logs (compliance)
+- Multi-cloud, multi-OS
+- Customer support
+
+**Weaknesses:**
+- Paid (per-host or per-MAU)
+- Vendor lock-in
+- Less control than OSS
+
+**When to choose:** large enterprise, compliance/audit requirements, dedicated chaos team, budget exists.
+
+## AWS FIS (Fault Injection Simulator)
+
+**What it is:** AWS-managed chaos service. Templates of "actions" (stop instance, throttle API) chained into experiments.
+
+**Strengths:**
+- IAM-integrated (proper auth/audit)
+- Native to AWS services (RDS failover, ECS/EKS, Network Manager)
+- Pay-per-experiment (no agents to maintain)
+
+**Weaknesses:**
+- AWS-only
+- Smaller attack library than Chaos Mesh / Gremlin
+- Multi-account is awkward
+
+**When to choose:** AWS-heavy team that wants chaos without managing the chaos infra.
+
+## Custom (DIY)
+
+**When to choose:**
+- Single-cloud, single-stack, low complexity
+- Budget = $0
+- Have engineering capacity to maintain the tool
+- Need a niche attack type that no tool covers
+
+**Implementation patterns:**
+- Bash scripts that wrap `tc` / iptables / kill / stress-ng
+- Application-level chaos via feature flags + middleware
+- Service mesh fault injection (Istio / Linkerd) — covers many cases without a chaos framework
+
+**Trade-offs:**
+- You build all the safety rails (abort, timeout, blast-radius)
+- You build the scheduler
+- You debug your own bugs
+
+For most teams, this is a starter path; once chaos becomes regular, switch to a real tool.
+
+## Pricing rule of thumb
+
+| Tool | Typical cost (annual) |
+|---|---|
+| Chaos Toolkit | $0 |
+| Chaos Mesh | $0 |
+| Litmus OSS | $0 |
+| Litmus Enterprise | $5-30k |
+| Gremlin | $20-100k+ |
+| AWS FIS | pay-per-action, ~$100-2000/mo for active use |
+| Custom | engineering time only |
+
+## Migration paths
+
+| From | To | Effort |
+|---|---|---|
+| Custom scripts | Chaos Toolkit | Low (wrap scripts as actions) |
+| Chaos Toolkit | Chaos Mesh | Medium (k8s-only; rewrite for CRDs) |
+| Chaos Mesh | Litmus | Medium (similar shape, different CRDs) |
+| Anything | Gremlin | Easy (Gremlin imports many formats) |
+
+## Selection checklist
+
+Before committing:
+- [ ] Stack matches (k8s vs multi-cloud vs AWS-only)
+- [ ] Required attack types covered (cross-reference `attack_taxonomy.md`)
+- [ ] Audit logging requirement met (Gremlin / AWS FIS only have full audit)
+- [ ] Self-hosting requirement met (OSS only)
+- [ ] Budget approved
+- [ ] Run a 30-day proof-of-concept; verify abort path works