aigroup-workflow 2.2.1 → 2.2.2

package/skills/infra/sre-engineer/references/incident-chaos.md

@@ -1,576 +1,576 @@
-# Incident Management and Chaos Engineering
-
-## Incident Response Framework
-
-Structured approach to incident management.
-
-```python
-from dataclasses import dataclass
-from datetime import datetime
-from enum import Enum
-from typing import List
-
-class Severity(Enum):
-    """Incident severity levels."""
-    SEV1 = "critical"     # Complete outage, major customer impact
-    SEV2 = "high"         # Partial outage, significant impact
-    SEV3 = "medium"       # Degraded performance, some users affected
-    SEV4 = "low"          # Minor issue, minimal impact
-
-@dataclass
-class Incident:
-    """Incident tracking."""
-    id: str
-    title: str
-    severity: Severity
-    started_at: datetime
-    detected_at: datetime
-    resolved_at: datetime | None = None
-    root_cause: str | None = None
-    impact: str | None = None
-
-    @property
-    def detection_time(self) -> float:
-        """Time from start to detection in minutes."""
-        delta = self.detected_at - self.started_at
-        return delta.total_seconds() / 60
-
-    @property
-    def mttr(self) -> float | None:
-        """Mean Time To Repair in minutes."""
-        if not self.resolved_at:
-            return None
-        delta = self.resolved_at - self.detected_at
-        return delta.total_seconds() / 60
-
-    @property
-    def total_duration(self) -> float | None:
-        """Total incident duration in minutes."""
-        if not self.resolved_at:
-            return None
-        delta = self.resolved_at - self.started_at
-        return delta.total_seconds() / 60
-
-# Example incident
-incident = Incident(
-    id="INC-2024-001",
-    title="Database connection pool exhaustion",
-    severity=Severity.SEV2,
-    started_at=datetime(2024, 1, 15, 14, 30),
-    detected_at=datetime(2024, 1, 15, 14, 35),
-    resolved_at=datetime(2024, 1, 15, 15, 10),
-    root_cause="Connection leak in payment service",
-    impact="Payment processing delayed for 15% of users"
-)
-
-print(f"Detection time: {incident.detection_time:.1f} minutes")
-print(f"MTTR: {incident.mttr:.1f} minutes")
-print(f"Total duration: {incident.total_duration:.1f} minutes")
-```
-
-## Incident Response Runbook
-
-```yaml
-# incident_response.yaml
-incident_response:
-  detection:
-    - "Acknowledge alert in PagerDuty"
-    - "Join #incident-response Slack channel"
-    - "Create incident doc from template"
-    - "Assess severity (SEV1-4)"
-
-  sev1_response:  # Critical - all hands
-    - "Page on-call lead + backup"
-    - "Notify VP Engineering immediately"
-    - "Start Zoom war room"
-    - "Assign incident commander"
-    - "Assign communication lead"
-    - "Post status update every 15 minutes"
-
-  sev2_response:  # High - team response
-    - "Page on-call engineer"
-    - "Notify team lead"
-    - "Create incident channel"
-    - "Post status update every 30 minutes"
-
-  roles:
-    incident_commander:
-      - "Coordinate response efforts"
-      - "Make decisions quickly"
-      - "Delegate tasks"
-      - "Communicate with stakeholders"
-
-    communication_lead:
-      - "Post regular status updates"
-      - "Notify affected customers"
-      - "Update status page"
-      - "Summarize timeline"
-
-    on_call_engineer:
-      - "Investigate root cause"
-      - "Implement fixes"
-      - "Verify resolution"
-      - "Document actions taken"
-
-  resolution:
-    - "Verify metrics returned to normal"
-    - "Monitor for 30 minutes"
-    - "Post final status update"
-    - "Schedule postmortem within 48 hours"
-    - "Close incident"
-```
-
-## Blameless Postmortem Template
-
-```markdown
-# Postmortem: [Incident Title]
-
-**Date:** 2024-01-15
-**Authors:** [Names]
-**Status:** Complete
-**Severity:** SEV2
-
-## Summary
-
-One-paragraph summary of what happened, impact, and resolution.
-
-## Impact
-
-- **Duration:** 40 minutes (14:30 - 15:10 UTC)
-- **Users affected:** ~15% of payment transactions
-- **Revenue impact:** Estimated $X delayed
-- **SLO impact:** Consumed 2.3% of monthly error budget
-
-## Timeline (all times UTC)
-
-| Time  | Event |
-|-------|-------|
-| 14:30 | Deployment of payment-service v2.3.0 completed |
-| 14:32 | Error rate begins increasing |
-| 14:35 | Alert fires: HighErrorRate |
-| 14:36 | On-call engineer acknowledges |
-| 14:40 | Incident declared SEV2 |
-| 14:45 | Root cause identified: connection leak |
-| 14:50 | Rollback initiated |
-| 14:55 | Rollback completed |
-| 15:00 | Error rate returns to normal |
-| 15:10 | Incident resolved, monitoring continued |
-
-## Root Cause
-
-The payment-service v2.3.0 deployment introduced a connection leak in the
-database connection pool. The new retry logic was not properly closing
-connections on timeout, causing the pool to exhaust after ~20 minutes.
-
-## Resolution
-
-Rolled back to payment-service v2.2.1, which immediately resolved the issue.
-
-## Detection
-
-**What went well:**
-- Alert fired within 5 minutes of issue start
-- Clear runbook helped quick diagnosis
-
-**What could be improved:**
-- Could have caught in staging with longer load test
-- Database connection pool metrics not monitored
-
-## Action Items
-
-| Action | Owner | Priority | Due Date |
-|--------|-------|----------|----------|
-| Add connection pool monitoring | @alice | P0 | 2024-01-20 |
-| Extend staging load tests to 30min | @bob | P1 | 2024-01-25 |
-| Review all resource cleanup in retry logic | @charlie | P1 | 2024-01-30 |
-| Add integration test for connection leaks | @dave | P2 | 2024-02-05 |
-
-## Lessons Learned
-
-**What went well:**
-- Quick detection and response
-- Effective team communication
-- Clear rollback procedure
-
-**What didn't go well:**
-- Issue not caught in pre-production testing
-- No monitoring for connection pool exhaustion
-
-**Where we got lucky:**
-- Issue occurred during low-traffic period
-- Only affected payment service, not critical systems
-```
-
-## Chaos Engineering
-
-Proactively test system resilience through controlled failure injection.
-
-```python
-# chaos_experiment.py
-from dataclasses import dataclass
-from datetime import datetime
-from enum import Enum
-from typing import Callable
-
-class ExperimentStatus(Enum):
-    """Chaos experiment lifecycle states."""
-    PLANNED = "planned"
-    RUNNING = "running"
-    SUCCESS = "success"
-    FAILED = "failed"
-    ABORTED = "aborted"
-
-@dataclass
-class ChaosExperiment:
-    """Define a chaos engineering experiment."""
-    name: str
-    hypothesis: str  # What we expect to happen
-    blast_radius: str  # Scope of impact
-    rollback_plan: str
-    success_criteria: str
-    status: ExperimentStatus = ExperimentStatus.PLANNED
-    started_at: datetime | None = None
-    completed_at: datetime | None = None
-    observations: list[str] | None = None
-
-    def should_abort(self, metrics: dict) -> bool:
-        """Check if experiment should be aborted.
-
-        Args:
-            metrics: Current system metrics
-
-        Returns:
-            bool: True if experiment should abort
-        """
-        # Abort if error rate exceeds 10%
-        if metrics.get('error_rate', 0) > 0.10:
-            return True
-
-        # Abort if latency p99 exceeds 2 seconds
-        if metrics.get('latency_p99', 0) > 2.0:
-            return True
-
-        return False
-
-# Example: Database failover experiment
-db_failover_experiment = ChaosExperiment(
-    name="Database Primary Failover",
-    hypothesis="System automatically fails over to replica within 30s with <1% error rate",
-    blast_radius="Single database instance, 50% of production traffic",
-    rollback_plan="Restore primary database immediately, redirect traffic",
-    success_criteria="- Failover completes in <30s\n- Error rate <1%\n- No data loss",
-)
-```
-
-## Chaos Testing Patterns
-
-```python
-# chaos_patterns.py - Common chaos engineering patterns
-import time
-import random
-from typing import Protocol
-
-class ChaosInjector(Protocol):
-    """Interface for chaos injection."""
-
-    def inject(self) -> None:
-        """Inject chaos into the system."""
-        ...
-
-    def rollback(self) -> None:
-        """Remove chaos and restore normal operation."""
-        ...
-
-class LatencyInjector:
-    """Inject artificial latency into requests."""
-
-    def __init__(self, target_service: str, latency_ms: int):
-        self.target_service = target_service
-        self.latency_ms = latency_ms
-
-    def inject(self) -> None:
-        """Add latency using iptables or proxy."""
-        # Example using tc (traffic control) on Linux
-        import subprocess
-        subprocess.run([
-            "tc", "qdisc", "add", "dev", "eth0",
-            "root", "netem", "delay", f"{self.latency_ms}ms"
-        ])
-
-    def rollback(self) -> None:
-        """Remove latency."""
-        import subprocess
-        subprocess.run(["tc", "qdisc", "del", "dev", "eth0", "root"])
-
-class PodKiller:
-    """Kill pods to test resilience."""
-
-    def __init__(self, namespace: str, label_selector: str, kill_percentage: float = 0.5):
-        self.namespace = namespace
-        self.label_selector = label_selector
-        self.kill_percentage = kill_percentage
-        self.killed_pods = []
-
-    def inject(self) -> None:
-        """Randomly kill pods matching selector."""
-        import subprocess
-
-        # Get pods
-        result = subprocess.run(
-            ["kubectl", "get", "pods", "-n", self.namespace,
-             "-l", self.label_selector, "-o", "name"],
-            capture_output=True,
-            text=True
-        )
-
-        pods = result.stdout.strip().split('\n')
-        num_to_kill = int(len(pods) * self.kill_percentage)
-        pods_to_kill = random.sample(pods, num_to_kill)
-
-        # Kill selected pods
-        for pod in pods_to_kill:
-            subprocess.run(["kubectl", "delete", pod, "-n", self.namespace])
-            self.killed_pods.append(pod)
-
-    def rollback(self) -> None:
-        """Pods will be recreated by deployment controller."""
-        # Wait for pods to be recreated
-        time.sleep(30)
-
-class NetworkPartition:
-    """Simulate network partition between services."""
-
-    def __init__(self, source_pod: str, target_service: str):
-        self.source_pod = source_pod
-        self.target_service = target_service
-
-    def inject(self) -> None:
-        """Block network traffic using iptables."""
-        import subprocess
-        subprocess.run([
-            "kubectl", "exec", self.source_pod, "--",
-            "iptables", "-A", "OUTPUT", "-d", self.target_service, "-j", "DROP"
-        ])
-
-    def rollback(self) -> None:
-        """Restore network traffic."""
-        import subprocess
-        subprocess.run([
-            "kubectl", "exec", self.source_pod, "--",
-            "iptables", "-D", "OUTPUT", "-d", self.target_service, "-j", "DROP"
-        ])
-```
-
-## Chaos Experiment Runner
-
-```python
-# chaos_runner.py - Safe chaos experiment execution
-from dataclasses import dataclass
-from datetime import datetime, timedelta
-import time
-
-@dataclass
-class SafetyConstraints:
-    """Safety constraints for chaos experiments."""
-    max_error_rate: float = 0.10  # 10%
-    max_latency_p99: float = 2.0  # 2 seconds
-    max_duration_minutes: int = 15
-    business_hours_only: bool = True
-
-class ChaosRunner:
-    """Safely execute chaos experiments with monitoring."""
-
-    def __init__(self, safety: SafetyConstraints):
-        self.safety = safety
-
-    def run_experiment(
-        self,
-        experiment: ChaosExperiment,
-        injector: ChaosInjector,
-        get_metrics: Callable[[], dict],
-    ) -> ChaosExperiment:
-        """Execute chaos experiment safely.
-
-        Args:
-            experiment: Experiment definition
-            injector: Chaos injector implementation
-            get_metrics: Function to fetch current metrics
-
-        Returns:
-            Updated experiment with results
-        """
-        # Pre-flight checks
-        if self.safety.business_hours_only:
-            current_hour = datetime.now().hour
-            if 9 <= current_hour <= 17:  # Business hours
-                experiment.status = ExperimentStatus.ABORTED
-                experiment.observations = ["Aborted: Business hours constraint"]
-                return experiment
-
-        # Baseline metrics
-        baseline_metrics = get_metrics()
-        print(f"Baseline metrics: {baseline_metrics}")
-
-        # Start experiment
-        experiment.status = ExperimentStatus.RUNNING
-        experiment.started_at = datetime.now()
-        experiment.observations = []
-
-        try:
-            # Inject chaos
-            print(f"Injecting chaos: {experiment.name}")
-            injector.inject()
-
-            # Monitor for max duration
-            start_time = datetime.now()
-            max_duration = timedelta(minutes=self.safety.max_duration_minutes)
-
-            while datetime.now() - start_time < max_duration:
-                time.sleep(10)  # Check every 10 seconds
-
-                current_metrics = get_metrics()
-                experiment.observations.append(
-                    f"{datetime.now().isoformat()}: {current_metrics}"
-                )
-
-                # Check safety constraints
-                if experiment.should_abort(current_metrics):
-                    print("ABORTING: Safety constraint violated")
-                    experiment.status = ExperimentStatus.ABORTED
-                    break
-
-            else:
-                # Completed successfully
-                experiment.status = ExperimentStatus.SUCCESS
-
-        except Exception as e:
-            print(f"ERROR: {e}")
-            experiment.status = ExperimentStatus.FAILED
-            experiment.observations.append(f"Exception: {str(e)}")
-
-        finally:
-            # Always rollback
-            print("Rolling back chaos injection")
-            injector.rollback()
-            experiment.completed_at = datetime.now()
-
-        return experiment
-
-# Example usage
-def get_current_metrics() -> dict:
-    """Fetch metrics from Prometheus."""
-    # In real implementation, query Prometheus
-    return {
-        'error_rate': 0.02,  # 2%
-        'latency_p99': 0.45,  # 450ms
-    }
-
-safety = SafetyConstraints(business_hours_only=False)
-runner = ChaosRunner(safety)
-
-experiment = ChaosExperiment(
-    name="Kill 50% of API pods",
-    hypothesis="API remains available with 50% pod loss",
-    blast_radius="50% of API pods",
-    rollback_plan="Pods auto-restart via deployment",
-    success_criteria="Error rate <5%, latency p99 <1s",
-)
-
-injector = PodKiller(
-    namespace="production",
-    label_selector="app=api",
-    kill_percentage=0.5,
-)
-
-result = runner.run_experiment(experiment, injector, get_current_metrics)
-print(f"Experiment status: {result.status.value}")
-```
-
-## Game Days
-
-Scheduled chaos engineering practice sessions.
-
-```yaml
-# gameday_plan.yaml
-gameday:
-  date: "2024-02-15"
-  duration: "2 hours"
-  participants:
-    - SRE team
-    - Backend engineers
-    - On-call rotation
-
-  objectives:
-    - Test incident response procedures
-    - Validate monitoring and alerting
-    - Practice communication protocols
-    - Identify gaps in runbooks
-
-  scenarios:
-    - scenario: "Database Primary Failure"
-      inject: "Terminate primary database pod"
-      expected: "Automatic failover to replica in <30s"
-
-    - scenario: "API Service Overload"
-      inject: "Generate 10x normal traffic"
-      expected: "Rate limiting activates, no errors"
-
-    - scenario: "Network Partition"
-      inject: "Block traffic between API and database"
-      expected: "Circuit breaker opens, graceful degradation"
-
-  success_criteria:
-    - "All scenarios handled without escalation"
-    - "MTTR <30 minutes for all scenarios"
-    - "Documentation updated with learnings"
-    - "Action items created for gaps"
-
-  safety:
-    - "Run in staging environment first"
-    - "VP Engineering notified beforehand"
-    - "Abort plan ready for each scenario"
-    - "Customer support team on standby"
-```
-
-## Chaos Engineering Maturity
-
-```python
-from enum import IntEnum
-
-class ChaosMaturity(IntEnum):
-    """Chaos engineering maturity levels."""
-    NONE = 0          # No chaos testing
-    AD_HOC = 1        # Occasional manual tests
-    SCHEDULED = 2     # Regular game days
-    CONTINUOUS = 3    # Automated in CI/CD
-    CULTURE = 4       # Embedded in development
-
-def assess_maturity(practices: dict[str, bool]) -> ChaosMaturity:
-    """Assess chaos engineering maturity level."""
-
-    if not practices.get('any_chaos_testing'):
-        return ChaosMaturity.NONE
-
-    if not practices.get('regular_game_days'):
-        return ChaosMaturity.AD_HOC
-
-    if not practices.get('automated_chaos'):
-        return ChaosMaturity.SCHEDULED
-
-    if not practices.get('chaos_in_cicd'):
-        return ChaosMaturity.CONTINUOUS
-
-    return ChaosMaturity.CULTURE
-
-# Example assessment
-current_practices = {
-    'any_chaos_testing': True,
-    'regular_game_days': True,
-    'automated_chaos': True,
-    'chaos_in_cicd': False,
-}
-
-maturity = assess_maturity(current_practices)
-print(f"Chaos maturity: {maturity.name}")
-# Target: CONTINUOUS or CULTURE
-```
+# Incident Management and Chaos Engineering
+
+## Incident Response Framework
+
+Structured approach to incident management.
+
+```python
+from dataclasses import dataclass
+from datetime import datetime
+from enum import Enum
+from typing import List
+
+class Severity(Enum):
+    """Incident severity levels."""
+    SEV1 = "critical"     # Complete outage, major customer impact
+    SEV2 = "high"         # Partial outage, significant impact
+    SEV3 = "medium"       # Degraded performance, some users affected
+    SEV4 = "low"          # Minor issue, minimal impact
+
+@dataclass
+class Incident:
+    """Incident tracking."""
+    id: str
+    title: str
+    severity: Severity
+    started_at: datetime
+    detected_at: datetime
+    resolved_at: datetime | None = None
+    root_cause: str | None = None
+    impact: str | None = None
+
+    @property
+    def detection_time(self) -> float:
+        """Time from start to detection in minutes."""
+        delta = self.detected_at - self.started_at
+        return delta.total_seconds() / 60
+
+    @property
+    def mttr(self) -> float | None:
+        """Mean Time To Repair in minutes."""
+        if not self.resolved_at:
+            return None
+        delta = self.resolved_at - self.detected_at
+        return delta.total_seconds() / 60
+
+    @property
+    def total_duration(self) -> float | None:
+        """Total incident duration in minutes."""
+        if not self.resolved_at:
+            return None
+        delta = self.resolved_at - self.started_at
+        return delta.total_seconds() / 60
+
+# Example incident
+incident = Incident(
+    id="INC-2024-001",
+    title="Database connection pool exhaustion",
+    severity=Severity.SEV2,
+    started_at=datetime(2024, 1, 15, 14, 30),
+    detected_at=datetime(2024, 1, 15, 14, 35),
+    resolved_at=datetime(2024, 1, 15, 15, 10),
+    root_cause="Connection leak in payment service",
+    impact="Payment processing delayed for 15% of users"
+)
+
+print(f"Detection time: {incident.detection_time:.1f} minutes")
+print(f"MTTR: {incident.mttr:.1f} minutes")
+print(f"Total duration: {incident.total_duration:.1f} minutes")
+```
+
+## Incident Response Runbook
+
+```yaml
+# incident_response.yaml
+incident_response:
+  detection:
+    - "Acknowledge alert in PagerDuty"
+    - "Join #incident-response Slack channel"
+    - "Create incident doc from template"
+    - "Assess severity (SEV1-4)"
+
+  sev1_response:  # Critical - all hands
+    - "Page on-call lead + backup"
+    - "Notify VP Engineering immediately"
+    - "Start Zoom war room"
+    - "Assign incident commander"
+    - "Assign communication lead"
+    - "Post status update every 15 minutes"
+
+  sev2_response:  # High - team response
+    - "Page on-call engineer"
+    - "Notify team lead"
+    - "Create incident channel"
+    - "Post status update every 30 minutes"
+
+  roles:
+    incident_commander:
+      - "Coordinate response efforts"
+      - "Make decisions quickly"
+      - "Delegate tasks"
+      - "Communicate with stakeholders"
+
+    communication_lead:
+      - "Post regular status updates"
+      - "Notify affected customers"
+      - "Update status page"
+      - "Summarize timeline"
+
+    on_call_engineer:
+      - "Investigate root cause"
+      - "Implement fixes"
+      - "Verify resolution"
+      - "Document actions taken"
+
+  resolution:
+    - "Verify metrics returned to normal"
+    - "Monitor for 30 minutes"
+    - "Post final status update"
+    - "Schedule postmortem within 48 hours"
+    - "Close incident"
+```
+
+## Blameless Postmortem Template
+
+```markdown
+# Postmortem: [Incident Title]
+
+**Date:** 2024-01-15
+**Authors:** [Names]
+**Status:** Complete
+**Severity:** SEV2
+
+## Summary
+
+One-paragraph summary of what happened, impact, and resolution.
+
+## Impact
+
+- **Duration:** 40 minutes (14:30 - 15:10 UTC)
+- **Users affected:** ~15% of payment transactions
+- **Revenue impact:** Estimated $X delayed
+- **SLO impact:** Consumed 2.3% of monthly error budget
+
+## Timeline (all times UTC)
+
+| Time  | Event |
+|-------|-------|
+| 14:30 | Deployment of payment-service v2.3.0 completed |
+| 14:32 | Error rate begins increasing |
+| 14:35 | Alert fires: HighErrorRate |
+| 14:36 | On-call engineer acknowledges |
+| 14:40 | Incident declared SEV2 |
+| 14:45 | Root cause identified: connection leak |
+| 14:50 | Rollback initiated |
+| 14:55 | Rollback completed |
+| 15:00 | Error rate returns to normal |
+| 15:10 | Incident resolved, monitoring continued |
+
+## Root Cause
+
+The payment-service v2.3.0 deployment introduced a connection leak in the
+database connection pool. The new retry logic was not properly closing
+connections on timeout, causing the pool to exhaust after ~20 minutes.
+
+## Resolution
+
+Rolled back to payment-service v2.2.1, which immediately resolved the issue.
+
+## Detection
+
+**What went well:**
+- Alert fired within 5 minutes of issue start
+- Clear runbook helped quick diagnosis
+
+**What could be improved:**
+- Could have caught in staging with longer load test
+- Database connection pool metrics not monitored
+
+## Action Items
+
+| Action | Owner | Priority | Due Date |
+|--------|-------|----------|----------|
+| Add connection pool monitoring | @alice | P0 | 2024-01-20 |
+| Extend staging load tests to 30min | @bob | P1 | 2024-01-25 |
+| Review all resource cleanup in retry logic | @charlie | P1 | 2024-01-30 |
+| Add integration test for connection leaks | @dave | P2 | 2024-02-05 |
+
+## Lessons Learned
+
+**What went well:**
+- Quick detection and response
+- Effective team communication
+- Clear rollback procedure
+
+**What didn't go well:**
+- Issue not caught in pre-production testing
+- No monitoring for connection pool exhaustion
+
+**Where we got lucky:**
+- Issue occurred during low-traffic period
+- Only affected payment service, not critical systems
+```
+
+## Chaos Engineering
+
+Proactively test system resilience through controlled failure injection.
+
+```python
+# chaos_experiment.py
+from dataclasses import dataclass
+from datetime import datetime
+from enum import Enum
+from typing import Callable
+
+class ExperimentStatus(Enum):
+    """Chaos experiment lifecycle states."""
+    PLANNED = "planned"
+    RUNNING = "running"
+    SUCCESS = "success"
+    FAILED = "failed"
+    ABORTED = "aborted"
+
+@dataclass
+class ChaosExperiment:
+    """Define a chaos engineering experiment."""
+    name: str
+    hypothesis: str  # What we expect to happen
+    blast_radius: str  # Scope of impact
+    rollback_plan: str
+    success_criteria: str
+    status: ExperimentStatus = ExperimentStatus.PLANNED
+    started_at: datetime | None = None
+    completed_at: datetime | None = None
+    observations: list[str] | None = None
+
+    def should_abort(self, metrics: dict) -> bool:
+        """Check if experiment should be aborted.
+
+        Args:
+            metrics: Current system metrics
+
+        Returns:
+            bool: True if experiment should abort
+        """
+        # Abort if error rate exceeds 10%
+        if metrics.get('error_rate', 0) > 0.10:
+            return True
+
+        # Abort if latency p99 exceeds 2 seconds
+        if metrics.get('latency_p99', 0) > 2.0:
+            return True
+
+        return False
+
+# Example: Database failover experiment
+db_failover_experiment = ChaosExperiment(
+    name="Database Primary Failover",
+    hypothesis="System automatically fails over to replica within 30s with <1% error rate",
+    blast_radius="Single database instance, 50% of production traffic",
+    rollback_plan="Restore primary database immediately, redirect traffic",
+    success_criteria="- Failover completes in <30s\n- Error rate <1%\n- No data loss",
+)
+```
+
+## Chaos Testing Patterns
+
+```python
+# chaos_patterns.py - Common chaos engineering patterns
+import time
+import random
+from typing import Protocol
+
+class ChaosInjector(Protocol):
+    """Interface for chaos injection."""
+
+    def inject(self) -> None:
+        """Inject chaos into the system."""
+        ...
+
+    def rollback(self) -> None:
+        """Remove chaos and restore normal operation."""
+        ...
+
+class LatencyInjector:
+    """Inject artificial latency into requests."""
+
+    def __init__(self, target_service: str, latency_ms: int):
+        self.target_service = target_service
+        self.latency_ms = latency_ms
+
+    def inject(self) -> None:
+        """Add latency using iptables or proxy."""
+        # Example using tc (traffic control) on Linux
+        import subprocess
+        subprocess.run([
+            "tc", "qdisc", "add", "dev", "eth0",
+            "root", "netem", "delay", f"{self.latency_ms}ms"
+        ])
+
+    def rollback(self) -> None:
+        """Remove latency."""
+        import subprocess
+        subprocess.run(["tc", "qdisc", "del", "dev", "eth0", "root"])
+
+class PodKiller:
+    """Kill pods to test resilience."""
+
+    def __init__(self, namespace: str, label_selector: str, kill_percentage: float = 0.5):
+        self.namespace = namespace
+        self.label_selector = label_selector
+        self.kill_percentage = kill_percentage
+        self.killed_pods = []
+
+    def inject(self) -> None:
+        """Randomly kill pods matching selector."""
+        import subprocess
+
+        # Get pods
+        result = subprocess.run(
+            ["kubectl", "get", "pods", "-n", self.namespace,
+             "-l", self.label_selector, "-o", "name"],
+            capture_output=True,
+            text=True
+        )
+
+        pods = result.stdout.strip().split('\n')
+        num_to_kill = int(len(pods) * self.kill_percentage)
+        pods_to_kill = random.sample(pods, num_to_kill)
+
+        # Kill selected pods
+        for pod in pods_to_kill:
+            subprocess.run(["kubectl", "delete", pod, "-n", self.namespace])
+            self.killed_pods.append(pod)
+
+    def rollback(self) -> None:
+        """Pods will be recreated by deployment controller."""
+        # Wait for pods to be recreated
+        time.sleep(30)
+
+class NetworkPartition:
+    """Simulate network partition between services."""
+
+    def __init__(self, source_pod: str, target_service: str):
+        self.source_pod = source_pod
+        self.target_service = target_service
+
+    def inject(self) -> None:
+        """Block network traffic using iptables."""
+        import subprocess
+        subprocess.run([
+            "kubectl", "exec", self.source_pod, "--",
+            "iptables", "-A", "OUTPUT", "-d", self.target_service, "-j", "DROP"
+        ])
+
+    def rollback(self) -> None:
+        """Restore network traffic."""
+        import subprocess
+        subprocess.run([
+            "kubectl", "exec", self.source_pod, "--",
+            "iptables", "-D", "OUTPUT", "-d", self.target_service, "-j", "DROP"
+        ])
+```
+
+## Chaos Experiment Runner
+
+```python
+# chaos_runner.py - Safe chaos experiment execution
+from dataclasses import dataclass
+from datetime import datetime, timedelta
+import time
+
+@dataclass
+class SafetyConstraints:
+    """Safety constraints for chaos experiments."""
+    max_error_rate: float = 0.10  # 10%
+    max_latency_p99: float = 2.0  # 2 seconds
+    max_duration_minutes: int = 15
+    business_hours_only: bool = True
+
+class ChaosRunner:
+    """Safely execute chaos experiments with monitoring."""
+
+    def __init__(self, safety: SafetyConstraints):
+        self.safety = safety
+
+    def run_experiment(
+        self,
+        experiment: ChaosExperiment,
+        injector: ChaosInjector,
+        get_metrics: Callable[[], dict],
+    ) -> ChaosExperiment:
+        """Execute chaos experiment safely.
+
+        Args:
+            experiment: Experiment definition
+            injector: Chaos injector implementation
+            get_metrics: Function to fetch current metrics
+
+        Returns:
+            Updated experiment with results
+        """
+        # Pre-flight checks
+        if self.safety.business_hours_only:
+            current_hour = datetime.now().hour
+            if 9 <= current_hour <= 17:  # Business hours
+                experiment.status = ExperimentStatus.ABORTED
+                experiment.observations = ["Aborted: Business hours constraint"]
+                return experiment
+
+        # Baseline metrics
+        baseline_metrics = get_metrics()
+        print(f"Baseline metrics: {baseline_metrics}")
+
+        # Start experiment
+        experiment.status = ExperimentStatus.RUNNING
+        experiment.started_at = datetime.now()
+        experiment.observations = []
+
+        try:
+            # Inject chaos
+            print(f"Injecting chaos: {experiment.name}")
+            injector.inject()
+
+            # Monitor for max duration
+            start_time = datetime.now()
+            max_duration = timedelta(minutes=self.safety.max_duration_minutes)
+
+            while datetime.now() - start_time < max_duration:
+                time.sleep(10)  # Check every 10 seconds
+
+                current_metrics = get_metrics()
+                experiment.observations.append(
+                    f"{datetime.now().isoformat()}: {current_metrics}"
+                )
+
+                # Check safety constraints
+                if experiment.should_abort(current_metrics):
+                    print("ABORTING: Safety constraint violated")
+                    experiment.status = ExperimentStatus.ABORTED
+                    break
+
+            else:
+                # Completed successfully
+                experiment.status = ExperimentStatus.SUCCESS
+
+        except Exception as e:
+            print(f"ERROR: {e}")
+            experiment.status = ExperimentStatus.FAILED
+            experiment.observations.append(f"Exception: {str(e)}")
+
+        finally:
+            # Always rollback
+            print("Rolling back chaos injection")
+            injector.rollback()
+            experiment.completed_at = datetime.now()
+
+        return experiment
+
+# Example usage
+def get_current_metrics() -> dict:
+    """Fetch metrics from Prometheus."""
+    # In real implementation, query Prometheus
+    return {
+        'error_rate': 0.02,  # 2%
+        'latency_p99': 0.45,  # 450ms
+    }
+
+safety = SafetyConstraints(business_hours_only=False)
+runner = ChaosRunner(safety)
+
+experiment = ChaosExperiment(
+    name="Kill 50% of API pods",
+    hypothesis="API remains available with 50% pod loss",
+    blast_radius="50% of API pods",
+    rollback_plan="Pods auto-restart via deployment",
+    success_criteria="Error rate <5%, latency p99 <1s",
+)
+
+injector = PodKiller(
+    namespace="production",
+    label_selector="app=api",
+    kill_percentage=0.5,
+)
+
+result = runner.run_experiment(experiment, injector, get_current_metrics)
+print(f"Experiment status: {result.status.value}")
+```
+
+## Game Days
+
+Scheduled chaos engineering practice sessions.
+
+```yaml
+# gameday_plan.yaml
+gameday:
+  date: "2024-02-15"
+  duration: "2 hours"
+  participants:
+    - SRE team
+    - Backend engineers
+    - On-call rotation
+
+  objectives:
+    - Test incident response procedures
+    - Validate monitoring and alerting
+    - Practice communication protocols
+    - Identify gaps in runbooks
+
+  scenarios:
+    - scenario: "Database Primary Failure"
+      inject: "Terminate primary database pod"
+      expected: "Automatic failover to replica in <30s"
+
+    - scenario: "API Service Overload"
+      inject: "Generate 10x normal traffic"
+      expected: "Rate limiting activates, no errors"
+
+    - scenario: "Network Partition"
+      inject: "Block traffic between API and database"
+      expected: "Circuit breaker opens, graceful degradation"
+
+  success_criteria:
+    - "All scenarios handled without escalation"
+    - "MTTR <30 minutes for all scenarios"
+    - "Documentation updated with learnings"
+    - "Action items created for gaps"
+
+  safety:
+    - "Run in staging environment first"
+    - "VP Engineering notified beforehand"
+    - "Abort plan ready for each scenario"
+    - "Customer support team on standby"
+```
+
+## Chaos Engineering Maturity
+
+```python
+from enum import IntEnum
+
+class ChaosMaturity(IntEnum):
+    """Chaos engineering maturity levels."""
+    NONE = 0          # No chaos testing
+    AD_HOC = 1        # Occasional manual tests
+    SCHEDULED = 2     # Regular game days
+    CONTINUOUS = 3    # Automated in CI/CD
+    CULTURE = 4       # Embedded in development
+
+def assess_maturity(practices: dict[str, bool]) -> ChaosMaturity:
+    """Assess chaos engineering maturity level."""
+
+    if not practices.get('any_chaos_testing'):
+        return ChaosMaturity.NONE
+
+    if not practices.get('regular_game_days'):
+        return ChaosMaturity.AD_HOC
+
+    if not practices.get('automated_chaos'):
+        return ChaosMaturity.SCHEDULED
+
+    if not practices.get('chaos_in_cicd'):
+        return ChaosMaturity.CONTINUOUS
+
+    return ChaosMaturity.CULTURE
+
+# Example assessment
+current_practices = {
+    'any_chaos_testing': True,
+    'regular_game_days': True,
+    'automated_chaos': True,
+    'chaos_in_cicd': False,
+}
+
+maturity = assess_maturity(current_practices)
+print(f"Chaos maturity: {maturity.name}")
+# Target: CONTINUOUS or CULTURE
+```