aigroup-workflow 2.1.1 → 2.2.0

package/skills/quality/chaos-engineer/references/experiment-design.md

@@ -0,0 +1,229 @@
+# Chaos Experiment Design
+
+## Experiment Template
+
+```yaml
+name: "Database Connection Pool Exhaustion"
+hypothesis: "When the database connection pool is exhausted, the application will gracefully degrade and return 503 errors without cascading failures"
+
+steady_state:
+  metrics:
+    - name: "Error Rate"
+      threshold: "< 0.1%"
+      source: "prometheus"
+      query: "rate(http_requests_total{status=~'5..'}[5m])"
+    - name: "Latency P99"
+      threshold: "< 500ms"
+      source: "datadog"
+    - name: "Active Connections"
+      threshold: "> 10"
+      query: "pg_stat_activity_count"
+
+blast_radius:
+  environment: "staging"
+  traffic_percentage: 10
+  duration_seconds: 300
+  max_error_rate: "5%"
+  auto_rollback: true
+
+injection:
+  type: "resource_exhaustion"
+  target: "database_connections"
+  method: "connection_leak"
+  parameters:
+    leak_rate: 5  # connections per second
+    max_leaked: 50
+
+safety:
+  rollback_triggers:
+    - "error_rate > 5%"
+    - "manual_kill_switch"
+    - "duration_exceeded"
+  rollback_time_limit_seconds: 30
+  alerts:
+    - slack: "#chaos-engineering"
+    - pagerduty: "chaos-team"
+
+success_criteria:
+  - "Circuit breakers activate within 10s"
+  - "503 errors returned (not 500)"
+  - "No cascading failures to other services"
+  - "System recovers within 60s of rollback"
+```
+
+## Hypothesis Formulation
+
+```python
+def create_hypothesis(component: str, failure: str, expected_behavior: str) -> dict:
+    """
+    Create well-formed chaos hypothesis.
+
+    Format: "Given [normal state], when [failure occurs],
+             then [expected behavior], measured by [metrics]"
+    """
+    return {
+        "given": f"System is in steady state with {component} functioning normally",
+        "when": f"{failure} occurs",
+        "then": expected_behavior,
+        "measured_by": [
+            "Error rate remains below threshold",
+            "Latency stays within SLO",
+            "No data loss or corruption",
+            "Recovery time within RTO"
+        ]
+    }
+
+# Example
+hypothesis = create_hypothesis(
+    component="payment service",
+    failure="50% packet loss to payment gateway",
+    expected_behavior="Requests timeout gracefully, retry queue activates, "
+                     "users see clear error messages"
+)
+```
+
+## Blast Radius Control
+
+```python
+from dataclasses import dataclass
+from enum import Enum
+
+class BlastRadiusLevel(Enum):
+    MINIMAL = "single_instance_dev"
+    LOW = "single_instance_staging"
+    MEDIUM = "percentage_staging"
+    HIGH = "percentage_production"
+    CRITICAL = "full_production"
+
+@dataclass
+class BlastRadiusConfig:
+    level: BlastRadiusLevel
+    environment: str
+    target_percentage: float  # 0-100
+    canary_users: list[str]
+    feature_flag: str
+    auto_rollback: bool
+    max_duration_seconds: int
+
+    def validate(self):
+        """Enforce safety rules."""
+        if self.level == BlastRadiusLevel.CRITICAL:
+            raise ValueError("CRITICAL blast radius requires explicit approval")
+
+        if self.environment == "production" and self.target_percentage > 10:
+            if not self.feature_flag or not self.auto_rollback:
+                raise ValueError("Production >10% requires feature flag AND auto-rollback")
+
+        if self.max_duration_seconds > 600:
+            raise ValueError("Max duration cannot exceed 10 minutes without approval")
+
+# Progressive blast radius expansion
+def progressive_rollout() -> list[BlastRadiusConfig]:
+    return [
+        BlastRadiusConfig(
+            level=BlastRadiusLevel.MINIMAL,
+            environment="dev",
+            target_percentage=100,
+            canary_users=[],
+            feature_flag="chaos_dev",
+            auto_rollback=True,
+            max_duration_seconds=300
+        ),
+        BlastRadiusConfig(
+            level=BlastRadiusLevel.LOW,
+            environment="staging",
+            target_percentage=100,
+            canary_users=[],
+            feature_flag="chaos_staging",
+            auto_rollback=True,
+            max_duration_seconds=600
+        ),
+        BlastRadiusConfig(
+            level=BlastRadiusLevel.MEDIUM,
+            environment="production",
+            target_percentage=1,
+            canary_users=["internal_team"],
+            feature_flag="chaos_prod_canary",
+            auto_rollback=True,
+            max_duration_seconds=300
+        )
+    ]
+```
+
+## Safety Mechanisms
+
+```python
+import asyncio
+from typing import Callable
+
+class ChaosExperimentSafety:
+    def __init__(self, config: dict):
+        self.config = config
+        self.kill_switch_active = False
+        self.metrics = {}
+
+    async def run_with_safety(self, chaos_fn: Callable):
+        """Execute chaos with automatic safety checks."""
+        # Pre-flight checks
+        if not await self.verify_steady_state():
+            raise Exception("System not in steady state - aborting")
+
+        # Set up rollback trigger
+        rollback_task = asyncio.create_task(self.monitor_for_rollback())
+        chaos_task = asyncio.create_task(chaos_fn())
+
+        try:
+            # Wait for either chaos completion or rollback trigger
+            done, pending = await asyncio.wait(
+                [chaos_task, rollback_task],
+                return_when=asyncio.FIRST_COMPLETED
+            )
+
+            if rollback_task in done:
+                # Rollback triggered - cancel chaos
+                chaos_task.cancel()
+                await self.rollback()
+
+        finally:
+            await self.ensure_system_recovery()
+
+    async def verify_steady_state(self) -> bool:
+        """Check all steady state metrics are within threshold."""
+        for metric in self.config['steady_state']['metrics']:
+            value = await self.query_metric(metric['query'])
+            if not self.within_threshold(value, metric['threshold']):
+                return False
+        return True
+
+    async def monitor_for_rollback(self):
+        """Continuously monitor for rollback triggers."""
+        start_time = asyncio.get_event_loop().time()
+
+        while True:
+            # Check duration limit
+            if asyncio.get_event_loop().time() - start_time > \
+               self.config['blast_radius']['duration_seconds']:
+                return "duration_exceeded"
+
+            # Check manual kill switch
+            if self.kill_switch_active:
+                return "manual_kill_switch"
+
+            # Check error rate
+            error_rate = await self.query_metric("error_rate")
+            if error_rate > float(self.config['blast_radius']['max_error_rate'].strip('%')):
+                return "error_rate_exceeded"
+
+            await asyncio.sleep(5)  # Check every 5 seconds
+```
+
+## Quick Reference
+
+| Phase | Key Actions | Time Limit |
+|-------|-------------|------------|
+| Design | Hypothesis, metrics, blast radius | 1 hour |
+| Review | Team review, safety check | 30 min |
+| Prepare | Setup monitoring, rollback | 1 hour |
+| Execute | Run experiment, monitor | 5-10 min |
+| Rollback | Restore steady state | < 30 sec |
+| Learn | Document findings, plan fixes | 2 hours |

package/skills/quality/chaos-engineer/references/game-days.md

@@ -0,0 +1,434 @@
+# Game Day Planning & Execution
+
+## Game Day Planning Template
+
+```yaml
+game_day:
+  name: "Database Failover Drill"
+  date: "2025-01-15"
+  time: "10:00-12:00 PST"
+  environment: "staging"  # Start in staging
+
+  objectives:
+    - "Verify RDS failover to standby in under 2 minutes"
+    - "Validate application auto-reconnect logic"
+    - "Test monitoring and alerting effectiveness"
+    - "Practice incident response procedures"
+
+  participants:
+    facilitator: "chaos-engineer@company.com"
+    observers:
+      - "sre-team@company.com"
+      - "dev-team@company.com"
+    responders:
+      - "on-call-engineer@company.com"
+      - "database-admin@company.com"
+    stakeholders:
+      - "engineering-manager@company.com"
+
+  scenarios:
+    - name: "Primary database instance failure"
+      duration_minutes: 30
+      steps:
+        - action: "Force RDS instance reboot with failover"
+          expected: "Failover to standby in <2 min"
+          success_criteria:
+            - "Downtime < 2 minutes"
+            - "No data loss"
+            - "Alerts fired correctly"
+
+    - name: "Network partition to database"
+      duration_minutes: 20
+      steps:
+        - action: "Block network traffic to RDS security group"
+          expected: "Application switches to read replica"
+          success_criteria:
+            - "Read-only mode activated"
+            - "User-facing error messages clear"
+
+  communication_plan:
+    announcement_channel: "#game-day-announcements"
+    war_room: "Zoom link: https://..."
+    status_updates_every: "5 minutes"
+    escalation_contacts:
+      - name: "VP Engineering"
+        phone: "+1-555-0100"
+        threshold: "downtime > 5 minutes"
+
+  rollback_plan:
+    automatic_rollback_triggers:
+      - "production traffic affected"
+      - "customer complaints received"
+      - "error_rate > 10%"
+    manual_rollback_command: "aws rds reboot-db-instance --db-instance-identifier primary --force-failover"
+    rollback_time_limit_seconds: 60
+
+  success_metrics:
+    - metric: "RTO (Recovery Time Objective)"
+      target: "< 2 minutes"
+      measurement: "time between failure and full recovery"
+    - metric: "Alert accuracy"
+      target: "100%"
+      measurement: "all expected alerts fired"
+    - metric: "Team response time"
+      target: "< 5 minutes"
+      measurement: "time to acknowledge incident"
+
+  post_mortem:
+    scheduled_for: "2025-01-16 14:00"
+    template: "game-day-retro.md"
+    required_attendees: "all participants"
+```
+
+## Game Day Runbook
+
+```markdown
+# Database Failover Game Day Runbook
+
+**Date**: January 15, 2025
+**Duration**: 2 hours
+**Environment**: Staging
+
+## Pre-Game Checklist (T-30 min)
+
+- [ ] Verify all participants joined war room
+- [ ] Confirm monitoring dashboards accessible
+- [ ] Test rollback procedures work
+- [ ] Announce game day start in #engineering
+- [ ] Verify staging environment healthy
+- [ ] Set up screen recording for timeline
+- [ ] Prepare incident timeline spreadsheet
+
+## Timeline
+
+### 10:00 - Introduction (10 min)
+- Facilitator explains objectives
+- Review scenarios and success criteria
+- Confirm roles and communication channels
+- Remind everyone: this is a learning exercise
+
+### 10:10 - Scenario 1: Primary DB Failure (30 min)
+
+**T+0 (10:10)** - Inject failure
+```bash
+aws rds reboot-db-instance \
+  --db-instance-identifier staging-primary \
+  --force-failover
+```
+
+**Expected Timeline**:
+- T+0: Reboot initiated
+- T+30s: Primary becomes unavailable
+- T+60s: DNS updated to standby
+- T+90s: Application reconnects
+- T+120s: Full recovery
+
+**Observer Tasks**:
+- [ ] Record exact time of failure injection
+- [ ] Monitor application error logs
+- [ ] Track alert notifications
+- [ ] Document team response actions
+- [ ] Screenshot dashboard states
+
+**Questions to Answer**:
+- How long until first alert?
+- Did application auto-reconnect?
+- Were customers impacted?
+- What manual interventions needed?
+
+### 10:40 - Debrief Scenario 1 (10 min)
+- What went well?
+- What could improve?
+- Any surprises?
+- Action items identified
+
+### 10:50 - Scenario 2: Network Partition (20 min)
+
+**T+0 (10:50)** - Inject failure
+```bash
+# Block database security group ingress
+aws ec2 revoke-security-group-ingress \
+  --group-id sg-xxxxx \
+  --protocol tcp \
+  --port 5432 \
+  --cidr 10.0.0.0/16
+```
+
+**Expected Behavior**:
+- Connection timeouts occur
+- Circuit breaker opens
+- Read-only mode activates
+- Clear error messages shown
+
+**Observer Tasks**:
+- [ ] Monitor circuit breaker state
+- [ ] Verify read-replica failover
+- [ ] Check user-facing error messages
+- [ ] Track degraded service duration
+
+### 11:10 - Debrief Scenario 2 (10 min)
+
+### 11:20 - Scenario 3: Surprise! (20 min)
+
+**Facilitator Note**: Don't announce this scenario details beforehand.
+Test true incident response capability.
+
+**Hidden Scenario**: Combination failure
+1. Database connection pool leak
+2. Simultaneous cache invalidation
+
+```python
+# Connection leak simulator
+import psycopg2
+connections = []
+for i in range(100):
+    conn = psycopg2.connect(DATABASE_URL)
+    connections.append(conn)
+    # Intentionally don't close
+```
+
+**Observer Tasks**:
+- [ ] How long to identify root cause?
+- [ ] Communication effectiveness
+- [ ] Cross-team coordination
+- [ ] Escalation decisions
+
+### 11:40 - Final Debrief & Wrap-up (20 min)
+
+**Debrief Questions**:
+1. What worked well?
+2. What didn't work?
+3. What surprised us?
+4. What are our top 3 action items?
+5. When should we run this again?
+
+## Post-Game Checklist
+
+- [ ] Restore all services to normal state
+- [ ] Verify no lingering issues
+- [ ] Collect all observer notes
+- [ ] Export metrics and dashboards
+- [ ] Schedule post-mortem meeting
+- [ ] Send thank-you to participants
+- [ ] Create action item tickets
+- [ ] Update runbooks based on learnings
+```
+
+## Game Day Observation Template
+
+```python
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import List
+
+@dataclass
+class GameDayObservation:
+    timestamp: datetime
+    observer: str
+    scenario: str
+    observation: str
+    category: str  # technical, process, communication, surprise
+    severity: str  # info, concern, critical
+    photo_url: str = ""
+
+@dataclass
+class GameDayMetrics:
+    scenario_name: str
+    start_time: datetime
+    end_time: datetime
+
+    # Technical metrics
+    time_to_detect_seconds: float
+    time_to_respond_seconds: float
+    time_to_recover_seconds: float
+    error_rate_peak: float
+    alerts_fired: List[str] = field(default_factory=list)
+    alerts_missed: List[str] = field(default_factory=list)
+
+    # Team metrics
+    responders_involved: int
+    escalations_needed: int
+    communication_gaps: List[str] = field(default_factory=list)
+
+    # Success criteria
+    met_rto: bool = False
+    met_rpo: bool = False
+    zero_customer_impact: bool = False
+
+    def calculate_mttr(self) -> float:
+        """Mean Time To Recovery"""
+        return (self.end_time - self.start_time).total_seconds()
+
+    def success_rate(self) -> float:
+        """Percentage of success criteria met"""
+        criteria = [
+            self.met_rto,
+            self.met_rpo,
+            self.zero_customer_impact,
+            len(self.alerts_missed) == 0
+        ]
+        return sum(criteria) / len(criteria) * 100
+
+# Example usage
+metrics = GameDayMetrics(
+    scenario_name="Database Failover",
+    start_time=datetime(2025, 1, 15, 10, 10, 0),
+    end_time=datetime(2025, 1, 15, 10, 12, 30),
+    time_to_detect_seconds=15.0,
+    time_to_respond_seconds=45.0,
+    time_to_recover_seconds=150.0,
+    error_rate_peak=0.05,
+    alerts_fired=["DatabaseConnectionError", "HighLatency"],
+    alerts_missed=["FailoverInitiated"],
+    responders_involved=3,
+    escalations_needed=0,
+    met_rto=True,
+    met_rpo=True,
+    zero_customer_impact=True
+)
+
+print(f"MTTR: {metrics.calculate_mttr()}s")
+print(f"Success Rate: {metrics.success_rate()}%")
+```
+
+## Surprise Scenarios Library
+
+```yaml
+# Keep these secret until game day!
+surprise_scenarios:
+  - name: "Cascading Failure"
+    description: "Primary failure triggers secondary issue"
+    injection:
+      - "Database failover (expected)"
+      - "Cache eviction due to new primary IP (surprise!)"
+    learning_goals:
+      - "Do we understand our dependencies?"
+      - "Can we handle multiple simultaneous issues?"
+
+  - name: "Monitoring Blind Spot"
+    description: "Failure that doesn't trigger alerts"
+    injection:
+      - "Gradual connection pool leak"
+      - "No immediate alerts fire"
+    learning_goals:
+      - "How do we discover issues without alerts?"
+      - "Do we have adequate monitoring coverage?"
+
+  - name: "Documentation Failure"
+    description: "Runbook is outdated or incorrect"
+    setup:
+      - "Modify runbook to have incorrect commands"
+      - "Or remove runbook entirely"
+    learning_goals:
+      - "Can team problem-solve without docs?"
+      - "How quickly can we update documentation?"
+
+  - name: "Key Person Unavailable"
+    description: "Subject matter expert is unreachable"
+    setup:
+      - "Ask SME to not respond for 15 minutes"
+    learning_goals:
+      - "Is knowledge properly distributed?"
+      - "Can team succeed without specific person?"
+
+  - name: "Partial Degradation"
+    description: "Service works but slowly"
+    injection:
+      - "Add 5 second latency instead of complete failure"
+    learning_goals:
+      - "Do we detect performance degradation?"
+      - "What are our latency SLOs?"
+```
+
+## Post-Game Report Template
+
+```markdown
+# Game Day Report: Database Failover
+
+**Date**: January 15, 2025
+**Participants**: 12
+**Duration**: 2 hours
+**Environment**: Staging
+
+## Executive Summary
+
+Conducted database failover game day to test RDS high availability and
+application resilience. Successfully failed over database in 2.5 minutes
+(target: 2 min). Discovered 3 critical gaps in monitoring and 2 process
+improvements needed.
+
+## Metrics
+
+| Metric | Target | Actual | Status |
+|--------|--------|--------|--------|
+| Time to Detect | < 30s | 15s | PASS |
+| Time to Respond | < 5min | 4min 20s | PASS |
+| Time to Recover | < 2min | 2min 30s | FAIL |
+| Alert Accuracy | 100% | 66% | FAIL |
+| Zero Customer Impact | Yes | Yes | PASS |
+
+## What Went Well
+
+1. Team responded quickly (4m 20s vs 5m target)
+2. Runbooks were accurate and helpful
+3. Communication was clear and frequent
+4. No customer impact during any scenario
+5. Application auto-reconnect worked perfectly
+
+## What Didn't Go Well
+
+1. Missing alert for failover initiation
+2. Took 30s longer than target to recover
+3. Connection pool exhaustion not detected
+4. Dashboard didn't show replica lag clearly
+5. Escalation contacts list was outdated
+
+## Surprises
+
+1. Cache invalidation cascaded from DB failover (unexpected)
+2. Read replica had 45s replication lag we didn't know about
+3. Application retried too aggressively during failover
+4. Team found a workaround we hadn't documented
+
+## Action Items
+
+| Action | Owner | Due Date | Priority |
+|--------|-------|----------|----------|
+| Add alert for RDS failover events | @sre-team | Jan 20 | P0 |
+| Update dashboard with replica lag | @platform | Jan 22 | P1 |
+| Document cache invalidation behavior | @dev-team | Jan 25 | P1 |
+| Add connection pool monitoring | @sre-team | Jan 27 | P0 |
+| Update escalation contact list | @manager | Jan 18 | P2 |
+| Tune application retry backoff | @dev-team | Feb 1 | P1 |
+
+## Lessons Learned
+
+1. **Monitoring Gaps**: We had blind spots in replica monitoring
+2. **Cascading Effects**: DB changes affect cache in non-obvious ways
+3. **Team Knowledge**: Cross-training is working well
+4. **Documentation**: Runbooks saved time, keep them updated
+
+## Next Game Day
+
+**Proposed Date**: March 15, 2025
+**Scenario**: Multi-region failover
+**Scope**: Production (with safeguards)
+
+## Appendix
+
+- Full timeline spreadsheet: [link]
+- Screen recordings: [link]
+- Metrics dashboard export: [link]
+- Raw observation notes: [link]
+```
+
+## Quick Reference
+
+| Phase | Duration | Key Activities |
+|-------|----------|----------------|
+| Planning | 2 weeks | Define scenarios, invite participants |
+| Pre-game | 30 min | Setup, verify environment, brief team |
+| Execution | 2 hours | Run scenarios, observe, document |
+| Debrief | 30 min | Immediate learnings, quick wins |
+| Post-mortem | 1 week later | Detailed analysis, action items |
+| Follow-up | 1 month | Verify improvements, plan next game day |