aigroup-workflow 2.1.2 → 2.2.0

package/skills/chaos-engineer/references/experiment-design.md

@@ -1,229 +0,0 @@
-# Chaos Experiment Design
-
-## Experiment Template
-
-```yaml
-name: "Database Connection Pool Exhaustion"
-hypothesis: "When the database connection pool is exhausted, the application will gracefully degrade and return 503 errors without cascading failures"
-
-steady_state:
-  metrics:
-    - name: "Error Rate"
-      threshold: "< 0.1%"
-      source: "prometheus"
-      query: "rate(http_requests_total{status=~'5..'}[5m])"
-    - name: "Latency P99"
-      threshold: "< 500ms"
-      source: "datadog"
-    - name: "Active Connections"
-      threshold: "> 10"
-      query: "pg_stat_activity_count"
-
-blast_radius:
-  environment: "staging"
-  traffic_percentage: 10
-  duration_seconds: 300
-  max_error_rate: "5%"
-  auto_rollback: true
-
-injection:
-  type: "resource_exhaustion"
-  target: "database_connections"
-  method: "connection_leak"
-  parameters:
-    leak_rate: 5  # connections per second
-    max_leaked: 50
-
-safety:
-  rollback_triggers:
-    - "error_rate > 5%"
-    - "manual_kill_switch"
-    - "duration_exceeded"
-  rollback_time_limit_seconds: 30
-  alerts:
-    - slack: "#chaos-engineering"
-    - pagerduty: "chaos-team"
-
-success_criteria:
-  - "Circuit breakers activate within 10s"
-  - "503 errors returned (not 500)"
-  - "No cascading failures to other services"
-  - "System recovers within 60s of rollback"
-```
-
-## Hypothesis Formulation
-
-```python
-def create_hypothesis(component: str, failure: str, expected_behavior: str) -> dict:
-    """
-    Create well-formed chaos hypothesis.
-
-    Format: "Given [normal state], when [failure occurs],
-             then [expected behavior], measured by [metrics]"
-    """
-    return {
-        "given": f"System is in steady state with {component} functioning normally",
-        "when": f"{failure} occurs",
-        "then": expected_behavior,
-        "measured_by": [
-            "Error rate remains below threshold",
-            "Latency stays within SLO",
-            "No data loss or corruption",
-            "Recovery time within RTO"
-        ]
-    }
-
-# Example
-hypothesis = create_hypothesis(
-    component="payment service",
-    failure="50% packet loss to payment gateway",
-    expected_behavior="Requests timeout gracefully, retry queue activates, "
-                     "users see clear error messages"
-)
-```
-
-## Blast Radius Control
-
-```python
-from dataclasses import dataclass
-from enum import Enum
-
-class BlastRadiusLevel(Enum):
-    MINIMAL = "single_instance_dev"
-    LOW = "single_instance_staging"
-    MEDIUM = "percentage_staging"
-    HIGH = "percentage_production"
-    CRITICAL = "full_production"
-
-@dataclass
-class BlastRadiusConfig:
-    level: BlastRadiusLevel
-    environment: str
-    target_percentage: float  # 0-100
-    canary_users: list[str]
-    feature_flag: str
-    auto_rollback: bool
-    max_duration_seconds: int
-
-    def validate(self):
-        """Enforce safety rules."""
-        if self.level == BlastRadiusLevel.CRITICAL:
-            raise ValueError("CRITICAL blast radius requires explicit approval")
-
-        if self.environment == "production" and self.target_percentage > 10:
-            if not self.feature_flag or not self.auto_rollback:
-                raise ValueError("Production >10% requires feature flag AND auto-rollback")
-
-        if self.max_duration_seconds > 600:
-            raise ValueError("Max duration cannot exceed 10 minutes without approval")
-
-# Progressive blast radius expansion
-def progressive_rollout() -> list[BlastRadiusConfig]:
-    return [
-        BlastRadiusConfig(
-            level=BlastRadiusLevel.MINIMAL,
-            environment="dev",
-            target_percentage=100,
-            canary_users=[],
-            feature_flag="chaos_dev",
-            auto_rollback=True,
-            max_duration_seconds=300
-        ),
-        BlastRadiusConfig(
-            level=BlastRadiusLevel.LOW,
-            environment="staging",
-            target_percentage=100,
-            canary_users=[],
-            feature_flag="chaos_staging",
-            auto_rollback=True,
-            max_duration_seconds=600
-        ),
-        BlastRadiusConfig(
-            level=BlastRadiusLevel.MEDIUM,
-            environment="production",
-            target_percentage=1,
-            canary_users=["internal_team"],
-            feature_flag="chaos_prod_canary",
-            auto_rollback=True,
-            max_duration_seconds=300
-        )
-    ]
-```
-
-## Safety Mechanisms
-
-```python
-import asyncio
-from typing import Callable
-
-class ChaosExperimentSafety:
-    def __init__(self, config: dict):
-        self.config = config
-        self.kill_switch_active = False
-        self.metrics = {}
-
-    async def run_with_safety(self, chaos_fn: Callable):
-        """Execute chaos with automatic safety checks."""
-        # Pre-flight checks
-        if not await self.verify_steady_state():
-            raise Exception("System not in steady state - aborting")
-
-        # Set up rollback trigger
-        rollback_task = asyncio.create_task(self.monitor_for_rollback())
-        chaos_task = asyncio.create_task(chaos_fn())
-
-        try:
-            # Wait for either chaos completion or rollback trigger
-            done, pending = await asyncio.wait(
-                [chaos_task, rollback_task],
-                return_when=asyncio.FIRST_COMPLETED
-            )
-
-            if rollback_task in done:
-                # Rollback triggered - cancel chaos
-                chaos_task.cancel()
-                await self.rollback()
-
-        finally:
-            await self.ensure_system_recovery()
-
-    async def verify_steady_state(self) -> bool:
-        """Check all steady state metrics are within threshold."""
-        for metric in self.config['steady_state']['metrics']:
-            value = await self.query_metric(metric['query'])
-            if not self.within_threshold(value, metric['threshold']):
-                return False
-        return True
-
-    async def monitor_for_rollback(self):
-        """Continuously monitor for rollback triggers."""
-        start_time = asyncio.get_event_loop().time()
-
-        while True:
-            # Check duration limit
-            if asyncio.get_event_loop().time() - start_time > \
-               self.config['blast_radius']['duration_seconds']:
-                return "duration_exceeded"
-
-            # Check manual kill switch
-            if self.kill_switch_active:
-                return "manual_kill_switch"
-
-            # Check error rate
-            error_rate = await self.query_metric("error_rate")
-            if error_rate > float(self.config['blast_radius']['max_error_rate'].strip('%')):
-                return "error_rate_exceeded"
-
-            await asyncio.sleep(5)  # Check every 5 seconds
-```
-
-## Quick Reference
-
-| Phase | Key Actions | Time Limit |
-|-------|-------------|------------|
-| Design | Hypothesis, metrics, blast radius | 1 hour |
-| Review | Team review, safety check | 30 min |
-| Prepare | Setup monitoring, rollback | 1 hour |
-| Execute | Run experiment, monitor | 5-10 min |
-| Rollback | Restore steady state | < 30 sec |
-| Learn | Document findings, plan fixes | 2 hours |

package/skills/chaos-engineer/references/game-days.md

@@ -1,434 +0,0 @@
-# Game Day Planning & Execution
-
-## Game Day Planning Template
-
-```yaml
-game_day:
-  name: "Database Failover Drill"
-  date: "2025-01-15"
-  time: "10:00-12:00 PST"
-  environment: "staging"  # Start in staging
-
-  objectives:
-    - "Verify RDS failover to standby in under 2 minutes"
-    - "Validate application auto-reconnect logic"
-    - "Test monitoring and alerting effectiveness"
-    - "Practice incident response procedures"
-
-  participants:
-    facilitator: "chaos-engineer@company.com"
-    observers:
-      - "sre-team@company.com"
-      - "dev-team@company.com"
-    responders:
-      - "on-call-engineer@company.com"
-      - "database-admin@company.com"
-    stakeholders:
-      - "engineering-manager@company.com"
-
-  scenarios:
-    - name: "Primary database instance failure"
-      duration_minutes: 30
-      steps:
-        - action: "Force RDS instance reboot with failover"
-          expected: "Failover to standby in <2 min"
-          success_criteria:
-            - "Downtime < 2 minutes"
-            - "No data loss"
-            - "Alerts fired correctly"
-
-    - name: "Network partition to database"
-      duration_minutes: 20
-      steps:
-        - action: "Block network traffic to RDS security group"
-          expected: "Application switches to read replica"
-          success_criteria:
-            - "Read-only mode activated"
-            - "User-facing error messages clear"
-
-  communication_plan:
-    announcement_channel: "#game-day-announcements"
-    war_room: "Zoom link: https://..."
-    status_updates_every: "5 minutes"
-    escalation_contacts:
-      - name: "VP Engineering"
-        phone: "+1-555-0100"
-        threshold: "downtime > 5 minutes"
-
-  rollback_plan:
-    automatic_rollback_triggers:
-      - "production traffic affected"
-      - "customer complaints received"
-      - "error_rate > 10%"
-    manual_rollback_command: "aws rds reboot-db-instance --db-instance-identifier primary --force-failover"
-    rollback_time_limit_seconds: 60
-
-  success_metrics:
-    - metric: "RTO (Recovery Time Objective)"
-      target: "< 2 minutes"
-      measurement: "time between failure and full recovery"
-    - metric: "Alert accuracy"
-      target: "100%"
-      measurement: "all expected alerts fired"
-    - metric: "Team response time"
-      target: "< 5 minutes"
-      measurement: "time to acknowledge incident"
-
-  post_mortem:
-    scheduled_for: "2025-01-16 14:00"
-    template: "game-day-retro.md"
-    required_attendees: "all participants"
-```
-
-## Game Day Runbook
-
-```markdown
-# Database Failover Game Day Runbook
-
-**Date**: January 15, 2025
-**Duration**: 2 hours
-**Environment**: Staging
-
-## Pre-Game Checklist (T-30 min)
-
-- [ ] Verify all participants joined war room
-- [ ] Confirm monitoring dashboards accessible
-- [ ] Test rollback procedures work
-- [ ] Announce game day start in #engineering
-- [ ] Verify staging environment healthy
-- [ ] Set up screen recording for timeline
-- [ ] Prepare incident timeline spreadsheet
-
-## Timeline
-
-### 10:00 - Introduction (10 min)
-- Facilitator explains objectives
-- Review scenarios and success criteria
-- Confirm roles and communication channels
-- Remind everyone: this is a learning exercise
-
-### 10:10 - Scenario 1: Primary DB Failure (30 min)
-
-**T+0 (10:10)** - Inject failure
-```bash
-aws rds reboot-db-instance \
-  --db-instance-identifier staging-primary \
-  --force-failover
-```
-
-**Expected Timeline**:
-- T+0: Reboot initiated
-- T+30s: Primary becomes unavailable
-- T+60s: DNS updated to standby
-- T+90s: Application reconnects
-- T+120s: Full recovery
-
-**Observer Tasks**:
-- [ ] Record exact time of failure injection
-- [ ] Monitor application error logs
-- [ ] Track alert notifications
-- [ ] Document team response actions
-- [ ] Screenshot dashboard states
-
-**Questions to Answer**:
-- How long until first alert?
-- Did application auto-reconnect?
-- Were customers impacted?
-- What manual interventions needed?
-
-### 10:40 - Debrief Scenario 1 (10 min)
-- What went well?
-- What could improve?
-- Any surprises?
-- Action items identified
-
-### 10:50 - Scenario 2: Network Partition (20 min)
-
-**T+0 (10:50)** - Inject failure
-```bash
-# Block database security group ingress
-aws ec2 revoke-security-group-ingress \
-  --group-id sg-xxxxx \
-  --protocol tcp \
-  --port 5432 \
-  --cidr 10.0.0.0/16
-```
-
-**Expected Behavior**:
-- Connection timeouts occur
-- Circuit breaker opens
-- Read-only mode activates
-- Clear error messages shown
-
-**Observer Tasks**:
-- [ ] Monitor circuit breaker state
-- [ ] Verify read-replica failover
-- [ ] Check user-facing error messages
-- [ ] Track degraded service duration
-
-### 11:10 - Debrief Scenario 2 (10 min)
-
-### 11:20 - Scenario 3: Surprise! (20 min)
-
-**Facilitator Note**: Don't announce this scenario details beforehand.
-Test true incident response capability.
-
-**Hidden Scenario**: Combination failure
-1. Database connection pool leak
-2. Simultaneous cache invalidation
-
-```python
-# Connection leak simulator
-import psycopg2
-connections = []
-for i in range(100):
-    conn = psycopg2.connect(DATABASE_URL)
-    connections.append(conn)
-    # Intentionally don't close
-```
-
-**Observer Tasks**:
-- [ ] How long to identify root cause?
-- [ ] Communication effectiveness
-- [ ] Cross-team coordination
-- [ ] Escalation decisions
-
-### 11:40 - Final Debrief & Wrap-up (20 min)
-
-**Debrief Questions**:
-1. What worked well?
-2. What didn't work?
-3. What surprised us?
-4. What are our top 3 action items?
-5. When should we run this again?
-
-## Post-Game Checklist
-
-- [ ] Restore all services to normal state
-- [ ] Verify no lingering issues
-- [ ] Collect all observer notes
-- [ ] Export metrics and dashboards
-- [ ] Schedule post-mortem meeting
-- [ ] Send thank-you to participants
-- [ ] Create action item tickets
-- [ ] Update runbooks based on learnings
-```
-
-## Game Day Observation Template
-
-```python
-from dataclasses import dataclass, field
-from datetime import datetime
-from typing import List
-
-@dataclass
-class GameDayObservation:
-    timestamp: datetime
-    observer: str
-    scenario: str
-    observation: str
-    category: str  # technical, process, communication, surprise
-    severity: str  # info, concern, critical
-    photo_url: str = ""
-
-@dataclass
-class GameDayMetrics:
-    scenario_name: str
-    start_time: datetime
-    end_time: datetime
-
-    # Technical metrics
-    time_to_detect_seconds: float
-    time_to_respond_seconds: float
-    time_to_recover_seconds: float
-    error_rate_peak: float
-    alerts_fired: List[str] = field(default_factory=list)
-    alerts_missed: List[str] = field(default_factory=list)
-
-    # Team metrics
-    responders_involved: int
-    escalations_needed: int
-    communication_gaps: List[str] = field(default_factory=list)
-
-    # Success criteria
-    met_rto: bool = False
-    met_rpo: bool = False
-    zero_customer_impact: bool = False
-
-    def calculate_mttr(self) -> float:
-        """Mean Time To Recovery"""
-        return (self.end_time - self.start_time).total_seconds()
-
-    def success_rate(self) -> float:
-        """Percentage of success criteria met"""
-        criteria = [
-            self.met_rto,
-            self.met_rpo,
-            self.zero_customer_impact,
-            len(self.alerts_missed) == 0
-        ]
-        return sum(criteria) / len(criteria) * 100
-
-# Example usage
-metrics = GameDayMetrics(
-    scenario_name="Database Failover",
-    start_time=datetime(2025, 1, 15, 10, 10, 0),
-    end_time=datetime(2025, 1, 15, 10, 12, 30),
-    time_to_detect_seconds=15.0,
-    time_to_respond_seconds=45.0,
-    time_to_recover_seconds=150.0,
-    error_rate_peak=0.05,
-    alerts_fired=["DatabaseConnectionError", "HighLatency"],
-    alerts_missed=["FailoverInitiated"],
-    responders_involved=3,
-    escalations_needed=0,
-    met_rto=True,
-    met_rpo=True,
-    zero_customer_impact=True
-)
-
-print(f"MTTR: {metrics.calculate_mttr()}s")
-print(f"Success Rate: {metrics.success_rate()}%")
-```
-
-## Surprise Scenarios Library
-
-```yaml
-# Keep these secret until game day!
-surprise_scenarios:
-  - name: "Cascading Failure"
-    description: "Primary failure triggers secondary issue"
-    injection:
-      - "Database failover (expected)"
-      - "Cache eviction due to new primary IP (surprise!)"
-    learning_goals:
-      - "Do we understand our dependencies?"
-      - "Can we handle multiple simultaneous issues?"
-
-  - name: "Monitoring Blind Spot"
-    description: "Failure that doesn't trigger alerts"
-    injection:
-      - "Gradual connection pool leak"
-      - "No immediate alerts fire"
-    learning_goals:
-      - "How do we discover issues without alerts?"
-      - "Do we have adequate monitoring coverage?"
-
-  - name: "Documentation Failure"
-    description: "Runbook is outdated or incorrect"
-    setup:
-      - "Modify runbook to have incorrect commands"
-      - "Or remove runbook entirely"
-    learning_goals:
-      - "Can team problem-solve without docs?"
-      - "How quickly can we update documentation?"
-
-  - name: "Key Person Unavailable"
-    description: "Subject matter expert is unreachable"
-    setup:
-      - "Ask SME to not respond for 15 minutes"
-    learning_goals:
-      - "Is knowledge properly distributed?"
-      - "Can team succeed without specific person?"
-
-  - name: "Partial Degradation"
-    description: "Service works but slowly"
-    injection:
-      - "Add 5 second latency instead of complete failure"
-    learning_goals:
-      - "Do we detect performance degradation?"
-      - "What are our latency SLOs?"
-```
-
-## Post-Game Report Template
-
-```markdown
-# Game Day Report: Database Failover
-
-**Date**: January 15, 2025
-**Participants**: 12
-**Duration**: 2 hours
-**Environment**: Staging
-
-## Executive Summary
-
-Conducted database failover game day to test RDS high availability and
-application resilience. Successfully failed over database in 2.5 minutes
-(target: 2 min). Discovered 3 critical gaps in monitoring and 2 process
-improvements needed.
-
-## Metrics
-
-| Metric | Target | Actual | Status |
-|--------|--------|--------|--------|
-| Time to Detect | < 30s | 15s | PASS |
-| Time to Respond | < 5min | 4min 20s | PASS |
-| Time to Recover | < 2min | 2min 30s | FAIL |
-| Alert Accuracy | 100% | 66% | FAIL |
-| Zero Customer Impact | Yes | Yes | PASS |
-
-## What Went Well
-
-1. Team responded quickly (4m 20s vs 5m target)
-2. Runbooks were accurate and helpful
-3. Communication was clear and frequent
-4. No customer impact during any scenario
-5. Application auto-reconnect worked perfectly
-
-## What Didn't Go Well
-
-1. Missing alert for failover initiation
-2. Took 30s longer than target to recover
-3. Connection pool exhaustion not detected
-4. Dashboard didn't show replica lag clearly
-5. Escalation contacts list was outdated
-
-## Surprises
-
-1. Cache invalidation cascaded from DB failover (unexpected)
-2. Read replica had 45s replication lag we didn't know about
-3. Application retried too aggressively during failover
-4. Team found a workaround we hadn't documented
-
-## Action Items
-
-| Action | Owner | Due Date | Priority |
-|--------|-------|----------|----------|
-| Add alert for RDS failover events | @sre-team | Jan 20 | P0 |
-| Update dashboard with replica lag | @platform | Jan 22 | P1 |
-| Document cache invalidation behavior | @dev-team | Jan 25 | P1 |
-| Add connection pool monitoring | @sre-team | Jan 27 | P0 |
-| Update escalation contact list | @manager | Jan 18 | P2 |
-| Tune application retry backoff | @dev-team | Feb 1 | P1 |
-
-## Lessons Learned
-
-1. **Monitoring Gaps**: We had blind spots in replica monitoring
-2. **Cascading Effects**: DB changes affect cache in non-obvious ways
-3. **Team Knowledge**: Cross-training is working well
-4. **Documentation**: Runbooks saved time, keep them updated
-
-## Next Game Day
-
-**Proposed Date**: March 15, 2025
-**Scenario**: Multi-region failover
-**Scope**: Production (with safeguards)
-
-## Appendix
-
-- Full timeline spreadsheet: [link]
-- Screen recordings: [link]
-- Metrics dashboard export: [link]
-- Raw observation notes: [link]
-```
-
-## Quick Reference
-
-| Phase | Duration | Key Activities |
-|-------|----------|----------------|
-| Planning | 2 weeks | Define scenarios, invite participants |
-| Pre-game | 30 min | Setup, verify environment, brief team |
-| Execution | 2 hours | Run scenarios, observe, document |
-| Debrief | 30 min | Immediate learnings, quick wins |
-| Post-mortem | 1 week later | Detailed analysis, action items |
-| Follow-up | 1 month | Verify improvements, plan next game day |