blockmine 1.21.0 → 1.23.0

package/.claude/skills/sre/resources/disaster-recovery.md

@@ -0,0 +1,232 @@
+# Disaster Recovery
+
+Backup strategies, RTO/RPO definitions, failover procedures, disaster recovery testing, and multi-region architectures.
+
+## Table of Contents
+
+- [RTO and RPO](#rto-and-rpo)
+- [Backup Strategies](#backup-strategies)
+- [Failover Procedures](#failover-procedures)
+- [DR Testing](#dr-testing)
+- [Multi-Region Architecture](#multi-region-architecture)
+
+## RTO and RPO
+
+**Definitions:**
+```
+RTO (Recovery Time Objective):
+  Maximum acceptable downtime
+  Example: 4 hours
+
+RPO (Recovery Point Objective):
+  Maximum acceptable data loss
+  Example: 1 hour (last backup)
+```
+
+**RTO/RPO Tiers:**
+```yaml
+tier_1_critical:
+  rto: 1 hour
+  rpo: 15 minutes
+  cost: High
+  examples: [payment processing, critical APIs]
+
+tier_2_important:
+  rto: 4 hours
+  rpo: 1 hour
+  cost: Medium
+  examples: [main application, databases]
+
+tier_3_standard:
+  rto: 24 hours
+  rpo: 24 hours
+  cost: Low
+  examples: [internal tools, analytics]
+```
+
+## Backup Strategies
+
+**3-2-1 Rule:**
+```
+3 copies of data
+2 different media types
+1 offsite backup
+```
+
+**Database Backups:**
+```yaml
+# PostgreSQL backup with WAL archiving
+postgresql_backup:
+  full_backup:
+    frequency: daily
+    retention: 30 days
+    command: |
+      pg_basebackup -h localhost -D /backup/$(date +%Y%m%d) -Ft -z -Xs
+
+  wal_archiving:
+    enabled: true
+    archive_command: |
+      aws s3 cp %p s3://backups/wal/%f
+    restore_command: |
+      aws s3 cp s3://backups/wal/%f %p
+
+  point_in_time_recovery:
+    enabled: true
+    max_recovery_window: 7 days
+```
+
+**Automated Backups (Velero for Kubernetes):**
+```yaml
+apiVersion: velero.io/v1
+kind: Schedule
+metadata:
+  name: daily-backup
+spec:
+  schedule: "0 1 * * *"  # 1 AM daily
+  template:
+    includedNamespaces:
+      - production
+    includedResources:
+      - "*"
+    snapshotVolumes: true
+    ttl: 720h  # 30 days
+```
+
+## Failover Procedures
+
+**Database Failover:**
+```yaml
+# Automated failover with Patroni
+patroni:
+  name: postgres01
+  scope: postgres-cluster
+
+  bootstrap:
+    dcs:
+      ttl: 30
+      loop_wait: 10
+      retry_timeout: 10
+      maximum_lag_on_failover: 1048576
+
+  postgresql:
+    parameters:
+      max_connections: 100
+      shared_buffers: 256MB
+
+# Failover process:
+# 1. Patroni detects primary failure
+# 2. Initiates leader election
+# 3. Promotes replica to primary
+# 4. Updates DNS/load balancer
+# 5. Old primary rejoins as replica
+```
+
+**Application Failover:**
+```yaml
+# Multi-region with Route53 health checks
+route53_failover:
+  primary:
+    region: us-east-1
+    health_check:
+      protocol: HTTPS
+      path: /health
+      interval: 30
+      failure_threshold: 3
+
+  secondary:
+    region: us-west-2
+    failover_mode: automatic
+    activate_when: primary_unhealthy
+```
+
+## DR Testing
+
+**DR Drill Schedule:**
+```yaml
+quarterly_dr_drill:
+  week_1:
+    - Review DR plan
+    - Update runbooks
+    - Verify backup integrity
+
+  week_2:
+    - Tabletop exercise
+    - Walk through procedures
+    - Identify gaps
+
+  week_3:
+    - Partial failover test
+    - Test database recovery
+    - Verify monitoring
+
+  week_4:
+    - Full DR drill
+    - Complete failover
+    - Document lessons learned
+```
+
+**DR Test Checklist:**
+```markdown
+- [ ] Backup restoration successful
+- [ ] RTO met (< 4 hours)
+- [ ] RPO met (< 1 hour data loss)
+- [ ] All services operational
+- [ ] Monitoring functional
+- [ ] Logs accessible
+- [ ] Team communication effective
+- [ ] Runbooks accurate
+- [ ] Action items documented
+```
+
+## Multi-Region Architecture
+
+**Active-Passive:**
+```
+Primary Region (Active)  ─────┐
+  - Handles all traffic       │
+  - Database writes           │ Replication
+                              │
+Secondary Region (Passive)  ◄─┘
+  - Standby for failover
+  - Read replicas only
+  - Activated manually
+```
+
+**Active-Active:**
+```
+Region 1 (Active)  ◄─────► Region 2 (Active)
+  - 50% traffic              - 50% traffic
+  - Full read/write          - Full read/write
+  - Bi-directional sync      - Bi-directional sync
+  - Auto-failover            - Auto-failover
+```
+
+**Implementation:**
+```yaml
+# Kubernetes multi-region with Cilium Cluster Mesh
+clusters:
+  us-east-1:
+    role: primary
+    services:
+      - api-service
+      - database (primary)
+      - cache
+
+  us-west-2:
+    role: secondary
+    services:
+      - api-service (read-only)
+      - database (replica)
+      - cache
+
+failover:
+  automatic: true
+  health_check_interval: 30s
+  failover_threshold: 3
+```
+
+---
+
+**Related Resources:**
+- [incident-management.md](incident-management.md)
+- [reliability-patterns.md](reliability-patterns.md)

package/.claude/skills/sre/resources/incident-management.md

@@ -0,0 +1,436 @@
+# Incident Management
+
+Incident response procedures, severity levels, escalation paths, communication protocols, postmortems, and on-call processes.
+
+## Table of Contents
+
+- [Incident Lifecycle](#incident-lifecycle)
+- [Severity Levels](#severity-levels)
+- [Roles and Responsibilities](#roles-and-responsibilities)
+- [Response Procedures](#response-procedures)
+- [Communication](#communication)
+- [Postmortems](#postmortems)
+- [Best Practices](#best-practices)
+
+## Incident Lifecycle
+
+```
+Detect → Respond → Mitigate → Resolve → Learn
+   ↓        ↓         ↓          ↓         ↓
+ Alert   Triage   Fix/Workaround Root   Postmortem
+                                 Cause
+```
+
+## Severity Levels
+
+```yaml
+SEV1 - Critical:
+  impact: Complete service outage or data loss
+  response_time: Immediate (< 15 minutes)
+  escalation: Page on-call + management
+  examples:
+    - Production database down
+    - Payment processing failed
+    - Security breach
+    - Data loss
+
+SEV2 - High:
+  impact: Major functionality impaired
+  response_time: 30 minutes
+  escalation: Page on-call
+  examples:
+    - API latency severely degraded
+    - Single region outage
+    - Critical feature unavailable
+
+SEV3 - Medium:
+  impact: Minor functionality impaired
+  response_time: 2 hours
+  escalation: Notify on-call via Slack
+  examples:
+    - Non-critical feature degraded
+    - Elevated error rates
+    - Performance slowdown
+
+SEV4 - Low:
+  impact: Minimal user impact
+  response_time: Next business day
+  escalation: Create ticket
+  examples:
+    - UI cosmetic issues
+    - Internal tool problems
+    - Low-priority bugs
+```
+
+## Roles and Responsibilities
+
+**Incident Commander:**
+- Owns the incident response
+- Makes decisions
+- Coordinates team
+- Manages communication
+
+**Technical Lead:**
+- Diagnoses root cause
+- Implements fixes
+- Validates resolution
+
+**Communications Lead:**
+- Updates status page
+- Notifies stakeholders
+- Manages customer communication
+
+**Scribe:**
+- Documents timeline
+- Records decisions
+- Tracks action items
+
+## Response Procedures
+
+### Detection and Triage
+
+```yaml
+# incident-response.yaml
+1_detection:
+  - Alert triggers
+  - Customer report
+  - Monitoring system
+  - Team member notice
+
+2_initial_triage:
+  - Assess severity
+  - Determine impact
+  - Page appropriate team
+  - Create incident channel
+
+3_form_response_team:
+  - Incident Commander
+  - Technical Lead(s)
+  - Communications Lead
+  - Subject Matter Experts
+```
+
+### Incident Command Structure
+
+```bash
+# Create incident Slack channel
+/incident create SEV1 "API Gateway Down"
+
+# Auto-creates:
+# - #incident-2024-001
+# - Zoom bridge
+# - Status page placeholder
+# - Timeline doc
+```
+
+### Response Playbooks
+
+**Database Outage:**
+```yaml
+playbook: database-outage
+severity: SEV1
+
+steps:
+  1_immediate:
+    - Check database health metrics
+    - Verify connectivity
+    - Check for locks/blocking queries
+    - Review recent changes
+
+  2_diagnosis:
+    - Check replication lag
+    - Review error logs
+    - Verify disk space
+    - Check connection pool
+
+  3_mitigation:
+    - Failover to replica
+    - Kill blocking queries
+    - Restart if necessary
+    - Scale resources
+
+  4_communication:
+    - Update status page
+    - Notify customers
+    - Inform stakeholders
+```
+
+**API Latency Degradation:**
+```yaml
+playbook: api-latency
+severity: SEV2
+
+steps:
+  1_gather_data:
+    - Check p95/p99 latency
+    - Review error rates
+    - Examine slow query logs
+    - Check downstream services
+
+  2_common_causes:
+    - Database slow queries
+    - Increased traffic
+    - Downstream dependency
+    - Resource exhaustion
+    - Code deployment
+
+  3_quick_fixes:
+    - Scale up instances
+    - Enable/adjust caching
+    - Rate limit traffic
+    - Rollback deployment
+```
+
+## Communication
+
+### Status Page Updates
+
+```yaml
+# Incident timeline
+14:05: Investigating - We're investigating reports of API errors
+14:15: Identified - Database connection issues identified
+14:30: Monitoring - Failover completed, monitoring recovery
+15:00: Resolved - All services restored, investigating root cause
+```
+
+### Customer Communication Template
+
+```markdown
+Subject: [RESOLVED] API Service Disruption - Jan 15, 2024
+
+Dear Customers,
+
+SUMMARY:
+Between 14:00-15:00 UTC today, our API service experienced elevated
+error rates affecting approximately 10% of requests.
+
+IMPACT:
+- API errors for 10% of requests
+- Average latency increased from 200ms to 2s
+- No data loss occurred
+
+ROOT CAUSE:
+Database connection pool exhaustion due to traffic spike
+
+RESOLUTION:
+- Scaled database connection pools
+- Implemented better connection management
+- Added auto-scaling triggers
+
+PREVENTION:
+- Enhanced monitoring and alerting
+- Implemented circuit breakers
+- Scheduled capacity review
+
+We apologize for the disruption. Please contact support@example.com
+with any questions.
+
+Status page: https://status.example.com/incidents/2024-001
+```
+
+### Internal Communication
+
+```markdown
+# Incident Slack Update Template
+:rotating_light: **SEV1 INCIDENT** :rotating_light:
+
+**Status:** Investigating
+**Impact:** API returning 500 errors
+**Started:** 14:05 UTC
+**Incident Commander:** @alice
+**Bridge:** https://zoom.us/j/123456789
+**Channel:** #incident-2024-001
+
+**Timeline:**
+14:05 - Alert triggered for high error rate
+14:07 - Incident declared SEV1
+14:10 - Team assembled
+14:15 - Root cause identified
+
+**Next Update:** 14:30 UTC or sooner if status changes
+```
+
+## Postmortems
+
+### Blameless Postmortem Template
+
+```markdown
+# Postmortem: API Outage - January 15, 2024
+
+## Incident Summary
+**Date:** 2024-01-15
+**Duration:** 55 minutes (14:05 - 15:00 UTC)
+**Severity:** SEV1
+**Impact:** 10% error rate, 500k affected requests
+**Root Cause:** Database connection pool exhaustion
+
+## Timeline (UTC)
+- 14:00: Traffic begins increasing (2x normal)
+- 14:05: Alert: High API error rate
+- 14:07: Incident declared SEV1
+- 14:10: Incident team assembled
+- 14:15: Root cause identified (connection pool exhausted)
+- 14:20: Mitigation started (scale connection pool)
+- 14:30: Error rates declining
+- 14:45: Monitoring recovery
+- 15:00: Incident resolved
+
+## Root Cause Analysis
+
+### What Happened
+A marketing campaign drove 2x normal traffic. Our database connection
+pool had a static size of 100 connections, which was exhausted. API
+servers could not acquire database connections, resulting in errors.
+
+### Why It Happened
+1. No auto-scaling for database connection pools
+2. Connection pool size not sized for peak traffic
+3. No circuit breaker to fail fast
+4. Insufficient load testing
+
+### Contributing Factors
+- Marketing campaign not coordinated with engineering
+- Connection pool metrics not monitored
+- No alerts on connection pool saturation
+
+## Impact
+- 500,000 failed API requests
+- 10% error rate for 55 minutes
+- Estimated revenue impact: $5,000
+- Customer complaints: 50
+
+## What Went Well
+- Fast detection (< 5 minutes)
+- Clear escalation path
+- Good team communication
+- Status page updated regularly
+- Fix deployed quickly
+
+## What Went Wrong
+- No advance warning of traffic spike
+- Connection pool not monitored
+- Manual scaling required
+- Customer notification delayed 10 minutes
+
+## Action Items
+
+### Immediate (This Week)
+- [ ] Implement connection pool auto-scaling (@alice, 2024-01-17)
+- [ ] Add connection pool metrics to dashboards (@bob, 2024-01-18)
+- [ ] Create alerts for pool saturation (@charlie, 2024-01-19)
+
+### Short-term (This Month)
+- [ ] Implement circuit breakers (@alice, 2024-01-25)
+- [ ] Load test at 3x normal traffic (@bob, 2024-01-30)
+- [ ] Create runbook for connection issues (@charlie, 2024-01-30)
+
+### Long-term (This Quarter)
+- [ ] Improve engineering/marketing coordination (@dave, 2024-03-31)
+- [ ] Implement capacity planning process (@eve, 2024-03-31)
+- [ ] Auto-notification system for incidents (@frank, 2024-03-31)
+
+## Lessons Learned
+1. Static resource limits are a failure point
+2. Cross-team coordination essential for major campaigns
+3. Observability gaps can hide brewing problems
+4. Circuit breakers prevent cascading failures
+
+## Related Incidents
+- INC-2023-089: Similar connection pool issue (resolved)
+- INC-2023-112: Traffic spike from campaign (different cause)
+
+## Appendix
+- [Grafana Dashboard](https://grafana.example.com/d/incident-2024-001)
+- [Logs](https://logs.example.com/incident-2024-001)
+- [Slack Channel](https://slack.com/archives/incident-2024-001)
+```
+
+### Postmortem Review Process
+
+```yaml
+postmortem_process:
+  1_draft:
+    owner: Incident Commander
+    deadline: 2 business days
+    content:
+      - Timeline
+      - Root cause
+      - Impact
+      - Action items
+
+  2_review:
+    participants:
+      - Incident team
+      - Engineering leadership
+      - Related teams
+    format: Meeting (30-60 min)
+    goals:
+      - Validate accuracy
+      - Identify additional learnings
+      - Prioritize action items
+
+  3_publish:
+    distribution:
+      - All engineering
+      - Product team
+      - Customer support
+      - Public (if appropriate)
+
+  4_followup:
+    - Track action items in project management tool
+    - Review progress in weekly meetings
+    - Update on completion
+```
+
+## Best Practices
+
+### 1. Blameless Culture
+
+```
+Focus on systems and processes, not individuals
+Ask "how" and "why", not "who"
+Encourage sharing mistakes openly
+```
+
+### 2. Clear Severity Definitions
+
+```yaml
+# Document and communicate
+# Train team on criteria
+# Review severity in retrospective
+```
+
+### 3. Designated Roles
+
+```
+Never have incident response without clear roles
+Incident Commander makes all decisions
+Scribe documents everything
+```
+
+### 4. Practice Incidents
+
+```yaml
+# Run incident simulations quarterly
+chaos_engineering:
+  - Simulate database failure
+  - Test failover procedures
+  - Practice communication
+  - Time the response
+```
+
+### 5. Action Item Follow-Through
+
+```
+Assign owners and deadlines
+Track in project management
+Report progress weekly
+Review in postmortem review
+```
+
+---
+
+**Related Resources:**
+- [on-call-runbooks.md](on-call-runbooks.md) - Diagnostic procedures
+- [observability-stack.md](observability-stack.md) - Monitoring and detection
+- [alerting-best-practices.md](alerting-best-practices.md) - Alert configuration