blockmine 1.21.0 → 1.23.0

package/.claude/skills/sre/SKILL.md

@@ -0,0 +1,464 @@
+# SRE - Site Reliability Engineering
+
+Comprehensive guide for implementing SRE practices: SLI/SLO/SLA definitions, error budgets, incident management, on-call best practices, chaos engineering, observability, and reliability patterns for production systems.
+
+## Purpose
+
+Enable teams to build and operate reliable, scalable systems by applying software engineering principles to infrastructure and operations challenges.
+
+## When to Use This Skill
+
+Automatically activates when working on:
+- Defining SLIs, SLOs, and SLAs
+- Managing error budgets
+- Incident response and management
+- On-call rotation and procedures
+- Observability and monitoring
+- Chaos engineering and resilience testing
+- Capacity planning and performance
+- Disaster recovery planning
+
+## Quick Start Checklist
+
+When implementing SRE practices:
+
+- [ ] Define Service Level Indicators (SLIs)
+- [ ] Set Service Level Objectives (SLOs) with stakeholders
+- [ ] Implement error budget tracking
+- [ ] Set up comprehensive monitoring and alerting
+- [ ] Create incident response runbooks
+- [ ] Establish on-call rotation
+- [ ] Implement automated remediation
+- [ ] Conduct post-incident reviews
+- [ ] Perform chaos engineering experiments
+- [ ] Document capacity planning process
+- [ ] Set up disaster recovery procedures
+
+## Core Concepts
+
+### SLI / SLO / SLA Hierarchy
+
+```
+SLA (Service Level Agreement)
+  ↓ Commits to customers
+  ↓
+SLO (Service Level Objective)
+  ↓ Internal target (stricter than SLA)
+  ↓
+SLI (Service Level Indicator)
+  ↓ Measurement that shows if you're meeting SLO
+  ↓
+Metrics (actual measurements)
+```
+
+**Example:**
+```
+SLA:  99.9% uptime guaranteed (3 customers)
+SLO:  99.95% uptime target (internal)
+SLI:  Percentage of successful HTTP requests
+      (2xx, 3xx responses / total requests)
+```
+
+### Error Budgets
+
+```
+Error Budget = 1 - SLO
+
+Example:
+SLO: 99.9% → Error Budget: 0.1%
+
+Monthly calculations (30 days):
+- Total time: 43,200 minutes
+- Allowed downtime: 43.2 minutes
+
+If error budget is:
+- Available → Can take risks, deploy features
+- Depleted → Focus on reliability, freeze features
+```
+
+### The Four Golden Signals
+
+```yaml
+1. Latency: How long requests take
+   - Good: p50, p95, p99 latencies
+   - Bad: Average latency (hides outliers)
+
+2. Traffic: Demand on your system
+   - Requests per second
+   - Transactions per second
+   - Bandwidth usage
+
+3. Errors: Rate of failed requests
+   - HTTP 5xx errors
+   - Failed operations
+   - Wrong results
+
+4. Saturation: How "full" your service is
+   - CPU usage
+   - Memory usage
+   - Disk I/O
+   - Queue depth
+```
+
+## Common Patterns
+
+### Pattern 1: SLO Definition
+
+```yaml
+# slos/api-service.yaml
+apiVersion: v1
+kind: SLO
+metadata:
+  name: api-service-availability
+  service: api-service
+spec:
+  description: "API service availability for customer requests"
+
+  # SLI definition
+  sli:
+    name: availability
+    description: "Percentage of successful requests"
+    query: |
+      sum(rate(http_requests_total{job="api-service",code=~"2.."}[5m]))
+      /
+      sum(rate(http_requests_total{job="api-service"}[5m]))
+
+  # SLO target
+  objectives:
+  - target: 0.999  # 99.9%
+    window: 30d    # Rolling 30 days
+
+  # Alert thresholds
+  alerting:
+    - burn_rate: 14.4   # Alert if burning budget 14.4x faster
+      short_window: 1h
+      long_window: 5m
+      severity: critical
+
+    - burn_rate: 6
+      short_window: 6h
+      long_window: 30m
+      severity: warning
+```
+
+**Latency SLO:**
+```yaml
+apiVersion: v1
+kind: SLO
+metadata:
+  name: api-service-latency
+spec:
+  sli:
+    name: latency
+    description: "95th percentile latency under 200ms"
+    query: |
+      histogram_quantile(0.95,
+        sum(rate(http_request_duration_seconds_bucket{job="api-service"}[5m])) by (le)
+      )
+
+  objectives:
+  - target: 0.200  # 200ms
+    window: 7d
+```
+
+### Pattern 2: Incident Response
+
+**Incident Severity Levels:**
+```yaml
+Severity 1 (Critical):
+  - Complete service outage
+  - Security breach
+  - Data loss
+  Response: Immediate, all hands
+  Escalation: 15 minutes
+
+Severity 2 (High):
+  - Partial outage
+  - Degraded performance
+  - Feature broken
+  Response: Within 30 minutes
+  Escalation: 1 hour
+
+Severity 3 (Medium):
+  - Minor issues
+  - Non-critical features affected
+  Response: Within 4 hours
+  Escalation: Next business day
+
+Severity 4 (Low):
+  - Cosmetic issues
+  - Questions
+  Response: Best effort
+```
+
+**Incident Runbook Template:**
+```markdown
+# Incident: API Service High Latency
+
+## Symptoms
+- API p95 latency > 500ms
+- Users reporting slow responses
+- Alert: "api-service-latency-high" firing
+
+## Impact
+- Degraded user experience
+- Potential timeout errors
+
+## Investigation Steps
+
+1. Check service status:
+   \`\`\`bash
+   kubectl get pods -n production -l app=api-service
+   kubectl top pods -n production -l app=api-service
+   \`\`\`
+
+2. Check recent deployments:
+   \`\`\`bash
+   kubectl rollout history deployment/api-service -n production
+   \`\`\`
+
+3. Check database performance:
+   - Query: `rate(pg_stat_database_tup_fetched[5m])`
+   - Check slow query log
+
+4. Check downstream dependencies:
+   - External API status
+   - Cache hit rate
+
+## Mitigation
+
+### Quick Fixes
+- Scale up: `kubectl scale deployment api-service --replicas=10`
+- Rollback: `kubectl rollout undo deployment/api-service`
+
+### Root Cause Fixes
+- Optimize slow queries
+- Add caching layer
+- Increase resource limits
+
+## Communication
+
+- Slack: #incidents channel
+- Status page: Update customer-facing status
+- Stakeholders: Notify via PagerDuty
+
+## Post-Incident
+
+- Create post-incident review
+- Update runbook with learnings
+- File tickets for improvements
+```
+
+### Pattern 3: Chaos Engineering
+
+```yaml
+# chaos-experiments/pod-failure.yaml
+apiVersion: chaos-mesh.org/v1alpha1
+kind: PodChaos
+metadata:
+  name: api-service-pod-failure
+  namespace: chaos-testing
+spec:
+  action: pod-failure
+  mode: one
+  selector:
+    namespaces:
+      - production
+    labelSelectors:
+      app: api-service
+  duration: "30s"
+  scheduler:
+    cron: "@every 2h"  # Run every 2 hours
+
+---
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+🎯 SKILL ACTIVATED: sre
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+# Network chaos
+apiVersion: chaos-mesh.org/v1alpha1
+kind: NetworkChaos
+metadata:
+  name: api-network-delay
+spec:
+  action: delay
+  mode: one
+  selector:
+    namespaces:
+      - production
+    labelSelectors:
+      app: api-service
+  delay:
+    latency: "100ms"
+    correlation: "100"
+    jitter: "0ms"
+  duration: "5m"
+```
+
+## Resource Files
+
+For detailed guidance on specific topics, see:
+
+### Core SRE Practices
+- **[slo-sli-sla.md](resources/slo-sli-sla.md)** - Service level definitions, SLO implementation, error budgets, SLA management
+- **[incident-management.md](resources/incident-management.md)** - Incident response procedures, runbooks, post-mortems, severity levels
+- **[toil-reduction.md](resources/toil-reduction.md)** - Identifying toil, automation strategies, measuring toil reduction
+
+### Observability
+- **[observability-stack.md](resources/observability-stack.md)** - Metrics, logs, traces, Prometheus, Grafana, Jaeger, distributed tracing
+- **[alerting-best-practices.md](resources/alerting-best-practices.md)** - Alert design, notification strategies, on-call alerting, alert fatigue
+- **[performance-optimization.md](resources/performance-optimization.md)** - Profiling, performance tuning, optimization techniques, bottleneck analysis
+
+### Operations
+- **[on-call-runbooks.md](resources/on-call-runbooks.md)** - On-call rotations, escalation procedures, runbook templates, burnout prevention
+- **[capacity-planning.md](resources/capacity-planning.md)** - Load testing, capacity modeling, growth forecasting, resource planning
+- **[disaster-recovery.md](resources/disaster-recovery.md)** - Backup strategies, RPO/RTO, DR testing, failover procedures, BC planning
+
+### Reliability Patterns
+- **[reliability-patterns.md](resources/reliability-patterns.md)** - Circuit breakers, retries, timeouts, bulkheads, rate limiting, graceful degradation
+- **[chaos-engineering.md](resources/chaos-engineering.md)** - Fault injection, resilience testing, game days, chaos experiments
+
+## Best Practices
+
+### 1. Start with User-Focused SLIs
+
+Measure what matters to users, not infrastructure metrics.
+
+```yaml
+# Good SLIs
+- Request success rate
+- Request latency
+- Data freshness
+
+# Poor SLIs
+- CPU usage
+- Memory usage
+- Pod count
+```
+
+### 2. Set Realistic SLOs
+
+```
+Don't aim for 100% - it's:
+- Impossible to achieve
+- Extremely expensive
+- Prevents risk-taking
+
+Typical SLOs:
+- User-facing: 99.9% - 99.99%
+- Internal services: 99% - 99.9%
+- Batch jobs: 95% - 99%
+```
+
+### 3. Embrace Error Budgets
+
+```
+Error budget available:
+✅ Deploy new features
+✅ Experiment
+✅ Take calculated risks
+
+Error budget depleted:
+❌ Feature freeze
+✅ Focus on reliability
+✅ Fix technical debt
+```
+
+### 4. Automate Toil
+
+```
+Toil = Manual, repetitive, automatable work
+
+Track toil:
+- Measure time spent on toil
+- Target: <50% of SRE time on toil
+- Automate high-frequency tasks
+```
+
+### 5. Blameless Post-Mortems
+
+```markdown
+# Post-Incident Review Template
+
+## Incident Summary
+- Date/Time
+- Duration
+- Severity
+- Services affected
+
+## Impact
+- Users affected
+- Revenue impact
+- SLO impact
+
+## Root Cause
+Technical explanation (not who)
+
+## Timeline
+Detailed chronology
+
+## Resolution
+What fixed it
+
+## Action Items
+- [ ] Short-term fixes
+- [ ] Long-term improvements
+- [ ] Process changes
+
+## Lessons Learned
+What went well
+What could be improved
+```
+
+### 6. Gradual Rollouts
+
+```
+Deployment stages:
+1. Canary (1% traffic) - 15 mins
+2. Small (10% traffic) - 30 mins
+3. Medium (50% traffic) - 1 hour
+4. Full (100% traffic)
+
+Automated rollback on:
+- Error rate increase >1%
+- Latency increase >20%
+- Failed health checks
+```
+
+## Anti-Patterns to Avoid
+
+❌ 100% availability target (unrealistic, expensive)
+❌ Ignoring error budgets (defeats the purpose)
+❌ Alert fatigue (too many noisy alerts)
+❌ Manual toil (should be automated)
+❌ Blame culture (prevents learning)
+❌ No runbooks (tribal knowledge)
+❌ Reactive only (should be proactive)
+❌ Monitoring without alerting (data without action)
+❌ No capacity planning (surprise outages)
+❌ Skipping post-mortems (missing learning opportunities)
+
+## Integration Points
+
+This skill integrates with:
+- **platform-engineering**: Infrastructure reliability, Kubernetes operations
+- **devsecops**: Security incident response, vulnerability management
+- **release-engineering**: Safe deployments, rollback strategies
+- **cloud-engineering**: Cloud monitoring, disaster recovery
+- **systems-engineering**: Performance tuning, system monitoring
+
+## Triggers and Activation
+
+This skill activates when you:
+- Define or modify SLOs/SLIs
+- Respond to incidents
+- Set up monitoring and alerting
+- Plan capacity or performance optimization
+- Conduct post-incident reviews
+- Implement chaos engineering
+
+---
+
+**Focus:** Reliability through engineering discipline
+**Key Principle:** Measure everything, improve continuously
+**Maintained by:** SRE team based on Google SRE practices and industry standards