blockmine 1.24.0 → 1.25.0

package/.claude/skills/sre/SKILL.md

@@ -1,464 +0,0 @@
-# SRE - Site Reliability Engineering
-
-Comprehensive guide for implementing SRE practices: SLI/SLO/SLA definitions, error budgets, incident management, on-call best practices, chaos engineering, observability, and reliability patterns for production systems.
-
-## Purpose
-
-Enable teams to build and operate reliable, scalable systems by applying software engineering principles to infrastructure and operations challenges.
-
-## When to Use This Skill
-
-Automatically activates when working on:
-- Defining SLIs, SLOs, and SLAs
-- Managing error budgets
-- Incident response and management
-- On-call rotation and procedures
-- Observability and monitoring
-- Chaos engineering and resilience testing
-- Capacity planning and performance
-- Disaster recovery planning
-
-## Quick Start Checklist
-
-When implementing SRE practices:
-
-- [ ] Define Service Level Indicators (SLIs)
-- [ ] Set Service Level Objectives (SLOs) with stakeholders
-- [ ] Implement error budget tracking
-- [ ] Set up comprehensive monitoring and alerting
-- [ ] Create incident response runbooks
-- [ ] Establish on-call rotation
-- [ ] Implement automated remediation
-- [ ] Conduct post-incident reviews
-- [ ] Perform chaos engineering experiments
-- [ ] Document capacity planning process
-- [ ] Set up disaster recovery procedures
-
-## Core Concepts
-
-### SLI / SLO / SLA Hierarchy
-
-```
-SLA (Service Level Agreement)
-  ↓ Commits to customers
-  ↓
-SLO (Service Level Objective)
-  ↓ Internal target (stricter than SLA)
-  ↓
-SLI (Service Level Indicator)
-  ↓ Measurement that shows if you're meeting SLO
-  ↓
-Metrics (actual measurements)
-```
-
-**Example:**
-```
-SLA:  99.9% uptime guaranteed (3 customers)
-SLO:  99.95% uptime target (internal)
-SLI:  Percentage of successful HTTP requests
-      (2xx, 3xx responses / total requests)
-```
-
-### Error Budgets
-
-```
-Error Budget = 1 - SLO
-
-Example:
-SLO: 99.9% → Error Budget: 0.1%
-
-Monthly calculations (30 days):
-- Total time: 43,200 minutes
-- Allowed downtime: 43.2 minutes
-
-If error budget is:
-- Available → Can take risks, deploy features
-- Depleted → Focus on reliability, freeze features
-```
-
-### The Four Golden Signals
-
-```yaml
-1. Latency: How long requests take
-   - Good: p50, p95, p99 latencies
-   - Bad: Average latency (hides outliers)
-
-2. Traffic: Demand on your system
-   - Requests per second
-   - Transactions per second
-   - Bandwidth usage
-
-3. Errors: Rate of failed requests
-   - HTTP 5xx errors
-   - Failed operations
-   - Wrong results
-
-4. Saturation: How "full" your service is
-   - CPU usage
-   - Memory usage
-   - Disk I/O
-   - Queue depth
-```
-
-## Common Patterns
-
-### Pattern 1: SLO Definition
-
-```yaml
-# slos/api-service.yaml
-apiVersion: v1
-kind: SLO
-metadata:
-  name: api-service-availability
-  service: api-service
-spec:
-  description: "API service availability for customer requests"
-
-  # SLI definition
-  sli:
-    name: availability
-    description: "Percentage of successful requests"
-    query: |
-      sum(rate(http_requests_total{job="api-service",code=~"2.."}[5m]))
-      /
-      sum(rate(http_requests_total{job="api-service"}[5m]))
-
-  # SLO target
-  objectives:
-  - target: 0.999  # 99.9%
-    window: 30d    # Rolling 30 days
-
-  # Alert thresholds
-  alerting:
-    - burn_rate: 14.4   # Alert if burning budget 14.4x faster
-      short_window: 1h
-      long_window: 5m
-      severity: critical
-
-    - burn_rate: 6
-      short_window: 6h
-      long_window: 30m
-      severity: warning
-```
-
-**Latency SLO:**
-```yaml
-apiVersion: v1
-kind: SLO
-metadata:
-  name: api-service-latency
-spec:
-  sli:
-    name: latency
-    description: "95th percentile latency under 200ms"
-    query: |
-      histogram_quantile(0.95,
-        sum(rate(http_request_duration_seconds_bucket{job="api-service"}[5m])) by (le)
-      )
-
-  objectives:
-  - target: 0.200  # 200ms
-    window: 7d
-```
-
-### Pattern 2: Incident Response
-
-**Incident Severity Levels:**
-```yaml
-Severity 1 (Critical):
-  - Complete service outage
-  - Security breach
-  - Data loss
-  Response: Immediate, all hands
-  Escalation: 15 minutes
-
-Severity 2 (High):
-  - Partial outage
-  - Degraded performance
-  - Feature broken
-  Response: Within 30 minutes
-  Escalation: 1 hour
-
-Severity 3 (Medium):
-  - Minor issues
-  - Non-critical features affected
-  Response: Within 4 hours
-  Escalation: Next business day
-
-Severity 4 (Low):
-  - Cosmetic issues
-  - Questions
-  Response: Best effort
-```
-
-**Incident Runbook Template:**
-```markdown
-# Incident: API Service High Latency
-
-## Symptoms
-- API p95 latency > 500ms
-- Users reporting slow responses
-- Alert: "api-service-latency-high" firing
-
-## Impact
-- Degraded user experience
-- Potential timeout errors
-
-## Investigation Steps
-
-1. Check service status:
-   \`\`\`bash
-   kubectl get pods -n production -l app=api-service
-   kubectl top pods -n production -l app=api-service
-   \`\`\`
-
-2. Check recent deployments:
-   \`\`\`bash
-   kubectl rollout history deployment/api-service -n production
-   \`\`\`
-
-3. Check database performance:
-   - Query: `rate(pg_stat_database_tup_fetched[5m])`
-   - Check slow query log
-
-4. Check downstream dependencies:
-   - External API status
-   - Cache hit rate
-
-## Mitigation
-
-### Quick Fixes
-- Scale up: `kubectl scale deployment api-service --replicas=10`
-- Rollback: `kubectl rollout undo deployment/api-service`
-
-### Root Cause Fixes
-- Optimize slow queries
-- Add caching layer
-- Increase resource limits
-
-## Communication
-
-- Slack: #incidents channel
-- Status page: Update customer-facing status
-- Stakeholders: Notify via PagerDuty
-
-## Post-Incident
-
-- Create post-incident review
-- Update runbook with learnings
-- File tickets for improvements
-```
-
-### Pattern 3: Chaos Engineering
-
-```yaml
-# chaos-experiments/pod-failure.yaml
-apiVersion: chaos-mesh.org/v1alpha1
-kind: PodChaos
-metadata:
-  name: api-service-pod-failure
-  namespace: chaos-testing
-spec:
-  action: pod-failure
-  mode: one
-  selector:
-    namespaces:
-      - production
-    labelSelectors:
-      app: api-service
-  duration: "30s"
-  scheduler:
-    cron: "@every 2h"  # Run every 2 hours
-
----
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-🎯 SKILL ACTIVATED: sre
-━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-# Network chaos
-apiVersion: chaos-mesh.org/v1alpha1
-kind: NetworkChaos
-metadata:
-  name: api-network-delay
-spec:
-  action: delay
-  mode: one
-  selector:
-    namespaces:
-      - production
-    labelSelectors:
-      app: api-service
-  delay:
-    latency: "100ms"
-    correlation: "100"
-    jitter: "0ms"
-  duration: "5m"
-```
-
-## Resource Files
-
-For detailed guidance on specific topics, see:
-
-### Core SRE Practices
-- **[slo-sli-sla.md](resources/slo-sli-sla.md)** - Service level definitions, SLO implementation, error budgets, SLA management
-- **[incident-management.md](resources/incident-management.md)** - Incident response procedures, runbooks, post-mortems, severity levels
-- **[toil-reduction.md](resources/toil-reduction.md)** - Identifying toil, automation strategies, measuring toil reduction
-
-### Observability
-- **[observability-stack.md](resources/observability-stack.md)** - Metrics, logs, traces, Prometheus, Grafana, Jaeger, distributed tracing
-- **[alerting-best-practices.md](resources/alerting-best-practices.md)** - Alert design, notification strategies, on-call alerting, alert fatigue
-- **[performance-optimization.md](resources/performance-optimization.md)** - Profiling, performance tuning, optimization techniques, bottleneck analysis
-
-### Operations
-- **[on-call-runbooks.md](resources/on-call-runbooks.md)** - On-call rotations, escalation procedures, runbook templates, burnout prevention
-- **[capacity-planning.md](resources/capacity-planning.md)** - Load testing, capacity modeling, growth forecasting, resource planning
-- **[disaster-recovery.md](resources/disaster-recovery.md)** - Backup strategies, RPO/RTO, DR testing, failover procedures, BC planning
-
-### Reliability Patterns
-- **[reliability-patterns.md](resources/reliability-patterns.md)** - Circuit breakers, retries, timeouts, bulkheads, rate limiting, graceful degradation
-- **[chaos-engineering.md](resources/chaos-engineering.md)** - Fault injection, resilience testing, game days, chaos experiments
-
-## Best Practices
-
-### 1. Start with User-Focused SLIs
-
-Measure what matters to users, not infrastructure metrics.
-
-```yaml
-# Good SLIs
-- Request success rate
-- Request latency
-- Data freshness
-
-# Poor SLIs
-- CPU usage
-- Memory usage
-- Pod count
-```
-
-### 2. Set Realistic SLOs
-
-```
-Don't aim for 100% - it's:
-- Impossible to achieve
-- Extremely expensive
-- Prevents risk-taking
-
-Typical SLOs:
-- User-facing: 99.9% - 99.99%
-- Internal services: 99% - 99.9%
-- Batch jobs: 95% - 99%
-```
-
-### 3. Embrace Error Budgets
-
-```
-Error budget available:
-✅ Deploy new features
-✅ Experiment
-✅ Take calculated risks
-
-Error budget depleted:
-❌ Feature freeze
-✅ Focus on reliability
-✅ Fix technical debt
-```
-
-### 4. Automate Toil
-
-```
-Toil = Manual, repetitive, automatable work
-
-Track toil:
-- Measure time spent on toil
-- Target: <50% of SRE time on toil
-- Automate high-frequency tasks
-```
-
-### 5. Blameless Post-Mortems
-
-```markdown
-# Post-Incident Review Template
-
-## Incident Summary
-- Date/Time
-- Duration
-- Severity
-- Services affected
-
-## Impact
-- Users affected
-- Revenue impact
-- SLO impact
-
-## Root Cause
-Technical explanation (not who)
-
-## Timeline
-Detailed chronology
-
-## Resolution
-What fixed it
-
-## Action Items
-- [ ] Short-term fixes
-- [ ] Long-term improvements
-- [ ] Process changes
-
-## Lessons Learned
-What went well
-What could be improved
-```
-
-### 6. Gradual Rollouts
-
-```
-Deployment stages:
-1. Canary (1% traffic) - 15 mins
-2. Small (10% traffic) - 30 mins
-3. Medium (50% traffic) - 1 hour
-4. Full (100% traffic)
-
-Automated rollback on:
-- Error rate increase >1%
-- Latency increase >20%
-- Failed health checks
-```
-
-## Anti-Patterns to Avoid
-
-❌ 100% availability target (unrealistic, expensive)
-❌ Ignoring error budgets (defeats the purpose)
-❌ Alert fatigue (too many noisy alerts)
-❌ Manual toil (should be automated)
-❌ Blame culture (prevents learning)
-❌ No runbooks (tribal knowledge)
-❌ Reactive only (should be proactive)
-❌ Monitoring without alerting (data without action)
-❌ No capacity planning (surprise outages)
-❌ Skipping post-mortems (missing learning opportunities)
-
-## Integration Points
-
-This skill integrates with:
-- **platform-engineering**: Infrastructure reliability, Kubernetes operations
-- **devsecops**: Security incident response, vulnerability management
-- **release-engineering**: Safe deployments, rollback strategies
-- **cloud-engineering**: Cloud monitoring, disaster recovery
-- **systems-engineering**: Performance tuning, system monitoring
-
-## Triggers and Activation
-
-This skill activates when you:
-- Define or modify SLOs/SLIs
-- Respond to incidents
-- Set up monitoring and alerting
-- Plan capacity or performance optimization
-- Conduct post-incident reviews
-- Implement chaos engineering
-
----
-
-**Focus:** Reliability through engineering discipline
-**Key Principle:** Measure everything, improve continuously
-**Maintained by:** SRE team based on Google SRE practices and industry standards