aigroup-workflow 1.0.0 → 1.1.0

package/skills/jarvis/sre-engineer/references/monitoring-alerting.md

@@ -0,0 +1,424 @@
+# Monitoring and Alerting
+
+## Golden Signals Monitoring
+
+Monitor the four golden signals for every service.
+
+```yaml
+# prometheus_rules.yaml - Golden signals recording rules
+groups:
+  - name: golden_signals
+    interval: 30s
+    rules:
+      # Latency: Request duration
+      - record: service:http_request_duration_seconds:p50
+        expr: |
+          histogram_quantile(0.50,
+            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
+          )
+
+      - record: service:http_request_duration_seconds:p95
+        expr: |
+          histogram_quantile(0.95,
+            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
+          )
+
+      - record: service:http_request_duration_seconds:p99
+        expr: |
+          histogram_quantile(0.99,
+            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
+          )
+
+      # Traffic: Requests per second
+      - record: service:http_requests:rate5m
+        expr: |
+          sum(rate(http_requests_total[5m])) by (service)
+
+      # Errors: Error rate
+      - record: service:http_requests:error_rate5m
+        expr: |
+          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
+          /
+          sum(rate(http_requests_total[5m])) by (service)
+
+      # Saturation: Resource utilization
+      - record: service:cpu_utilization
+        expr: |
+          avg(rate(container_cpu_usage_seconds_total[5m])) by (service)
+
+      - record: service:memory_utilization
+        expr: |
+          avg(container_memory_working_set_bytes / container_spec_memory_limit_bytes)
+          by (service)
+```
+
+## Alert Design Principles
+
+Good alerts are actionable, not just informative.
+
+```yaml
+# alerts.yaml - SLO-based alerting
+groups:
+  - name: slo_alerts
+    rules:
+      # Multi-window burn rate alert (fast burn)
+      - alert: ErrorBudgetBurnRateFast
+        expr: |
+          (
+            service:http_requests:error_rate5m > (14.4 * 0.001)
+            and
+            service:http_requests:error_rate1h > (14.4 * 0.001)
+          )
+        for: 2m
+        labels:
+          severity: critical
+          slo: availability
+        annotations:
+          summary: "Fast error budget burn on {{ $labels.service }}"
+          description: |
+            Service {{ $labels.service }} is burning error budget at 14.4x rate.
+            At this rate, 30-day budget will exhaust in 2 days.
+
+            Current error rate: {{ $value | humanizePercentage }}
+            Threshold: 1.44%
+
+            RUNBOOK: https://runbooks.example.com/error-budget-burn
+
+      # Slow burn rate alert
+      - alert: ErrorBudgetBurnRateSlow
+        expr: |
+          (
+            service:http_requests:error_rate6h > (6 * 0.001)
+            and
+            service:http_requests:error_rate1d > (6 * 0.001)
+          )
+        for: 15m
+        labels:
+          severity: warning
+          slo: availability
+        annotations:
+          summary: "Slow error budget burn on {{ $labels.service }}"
+          description: |
+            Service {{ $labels.service }} is burning error budget at 6x rate.
+
+            RUNBOOK: https://runbooks.example.com/error-budget-burn
+
+      # Latency SLO violation
+      - alert: LatencySLOViolation
+        expr: |
+          service:http_request_duration_seconds:p99 > 0.5
+        for: 5m
+        labels:
+          severity: warning
+          slo: latency
+        annotations:
+          summary: "P99 latency exceeds 500ms on {{ $labels.service }}"
+          description: |
+            P99 latency is {{ $value }}s, exceeding 500ms threshold.
+
+            Check:
+            1. Database query performance
+            2. External API latency
+            3. Resource saturation (CPU/memory)
+
+            RUNBOOK: https://runbooks.example.com/high-latency
+
+      # Saturation alert
+      - alert: HighMemoryUtilization
+        expr: |
+          service:memory_utilization > 0.85
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High memory usage on {{ $labels.service }}"
+          description: |
+            Memory utilization is {{ $value | humanizePercentage }}.
+
+            Actions:
+            1. Check for memory leaks
+            2. Review recent deployments
+            3. Consider scaling up
+
+            RUNBOOK: https://runbooks.example.com/high-memory
+```
+
+## Alert Runbook Template
+
+Every alert must link to a runbook with clear remediation steps.
+
+```markdown
+# Runbook: Error Budget Burn Rate
+
+## Alert: ErrorBudgetBurnRateFast
+
+### Description
+The service is consuming error budget faster than sustainable rate.
+At current rate, the 30-day error budget will be exhausted within 2 days.
+
+### Severity: Critical
+
+### Impact
+- Users experiencing elevated error rates
+- Risk of SLO violation and feature freeze
+- Potential customer impact
+
+### Triage Steps
+
+1. **Check current error rate**
+   ```promql
+   rate(http_requests_total{status=~"5..", service="api"}[5m])
+   ```
+
+2. **Identify error types**
+   ```bash
+   kubectl logs -l app=api --tail=100 | grep ERROR
+   ```
+
+3. **Check recent deployments**
+   ```bash
+   kubectl rollout history deployment/api
+   ```
+
+4. **Review dependencies**
+   - Database health
+   - External API status
+   - Infrastructure issues
+
+### Remediation
+
+**If caused by recent deployment:**
+```bash
+# Rollback to previous version
+kubectl rollout undo deployment/api
+
+# Verify rollback
+kubectl rollout status deployment/api
+```
+
+**If database issue:**
+```bash
+# Check database connections
+kubectl exec -it postgres-0 -- psql -c "SELECT count(*) FROM pg_stat_activity;"
+
+# Check slow queries
+kubectl exec -it postgres-0 -- psql -c "SELECT * FROM pg_stat_statements ORDER BY mean_time DESC LIMIT 10;"
+```
+
+**If traffic spike:**
+```bash
+# Scale up replicas
+kubectl scale deployment/api --replicas=10
+
+# Enable rate limiting
+kubectl apply -f rate-limit-config.yaml
+```
+
+### Communication
+
+**Slack template:**
+```
+:fire: INCIDENT: Error budget burn rate critical
+
+Service: api
+Error rate: [X]%
+Impact: [describe user impact]
+ETA: [when will it be resolved]
+
+Incident doc: [link]
+```
+
+### Prevention
+- Add integration tests for this failure mode
+- Implement circuit breaker for external dependencies
+- Add capacity planning for traffic spikes
+```
+
+## Dashboard Configuration
+
+```python
+# grafana_dashboard.py - Generate SLO dashboard using Grafana SDK
+from grafana_dashboard import Dashboard, Panel, Target
+
+def create_slo_dashboard(service: str) -> dict:
+    """Create SLO monitoring dashboard for a service."""
+
+    dashboard = Dashboard(
+        title=f"{service} - SLO Dashboard",
+        tags=["slo", "sre", service],
+        refresh="1m",
+    )
+
+    # SLI Current Value
+    dashboard.add_panel(
+        Panel(
+            title="Availability SLI (30d)",
+            targets=[
+                Target(
+                    expr=f"""
+                    sum(rate(http_requests_total{{
+                        status=~"2..",
+                        service="{service}"
+                    }}[30d]))
+                    /
+                    sum(rate(http_requests_total{{service="{service}"}}[30d]))
+                    """,
+                    legendFormat="Current SLI",
+                ),
+            ],
+            thresholds=[
+                {"value": 0.999, "color": "red"},
+                {"value": 0.9995, "color": "yellow"},
+                {"value": 1.0, "color": "green"},
+            ],
+        )
+    )
+
+    # Error Budget Remaining
+    dashboard.add_panel(
+        Panel(
+            title="Error Budget Remaining",
+            targets=[
+                Target(
+                    expr=f"""
+                    (0.001 - (1 - (
+                      sum(rate(http_requests_total{{
+                          status=~"2..",
+                          service="{service}"
+                      }}[30d]))
+                      /
+                      sum(rate(http_requests_total{{service="{service}"}}[30d]))
+                    ))) / 0.001 * 100
+                    """,
+                    legendFormat="Budget Remaining %",
+                ),
+            ],
+            unit="percent",
+        )
+    )
+
+    # Burn Rate
+    dashboard.add_panel(
+        Panel(
+            title="Error Budget Burn Rate",
+            targets=[
+                Target(
+                    expr=f"""
+                    (1 - (
+                      sum(rate(http_requests_total{{
+                          status=~"2..",
+                          service="{service}"
+                      }}[1h]))
+                      /
+                      sum(rate(http_requests_total{{service="{service}"}}[1h]))
+                    )) / 0.001
+                    """,
+                    legendFormat="1h burn rate",
+                ),
+            ],
+            thresholds=[
+                {"value": 1.0, "color": "green"},
+                {"value": 6.0, "color": "yellow"},
+                {"value": 14.4, "color": "red"},
+            ],
+        )
+    )
+
+    # Golden Signals
+    dashboard.add_row("Golden Signals")
+
+    dashboard.add_panel(
+        Panel(
+            title="Latency (P50, P95, P99)",
+            targets=[
+                Target(
+                    expr=f'service:http_request_duration_seconds:p50{{service="{service}"}}',
+                    legendFormat="p50",
+                ),
+                Target(
+                    expr=f'service:http_request_duration_seconds:p95{{service="{service}"}}',
+                    legendFormat="p95",
+                ),
+                Target(
+                    expr=f'service:http_request_duration_seconds:p99{{service="{service}"}}',
+                    legendFormat="p99",
+                ),
+            ],
+            unit="s",
+        )
+    )
+
+    return dashboard.to_json()
+```
+
+## Alert Fatigue Prevention
+
+```python
+from dataclasses import dataclass
+from typing import List
+
+@dataclass
+class AlertQualityMetrics:
+    """Track alert quality to prevent fatigue."""
+    total_alerts: int
+    actionable_alerts: int  # Required manual intervention
+    false_positives: int
+    auto_resolved: int  # Resolved before human action
+
+    @property
+    def precision(self) -> float:
+        """Percentage of alerts that were actionable."""
+        if self.total_alerts == 0:
+            return 0.0
+        return (self.actionable_alerts / self.total_alerts) * 100
+
+    @property
+    def toil_ratio(self) -> float:
+        """Percentage of alerts that required manual work."""
+        if self.total_alerts == 0:
+            return 0.0
+        return ((self.actionable_alerts + self.false_positives) / self.total_alerts) * 100
+
+# Target: >90% precision, <30% toil
+metrics = AlertQualityMetrics(
+    total_alerts=100,
+    actionable_alerts=85,
+    false_positives=5,
+    auto_resolved=10,
+)
+
+print(f"Alert precision: {metrics.precision}%")
+print(f"Toil ratio: {metrics.toil_ratio}%")
+```
+
+## On-Call Alert Guidelines
+
+```yaml
+# on_call_alert_standards.yaml
+alert_standards:
+  page_worthy:
+    - "Immediate user impact (>5% of users affected)"
+    - "SLO violation in progress"
+    - "Error budget burn rate critical (>10x)"
+    - "Security incident"
+    - "Data loss risk"
+
+  not_page_worthy:
+    - "Predictive alerts without current impact"
+    - "Informational metrics"
+    - "Non-user-facing issues"
+    - "Slow trends (address during business hours)"
+
+  alert_routing:
+    critical:
+      - page: on-call engineer
+      - slack: "#incidents"
+      - create: incident doc
+
+    warning:
+      - slack: "#alerts"
+      - ticket: auto-create if persists >1h
+
+    info:
+      - dashboard: only
+```

package/skills/jarvis/sre-engineer/references/slo-sli-management.md

@@ -0,0 +1,238 @@
+# SLO/SLI Management
+
+## SLI Definition Patterns
+
+Service Level Indicators are quantitative measurements of service behavior.
+
+### Request-Based SLIs
+
+```python
+# Availability SLI: Proportion of successful requests
+# Good events: HTTP 200-299, 4XX (client errors don't count against SLI)
+# Total events: All requests
+
+def calculate_availability_sli(metrics):
+    """Calculate availability SLI from request metrics."""
+    successful_requests = metrics['http_2xx'] + metrics['http_4xx']
+    total_requests = metrics['total_requests']
+
+    if total_requests == 0:
+        return 1.0  # No traffic = 100% available
+
+    return successful_requests / total_requests
+
+# Example: 99.9% of requests return successfully
+# SLI = successful_requests / total_requests
+```
+
+### Latency-Based SLIs
+
+```python
+def calculate_latency_sli(latency_histogram, threshold_ms=500):
+    """Calculate latency SLI from histogram.
+
+    Args:
+        latency_histogram: dict mapping latency buckets to request counts
+        threshold_ms: latency threshold in milliseconds
+
+    Returns:
+        float: Proportion of requests faster than threshold
+    """
+    fast_requests = sum(
+        count for bucket, count in latency_histogram.items()
+        if bucket <= threshold_ms
+    )
+    total_requests = sum(latency_histogram.values())
+
+    return fast_requests / total_requests if total_requests > 0 else 1.0
+
+# Example: 99% of requests complete in <500ms
+# SLI = requests_under_500ms / total_requests
+```
+
+## SLO Configuration
+
+```yaml
+# slo_config.yaml - Production API SLO definitions
+apiVersion: sre/v1
+kind: ServiceLevelObjective
+metadata:
+  service: payment-api
+  environment: production
+spec:
+  slos:
+    - name: availability
+      description: "Users can successfully complete payment requests"
+      sli:
+        metric: http_requests_total
+        query: |
+          sum(rate(http_requests_total{status=~"2..|4..", service="payment-api"}[30d]))
+          /
+          sum(rate(http_requests_total{service="payment-api"}[30d]))
+      target: 0.999  # 99.9%
+      window: 30d
+
+    - name: latency
+      description: "Payment requests complete quickly"
+      sli:
+        metric: http_request_duration_seconds
+        query: |
+          histogram_quantile(0.95,
+            sum(rate(http_request_duration_seconds_bucket{service="payment-api"}[30d]))
+            by (le)
+          ) < 0.5
+      target: 0.99  # 99% of requests under 500ms
+      window: 30d
+```
+
+## Golden Signals
+
+The four golden signals every service should measure:
+
+```python
+from dataclasses import dataclass
+from typing import Dict
+
+@dataclass
+class GoldenSignals:
+    """Four golden signals of monitoring."""
+
+    # Latency: Time to service requests (success vs failure)
+    latency_p50_ms: float
+    latency_p95_ms: float
+    latency_p99_ms: float
+
+    # Traffic: Demand on your system (requests/sec)
+    requests_per_second: float
+
+    # Errors: Rate of failed requests
+    error_rate: float  # 0.0 to 1.0
+
+    # Saturation: How "full" is your service (CPU, memory, disk)
+    cpu_utilization: float  # 0.0 to 1.0
+    memory_utilization: float  # 0.0 to 1.0
+
+    def is_healthy(self, slo_targets: Dict[str, float]) -> bool:
+        """Check if all signals are within SLO targets."""
+        return (
+            self.latency_p99_ms <= slo_targets['latency_p99_ms'] and
+            self.error_rate <= (1 - slo_targets['availability']) and
+            self.cpu_utilization <= slo_targets['max_cpu'] and
+            self.memory_utilization <= slo_targets['max_memory']
+        )
+```
+
+## SLO Calculation Examples
+
+```python
+from datetime import timedelta
+from typing import NamedTuple
+
+class SLOTarget(NamedTuple):
+    """SLO target configuration."""
+    target: float  # 0.999 for 99.9%
+    window: timedelta  # 30 days
+
+    @property
+    def error_budget(self) -> float:
+        """Calculate error budget (1 - target)."""
+        return 1 - self.target
+
+    @property
+    def allowed_downtime(self) -> timedelta:
+        """Calculate allowed downtime in window."""
+        total_seconds = self.window.total_seconds()
+        allowed_seconds = total_seconds * self.error_budget
+        return timedelta(seconds=allowed_seconds)
+
+# Example SLOs
+availability_slo = SLOTarget(target=0.999, window=timedelta(days=30))
+print(f"Error budget: {availability_slo.error_budget * 100}%")
+print(f"Allowed downtime: {availability_slo.allowed_downtime}")
+# Output:
+# Error budget: 0.1%
+# Allowed downtime: 43.2 minutes per 30 days
+
+latency_slo = SLOTarget(target=0.99, window=timedelta(days=30))
+print(f"99% of requests must be fast")
+print(f"1% can be slow: {latency_slo.error_budget * 100}%")
+```
+
+## Multi-Window SLO Tracking
+
+```python
+class MultiWindowSLO:
+    """Track SLO compliance across multiple time windows."""
+
+    def __init__(self, target: float):
+        self.target = target
+        self.windows = {
+            '1h': timedelta(hours=1),
+            '24h': timedelta(hours=24),
+            '7d': timedelta(days=7),
+            '30d': timedelta(days=30),
+        }
+
+    def check_compliance(self, sli_values: Dict[str, float]) -> Dict[str, bool]:
+        """Check if SLI meets target in each window.
+
+        Args:
+            sli_values: Dict mapping window name to measured SLI
+
+        Returns:
+            Dict mapping window name to compliance boolean
+        """
+        return {
+            window: sli >= self.target
+            for window, sli in sli_values.items()
+        }
+
+    def get_burn_rate(self, current_sli: float) -> float:
+        """Calculate error budget burn rate.
+
+        Burn rate > 1.0 means burning budget faster than sustainable.
+        """
+        error_budget = 1 - self.target
+        current_error_rate = 1 - current_sli
+
+        if error_budget == 0:
+            return float('inf')
+
+        return current_error_rate / error_budget
+
+# Usage
+slo = MultiWindowSLO(target=0.999)
+current_sli = 0.997  # 99.7% availability
+
+burn_rate = slo.get_burn_rate(current_sli)
+print(f"Burn rate: {burn_rate}x")
+# If burn_rate = 3.0, burning budget 3x faster than sustainable
+```
+
+## SLO Review Checklist
+
+Before finalizing SLOs:
+
+1. **User-centric**: Does it measure user-facing impact?
+2. **Achievable**: Can you meet it with current architecture?
+3. **Measurable**: Can you accurately track the SLI?
+4. **Meaningful**: Does violating it require action?
+5. **Documented**: Is the calculation clear and agreed upon?
+6. **Budgeted**: Is there an error budget policy?
+
+## Common SLO Targets
+
+```yaml
+# Typical SLO targets by service tier
+tier_1_critical:
+  availability: 99.99%  # 4m 23s downtime/month
+  latency_p99: 100ms
+
+tier_2_important:
+  availability: 99.9%   # 43m 28s downtime/month
+  latency_p99: 500ms
+
+tier_3_standard:
+  availability: 99.5%   # 3h 37m downtime/month
+  latency_p99: 1000ms
+```