aigroup-workflow 2.1.2 → 2.2.0

package/skills/sre-engineer/references/monitoring-alerting.md

@@ -1,424 +0,0 @@
-# Monitoring and Alerting
-
-## Golden Signals Monitoring
-
-Monitor the four golden signals for every service.
-
-```yaml
-# prometheus_rules.yaml - Golden signals recording rules
-groups:
-  - name: golden_signals
-    interval: 30s
-    rules:
-      # Latency: Request duration
-      - record: service:http_request_duration_seconds:p50
-        expr: |
-          histogram_quantile(0.50,
-            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
-          )
-
-      - record: service:http_request_duration_seconds:p95
-        expr: |
-          histogram_quantile(0.95,
-            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
-          )
-
-      - record: service:http_request_duration_seconds:p99
-        expr: |
-          histogram_quantile(0.99,
-            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
-          )
-
-      # Traffic: Requests per second
-      - record: service:http_requests:rate5m
-        expr: |
-          sum(rate(http_requests_total[5m])) by (service)
-
-      # Errors: Error rate
-      - record: service:http_requests:error_rate5m
-        expr: |
-          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
-          /
-          sum(rate(http_requests_total[5m])) by (service)
-
-      # Saturation: Resource utilization
-      - record: service:cpu_utilization
-        expr: |
-          avg(rate(container_cpu_usage_seconds_total[5m])) by (service)
-
-      - record: service:memory_utilization
-        expr: |
-          avg(container_memory_working_set_bytes / container_spec_memory_limit_bytes)
-          by (service)
-```
-
-## Alert Design Principles
-
-Good alerts are actionable, not just informative.
-
-```yaml
-# alerts.yaml - SLO-based alerting
-groups:
-  - name: slo_alerts
-    rules:
-      # Multi-window burn rate alert (fast burn)
-      - alert: ErrorBudgetBurnRateFast
-        expr: |
-          (
-            service:http_requests:error_rate5m > (14.4 * 0.001)
-            and
-            service:http_requests:error_rate1h > (14.4 * 0.001)
-          )
-        for: 2m
-        labels:
-          severity: critical
-          slo: availability
-        annotations:
-          summary: "Fast error budget burn on {{ $labels.service }}"
-          description: |
-            Service {{ $labels.service }} is burning error budget at 14.4x rate.
-            At this rate, 30-day budget will exhaust in 2 days.
-
-            Current error rate: {{ $value | humanizePercentage }}
-            Threshold: 1.44%
-
-            RUNBOOK: https://runbooks.example.com/error-budget-burn
-
-      # Slow burn rate alert
-      - alert: ErrorBudgetBurnRateSlow
-        expr: |
-          (
-            service:http_requests:error_rate6h > (6 * 0.001)
-            and
-            service:http_requests:error_rate1d > (6 * 0.001)
-          )
-        for: 15m
-        labels:
-          severity: warning
-          slo: availability
-        annotations:
-          summary: "Slow error budget burn on {{ $labels.service }}"
-          description: |
-            Service {{ $labels.service }} is burning error budget at 6x rate.
-
-            RUNBOOK: https://runbooks.example.com/error-budget-burn
-
-      # Latency SLO violation
-      - alert: LatencySLOViolation
-        expr: |
-          service:http_request_duration_seconds:p99 > 0.5
-        for: 5m
-        labels:
-          severity: warning
-          slo: latency
-        annotations:
-          summary: "P99 latency exceeds 500ms on {{ $labels.service }}"
-          description: |
-            P99 latency is {{ $value }}s, exceeding 500ms threshold.
-
-            Check:
-            1. Database query performance
-            2. External API latency
-            3. Resource saturation (CPU/memory)
-
-            RUNBOOK: https://runbooks.example.com/high-latency
-
-      # Saturation alert
-      - alert: HighMemoryUtilization
-        expr: |
-          service:memory_utilization > 0.85
-        for: 10m
-        labels:
-          severity: warning
-        annotations:
-          summary: "High memory usage on {{ $labels.service }}"
-          description: |
-            Memory utilization is {{ $value | humanizePercentage }}.
-
-            Actions:
-            1. Check for memory leaks
-            2. Review recent deployments
-            3. Consider scaling up
-
-            RUNBOOK: https://runbooks.example.com/high-memory
-```
-
-## Alert Runbook Template
-
-Every alert must link to a runbook with clear remediation steps.
-
-```markdown
-# Runbook: Error Budget Burn Rate
-
-## Alert: ErrorBudgetBurnRateFast
-
-### Description
-The service is consuming error budget faster than sustainable rate.
-At current rate, the 30-day error budget will be exhausted within 2 days.
-
-### Severity: Critical
-
-### Impact
-- Users experiencing elevated error rates
-- Risk of SLO violation and feature freeze
-- Potential customer impact
-
-### Triage Steps
-
-1. **Check current error rate**
-   ```promql
-   rate(http_requests_total{status=~"5..", service="api"}[5m])
-   ```
-
-2. **Identify error types**
-   ```bash
-   kubectl logs -l app=api --tail=100 | grep ERROR
-   ```
-
-3. **Check recent deployments**
-   ```bash
-   kubectl rollout history deployment/api
-   ```
-
-4. **Review dependencies**
-   - Database health
-   - External API status
-   - Infrastructure issues
-
-### Remediation
-
-**If caused by recent deployment:**
-```bash
-# Rollback to previous version
-kubectl rollout undo deployment/api
-
-# Verify rollback
-kubectl rollout status deployment/api
-```
-
-**If database issue:**
-```bash
-# Check database connections
-kubectl exec -it postgres-0 -- psql -c "SELECT count(*) FROM pg_stat_activity;"
-
-# Check slow queries
-kubectl exec -it postgres-0 -- psql -c "SELECT * FROM pg_stat_statements ORDER BY mean_time DESC LIMIT 10;"
-```
-
-**If traffic spike:**
-```bash
-# Scale up replicas
-kubectl scale deployment/api --replicas=10
-
-# Enable rate limiting
-kubectl apply -f rate-limit-config.yaml
-```
-
-### Communication
-
-**Slack template:**
-```
-:fire: INCIDENT: Error budget burn rate critical
-
-Service: api
-Error rate: [X]%
-Impact: [describe user impact]
-ETA: [when will it be resolved]
-
-Incident doc: [link]
-```
-
-### Prevention
-- Add integration tests for this failure mode
-- Implement circuit breaker for external dependencies
-- Add capacity planning for traffic spikes
-```
-
-## Dashboard Configuration
-
-```python
-# grafana_dashboard.py - Generate SLO dashboard using Grafana SDK
-from grafana_dashboard import Dashboard, Panel, Target
-
-def create_slo_dashboard(service: str) -> dict:
-    """Create SLO monitoring dashboard for a service."""
-
-    dashboard = Dashboard(
-        title=f"{service} - SLO Dashboard",
-        tags=["slo", "sre", service],
-        refresh="1m",
-    )
-
-    # SLI Current Value
-    dashboard.add_panel(
-        Panel(
-            title="Availability SLI (30d)",
-            targets=[
-                Target(
-                    expr=f"""
-                    sum(rate(http_requests_total{{
-                        status=~"2..",
-                        service="{service}"
-                    }}[30d]))
-                    /
-                    sum(rate(http_requests_total{{service="{service}"}}[30d]))
-                    """,
-                    legendFormat="Current SLI",
-                ),
-            ],
-            thresholds=[
-                {"value": 0.999, "color": "red"},
-                {"value": 0.9995, "color": "yellow"},
-                {"value": 1.0, "color": "green"},
-            ],
-        )
-    )
-
-    # Error Budget Remaining
-    dashboard.add_panel(
-        Panel(
-            title="Error Budget Remaining",
-            targets=[
-                Target(
-                    expr=f"""
-                    (0.001 - (1 - (
-                      sum(rate(http_requests_total{{
-                          status=~"2..",
-                          service="{service}"
-                      }}[30d]))
-                      /
-                      sum(rate(http_requests_total{{service="{service}"}}[30d]))
-                    ))) / 0.001 * 100
-                    """,
-                    legendFormat="Budget Remaining %",
-                ),
-            ],
-            unit="percent",
-        )
-    )
-
-    # Burn Rate
-    dashboard.add_panel(
-        Panel(
-            title="Error Budget Burn Rate",
-            targets=[
-                Target(
-                    expr=f"""
-                    (1 - (
-                      sum(rate(http_requests_total{{
-                          status=~"2..",
-                          service="{service}"
-                      }}[1h]))
-                      /
-                      sum(rate(http_requests_total{{service="{service}"}}[1h]))
-                    )) / 0.001
-                    """,
-                    legendFormat="1h burn rate",
-                ),
-            ],
-            thresholds=[
-                {"value": 1.0, "color": "green"},
-                {"value": 6.0, "color": "yellow"},
-                {"value": 14.4, "color": "red"},
-            ],
-        )
-    )
-
-    # Golden Signals
-    dashboard.add_row("Golden Signals")
-
-    dashboard.add_panel(
-        Panel(
-            title="Latency (P50, P95, P99)",
-            targets=[
-                Target(
-                    expr=f'service:http_request_duration_seconds:p50{{service="{service}"}}',
-                    legendFormat="p50",
-                ),
-                Target(
-                    expr=f'service:http_request_duration_seconds:p95{{service="{service}"}}',
-                    legendFormat="p95",
-                ),
-                Target(
-                    expr=f'service:http_request_duration_seconds:p99{{service="{service}"}}',
-                    legendFormat="p99",
-                ),
-            ],
-            unit="s",
-        )
-    )
-
-    return dashboard.to_json()
-```
-
-## Alert Fatigue Prevention
-
-```python
-from dataclasses import dataclass
-from typing import List
-
-@dataclass
-class AlertQualityMetrics:
-    """Track alert quality to prevent fatigue."""
-    total_alerts: int
-    actionable_alerts: int  # Required manual intervention
-    false_positives: int
-    auto_resolved: int  # Resolved before human action
-
-    @property
-    def precision(self) -> float:
-        """Percentage of alerts that were actionable."""
-        if self.total_alerts == 0:
-            return 0.0
-        return (self.actionable_alerts / self.total_alerts) * 100
-
-    @property
-    def toil_ratio(self) -> float:
-        """Percentage of alerts that required manual work."""
-        if self.total_alerts == 0:
-            return 0.0
-        return ((self.actionable_alerts + self.false_positives) / self.total_alerts) * 100
-
-# Target: >90% precision, <30% toil
-metrics = AlertQualityMetrics(
-    total_alerts=100,
-    actionable_alerts=85,
-    false_positives=5,
-    auto_resolved=10,
-)
-
-print(f"Alert precision: {metrics.precision}%")
-print(f"Toil ratio: {metrics.toil_ratio}%")
-```
-
-## On-Call Alert Guidelines
-
-```yaml
-# on_call_alert_standards.yaml
-alert_standards:
-  page_worthy:
-    - "Immediate user impact (>5% of users affected)"
-    - "SLO violation in progress"
-    - "Error budget burn rate critical (>10x)"
-    - "Security incident"
-    - "Data loss risk"
-
-  not_page_worthy:
-    - "Predictive alerts without current impact"
-    - "Informational metrics"
-    - "Non-user-facing issues"
-    - "Slow trends (address during business hours)"
-
-  alert_routing:
-    critical:
-      - page: on-call engineer
-      - slack: "#incidents"
-      - create: incident doc
-
-    warning:
-      - slack: "#alerts"
-      - ticket: auto-create if persists >1h
-
-    info:
-      - dashboard: only
-```

package/skills/sre-engineer/references/slo-sli-management.md

@@ -1,238 +0,0 @@
-# SLO/SLI Management
-
-## SLI Definition Patterns
-
-Service Level Indicators are quantitative measurements of service behavior.
-
-### Request-Based SLIs
-
-```python
-# Availability SLI: Proportion of successful requests
-# Good events: HTTP 200-299, 4XX (client errors don't count against SLI)
-# Total events: All requests
-
-def calculate_availability_sli(metrics):
-    """Calculate availability SLI from request metrics."""
-    successful_requests = metrics['http_2xx'] + metrics['http_4xx']
-    total_requests = metrics['total_requests']
-
-    if total_requests == 0:
-        return 1.0  # No traffic = 100% available
-
-    return successful_requests / total_requests
-
-# Example: 99.9% of requests return successfully
-# SLI = successful_requests / total_requests
-```
-
-### Latency-Based SLIs
-
-```python
-def calculate_latency_sli(latency_histogram, threshold_ms=500):
-    """Calculate latency SLI from histogram.
-
-    Args:
-        latency_histogram: dict mapping latency buckets to request counts
-        threshold_ms: latency threshold in milliseconds
-
-    Returns:
-        float: Proportion of requests faster than threshold
-    """
-    fast_requests = sum(
-        count for bucket, count in latency_histogram.items()
-        if bucket <= threshold_ms
-    )
-    total_requests = sum(latency_histogram.values())
-
-    return fast_requests / total_requests if total_requests > 0 else 1.0
-
-# Example: 99% of requests complete in <500ms
-# SLI = requests_under_500ms / total_requests
-```
-
-## SLO Configuration
-
-```yaml
-# slo_config.yaml - Production API SLO definitions
-apiVersion: sre/v1
-kind: ServiceLevelObjective
-metadata:
-  service: payment-api
-  environment: production
-spec:
-  slos:
-    - name: availability
-      description: "Users can successfully complete payment requests"
-      sli:
-        metric: http_requests_total
-        query: |
-          sum(rate(http_requests_total{status=~"2..|4..", service="payment-api"}[30d]))
-          /
-          sum(rate(http_requests_total{service="payment-api"}[30d]))
-      target: 0.999  # 99.9%
-      window: 30d
-
-    - name: latency
-      description: "Payment requests complete quickly"
-      sli:
-        metric: http_request_duration_seconds
-        query: |
-          histogram_quantile(0.95,
-            sum(rate(http_request_duration_seconds_bucket{service="payment-api"}[30d]))
-            by (le)
-          ) < 0.5
-      target: 0.99  # 99% of requests under 500ms
-      window: 30d
-```
-
-## Golden Signals
-
-The four golden signals every service should measure:
-
-```python
-from dataclasses import dataclass
-from typing import Dict
-
-@dataclass
-class GoldenSignals:
-    """Four golden signals of monitoring."""
-
-    # Latency: Time to service requests (success vs failure)
-    latency_p50_ms: float
-    latency_p95_ms: float
-    latency_p99_ms: float
-
-    # Traffic: Demand on your system (requests/sec)
-    requests_per_second: float
-
-    # Errors: Rate of failed requests
-    error_rate: float  # 0.0 to 1.0
-
-    # Saturation: How "full" is your service (CPU, memory, disk)
-    cpu_utilization: float  # 0.0 to 1.0
-    memory_utilization: float  # 0.0 to 1.0
-
-    def is_healthy(self, slo_targets: Dict[str, float]) -> bool:
-        """Check if all signals are within SLO targets."""
-        return (
-            self.latency_p99_ms <= slo_targets['latency_p99_ms'] and
-            self.error_rate <= (1 - slo_targets['availability']) and
-            self.cpu_utilization <= slo_targets['max_cpu'] and
-            self.memory_utilization <= slo_targets['max_memory']
-        )
-```
-
-## SLO Calculation Examples
-
-```python
-from datetime import timedelta
-from typing import NamedTuple
-
-class SLOTarget(NamedTuple):
-    """SLO target configuration."""
-    target: float  # 0.999 for 99.9%
-    window: timedelta  # 30 days
-
-    @property
-    def error_budget(self) -> float:
-        """Calculate error budget (1 - target)."""
-        return 1 - self.target
-
-    @property
-    def allowed_downtime(self) -> timedelta:
-        """Calculate allowed downtime in window."""
-        total_seconds = self.window.total_seconds()
-        allowed_seconds = total_seconds * self.error_budget
-        return timedelta(seconds=allowed_seconds)
-
-# Example SLOs
-availability_slo = SLOTarget(target=0.999, window=timedelta(days=30))
-print(f"Error budget: {availability_slo.error_budget * 100}%")
-print(f"Allowed downtime: {availability_slo.allowed_downtime}")
-# Output:
-# Error budget: 0.1%
-# Allowed downtime: 43.2 minutes per 30 days
-
-latency_slo = SLOTarget(target=0.99, window=timedelta(days=30))
-print(f"99% of requests must be fast")
-print(f"1% can be slow: {latency_slo.error_budget * 100}%")
-```
-
-## Multi-Window SLO Tracking
-
-```python
-class MultiWindowSLO:
-    """Track SLO compliance across multiple time windows."""
-
-    def __init__(self, target: float):
-        self.target = target
-        self.windows = {
-            '1h': timedelta(hours=1),
-            '24h': timedelta(hours=24),
-            '7d': timedelta(days=7),
-            '30d': timedelta(days=30),
-        }
-
-    def check_compliance(self, sli_values: Dict[str, float]) -> Dict[str, bool]:
-        """Check if SLI meets target in each window.
-
-        Args:
-            sli_values: Dict mapping window name to measured SLI
-
-        Returns:
-            Dict mapping window name to compliance boolean
-        """
-        return {
-            window: sli >= self.target
-            for window, sli in sli_values.items()
-        }
-
-    def get_burn_rate(self, current_sli: float) -> float:
-        """Calculate error budget burn rate.
-
-        Burn rate > 1.0 means burning budget faster than sustainable.
-        """
-        error_budget = 1 - self.target
-        current_error_rate = 1 - current_sli
-
-        if error_budget == 0:
-            return float('inf')
-
-        return current_error_rate / error_budget
-
-# Usage
-slo = MultiWindowSLO(target=0.999)
-current_sli = 0.997  # 99.7% availability
-
-burn_rate = slo.get_burn_rate(current_sli)
-print(f"Burn rate: {burn_rate}x")
-# If burn_rate = 3.0, burning budget 3x faster than sustainable
-```
-
-## SLO Review Checklist
-
-Before finalizing SLOs:
-
-1. **User-centric**: Does it measure user-facing impact?
-2. **Achievable**: Can you meet it with current architecture?
-3. **Measurable**: Can you accurately track the SLI?
-4. **Meaningful**: Does violating it require action?
-5. **Documented**: Is the calculation clear and agreed upon?
-6. **Budgeted**: Is there an error budget policy?
-
-## Common SLO Targets
-
-```yaml
-# Typical SLO targets by service tier
-tier_1_critical:
-  availability: 99.99%  # 4m 23s downtime/month
-  latency_p99: 100ms
-
-tier_2_important:
-  availability: 99.9%   # 43m 28s downtime/month
-  latency_p99: 500ms
-
-tier_3_standard:
-  availability: 99.5%   # 3h 37m downtime/month
-  latency_p99: 1000ms
-```