specweave 0.3.13 → 0.4.0

package/src/skills/prometheus-configuration/SKILL.md

@@ -0,0 +1,392 @@
+---
+name: prometheus-configuration
+description: Set up Prometheus for comprehensive metric collection, storage, and monitoring of infrastructure and applications. Use when implementing metrics collection, setting up monitoring infrastructure, or configuring alerting systems.
+---
+
+# Prometheus Configuration
+
+Complete guide to Prometheus setup, metric collection, scrape configuration, and recording rules.
+
+## Purpose
+
+Configure Prometheus for comprehensive metric collection, alerting, and monitoring of infrastructure and applications.
+
+## When to Use
+
+- Set up Prometheus monitoring
+- Configure metric scraping
+- Create recording rules
+- Design alert rules
+- Implement service discovery
+
+## Prometheus Architecture
+
+```
+┌──────────────┐
+│ Applications │ ← Instrumented with client libraries
+└──────┬───────┘
+       │ /metrics endpoint
+       ↓
+┌──────────────┐
+│  Prometheus  │ ← Scrapes metrics periodically
+│    Server    │
+└──────┬───────┘
+       │
+       ├─→ AlertManager (alerts)
+       ├─→ Grafana (visualization)
+       └─→ Long-term storage (Thanos/Cortex)
+```
+
+## Installation
+
+### Kubernetes with Helm
+
+```bash
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+
+helm install prometheus prometheus-community/kube-prometheus-stack \
+  --namespace monitoring \
+  --create-namespace \
+  --set prometheus.prometheusSpec.retention=30d \
+  --set prometheus.prometheusSpec.storageVolumeSize=50Gi
+```
+
+### Docker Compose
+
+```yaml
+version: '3.8'
+services:
+  prometheus:
+    image: prom/prometheus:latest
+    ports:
+      - "9090:9090"
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml
+      - prometheus-data:/prometheus
+    command:
+      - '--config.file=/etc/prometheus/prometheus.yml'
+      - '--storage.tsdb.path=/prometheus'
+      - '--storage.tsdb.retention.time=30d'
+
+volumes:
+  prometheus-data:
+```
+
+## Configuration File
+
+**prometheus.yml:**
+```yaml
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+  external_labels:
+    cluster: 'production'
+    region: 'us-west-2'
+
+# Alertmanager configuration
+alerting:
+  alertmanagers:
+    - static_configs:
+        - targets:
+          - alertmanager:9093
+
+# Load rules files
+rule_files:
+  - /etc/prometheus/rules/*.yml
+
+# Scrape configurations
+scrape_configs:
+  # Prometheus itself
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: ['localhost:9090']
+
+  # Node exporters
+  - job_name: 'node-exporter'
+    static_configs:
+      - targets:
+        - 'node1:9100'
+        - 'node2:9100'
+        - 'node3:9100'
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: instance
+        regex: '([^:]+)(:[0-9]+)?'
+        replacement: '${1}'
+
+  # Kubernetes pods with annotations
+  - job_name: 'kubernetes-pods'
+    kubernetes_sd_configs:
+      - role: pod
+    relabel_configs:
+      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
+        action: keep
+        regex: true
+      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
+        action: replace
+        target_label: __metrics_path__
+        regex: (.+)
+      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
+        action: replace
+        regex: ([^:]+)(?::\d+)?;(\d+)
+        replacement: $1:$2
+        target_label: __address__
+      - source_labels: [__meta_kubernetes_namespace]
+        action: replace
+        target_label: namespace
+      - source_labels: [__meta_kubernetes_pod_name]
+        action: replace
+        target_label: pod
+
+  # Application metrics
+  - job_name: 'my-app'
+    static_configs:
+      - targets:
+        - 'app1.example.com:9090'
+        - 'app2.example.com:9090'
+    metrics_path: '/metrics'
+    scheme: 'https'
+    tls_config:
+      ca_file: /etc/prometheus/ca.crt
+      cert_file: /etc/prometheus/client.crt
+      key_file: /etc/prometheus/client.key
+```
+
+**Reference:** See `assets/prometheus.yml.template`
+
+## Scrape Configurations
+
+### Static Targets
+
+```yaml
+scrape_configs:
+  - job_name: 'static-targets'
+    static_configs:
+      - targets: ['host1:9100', 'host2:9100']
+        labels:
+          env: 'production'
+          region: 'us-west-2'
+```
+
+### File-based Service Discovery
+
+```yaml
+scrape_configs:
+  - job_name: 'file-sd'
+    file_sd_configs:
+      - files:
+        - /etc/prometheus/targets/*.json
+        - /etc/prometheus/targets/*.yml
+        refresh_interval: 5m
+```
+
+**targets/production.json:**
+```json
+[
+  {
+    "targets": ["app1:9090", "app2:9090"],
+    "labels": {
+      "env": "production",
+      "service": "api"
+    }
+  }
+]
+```
+
+### Kubernetes Service Discovery
+
+```yaml
+scrape_configs:
+  - job_name: 'kubernetes-services'
+    kubernetes_sd_configs:
+      - role: service
+    relabel_configs:
+      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
+        action: keep
+        regex: true
+      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
+        action: replace
+        target_label: __scheme__
+        regex: (https?)
+      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
+        action: replace
+        target_label: __metrics_path__
+        regex: (.+)
+```
+
+**Reference:** See `references/scrape-configs.md`
+
+## Recording Rules
+
+Create pre-computed metrics for frequently queried expressions:
+
+```yaml
+# /etc/prometheus/rules/recording_rules.yml
+groups:
+  - name: api_metrics
+    interval: 15s
+    rules:
+      # HTTP request rate per service
+      - record: job:http_requests:rate5m
+        expr: sum by (job) (rate(http_requests_total[5m]))
+
+      # Error rate percentage
+      - record: job:http_requests_errors:rate5m
+        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
+
+      - record: job:http_requests_error_rate:percentage
+        expr: |
+          (job:http_requests_errors:rate5m / job:http_requests:rate5m) * 100
+
+      # P95 latency
+      - record: job:http_request_duration:p95
+        expr: |
+          histogram_quantile(0.95,
+            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
+          )
+
+  - name: resource_metrics
+    interval: 30s
+    rules:
+      # CPU utilization percentage
+      - record: instance:node_cpu:utilization
+        expr: |
+          100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+
+      # Memory utilization percentage
+      - record: instance:node_memory:utilization
+        expr: |
+          100 - ((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100)
+
+      # Disk usage percentage
+      - record: instance:node_disk:utilization
+        expr: |
+          100 - ((node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100)
+```
+
+**Reference:** See `references/recording-rules.md`
+
+## Alert Rules
+
+```yaml
+# /etc/prometheus/rules/alert_rules.yml
+groups:
+  - name: availability
+    interval: 30s
+    rules:
+      - alert: ServiceDown
+        expr: up{job="my-app"} == 0
+        for: 1m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Service {{ $labels.instance }} is down"
+          description: "{{ $labels.job }} has been down for more than 1 minute"
+
+      - alert: HighErrorRate
+        expr: job:http_requests_error_rate:percentage > 5
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High error rate for {{ $labels.job }}"
+          description: "Error rate is {{ $value }}% (threshold: 5%)"
+
+      - alert: HighLatency
+        expr: job:http_request_duration:p95 > 1
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High latency for {{ $labels.job }}"
+          description: "P95 latency is {{ $value }}s (threshold: 1s)"
+
+  - name: resources
+    interval: 1m
+    rules:
+      - alert: HighCPUUsage
+        expr: instance:node_cpu:utilization > 80
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High CPU usage on {{ $labels.instance }}"
+          description: "CPU usage is {{ $value }}%"
+
+      - alert: HighMemoryUsage
+        expr: instance:node_memory:utilization > 85
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High memory usage on {{ $labels.instance }}"
+          description: "Memory usage is {{ $value }}%"
+
+      - alert: DiskSpaceLow
+        expr: instance:node_disk:utilization > 90
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Low disk space on {{ $labels.instance }}"
+          description: "Disk usage is {{ $value }}%"
+```
+
+## Validation
+
+```bash
+# Validate configuration
+promtool check config prometheus.yml
+
+# Validate rules
+promtool check rules /etc/prometheus/rules/*.yml
+
+# Test query
+promtool query instant http://localhost:9090 'up'
+```
+
+**Reference:** See `scripts/validate-prometheus.sh`
+
+## Best Practices
+
+1. **Use consistent naming** for metrics (prefix_name_unit)
+2. **Set appropriate scrape intervals** (15-60s typical)
+3. **Use recording rules** for expensive queries
+4. **Implement high availability** (multiple Prometheus instances)
+5. **Configure retention** based on storage capacity
+6. **Use relabeling** for metric cleanup
+7. **Monitor Prometheus itself**
+8. **Implement federation** for large deployments
+9. **Use Thanos/Cortex** for long-term storage
+10. **Document custom metrics**
+
+## Troubleshooting
+
+**Check scrape targets:**
+```bash
+curl http://localhost:9090/api/v1/targets
+```
+
+**Check configuration:**
+```bash
+curl http://localhost:9090/api/v1/status/config
+```
+
+**Test query:**
+```bash
+curl 'http://localhost:9090/api/v1/query?query=up'
+```
+
+## Reference Files
+
+- `assets/prometheus.yml.template` - Complete configuration template
+- `references/scrape-configs.md` - Scrape configuration patterns
+- `references/recording-rules.md` - Recording rule examples
+- `scripts/validate-prometheus.sh` - Validation script
+
+## Related Skills
+
+- `grafana-dashboards` - For visualization
+- `slo-implementation` - For SLO monitoring
+- `distributed-tracing` - For request tracing

package/src/skills/slo-implementation/SKILL.md

@@ -0,0 +1,329 @@
+---
+name: slo-implementation
+description: Define and implement Service Level Indicators (SLIs) and Service Level Objectives (SLOs) with error budgets and alerting. Use when establishing reliability targets, implementing SRE practices, or measuring service performance.
+---
+
+# SLO Implementation
+
+Framework for defining and implementing Service Level Indicators (SLIs), Service Level Objectives (SLOs), and error budgets.
+
+## Purpose
+
+Implement measurable reliability targets using SLIs, SLOs, and error budgets to balance reliability with innovation velocity.
+
+## When to Use
+
+- Define service reliability targets
+- Measure user-perceived reliability
+- Implement error budgets
+- Create SLO-based alerts
+- Track reliability goals
+
+## SLI/SLO/SLA Hierarchy
+
+```
+SLA (Service Level Agreement)
+  ↓ Contract with customers
+SLO (Service Level Objective)
+  ↓ Internal reliability target
+SLI (Service Level Indicator)
+  ↓ Actual measurement
+```
+
+## Defining SLIs
+
+### Common SLI Types
+
+#### 1. Availability SLI
+```promql
+# Successful requests / Total requests
+sum(rate(http_requests_total{status!~"5.."}[28d]))
+/
+sum(rate(http_requests_total[28d]))
+```
+
+#### 2. Latency SLI
+```promql
+# Requests below latency threshold / Total requests
+sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
+/
+sum(rate(http_request_duration_seconds_count[28d]))
+```
+
+#### 3. Durability SLI
+```
+# Successful writes / Total writes
+sum(storage_writes_successful_total)
+/
+sum(storage_writes_total)
+```
+
+**Reference:** See `references/slo-definitions.md`
+
+## Setting SLO Targets
+
+### Availability SLO Examples
+
+| SLO % | Downtime/Month | Downtime/Year |
+|-------|----------------|---------------|
+| 99%   | 7.2 hours      | 3.65 days     |
+| 99.9% | 43.2 minutes   | 8.76 hours    |
+| 99.95%| 21.6 minutes   | 4.38 hours    |
+| 99.99%| 4.32 minutes   | 52.56 minutes |
+
+### Choose Appropriate SLOs
+
+**Consider:**
+- User expectations
+- Business requirements
+- Current performance
+- Cost of reliability
+- Competitor benchmarks
+
+**Example SLOs:**
+```yaml
+slos:
+  - name: api_availability
+    target: 99.9
+    window: 28d
+    sli: |
+      sum(rate(http_requests_total{status!~"5.."}[28d]))
+      /
+      sum(rate(http_requests_total[28d]))
+
+  - name: api_latency_p95
+    target: 99
+    window: 28d
+    sli: |
+      sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
+      /
+      sum(rate(http_request_duration_seconds_count[28d]))
+```
+
+## Error Budget Calculation
+
+### Error Budget Formula
+
+```
+Error Budget = 1 - SLO Target
+```
+
+**Example:**
+- SLO: 99.9% availability
+- Error Budget: 0.1% = 43.2 minutes/month
+- Current Error: 0.05% = 21.6 minutes/month
+- Remaining Budget: 50%
+
+### Error Budget Policy
+
+```yaml
+error_budget_policy:
+  - remaining_budget: 100%
+    action: Normal development velocity
+  - remaining_budget: 50%
+    action: Consider postponing risky changes
+  - remaining_budget: 10%
+    action: Freeze non-critical changes
+  - remaining_budget: 0%
+    action: Feature freeze, focus on reliability
+```
+
+**Reference:** See `references/error-budget.md`
+
+## SLO Implementation
+
+### Prometheus Recording Rules
+
+```yaml
+# SLI Recording Rules
+groups:
+  - name: sli_rules
+    interval: 30s
+    rules:
+      # Availability SLI
+      - record: sli:http_availability:ratio
+        expr: |
+          sum(rate(http_requests_total{status!~"5.."}[28d]))
+          /
+          sum(rate(http_requests_total[28d]))
+
+      # Latency SLI (requests < 500ms)
+      - record: sli:http_latency:ratio
+        expr: |
+          sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
+          /
+          sum(rate(http_request_duration_seconds_count[28d]))
+
+  - name: slo_rules
+    interval: 5m
+    rules:
+      # SLO compliance (1 = meeting SLO, 0 = violating)
+      - record: slo:http_availability:compliance
+        expr: sli:http_availability:ratio >= bool 0.999
+
+      - record: slo:http_latency:compliance
+        expr: sli:http_latency:ratio >= bool 0.99
+
+      # Error budget remaining (percentage)
+      - record: slo:http_availability:error_budget_remaining
+        expr: |
+          (sli:http_availability:ratio - 0.999) / (1 - 0.999) * 100
+
+      # Error budget burn rate
+      - record: slo:http_availability:burn_rate_5m
+        expr: |
+          (1 - (
+            sum(rate(http_requests_total{status!~"5.."}[5m]))
+            /
+            sum(rate(http_requests_total[5m]))
+          )) / (1 - 0.999)
+```
+
+### SLO Alerting Rules
+
+```yaml
+groups:
+  - name: slo_alerts
+    interval: 1m
+    rules:
+      # Fast burn: 14.4x rate, 1 hour window
+      # Consumes 2% error budget in 1 hour
+      - alert: SLOErrorBudgetBurnFast
+        expr: |
+          slo:http_availability:burn_rate_1h > 14.4
+          and
+          slo:http_availability:burn_rate_5m > 14.4
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Fast error budget burn detected"
+          description: "Error budget burning at {{ $value }}x rate"
+
+      # Slow burn: 6x rate, 6 hour window
+      # Consumes 5% error budget in 6 hours
+      - alert: SLOErrorBudgetBurnSlow
+        expr: |
+          slo:http_availability:burn_rate_6h > 6
+          and
+          slo:http_availability:burn_rate_30m > 6
+        for: 15m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Slow error budget burn detected"
+          description: "Error budget burning at {{ $value }}x rate"
+
+      # Error budget exhausted
+      - alert: SLOErrorBudgetExhausted
+        expr: slo:http_availability:error_budget_remaining < 0
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "SLO error budget exhausted"
+          description: "Error budget remaining: {{ $value }}%"
+```
+
+## SLO Dashboard
+
+**Grafana Dashboard Structure:**
+
+```
+┌────────────────────────────────────┐
+│ SLO Compliance (Current)           │
+│ ✓ 99.95% (Target: 99.9%)          │
+├────────────────────────────────────┤
+│ Error Budget Remaining: 65%        │
+│ ████████░░ 65%                     │
+├────────────────────────────────────┤
+│ SLI Trend (28 days)                │
+│ [Time series graph]                │
+├────────────────────────────────────┤
+│ Burn Rate Analysis                 │
+│ [Burn rate by time window]         │
+└────────────────────────────────────┘
+```
+
+**Example Queries:**
+
+```promql
+# Current SLO compliance
+sli:http_availability:ratio * 100
+
+# Error budget remaining
+slo:http_availability:error_budget_remaining
+
+# Days until error budget exhausted (at current burn rate)
+(slo:http_availability:error_budget_remaining / 100)
+*
+28
+/
+(1 - sli:http_availability:ratio) * (1 - 0.999)
+```
+
+## Multi-Window Burn Rate Alerts
+
+```yaml
+# Combination of short and long windows reduces false positives
+rules:
+  - alert: SLOBurnRateHigh
+    expr: |
+      (
+        slo:http_availability:burn_rate_1h > 14.4
+        and
+        slo:http_availability:burn_rate_5m > 14.4
+      )
+      or
+      (
+        slo:http_availability:burn_rate_6h > 6
+        and
+        slo:http_availability:burn_rate_30m > 6
+      )
+    labels:
+      severity: critical
+```
+
+## SLO Review Process
+
+### Weekly Review
+- Current SLO compliance
+- Error budget status
+- Trend analysis
+- Incident impact
+
+### Monthly Review
+- SLO achievement
+- Error budget usage
+- Incident postmortems
+- SLO adjustments
+
+### Quarterly Review
+- SLO relevance
+- Target adjustments
+- Process improvements
+- Tooling enhancements
+
+## Best Practices
+
+1. **Start with user-facing services**
+2. **Use multiple SLIs** (availability, latency, etc.)
+3. **Set achievable SLOs** (don't aim for 100%)
+4. **Implement multi-window alerts** to reduce noise
+5. **Track error budget** consistently
+6. **Review SLOs regularly**
+7. **Document SLO decisions**
+8. **Align with business goals**
+9. **Automate SLO reporting**
+10. **Use SLOs for prioritization**
+
+## Reference Files
+
+- `assets/slo-template.md` - SLO definition template
+- `references/slo-definitions.md` - SLO definition patterns
+- `references/error-budget.md` - Error budget calculations
+
+## Related Skills
+
+- `prometheus-configuration` - For metric collection
+- `grafana-dashboards` - For SLO visualization