@sylix/coworker 2.0.10 → 2.0.12

package/dist/skills/defaults/devops/postmortem-writing.md

@@ -0,0 +1,396 @@
+---
+name: postmortem-writing
+description: Write effective blameless postmortems with root cause analysis, timelines, and action items. Use when conducting incident reviews, writing postmortem documents, or improving incident response processes.
+---
+
+# Postmortem Writing
+
+Comprehensive guide to writing effective, blameless postmortems that drive organizational learning and prevent incident recurrence.
+
+## When to Use This Skill
+
+- Conducting post-incident reviews
+- Writing postmortem documents
+- Facilitating blameless postmortem meetings
+- Identifying root causes and contributing factors
+- Creating actionable follow-up items
+- Building organizational learning culture
+
+## Core Concepts
+
+### 1. Blameless Culture
+
+| Blame-Focused            | Blameless                         |
+| ------------------------ | --------------------------------- |
+| "Who caused this?"       | "What conditions allowed this?"   |
+| "Someone made a mistake" | "The system allowed this mistake" |
+| Punish individuals       | Improve systems                   |
+| Hide information         | Share learnings                   |
+| Fear of speaking up      | Psychological safety              |
+
+### 2. Postmortem Triggers
+
+- SEV1 or SEV2 incidents
+- Customer-facing outages > 15 minutes
+- Data loss or security incidents
+- Near-misses that could have been severe
+- Novel failure modes
+- Incidents requiring unusual intervention
+
+## Quick Start
+
+### Postmortem Timeline
+
+```
+Day 0: Incident occurs
+Day 1-2: Draft postmortem document
+Day 3-5: Postmortem meeting
+Day 5-7: Finalize document, create tickets
+Week 2+: Action item completion
+Quarterly: Review patterns across incidents
+```
+
+## Templates
+
+### Template 1: Standard Postmortem
+
+```markdown
+# Postmortem: [Incident Title]
+
+**Date**: 2024-01-15
+**Authors**: @alice, @bob
+**Status**: Draft | In Review | Final
+**Incident Severity**: SEV2
+**Incident Duration**: 47 minutes
+
+## Executive Summary
+
+On January 15, 2024, the payment processing service experienced a 47-minute outage affecting approximately 12,000 customers. The root cause was a database connection pool exhaustion triggered by a configuration change in deployment v2.3.4. The incident was resolved by rolling back to v2.3.3 and increasing connection pool limits.
+
+**Impact**:
+
+- 12,000 customers unable to complete purchases
+- Estimated revenue loss: $45,000
+- 847 support tickets created
+- No data loss or security implications
+
+## Timeline (All times UTC)
+
+| Time  | Event                                           |
+| ----- | ----------------------------------------------- |
+| 14:23 | Deployment v2.3.4 completed to production       |
+| 14:31 | First alert: `payment_error_rate > 5%`          |
+| 14:33 | On-call engineer @alice acknowledges alert      |
+| 14:35 | Initial investigation begins, error rate at 23% |
+| 14:41 | Incident declared SEV2, @bob joins              |
+| 14:45 | Database connection exhaustion identified       |
+| 14:52 | Decision to rollback deployment                 |
+| 14:58 | Rollback to v2.3.3 initiated                    |
+| 15:10 | Rollback complete, error rate dropping          |
+| 15:18 | Service fully recovered, incident resolved      |
+
+## Root Cause Analysis
+
+### What Happened
+
+The v2.3.4 deployment included a change to the database query pattern that inadvertently removed connection pooling for a frequently-called endpoint. Each request opened a new database connection instead of reusing pooled connections.
+
+### Why It Happened
+
+1. **Proximate Cause**: Code change in `PaymentRepository.java` replaced pooled `DataSource` with direct `DriverManager.getConnection()` calls.
+
+2. **Contributing Factors**:
+   - Code review did not catch the connection handling change
+   - No integration tests specifically for connection pool behavior
+   - Staging environment has lower traffic, masking the issue
+   - Database connection metrics alert threshold was too high (90%)
+
+3. **5 Whys Analysis**:
+   - Why did the service fail? → Database connections exhausted
+   - Why were connections exhausted? → Each request opened new connection
+   - Why did each request open new connection? → Code bypassed connection pool
+   - Why did code bypass connection pool? → Developer unfamiliar with codebase patterns
+   - Why was developer unfamiliar? → No documentation on connection management patterns
+
+### System Diagram
+```
+
+[Client] → [Load Balancer] → [Payment Service] → [Database]
+↓
+Connection Pool (broken)
+↓
+Direct connections (cause)
+
+```
+
+## Detection
+
+### What Worked
+- Error rate alert fired within 8 minutes of deployment
+- Grafana dashboard clearly showed connection spike
+- On-call response was swift (2 minute acknowledgment)
+
+### What Didn't Work
+- Database connection metric alert threshold too high
+- No deployment-correlated alerting
+- Canary deployment would have caught this earlier
+
+### Detection Gap
+The deployment completed at 14:23, but the first alert didn't fire until 14:31 (8 minutes). A deployment-aware alert could have detected the issue faster.
+
+## Response
+
+### What Worked
+- On-call engineer quickly identified database as the issue
+- Rollback decision was made decisively
+- Clear communication in incident channel
+
+### What Could Be Improved
+- Took 10 minutes to correlate issue with recent deployment
+- Had to manually check deployment history
+- Rollback took 12 minutes (could be faster)
+
+## Impact
+
+### Customer Impact
+- 12,000 unique customers affected
+- Average impact duration: 35 minutes
+- 847 support tickets (23% of affected users)
+- Customer satisfaction score dropped 12 points
+
+### Business Impact
+- Estimated revenue loss: $45,000
+- Support cost: ~$2,500 (agent time)
+- Engineering time: ~8 person-hours
+
+### Technical Impact
+- Database primary experienced elevated load
+- Some replica lag during incident
+- No permanent damage to systems
+
+## Lessons Learned
+
+### What Went Well
+1. Alerting detected the issue before customer reports
+2. Team collaborated effectively under pressure
+3. Rollback procedure worked smoothly
+4. Communication was clear and timely
+
+### What Went Wrong
+1. Code review missed critical change
+2. Test coverage gap for connection pooling
+3. Staging environment doesn't reflect production traffic
+4. Alert thresholds were not tuned properly
+
+### Where We Got Lucky
+1. Incident occurred during business hours with full team available
+2. Database handled the load without failing completely
+3. No other incidents occurred simultaneously
+
+## Action Items
+
+| Priority | Action | Owner | Due Date | Ticket |
+|----------|--------|-------|----------|--------|
+| P0 | Add integration test for connection pool behavior | @alice | 2024-01-22 | ENG-1234 |
+| P0 | Lower database connection alert threshold to 70% | @bob | 2024-01-17 | OPS-567 |
+| P1 | Document connection management patterns | @alice | 2024-01-29 | DOC-89 |
+| P1 | Implement deployment-correlated alerting | @bob | 2024-02-05 | OPS-568 |
+| P2 | Evaluate canary deployment strategy | @charlie | 2024-02-15 | ENG-1235 |
+| P2 | Load test staging with production-like traffic | @dave | 2024-02-28 | QA-123 |
+
+## Appendix
+
+### Supporting Data
+
+#### Error Rate Graph
+[Link to Grafana dashboard snapshot]
+
+#### Database Connection Graph
+[Link to metrics]
+
+### Related Incidents
+- 2023-11-02: Similar connection issue in User Service (POSTMORTEM-42)
+
+### References
+- [Connection Pool Best Practices](internal-wiki/connection-pools)
+- [Deployment Runbook](internal-wiki/deployment-runbook)
+```
+
+### Template 2: 5 Whys Analysis
+
+```markdown
+# 5 Whys Analysis: [Incident]
+
+## Problem Statement
+
+Payment service experienced 47-minute outage due to database connection exhaustion.
+
+## Analysis
+
+### Why #1: Why did the service fail?
+
+**Answer**: Database connections were exhausted, causing all new requests to fail.
+
+**Evidence**: Metrics showed connection count at 100/100 (max), with 500+ pending requests.
+
+---
+
+### Why #2: Why were database connections exhausted?
+
+**Answer**: Each incoming request opened a new database connection instead of using the connection pool.
+
+**Evidence**: Code diff shows direct `DriverManager.getConnection()` instead of pooled `DataSource`.
+
+---
+
+### Why #3: Why did the code bypass the connection pool?
+
+**Answer**: A developer refactored the repository class and inadvertently changed the connection acquisition method.
+
+**Evidence**: PR #1234 shows the change, made while fixing a different bug.
+
+---
+
+### Why #4: Why wasn't this caught in code review?
+
+**Answer**: The reviewer focused on the functional change (the bug fix) and didn't notice the infrastructure change.
+
+**Evidence**: Review comments only discuss business logic.
+
+---
+
+### Why #5: Why isn't there a safety net for this type of change?
+
+**Answer**: We lack automated tests that verify connection pool behavior and lack documentation about our connection patterns.
+
+**Evidence**: Test suite has no tests for connection handling; wiki has no article on database connections.
+
+## Root Causes Identified
+
+1. **Primary**: Missing automated tests for infrastructure behavior
+2. **Secondary**: Insufficient documentation of architectural patterns
+3. **Tertiary**: Code review checklist doesn't include infrastructure considerations
+
+## Systemic Improvements
+
+| Root Cause    | Improvement                       | Type       |
+| ------------- | --------------------------------- | ---------- |
+| Missing tests | Add infrastructure behavior tests | Prevention |
+| Missing docs  | Document connection patterns      | Prevention |
+| Review gaps   | Update review checklist           | Detection  |
+| No canary     | Implement canary deployments      | Mitigation |
+```
+
+### Template 3: Quick Postmortem (Minor Incidents)
+
+```markdown
+# Quick Postmortem: [Brief Title]
+
+**Date**: 2024-01-15 | **Duration**: 12 min | **Severity**: SEV3
+
+## What Happened
+
+API latency spiked to 5s due to cache miss storm after cache flush.
+
+## Timeline
+
+- 10:00 - Cache flush initiated for config update
+- 10:02 - Latency alerts fire
+- 10:05 - Identified as cache miss storm
+- 10:08 - Enabled cache warming
+- 10:12 - Latency normalized
+
+## Root Cause
+
+Full cache flush for minor config update caused thundering herd.
+
+## Fix
+
+- Immediate: Enabled cache warming
+- Long-term: Implement partial cache invalidation (ENG-999)
+
+## Lessons
+
+Don't full-flush cache in production; use targeted invalidation.
+```
+
+## Facilitation Guide
+
+### Running a Postmortem Meeting
+
+```markdown
+## Meeting Structure (60 minutes)
+
+### 1. Opening (5 min)
+
+- Remind everyone of blameless culture
+- "We're here to learn, not to blame"
+- Review meeting norms
+
+### 2. Timeline Review (15 min)
+
+- Walk through events chronologically
+- Ask clarifying questions
+- Identify gaps in timeline
+
+### 3. Analysis Discussion (20 min)
+
+- What failed?
+- Why did it fail?
+- What conditions allowed this?
+- What would have prevented it?
+
+### 4. Action Items (15 min)
+
+- Brainstorm improvements
+- Prioritize by impact and effort
+- Assign owners and due dates
+
+### 5. Closing (5 min)
+
+- Summarize key learnings
+- Confirm action item owners
+- Schedule follow-up if needed
+
+## Facilitation Tips
+
+- Keep discussion on track
+- Redirect blame to systems
+- Encourage quiet participants
+- Document dissenting views
+- Time-box tangents
+```
+
+## Anti-Patterns to Avoid
+
+| Anti-Pattern            | Problem                    | Better Approach                 |
+| ----------------------- | -------------------------- | ------------------------------- |
+| **Blame game**          | Shuts down learning        | Focus on systems                |
+| **Shallow analysis**    | Doesn't prevent recurrence | Ask "why" 5 times              |
+| **No action items**    | Waste of time             | Always have concrete next steps |
+| **Unrealistic actions** | Never completed           | Scope to achievable tasks       |
+| **No follow-up**       | Actions forgotten         | Track in ticketing system       |
+
+## Best Practices
+
+### Do's
+
+- **Start immediately** - Memory fades fast
+- **Be specific** - Exact times, exact errors
+- **Include graphs** - Visual evidence
+- **Assign owners** - No orphan action items
+- **Share widely** - Organizational learning
+
+### Don'ts
+
+- **Don't name and shame** - Ever
+- **Don't skip small incidents** - They reveal patterns
+- **Don't make it a blame doc** - That kills learning
+- **Don't create busywork** - Actions should be meaningful
+- **Don't skip follow-up** - Verify actions completed
+
+## Resources
+
+- [Google SRE - Postmortem Culture](https://sre.google/sre-book/postmortem-culture/)
+- [Etsy's Blameless Postmortems](https://codeascraft.com/2012/05/22/blameless-postmortems/)
+- [PagerDuty Postmortem Guide](https://postmortems.pagerduty.com/)

package/dist/skills/defaults/devops/prometheus-configuration.md

@@ -0,0 +1,265 @@
+---
+name: prometheus-configuration
+description: Set up Prometheus for comprehensive metric collection, storage, and monitoring of infrastructure and applications. Use when implementing metrics collection, setting up monitoring infrastructure, or configuring alerting systems.
+---
+
+# Prometheus Configuration
+
+Complete guide to Prometheus setup, metric collection, scrape configuration, and recording rules.
+
+## Purpose
+
+Configure Prometheus for comprehensive metric collection, alerting, and monitoring of infrastructure and applications.
+
+## When to Use
+
+- Set up Prometheus monitoring
+- Configure metric scraping
+- Create recording rules
+- Design alert rules
+- Implement service discovery
+
+## Prometheus Architecture
+
+```
+┌──────────────┐
+│ Applications │ ← Instrumented with client libraries
+└──────┬───────┘
+       │ /metrics endpoint
+       ↓
+┌──────────────┐
+│  Prometheus  │ ← Scrapes metrics periodically
+│    Server    │
+└──────┬───────┘
+       │
+       ├─→ AlertManager (alerts)
+       ├─→ Grafana (visualization)
+       └─→ Long-term storage (Thanos/Cortex)
+```
+
+## Installation
+
+### Kubernetes with Helm
+
+```bash
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm install prometheus prometheus-community/kube-prometheus-stack \
+  --namespace monitoring \
+  --create-namespace \
+  --set prometheus.prometheusSpec.retention=30d
+```
+
+### Docker Compose
+
+```yaml
+version: "3.8"
+services:
+  prometheus:
+    image: prom/prometheus:latest
+    ports:
+      - "9090:9090"
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml
+      - prometheus-data:/prometheus
+    command:
+      - "--config.file=/etc/prometheus/prometheus.yml"
+      - "--storage.tsdb.path=/prometheus"
+```
+
+## Configuration File
+
+```yaml
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+alerting:
+  alertmanagers:
+    - static_configs:
+        - targets:
+            - alertmanager:9093
+
+rule_files:
+  - /etc/prometheus/rules/*.yml
+
+scrape_configs:
+  - job_name: "prometheus"
+    static_configs:
+      - targets: ["localhost:9090"]
+
+  - job_name: "node-exporter"
+    static_configs:
+      - targets:
+          - "node1:9100"
+          - "node2:9100"
+
+  - job_name: "kubernetes-pods"
+    kubernetes_sd_configs:
+      - role: pod
+    relabel_configs:
+      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
+        action: keep
+        regex: true
+      - source_labels: [__meta_kubernetes_namespace]
+        action: replace
+        target_label: namespace
+```
+
+## Scrape Configurations
+
+### Static Targets
+
+```yaml
+scrape_configs:
+  - job_name: "static-targets"
+    static_configs:
+      - targets: ["host1:9100", "host2:9100"]
+        labels:
+          env: "production"
+```
+
+### File-based Service Discovery
+
+```yaml
+scrape_configs:
+  - job_name: "file-sd"
+    file_sd_configs:
+      - files:
+          - /etc/prometheus/targets/*.json
+        refresh_interval: 5m
+```
+
+**targets/production.json:**
+
+```json
+[
+  {
+    "targets": ["app1:9090", "app2:9090"],
+    "labels": {
+      "env": "production"
+    }
+  }
+]
+```
+
+### Kubernetes Service Discovery
+
+```yaml
+scrape_configs:
+  - job_name: "kubernetes-services"
+    kubernetes_sd_configs:
+      - role: service
+    relabel_configs:
+      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
+        action: keep
+        regex: true
+```
+
+## Recording Rules
+
+Create pre-computed metrics for frequently queried expressions:
+
+```yaml
+groups:
+  - name: api_metrics
+    interval: 15s
+    rules:
+      # HTTP request rate per service
+      - record: job:http_requests:rate5m
+        expr: sum by (job) (rate(http_requests_total[5m]))
+
+      # Error rate percentage
+      - record: job:http_requests_error_rate:percentage
+        expr: |
+          (sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) / job:http_requests:rate5m) * 100
+
+      # P95 latency
+      - record: job:http_request_duration:p95
+        expr: |
+          histogram_quantile(0.95,
+            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
+          )
+
+  - name: resource_metrics
+    rules:
+      # CPU utilization percentage
+      - record: instance:node_cpu:utilization
+        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+
+      # Memory utilization percentage
+      - record: instance:node_memory:utilization
+        expr: 100 - ((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100)
+```
+
+## Alert Rules
+
+```yaml
+groups:
+  - name: availability
+    rules:
+      - alert: ServiceDown
+        expr: up{job="my-app"} == 0
+        for: 1m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Service {{ $labels.instance }} is down"
+          description: "{{ $labels.job }} has been down for more than 1 minute"
+
+      - alert: HighErrorRate
+        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High error rate for {{ $labels.job }}"
+          description: "Error rate is {{ $value }}%"
+
+      - alert: HighCPUUsage
+        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High CPU usage on {{ $labels.instance }}"
+```
+
+## Validation
+
+```bash
+# Validate configuration
+promtool check config prometheus.yml
+
+# Validate rules
+promtool check rules /etc/prometheus/rules/*.yml
+```
+
+## Best Practices
+
+1. **Use consistent naming** for metrics (prefix_name_unit)
+2. **Set appropriate scrape intervals** (15-60s typical)
+3. **Use recording rules** for expensive queries
+4. **Implement high availability** (multiple Prometheus instances)
+5. **Configure retention** based on storage capacity
+6. **Use relabeling** for metric cleanup
+7. **Monitor Prometheus itself**
+8. **Use Thanos/Cortex** for long-term storage
+
+## Troubleshooting
+
+**Check scrape targets:**
+
+```bash
+curl http://localhost:9090/api/v1/targets
+```
+
+**Test query:**
+
+```bash
+curl 'http://localhost:9090/api/v1/query?query=up'
+```
+
+## Related Skills
+
+- `observability` - For general observability
+- `grafana-dashboards` - For visualization