@sylix/coworker 2.0.10 → 2.0.12

package/dist/skills/defaults/devops/observability.md

@@ -0,0 +1,243 @@
+---
+name: observability
+description: Implement comprehensive observability with logging, metrics, tracing, and alerting.
+---
+
+# Observability — CoWorker Edition
+
+Build observable systems with comprehensive monitoring.
+
+## When to Use This Skill
+
+- Setting up monitoring
+- Debugging production issues
+- Creating dashboards
+- Alert configuration
+
+## Core Concepts
+
+### 1. Structured Logging
+
+```python
+import structlog
+import logging
+
+# Configure structlog
+structlog.configure(
+    processors=[
+        structlog.stdlib.filter_by_level,
+        structlog.stdlib.add_logger_name,
+        structlog.stdlib.add_log_level,
+        structlog.processors.TimeStamper(fmt="iso"),
+        structlog.processors.StackInfoRenderer(),
+        structlog.processors.format_exc_info,
+        structlog.processors.JSONRenderer()
+    ],
+    wrapper_class=structlog.stdlib.BoundLogger,
+    context_class=dict,
+    logger_factory=structlog.stdlib.LoggerFactory(),
+)
+
+logger = structlog.get_logger()
+
+# Usage
+logger.info(
+    "order_created",
+    order_id="123",
+    user_id="456",
+    amount=99.99
+)
+
+# Output:
+# {"event": "order_created", "order_id": "123", "user_id": "456", 
+#  "amount": 99.99, "timestamp": "2024-01-15T10:30:00Z"}
+```
+
+### 2. Prometheus Metrics
+
+```python
+from prometheus_client import Counter, Histogram, Gauge
+
+# Request metrics
+http_requests_total = Counter(
+    'http_requests_total',
+    'Total HTTP requests',
+    ['method', 'endpoint', 'status']
+)
+
+http_request_duration_seconds = Histogram(
+    'http_request_duration_seconds',
+    'HTTP request duration',
+    ['method', 'endpoint'],
+    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0]
+)
+
+# Business metrics
+orders_placed = Counter(
+    'orders_placed_total',
+    'Total orders placed',
+    ['status']
+)
+
+active_users = Gauge(
+    'active_users',
+    'Number of active users'
+)
+
+# Usage in FastAPI
+@app.middleware("http")
+async def metrics_middleware(request, call_next):
+    start = time.time()
+    response = await call_next(request)
+    duration = time.time() - start
+    
+    http_requests_total.labels(
+        method=request.method,
+        endpoint=request.url.path,
+        status=response.status_code
+    ).inc()
+    
+    http_request_duration_seconds.labels(
+        method=request.method,
+        endpoint=request.url.path
+    ).observe(duration)
+    
+    return response
+```
+
+### 3. Distributed Tracing
+
+```python
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.exporter.jaeger.thrift import JaegerExporter
+
+# Setup
+provider = TracerProvider()
+processor = BatchSpanProcessor(JaegerExporter(
+    agent_host_name="jaeger",
+    agent_port=6831,
+))
+provider.add_span_processor(processor)
+trace.set_tracer_provider(provider)
+
+tracer = trace.get_tracer(__name__)
+
+# Usage
+class OrderService:
+    async def create_order(self, user_id: str, items: list):
+        with tracer.start_as_current_span("create_order") as span:
+            span.set_attribute("user_id", user_id)
+            span.set_attribute("item_count", len(items))
+            
+            # Validate order
+            with tracer.start_as_current_span("validate_order"):
+                await self.validate_order(user_id, items)
+            
+            # Process payment
+            with tracer.start_as_current_span("process_payment"):
+                payment = await self.payment.charge(user_id, items)
+            
+            # Save order
+            with tracer.start_as_current_span("save_order"):
+                order = await self.orders.save(...)
+            
+            span.set_attribute("order_id", order.id)
+            return order
+```
+
+### 4. Grafana Dashboard
+
+```yaml
+# dashboard.json
+{
+  "dashboard": {
+    "title": "API Performance",
+    "panels": [
+      {
+        "title": "Request Rate",
+        "type": "graph",
+        "targets": [
+          {
+            "expr": "rate(http_requests_total[5m])",
+            "legendFormat": "{{method}} {{endpoint}}"
+          }
+        ]
+      },
+      {
+        "title": "Error Rate",
+        "type": "graph",
+        "targets": [
+          {
+            "expr": "rate(http_requests_total{status=~'5..'}[5m])",
+            "legendFormat": "5xx - {{endpoint}}"
+          }
+        ]
+      },
+      {
+        "title": "P99 Latency",
+        "type": "graph",
+        "targets": [
+          {
+            "expr": "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))",
+            "legendFormat": "P99"
+          }
+        ]
+      }
+    ]
+  }
+}
+```
+
+### 5. Alerting Rules
+
+```yaml
+# alerts.yaml
+groups:
+  - name: api_alerts
+    rules:
+      - alert: HighErrorRate
+        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: "High error rate on {{ $labels.endpoint }}"
+          description: "Error rate is {{ $value | humanizePercentage }}"
+      
+      - alert: HighLatency
+        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 1
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High latency on {{ $labels.endpoint }}"
+          description: "P99 latency is {{ $value }}s"
+      
+      - alert: LowDiskSpace
+        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.1
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Low disk space on {{ $labels.instance }}"
+```
+
+## Best Practices
+
+1. **Structured logs** - JSON with context
+2. **Metric cardinality** - Don't over-tag
+3. **Trace everything** - Distributed systems
+4. **Alert thresholds** - Based on SLIs
+5. **Dashboards** - Actionable, not just data
+6. **SLO tracking** - Error budgets
+
+## Common Mistakes
+
+- No logging in production
+- Too many alerts
+- No alert tuning
+- Missing context in logs
+- Not tracking business metrics
+- No error budgeting