aigroup-workflow 2.1.2 → 2.2.0

package/skills/backend/microservices-architect/references/observability.md

@@ -0,0 +1,805 @@
+# Observability in Microservices
+
+Comprehensive guide for monitoring, tracing, and debugging distributed systems.
+
+## The Three Pillars
+
+### 1. Metrics
+
+**Purpose:** Quantitative measurements of system behavior over time.
+
+**Categories:**
+
+**Business Metrics:**
+```
+Examples:
+- Orders per minute
+- Revenue per hour
+- Active users
+- Conversion rate
+- Cart abandonment rate
+
+Why Important:
+- Align with business goals
+- Detect business anomalies
+- Inform scaling decisions
+
+Implementation:
+from prometheus_client import Counter, Histogram
+
+orders_total = Counter(
+    'orders_total',
+    'Total number of orders',
+    ['status', 'payment_method']
+)
+
+order_value = Histogram(
+    'order_value_dollars',
+    'Order value in dollars',
+    buckets=[10, 50, 100, 500, 1000, 5000]
+)
+
+# In code
+orders_total.labels(status='completed', payment_method='credit_card').inc()
+order_value.observe(order.total_amount)
+```
+
+**System Metrics:**
+```
+Infrastructure:
+- CPU usage
+- Memory usage
+- Disk I/O
+- Network throughput
+
+Application:
+- Request rate
+- Error rate
+- Request duration (latency)
+- Active connections
+- Thread pool utilization
+
+Database:
+- Query duration
+- Connection pool usage
+- Slow queries
+- Deadlocks
+
+Message Queue:
+- Queue depth
+- Message processing rate
+- Consumer lag
+- Dead letter queue size
+```
+
+**The Four Golden Signals (Google SRE):**
+```
+1. Latency:
+   - Time to serve requests
+   - Track p50, p95, p99, p99.9
+   - Separate success vs error latency
+
+   request_duration = Histogram(
+       'http_request_duration_seconds',
+       'HTTP request duration',
+       ['method', 'endpoint', 'status']
+   )
+
+2. Traffic:
+   - Requests per second
+   - Transactions per second
+   - Concurrent users
+
+   requests_total = Counter(
+       'http_requests_total',
+       'Total HTTP requests',
+       ['method', 'endpoint', 'status']
+   )
+
+3. Errors:
+   - Rate of failed requests
+   - 4xx vs 5xx errors
+   - Exception types
+
+   errors_total = Counter(
+       'errors_total',
+       'Total errors',
+       ['service', 'error_type']
+   )
+
+4. Saturation:
+   - Resource utilization
+   - Queue depth
+   - Thread pool usage
+
+   connection_pool_usage = Gauge(
+       'db_connection_pool_active',
+       'Active database connections'
+   )
+```
+
+**RED Method (for services):**
+```
+- Rate: Requests per second
+- Errors: Failed requests per second
+- Duration: Request latency distribution
+
+Perfect for microservices dashboards
+```
+
+**USE Method (for resources):**
+```
+- Utilization: Percentage of time resource busy
+- Saturation: Queue depth or waiting threads
+- Errors: Error count
+
+Perfect for infrastructure monitoring
+```
+
+### 2. Logs
+
+**Purpose:** Discrete event records with context.
+
+**Structured Logging:**
+```json
+{
+  "timestamp": "2025-12-14T15:30:45.123Z",
+  "level": "INFO",
+  "service": "order-service",
+  "version": "1.2.3",
+  "traceId": "abc123def456",
+  "spanId": "span789",
+  "userId": "user-123",
+  "message": "Order created successfully",
+  "orderId": "order-456",
+  "totalAmount": 99.99,
+  "currency": "USD",
+  "duration_ms": 45,
+  "endpoint": "/api/v1/orders",
+  "method": "POST",
+  "statusCode": 201
+}
+```
+
+**Log Levels:**
+```
+ERROR:
+- Application errors
+- Failed operations
+- Exceptions
+Use: Alerts, immediate attention
+
+WARN:
+- Degraded functionality
+- Retry attempts
+- Deprecated API usage
+Use: Investigation, potential issues
+
+INFO:
+- Business events (order created, user logged in)
+- System events (service started, configuration loaded)
+Use: Audit trail, business analytics
+
+DEBUG:
+- Detailed execution flow
+- Variable values
+- Function entry/exit
+Use: Development, troubleshooting
+
+TRACE:
+- Very detailed debugging
+Use: Deep troubleshooting (disabled in production usually)
+```
+
+**Correlation IDs:**
+```
+Request flow across services:
+
+Client Request → API Gateway
+                 ↓ (correlationId: corr-123)
+                 Order Service
+                 ↓ (correlationId: corr-123)
+                 Payment Service
+                 ↓ (correlationId: corr-123)
+                 Notification Service
+
+All logs include correlationId: corr-123
+Easy to trace entire request flow
+
+Implementation:
+import logging
+from contextvars import ContextVar
+
+correlation_id_var = ContextVar('correlation_id', default=None)
+
+class CorrelationIdFilter(logging.Filter):
+    def filter(self, record):
+        record.correlation_id = correlation_id_var.get()
+        return True
+
+# Middleware
+async def correlation_middleware(request, call_next):
+    correlation_id = request.headers.get('X-Correlation-ID', str(uuid4()))
+    correlation_id_var.set(correlation_id)
+    response = await call_next(request)
+    response.headers['X-Correlation-ID'] = correlation_id
+    return response
+```
+
+**Log Aggregation:**
+```
+Services → Log Shipper → Centralized Log Storage → Visualization
+
+Tools:
+- ELK Stack (Elasticsearch, Logstash, Kibana)
+- EFK Stack (Elasticsearch, Fluentd, Kibana)
+- Loki (from Grafana)
+- CloudWatch Logs (AWS)
+- Stackdriver (GCP)
+
+Query Examples:
+# Find all errors for specific user
+service:"order-service" AND level:"ERROR" AND userId:"user-123"
+
+# Find slow requests
+service:"payment-service" AND duration_ms:>5000
+
+# Find requests with specific correlation ID
+correlationId:"corr-123"
+```
+
+### 3. Distributed Tracing
+
+**Purpose:** Visualize request flow across services, identify bottlenecks.
+
+**Concepts:**
+
+**Trace:**
+```
+Entire request journey across all services
+
+Example: User places order
+Trace ID: trace-abc123
+
+Spans in trace:
+1. api-gateway: /checkout (200ms)
+2. order-service: createOrder (150ms)
+3. payment-service: processPayment (80ms)
+4. inventory-service: reserveItems (40ms)
+5. notification-service: sendEmail (30ms)
+
+Total: 200ms (some parallel execution)
+```
+
+**Span:**
+```
+Single operation within a trace
+
+Span attributes:
+{
+  "traceId": "trace-abc123",
+  "spanId": "span-456",
+  "parentSpanId": "span-123",
+  "name": "POST /api/v1/orders",
+  "startTime": "2025-12-14T15:30:45.000Z",
+  "endTime": "2025-12-14T15:30:45.150Z",
+  "duration": 150,
+  "status": "OK",
+  "attributes": {
+    "http.method": "POST",
+    "http.url": "/api/v1/orders",
+    "http.status_code": 201,
+    "user.id": "user-123",
+    "order.id": "order-456",
+    "order.total": 99.99
+  },
+  "events": [
+    {
+      "timestamp": "2025-12-14T15:30:45.050Z",
+      "name": "Validating order items"
+    },
+    {
+      "timestamp": "2025-12-14T15:30:45.100Z",
+      "name": "Calling payment service"
+    }
+  ]
+}
+```
+
+**Implementation (OpenTelemetry):**
+```python
+from opentelemetry import trace
+from opentelemetry.exporter.jaeger.thrift import JaegerExporter
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
+
+# Setup tracing
+provider = TracerProvider()
+jaeger_exporter = JaegerExporter(
+    agent_host_name="jaeger",
+    agent_port=6831
+)
+provider.add_span_processor(BatchSpanProcessor(jaeger_exporter))
+trace.set_tracer_provider(provider)
+
+# Instrument FastAPI
+app = FastAPI()
+FastAPIInstrumentor.instrument_app(app)
+
+# Manual span creation
+tracer = trace.get_tracer(__name__)
+
+async def create_order(order_data):
+    with tracer.start_as_current_span("create_order") as span:
+        span.set_attribute("order.items_count", len(order_data.items))
+        span.set_attribute("order.total", order_data.total)
+
+        # Database operation
+        with tracer.start_as_current_span("db.insert_order"):
+            order_id = await db.insert_order(order_data)
+
+        # Call payment service
+        with tracer.start_as_current_span("http.payment_service") as payment_span:
+            payment_span.set_attribute("http.url", f"{PAYMENT_URL}/payments")
+            result = await payment_service.charge(order_id, order_data.total)
+
+        return order_id
+```
+
+**Trace Visualization:**
+```
+Jaeger UI shows:
+
+Timeline view:
+|-- api-gateway (200ms) ----------------------------------|
+    |-- order-service (150ms) ------------------------|
+        |-- db.insert_order (30ms) --|
+        |-- payment-service (80ms) -----------------|
+            |-- db.create_transaction (20ms) ----|
+        |-- notification-service (30ms) ----------|
+
+Critical path highlighted
+Bottlenecks identified (payment-service taking 80ms)
+Parallel operations visible
+```
+
+**Sampling Strategies:**
+```
+Problem: Tracing every request is expensive
+
+Solutions:
+
+1. Probabilistic Sampling:
+   - Trace 1% of requests
+   - Good for high-volume services
+
+2. Rate Limiting Sampling:
+   - Max 100 traces per second
+   - Prevents overwhelming trace backend
+
+3. Tail-Based Sampling:
+   - Trace all errors
+   - Trace slow requests (>5s)
+   - Sample 1% of fast successful requests
+
+4. Priority Sampling:
+   - Always trace premium users
+   - Always trace critical endpoints
+   - Sample others
+
+Implementation:
+from opentelemetry.sdk.trace.sampling import (
+    ParentBasedTraceIdRatioBased,
+    ALWAYS_ON,
+    ALWAYS_OFF
+)
+
+# Sample 1% of traces
+sampler = ParentBasedTraceIdRatioBased(0.01)
+
+# Or custom sampler
+class CustomSampler:
+    def should_sample(self, context, trace_id, name, attributes):
+        # Always sample errors
+        if attributes.get("http.status_code", 0) >= 500:
+            return ALWAYS_ON
+
+        # Always sample slow requests
+        if attributes.get("duration_ms", 0) > 5000:
+            return ALWAYS_ON
+
+        # Sample 1% of others
+        return ParentBasedTraceIdRatioBased(0.01).should_sample(...)
+```
+
+## Service Level Objectives (SLOs)
+
+### Defining SLOs
+
+**SLI (Service Level Indicator):**
+```
+Quantitative measure of service level
+
+Examples:
+- Request latency: p99 < 200ms
+- Availability: 99.9% of requests succeed
+- Throughput: Handle 10,000 requests/sec
+```
+
+**SLO (Service Level Objective):**
+```
+Target value for SLI
+
+Examples:
+- 99.9% of requests complete in < 200ms
+- 99.95% availability over 30 days
+- Zero data loss
+
+SLO Components:
+- Metric: What you measure (latency, availability)
+- Target: Threshold (99.9%, 200ms)
+- Time window: Evaluation period (30 days, weekly)
+```
+
+**SLA (Service Level Agreement):**
+```
+Contract with consequences if SLO not met
+
+Example:
+- SLO: 99.9% availability
+- SLA: If availability < 99.9%, customers get 10% credit
+
+SLA ≤ SLO (leave buffer for incidents)
+```
+
+**Error Budget:**
+```
+Allowed failure to meet SLO = (100% - SLO target)
+
+Example:
+SLO: 99.9% availability
+Error budget: 0.1% = 43.8 minutes downtime per month
+
+Error budget consumed:
+- Outages
+- Slow responses
+- Failed requests
+
+When error budget exhausted:
+- Freeze feature deployments
+- Focus on reliability
+- Only critical fixes deployed
+
+Benefits:
+- Balances innovation vs stability
+- Data-driven deployment decisions
+- Aligns engineering priorities
+```
+
+### Implementing SLO Monitoring
+
+**Prometheus + Grafana:**
+```
+# SLI: Availability
+availability_sli = (
+    sum(rate(http_requests_total{status!~"5.."}[30d]))
+    /
+    sum(rate(http_requests_total[30d]))
+) * 100
+
+# SLI: Latency
+latency_sli = histogram_quantile(
+    0.99,
+    rate(http_request_duration_seconds_bucket[30d])
+)
+
+# Error Budget
+error_budget_remaining = (
+    1 - (target_slo / 100)
+) - (
+    1 - (availability_sli / 100)
+)
+
+Alert when error budget < 10%:
+alert: ErrorBudgetCritical
+expr: error_budget_remaining < 0.1
+annotations:
+  summary: "Error budget critically low"
+  description: "Only 10% error budget remaining. Freeze deployments."
+```
+
+## Alerting Strategies
+
+### Alert Levels
+
+**Critical (Page immediately):**
+```
+Conditions:
+- Service completely down
+- Error rate > 50%
+- Data loss occurring
+- SLO burn rate critical
+
+Actions:
+- Page on-call engineer
+- Incident created automatically
+- Escalate if not acknowledged in 5 min
+
+Example:
+alert: ServiceDown
+expr: up{service="payment-service"} == 0
+for: 1m
+severity: critical
+```
+
+**Warning (Investigate soon):**
+```
+Conditions:
+- Elevated error rate (5-10%)
+- Latency degraded (p99 > 500ms)
+- Queue depth increasing
+- Error budget < 25%
+
+Actions:
+- Slack notification
+- Create ticket
+- Investigate during business hours
+
+Example:
+alert: HighErrorRate
+expr: rate(http_requests_total{status="500"}[5m]) > 0.05
+for: 10m
+severity: warning
+```
+
+**Info (Awareness):**
+```
+Conditions:
+- Deployment completed
+- Scaling event
+- Configuration changed
+- Capacity threshold reached
+
+Actions:
+- Log to monitoring system
+- Dashboard annotation
+- Optional Slack notification
+```
+
+### Alert Best Practices
+
+**Actionable Alerts:**
+```
+Bad Alert:
+"High CPU usage"
+
+Good Alert:
+"CPU usage > 80% on order-service-pod-abc for 10 minutes
+Runbook: https://wiki.company.com/runbooks/high-cpu
+Likely cause: Memory leak or infinite loop
+Actions: 1) Check recent deployments 2) Review logs for exceptions 3) Consider rolling back"
+
+Include:
+✓ What is wrong
+✓ Why it matters
+✓ How to investigate
+✓ Runbook link
+✓ Suggested actions
+```
+
+**Avoid Alert Fatigue:**
+```
+Problems:
+- Too many alerts
+- False positives
+- Non-actionable alerts
+- Duplicate alerts
+
+Solutions:
+- Alert on symptoms, not causes
+- Proper thresholds and durations
+- Alert aggregation (don't alert per pod, alert per service)
+- Regular alert review and tuning
+- Auto-resolve alerts
+- Silence during maintenance
+
+Good Practice:
+for: 5m  # Don't alert on transient spikes
+group_by: [service]  # Aggregate per service
+group_wait: 30s  # Wait before sending
+group_interval: 5m  # Batch notifications
+```
+
+## Observability Stack
+
+### Recommended Tools
+
+**Metrics:**
+```
+Collection: Prometheus
+- Pull-based metrics
+- Time-series database
+- Powerful query language (PromQL)
+- Service discovery
+
+Visualization: Grafana
+- Beautiful dashboards
+- Alerting integration
+- Multiple data sources
+- Template variables
+
+Alternative: Datadog, New Relic, CloudWatch
+```
+
+**Logs:**
+```
+Aggregation: ELK Stack
+- Elasticsearch (storage & search)
+- Logstash / Fluentd (collection)
+- Kibana (visualization)
+
+Or: Loki (lightweight alternative)
+- Integrates with Grafana
+- Labels instead of full-text indexing
+- Lower resource usage
+
+Alternative: Splunk, Datadog, CloudWatch Logs
+```
+
+**Tracing:**
+```
+Backend: Jaeger or Zipkin
+- Trace storage
+- Trace visualization
+- Dependency graphs
+- Performance analysis
+
+Instrumentation: OpenTelemetry
+- Vendor-neutral standard
+- Auto-instrumentation for common frameworks
+- Manual instrumentation API
+- Export to any backend
+
+Alternative: Datadog APM, New Relic, Lightstep
+```
+
+**All-in-One:**
+```
+Observability platforms:
+- Datadog (metrics, logs, traces, RUM)
+- New Relic (APM, logs, infrastructure)
+- Dynatrace (auto-instrumentation, AI)
+
+Pros:
+- Unified experience
+- Correlated data
+- Easier setup
+
+Cons:
+- Vendor lock-in
+- Higher cost
+- Less flexibility
+```
+
+### Implementation Checklist
+
+**For Each Service:**
+```
+✓ Structured logging with correlation IDs
+✓ Metrics exported (Prometheus format)
+✓ Distributed tracing instrumented
+✓ Health check endpoints (/health/live, /health/ready)
+✓ Graceful shutdown handling
+✓ Resource limits set (CPU, memory)
+✓ Alerts configured for critical paths
+✓ Dashboards created
+✓ Runbooks documented
+✓ On-call rotation established
+```
+
+**For System-Wide:**
+```
+✓ Centralized log aggregation
+✓ Distributed tracing backend
+✓ Metrics aggregation and storage
+✓ Unified dashboards (service overview)
+✓ Alert routing configured
+✓ Incident management process
+✓ Post-mortem template
+✓ SLO definitions and tracking
+✓ Dependency mapping
+✓ Chaos engineering experiments
+```
+
+## Troubleshooting Workflow
+
+**Incident Response:**
+```
+1. Detect (Alert fires)
+   - Check dashboard
+   - Verify alert is valid
+   - Assess impact
+
+2. Triage (Determine severity)
+   - Critical: Page on-call
+   - Warning: Create ticket
+   - How many users affected?
+   - What functionality broken?
+
+3. Investigate (Find root cause)
+   - Check recent deployments
+   - Review logs (search by correlation ID)
+   - Analyze traces (slow operations)
+   - Check metrics (resource saturation)
+   - Examine dependencies
+
+4. Mitigate (Stop the bleeding)
+   - Rollback deployment
+   - Scale up resources
+   - Failover to backup
+   - Enable circuit breakers
+   - Rate limit traffic
+
+5. Resolve (Fix root cause)
+   - Deploy fix
+   - Verify resolution
+   - Monitor for recurrence
+
+6. Post-mortem (Learn and improve)
+   - Timeline of events
+   - Root cause analysis
+   - Action items
+   - Update runbooks
+```
+
+**Using Traces to Debug:**
+```
+Scenario: API returning 500 errors
+
+1. Find failing trace:
+   - Filter: status = error, service = api-gateway
+   - Sort by timestamp (most recent)
+
+2. Analyze span waterfall:
+   - Identify which service failed (order-service returned 500)
+   - Check error message in span
+   - Review span attributes
+
+3. Correlate with logs:
+   - Extract trace ID from failed trace
+   - Search logs: traceId:"trace-abc123"
+   - Find exception stack trace
+
+4. Check related metrics:
+   - order-service error rate spiked 10 min ago
+   - Corresponds with deployment
+   - Likely cause: Bad deployment
+
+5. Remediate:
+   - Rollback order-service
+   - Verify errors stopped
+   - Create ticket for bug fix
+```
+
+## Summary
+
+Observability is non-negotiable in microservices:
+
+**Must-Haves:**
+- Structured logging with correlation IDs
+- Metrics (RED/USE methodology)
+- Distributed tracing (OpenTelemetry)
+- Centralized log aggregation
+- SLO tracking with error budgets
+- Actionable alerts with runbooks
+
+**Best Practices:**
+- Correlate metrics, logs, and traces
+- Define SLOs based on user experience
+- Alert on symptoms, not causes
+- Maintain runbooks for common issues
+- Regular post-mortems and learning
+- Practice incident response with game days
+
+Without observability, you're flying blind in production.