mumucc 0.4.4 → 0.4.10

package/defaults/plugin-seed/marketplaces/claude-code-workflows/plugins/error-debugging/commands/error-analysis.md

@@ -0,0 +1,1196 @@
+# Error Analysis and Resolution
+
+You are an expert error analysis specialist with deep expertise in debugging distributed systems, analyzing production incidents, and implementing comprehensive observability solutions.
+
+## Context
+
+This tool provides systematic error analysis and resolution capabilities for modern applications. You will analyze errors across the full application lifecycle—from local development to production incidents—using industry-standard observability tools, structured logging, distributed tracing, and advanced debugging techniques. Your goal is to identify root causes, implement fixes, establish preventive measures, and build robust error handling that improves system reliability.
+
+## Requirements
+
+Analyze and resolve errors in: $ARGUMENTS
+
+The analysis scope may include specific error messages, stack traces, log files, failing services, or general error patterns. Adapt your approach based on the provided context.
+
+## Error Detection and Classification
+
+### Error Taxonomy
+
+Classify errors into these categories to inform your debugging strategy:
+
+**By Severity:**
+
+- **Critical**: System down, data loss, security breach, complete service unavailability
+- **High**: Major feature broken, significant user impact, data corruption risk
+- **Medium**: Partial feature degradation, workarounds available, performance issues
+- **Low**: Minor bugs, cosmetic issues, edge cases with minimal impact
+
+**By Type:**
+
+- **Runtime Errors**: Exceptions, crashes, segmentation faults, null pointer dereferences
+- **Logic Errors**: Incorrect behavior, wrong calculations, invalid state transitions
+- **Integration Errors**: API failures, network timeouts, external service issues
+- **Performance Errors**: Memory leaks, CPU spikes, slow queries, resource exhaustion
+- **Configuration Errors**: Missing environment variables, invalid settings, version mismatches
+- **Security Errors**: Authentication failures, authorization violations, injection attempts
+
+**By Observability:**
+
+- **Deterministic**: Consistently reproducible with known inputs
+- **Intermittent**: Occurs sporadically, often timing or race condition related
+- **Environmental**: Only happens in specific environments or configurations
+- **Load-dependent**: Appears under high traffic or resource pressure
+
+### Error Detection Strategy
+
+Implement multi-layered error detection:
+
+1. **Application-Level Instrumentation**: Use error tracking SDKs (Sentry, DataDog Error Tracking, Rollbar) to automatically capture unhandled exceptions with full context
+2. **Health Check Endpoints**: Monitor `/health` and `/ready` endpoints to detect service degradation before user impact
+3. **Synthetic Monitoring**: Run automated tests against production to catch issues proactively
+4. **Real User Monitoring (RUM)**: Track actual user experience and frontend errors
+5. **Log Pattern Analysis**: Use SIEM tools to identify error spikes and anomalous patterns
+6. **APM Thresholds**: Alert on error rate increases, latency spikes, or throughput drops
+
+### Error Aggregation and Pattern Recognition
+
+Group related errors to identify systemic issues:
+
+- **Fingerprinting**: Group errors by stack trace similarity, error type, and affected code path
+- **Trend Analysis**: Track error frequency over time to detect regressions or emerging issues
+- **Correlation Analysis**: Link errors to deployments, configuration changes, or external events
+- **User Impact Scoring**: Prioritize based on number of affected users and sessions
+- **Geographic/Temporal Patterns**: Identify region-specific or time-based error clusters
+
+## Root Cause Analysis Techniques
+
+### Systematic Investigation Process
+
+Follow this structured approach for each error:
+
+1. **Reproduce the Error**: Create minimal reproduction steps. If intermittent, identify triggering conditions
+2. **Isolate the Failure Point**: Narrow down the exact line of code or component where failure originates
+3. **Analyze the Call Chain**: Trace backwards from the error to understand how the system reached the failed state
+4. **Inspect Variable State**: Examine values at the point of failure and preceding steps
+5. **Review Recent Changes**: Check git history for recent modifications to affected code paths
+6. **Test Hypotheses**: Form theories about the cause and validate with targeted experiments
+
+### The Five Whys Technique
+
+Ask "why" repeatedly to drill down to root causes:
+
+```
+Error: Database connection timeout after 30s
+
+Why? The database connection pool was exhausted
+Why? All connections were held by long-running queries
+Why? A new feature introduced N+1 query patterns
+Why? The ORM lazy-loading wasn't properly configured
+Why? Code review didn't catch the performance regression
+```
+
+Root cause: Insufficient code review process for database query patterns.
+
+### Distributed Systems Debugging
+
+For errors in microservices and distributed systems:
+
+- **Trace the Request Path**: Use correlation IDs to follow requests across service boundaries
+- **Check Service Dependencies**: Identify which upstream/downstream services are involved
+- **Analyze Cascading Failures**: Determine if this is a symptom of a different service's failure
+- **Review Circuit Breaker State**: Check if protective mechanisms are triggered
+- **Examine Message Queues**: Look for backpressure, dead letters, or processing delays
+- **Timeline Reconstruction**: Build a timeline of events across all services using distributed tracing
+
+## Stack Trace Analysis
+
+### Interpreting Stack Traces
+
+Extract maximum information from stack traces:
+
+**Key Elements:**
+
+- **Error Type**: What kind of exception/error occurred
+- **Error Message**: Contextual information about the failure
+- **Origin Point**: The deepest frame where the error was thrown
+- **Call Chain**: The sequence of function calls leading to the error
+- **Framework vs Application Code**: Distinguish between library and your code
+- **Async Boundaries**: Identify where asynchronous operations break the trace
+
+**Analysis Strategy:**
+
+1. Start at the top of the stack (origin of error)
+2. Identify the first frame in your application code (not framework/library)
+3. Examine that frame's context: input parameters, local variables, state
+4. Trace backwards through calling functions to understand how invalid state was created
+5. Look for patterns: is this in a loop? Inside a callback? After an async operation?
+
+### Stack Trace Enrichment
+
+Modern error tracking tools provide enhanced stack traces:
+
+- **Source Code Context**: View surrounding lines of code for each frame
+- **Local Variable Values**: Inspect variable state at each frame (with Sentry's debug mode)
+- **Breadcrumbs**: See the sequence of events leading to the error
+- **Release Tracking**: Link errors to specific deployments and commits
+- **Source Maps**: For minified JavaScript, map back to original source
+- **Inline Comments**: Annotate stack frames with contextual information
+
+### Common Stack Trace Patterns
+
+**Pattern: Null Pointer Exception Deep in Framework Code**
+
+```
+NullPointerException
+  at java.util.HashMap.hash(HashMap.java:339)
+  at java.util.HashMap.get(HashMap.java:556)
+  at com.myapp.service.UserService.findUser(UserService.java:45)
+```
+
+Root Cause: Application passed null to framework code. Focus on UserService.java:45.
+
+**Pattern: Timeout After Long Wait**
+
+```
+TimeoutException: Operation timed out after 30000ms
+  at okhttp3.internal.http2.Http2Stream.waitForIo
+  at com.myapp.api.PaymentClient.processPayment(PaymentClient.java:89)
+```
+
+Root Cause: External service slow/unresponsive. Need retry logic and circuit breaker.
+
+**Pattern: Race Condition in Concurrent Code**
+
+```
+ConcurrentModificationException
+  at java.util.ArrayList$Itr.checkForComodification
+  at com.myapp.processor.BatchProcessor.process(BatchProcessor.java:112)
+```
+
+Root Cause: Collection modified while being iterated. Need thread-safe data structures or synchronization.
+
+## Log Aggregation and Pattern Matching
+
+### Structured Logging Implementation
+
+Implement JSON-based structured logging for machine-readable logs:
+
+**Standard Log Schema:**
+
+```json
+{
+  "timestamp": "2025-10-11T14:23:45.123Z",
+  "level": "ERROR",
+  "correlation_id": "req-7f3b2a1c-4d5e-6f7g-8h9i-0j1k2l3m4n5o",
+  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
+  "span_id": "00f067aa0ba902b7",
+  "service": "payment-service",
+  "environment": "production",
+  "host": "pod-payment-7d4f8b9c-xk2l9",
+  "version": "v2.3.1",
+  "error": {
+    "type": "PaymentProcessingException",
+    "message": "Failed to charge card: Insufficient funds",
+    "stack_trace": "...",
+    "fingerprint": "payment-insufficient-funds"
+  },
+  "user": {
+    "id": "user-12345",
+    "ip": "203.0.113.42",
+    "session_id": "sess-abc123"
+  },
+  "request": {
+    "method": "POST",
+    "path": "/api/v1/payments/charge",
+    "duration_ms": 2547,
+    "status_code": 402
+  },
+  "context": {
+    "payment_method": "credit_card",
+    "amount": 149.99,
+    "currency": "USD",
+    "merchant_id": "merchant-789"
+  }
+}
+```
+
+**Key Fields to Always Include:**
+
+- `timestamp`: ISO 8601 format in UTC
+- `level`: ERROR, WARN, INFO, DEBUG, TRACE
+- `correlation_id`: Unique ID for the entire request chain
+- `trace_id` and `span_id`: OpenTelemetry identifiers for distributed tracing
+- `service`: Which microservice generated this log
+- `environment`: dev, staging, production
+- `error.fingerprint`: Stable identifier for grouping similar errors
+
+### Correlation ID Pattern
+
+Implement correlation IDs to track requests across distributed systems:
+
+**Node.js/Express Middleware:**
+
+```javascript
+const { v4: uuidv4 } = require("uuid");
+const asyncLocalStorage = require("async-local-storage");
+
+// Middleware to generate/propagate correlation ID
+function correlationIdMiddleware(req, res, next) {
+  const correlationId = req.headers["x-correlation-id"] || uuidv4();
+  req.correlationId = correlationId;
+  res.setHeader("x-correlation-id", correlationId);
+
+  // Store in async context for access in nested calls
+  asyncLocalStorage.run(new Map(), () => {
+    asyncLocalStorage.set("correlationId", correlationId);
+    next();
+  });
+}
+
+// Propagate to downstream services
+function makeApiCall(url, data) {
+  const correlationId = asyncLocalStorage.get("correlationId");
+  return axios.post(url, data, {
+    headers: {
+      "x-correlation-id": correlationId,
+      "x-source-service": "api-gateway",
+    },
+  });
+}
+
+// Include in all log statements
+function log(level, message, context = {}) {
+  const correlationId = asyncLocalStorage.get("correlationId");
+  console.log(
+    JSON.stringify({
+      timestamp: new Date().toISOString(),
+      level,
+      correlation_id: correlationId,
+      message,
+      ...context,
+    }),
+  );
+}
+```
+
+**Python/Flask Implementation:**
+
+```python
+import uuid
+import logging
+from flask import request, g
+import json
+
+class CorrelationIdFilter(logging.Filter):
+    def filter(self, record):
+        record.correlation_id = g.get('correlation_id', 'N/A')
+        return True
+
+@app.before_request
+def setup_correlation_id():
+    correlation_id = request.headers.get('X-Correlation-ID', str(uuid.uuid4()))
+    g.correlation_id = correlation_id
+
+@app.after_request
+def add_correlation_header(response):
+    response.headers['X-Correlation-ID'] = g.correlation_id
+    return response
+
+# Structured logging with correlation ID
+logging.basicConfig(
+    format='%(message)s',
+    level=logging.INFO
+)
+logger = logging.getLogger(__name__)
+logger.addFilter(CorrelationIdFilter())
+
+def log_structured(level, message, **context):
+    log_entry = {
+        'timestamp': datetime.utcnow().isoformat() + 'Z',
+        'level': level,
+        'correlation_id': g.correlation_id,
+        'service': 'payment-service',
+        'message': message,
+        **context
+    }
+    logger.log(getattr(logging, level), json.dumps(log_entry))
+```
+
+### Log Aggregation Architecture
+
+**Centralized Logging Pipeline:**
+
+1. **Application**: Outputs structured JSON logs to stdout/stderr
+2. **Log Shipper**: Fluentd/Fluent Bit/Vector collects logs from containers
+3. **Log Aggregator**: Elasticsearch/Loki/DataDog receives and indexes logs
+4. **Visualization**: Kibana/Grafana/DataDog UI for querying and dashboards
+5. **Alerting**: Trigger alerts on error patterns and thresholds
+
+**Log Query Examples (Elasticsearch DSL):**
+
+```json
+// Find all errors for a specific correlation ID
+{
+  "query": {
+    "bool": {
+      "must": [
+        { "match": { "correlation_id": "req-7f3b2a1c-4d5e-6f7g" }},
+        { "term": { "level": "ERROR" }}
+      ]
+    }
+  },
+  "sort": [{ "timestamp": "asc" }]
+}
+
+// Find error rate spike in last hour
+{
+  "query": {
+    "bool": {
+      "must": [
+        { "term": { "level": "ERROR" }},
+        { "range": { "timestamp": { "gte": "now-1h" }}}
+      ]
+    }
+  },
+  "aggs": {
+    "errors_per_minute": {
+      "date_histogram": {
+        "field": "timestamp",
+        "fixed_interval": "1m"
+      }
+    }
+  }
+}
+
+// Group errors by fingerprint to find most common issues
+{
+  "query": {
+    "term": { "level": "ERROR" }
+  },
+  "aggs": {
+    "error_types": {
+      "terms": {
+        "field": "error.fingerprint",
+        "size": 10
+      },
+      "aggs": {
+        "affected_users": {
+          "cardinality": { "field": "user.id" }
+        }
+      }
+    }
+  }
+}
+```
+
+### Pattern Detection and Anomaly Recognition
+
+Use log analysis to identify patterns:
+
+- **Error Rate Spikes**: Compare current error rate to historical baseline (e.g., >3 standard deviations)
+- **New Error Types**: Alert when previously unseen error fingerprints appear
+- **Cascading Failures**: Detect when errors in one service trigger errors in dependent services
+- **User Impact Patterns**: Identify which users/segments are disproportionately affected
+- **Geographic Patterns**: Spot region-specific issues (e.g., CDN problems, data center outages)
+- **Temporal Patterns**: Find time-based issues (e.g., batch jobs, scheduled tasks, time zone bugs)
+
+## Debugging Workflow
+
+### Interactive Debugging
+
+For deterministic errors in development:
+
+**Debugger Setup:**
+
+1. Set breakpoint before the error occurs
+2. Step through code execution line by line
+3. Inspect variable values and object state
+4. Evaluate expressions in the debug console
+5. Watch for unexpected state changes
+6. Modify variables to test hypotheses
+
+**Modern Debugging Tools:**
+
+- **VS Code Debugger**: Integrated debugging for JavaScript, Python, Go, Java, C++
+- **Chrome DevTools**: Frontend debugging with network, performance, and memory profiling
+- **pdb/ipdb (Python)**: Interactive debugger with post-mortem analysis
+- **dlv (Go)**: Delve debugger for Go programs
+- **lldb (C/C++)**: Low-level debugger with reverse debugging capabilities
+
+### Production Debugging
+
+For errors in production environments where debuggers aren't available:
+
+**Safe Production Debugging Techniques:**
+
+1. **Enhanced Logging**: Add strategic log statements around suspected failure points
+2. **Feature Flags**: Enable verbose logging for specific users/requests
+3. **Sampling**: Log detailed context for a percentage of requests
+4. **APM Transaction Traces**: Use DataDog APM or New Relic to see detailed transaction flows
+5. **Distributed Tracing**: Leverage OpenTelemetry traces to understand cross-service interactions
+6. **Profiling**: Use continuous profilers (DataDog Profiler, Pyroscope) to identify hot spots
+7. **Heap Dumps**: Capture memory snapshots for analysis of memory leaks
+8. **Traffic Mirroring**: Replay production traffic in staging for safe investigation
+
+**Remote Debugging (Use Cautiously):**
+
+- Attach debugger to running process only in non-critical services
+- Use read-only breakpoints that don't pause execution
+- Time-box debugging sessions strictly
+- Always have rollback plan ready
+
+### Memory and Performance Debugging
+
+**Memory Leak Detection:**
+
+```javascript
+// Node.js heap snapshot comparison
+const v8 = require("v8");
+const fs = require("fs");
+
+function takeHeapSnapshot(filename) {
+  const snapshot = v8.writeHeapSnapshot(filename);
+  console.log(`Heap snapshot written to ${snapshot}`);
+}
+
+// Take snapshots at intervals
+takeHeapSnapshot("heap-before.heapsnapshot");
+// ... run operations that might leak ...
+takeHeapSnapshot("heap-after.heapsnapshot");
+
+// Analyze in Chrome DevTools Memory profiler
+// Look for objects with increasing retained size
+```
+
+**Performance Profiling:**
+
+```python
+# Python profiling with cProfile
+import cProfile
+import pstats
+from pstats import SortKey
+
+def profile_function():
+    profiler = cProfile.Profile()
+    profiler.enable()
+
+    # Your code here
+    process_large_dataset()
+
+    profiler.disable()
+
+    stats = pstats.Stats(profiler)
+    stats.sort_stats(SortKey.CUMULATIVE)
+    stats.print_stats(20)  # Top 20 time-consuming functions
+```
+
+## Error Prevention Strategies
+
+### Input Validation and Type Safety
+
+**Defensive Programming:**
+
+```typescript
+// TypeScript: Leverage type system for compile-time safety
+interface PaymentRequest {
+  amount: number;
+  currency: string;
+  customerId: string;
+  paymentMethodId: string;
+}
+
+function processPayment(request: PaymentRequest): PaymentResult {
+  // Runtime validation for external inputs
+  if (request.amount <= 0) {
+    throw new ValidationError("Amount must be positive");
+  }
+
+  if (!["USD", "EUR", "GBP"].includes(request.currency)) {
+    throw new ValidationError("Unsupported currency");
+  }
+
+  // Use Zod or Yup for complex validation
+  const schema = z.object({
+    amount: z.number().positive().max(1000000),
+    currency: z.enum(["USD", "EUR", "GBP"]),
+    customerId: z.string().uuid(),
+    paymentMethodId: z.string().min(1),
+  });
+
+  const validated = schema.parse(request);
+
+  // Now safe to process
+  return chargeCustomer(validated);
+}
+```
+
+**Python Type Hints and Validation:**
+
+```python
+from typing import Optional
+from pydantic import BaseModel, validator, Field
+from decimal import Decimal
+
+class PaymentRequest(BaseModel):
+    amount: Decimal = Field(..., gt=0, le=1000000)
+    currency: str
+    customer_id: str
+    payment_method_id: str
+
+    @validator('currency')
+    def validate_currency(cls, v):
+        if v not in ['USD', 'EUR', 'GBP']:
+            raise ValueError('Unsupported currency')
+        return v
+
+    @validator('customer_id', 'payment_method_id')
+    def validate_ids(cls, v):
+        if not v or len(v) < 1:
+            raise ValueError('ID cannot be empty')
+        return v
+
+def process_payment(request: PaymentRequest) -> PaymentResult:
+    # Pydantic validates automatically on instantiation
+    # Type hints provide IDE support and static analysis
+    return charge_customer(request)
+```
+
+### Error Boundaries and Graceful Degradation
+
+**React Error Boundaries:**
+
+```typescript
+import React, { Component, ErrorInfo, ReactNode } from 'react';
+import * as Sentry from '@sentry/react';
+
+interface Props {
+  children: ReactNode;
+  fallback?: ReactNode;
+}
+
+interface State {
+  hasError: boolean;
+  error?: Error;
+}
+
+class ErrorBoundary extends Component<Props, State> {
+  public state: State = {
+    hasError: false
+  };
+
+  public static getDerivedStateFromError(error: Error): State {
+    return { hasError: true, error };
+  }
+
+  public componentDidCatch(error: Error, errorInfo: ErrorInfo) {
+    // Log to error tracking service
+    Sentry.captureException(error, {
+      contexts: {
+        react: {
+          componentStack: errorInfo.componentStack
+        }
+      }
+    });
+
+    console.error('Uncaught error:', error, errorInfo);
+  }
+
+  public render() {
+    if (this.state.hasError) {
+      return this.props.fallback || (
+        <div role="alert">
+          <h2>Something went wrong</h2>
+          <details>
+            <summary>Error details</summary>
+            <pre>{this.state.error?.message}</pre>
+          </details>
+        </div>
+      );
+    }
+
+    return this.props.children;
+  }
+}
+
+export default ErrorBoundary;
+```
+
+**Circuit Breaker Pattern:**
+
+```python
+from datetime import datetime, timedelta
+from enum import Enum
+import time
+
+class CircuitState(Enum):
+    CLOSED = "closed"      # Normal operation
+    OPEN = "open"          # Failing, reject requests
+    HALF_OPEN = "half_open"  # Testing if service recovered
+
+class CircuitBreaker:
+    def __init__(self, failure_threshold=5, timeout=60, success_threshold=2):
+        self.failure_threshold = failure_threshold
+        self.timeout = timeout
+        self.success_threshold = success_threshold
+        self.failure_count = 0
+        self.success_count = 0
+        self.last_failure_time = None
+        self.state = CircuitState.CLOSED
+
+    def call(self, func, *args, **kwargs):
+        if self.state == CircuitState.OPEN:
+            if self._should_attempt_reset():
+                self.state = CircuitState.HALF_OPEN
+            else:
+                raise CircuitBreakerOpenError("Circuit breaker is OPEN")
+
+        try:
+            result = func(*args, **kwargs)
+            self._on_success()
+            return result
+        except Exception as e:
+            self._on_failure()
+            raise
+
+    def _on_success(self):
+        self.failure_count = 0
+        if self.state == CircuitState.HALF_OPEN:
+            self.success_count += 1
+            if self.success_count >= self.success_threshold:
+                self.state = CircuitState.CLOSED
+                self.success_count = 0
+
+    def _on_failure(self):
+        self.failure_count += 1
+        self.last_failure_time = datetime.now()
+        if self.failure_count >= self.failure_threshold:
+            self.state = CircuitState.OPEN
+
+    def _should_attempt_reset(self):
+        return (datetime.now() - self.last_failure_time) > timedelta(seconds=self.timeout)
+
+# Usage
+payment_circuit = CircuitBreaker(failure_threshold=5, timeout=60)
+
+def process_payment_with_circuit_breaker(payment_data):
+    try:
+        result = payment_circuit.call(external_payment_api.charge, payment_data)
+        return result
+    except CircuitBreakerOpenError:
+        # Graceful degradation: queue for later processing
+        payment_queue.enqueue(payment_data)
+        return {"status": "queued", "message": "Payment will be processed shortly"}
+```
+
+### Retry Logic with Exponential Backoff
+
+```typescript
+// TypeScript retry implementation
+interface RetryOptions {
+  maxAttempts: number;
+  baseDelayMs: number;
+  maxDelayMs: number;
+  exponentialBase: number;
+  retryableErrors?: string[];
+}
+
+async function retryWithBackoff<T>(
+  fn: () => Promise<T>,
+  options: RetryOptions = {
+    maxAttempts: 3,
+    baseDelayMs: 1000,
+    maxDelayMs: 30000,
+    exponentialBase: 2,
+  },
+): Promise<T> {
+  let lastError: Error;
+
+  for (let attempt = 0; attempt < options.maxAttempts; attempt++) {
+    try {
+      return await fn();
+    } catch (error) {
+      lastError = error as Error;
+
+      // Check if error is retryable
+      if (
+        options.retryableErrors &&
+        !options.retryableErrors.includes(error.name)
+      ) {
+        throw error; // Don't retry non-retryable errors
+      }
+
+      if (attempt < options.maxAttempts - 1) {
+        const delay = Math.min(
+          options.baseDelayMs * Math.pow(options.exponentialBase, attempt),
+          options.maxDelayMs,
+        );
+
+        // Add jitter to prevent thundering herd
+        const jitter = Math.random() * 0.1 * delay;
+        const actualDelay = delay + jitter;
+
+        console.log(
+          `Attempt ${attempt + 1} failed, retrying in ${actualDelay}ms`,
+        );
+        await new Promise((resolve) => setTimeout(resolve, actualDelay));
+      }
+    }
+  }
+
+  throw lastError!;
+}
+
+// Usage
+const result = await retryWithBackoff(
+  () => fetch("https://api.example.com/data"),
+  {
+    maxAttempts: 3,
+    baseDelayMs: 1000,
+    maxDelayMs: 10000,
+    exponentialBase: 2,
+    retryableErrors: ["NetworkError", "TimeoutError"],
+  },
+);
+```
+
+## Monitoring and Alerting Integration
+
+### Modern Observability Stack (2025)
+
+**Recommended Architecture:**
+
+- **Metrics**: Prometheus + Grafana or DataDog
+- **Logs**: Elasticsearch/Loki + Fluentd or DataDog Logs
+- **Traces**: OpenTelemetry + Jaeger/Tempo or DataDog APM
+- **Errors**: Sentry or DataDog Error Tracking
+- **Frontend**: Sentry Browser SDK or DataDog RUM
+- **Synthetics**: DataDog Synthetics or Checkly
+
+### Sentry Integration
+
+**Node.js/Express Setup:**
+
+```javascript
+const Sentry = require("@sentry/node");
+const { ProfilingIntegration } = require("@sentry/profiling-node");
+
+Sentry.init({
+  dsn: process.env.SENTRY_DSN,
+  environment: process.env.NODE_ENV,
+  release: process.env.GIT_COMMIT_SHA,
+
+  // Performance monitoring
+  tracesSampleRate: 0.1, // 10% of transactions
+  profilesSampleRate: 0.1,
+
+  integrations: [
+    new ProfilingIntegration(),
+    new Sentry.Integrations.Http({ tracing: true }),
+    new Sentry.Integrations.Express({ app }),
+  ],
+
+  beforeSend(event, hint) {
+    // Scrub sensitive data
+    if (event.request) {
+      delete event.request.cookies;
+      delete event.request.headers?.authorization;
+    }
+
+    // Add custom context
+    event.tags = {
+      ...event.tags,
+      region: process.env.AWS_REGION,
+      instance_id: process.env.INSTANCE_ID,
+    };
+
+    return event;
+  },
+});
+
+// Express middleware
+app.use(Sentry.Handlers.requestHandler());
+app.use(Sentry.Handlers.tracingHandler());
+
+// Routes here...
+
+// Error handler (must be last)
+app.use(Sentry.Handlers.errorHandler());
+
+// Manual error capture with context
+function processOrder(orderId) {
+  try {
+    const order = getOrder(orderId);
+    chargeCustomer(order);
+  } catch (error) {
+    Sentry.captureException(error, {
+      tags: {
+        operation: "process_order",
+        order_id: orderId,
+      },
+      contexts: {
+        order: {
+          id: orderId,
+          status: order?.status,
+          amount: order?.amount,
+        },
+      },
+      user: {
+        id: order?.customerId,
+      },
+    });
+    throw error;
+  }
+}
+```
+
+### DataDog APM Integration
+
+**Python/Flask Setup:**
+
+```python
+from ddtrace import patch_all, tracer
+from ddtrace.contrib.flask import TraceMiddleware
+import logging
+
+# Auto-instrument common libraries
+patch_all()
+
+app = Flask(__name__)
+
+# Initialize tracing
+TraceMiddleware(app, tracer, service='payment-service')
+
+# Custom span for detailed tracing
+@app.route('/api/v1/payments/charge', methods=['POST'])
+def charge_payment():
+    with tracer.trace('payment.charge', service='payment-service') as span:
+        payment_data = request.json
+
+        # Add custom tags
+        span.set_tag('payment.amount', payment_data['amount'])
+        span.set_tag('payment.currency', payment_data['currency'])
+        span.set_tag('customer.id', payment_data['customer_id'])
+
+        try:
+            result = payment_processor.charge(payment_data)
+            span.set_tag('payment.status', 'success')
+            return jsonify(result), 200
+        except InsufficientFundsError as e:
+            span.set_tag('payment.status', 'insufficient_funds')
+            span.set_tag('error', True)
+            return jsonify({'error': 'Insufficient funds'}), 402
+        except Exception as e:
+            span.set_tag('payment.status', 'error')
+            span.set_tag('error', True)
+            span.set_tag('error.message', str(e))
+            raise
+```
+
+### OpenTelemetry Implementation
+
+**Go Service with OpenTelemetry:**
+
+```go
+package main
+
+import (
+    "context"
+    "go.opentelemetry.io/otel"
+    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
+    "go.opentelemetry.io/otel/sdk/trace"
+    sdktrace "go.opentelemetry.io/otel/sdk/trace"
+    "go.opentelemetry.io/otel/attribute"
+    "go.opentelemetry.io/otel/codes"
+)
+
+func initTracer() (*sdktrace.TracerProvider, error) {
+    exporter, err := otlptracegrpc.New(
+        context.Background(),
+        otlptracegrpc.WithEndpoint("otel-collector:4317"),
+        otlptracegrpc.WithInsecure(),
+    )
+    if err != nil {
+        return nil, err
+    }
+
+    tp := sdktrace.NewTracerProvider(
+        sdktrace.WithBatcher(exporter),
+        sdktrace.WithResource(resource.NewWithAttributes(
+            semconv.SchemaURL,
+            semconv.ServiceNameKey.String("payment-service"),
+            semconv.ServiceVersionKey.String("v2.3.1"),
+            attribute.String("environment", "production"),
+        )),
+    )
+
+    otel.SetTracerProvider(tp)
+    return tp, nil
+}
+
+func processPayment(ctx context.Context, paymentReq PaymentRequest) error {
+    tracer := otel.Tracer("payment-service")
+    ctx, span := tracer.Start(ctx, "processPayment")
+    defer span.End()
+
+    // Add attributes
+    span.SetAttributes(
+        attribute.Float64("payment.amount", paymentReq.Amount),
+        attribute.String("payment.currency", paymentReq.Currency),
+        attribute.String("customer.id", paymentReq.CustomerID),
+    )
+
+    // Call downstream service
+    err := chargeCard(ctx, paymentReq)
+    if err != nil {
+        span.RecordError(err)
+        span.SetStatus(codes.Error, err.Error())
+        return err
+    }
+
+    span.SetStatus(codes.Ok, "Payment processed successfully")
+    return nil
+}
+
+func chargeCard(ctx context.Context, paymentReq PaymentRequest) error {
+    tracer := otel.Tracer("payment-service")
+    ctx, span := tracer.Start(ctx, "chargeCard")
+    defer span.End()
+
+    // Simulate external API call
+    result, err := paymentGateway.Charge(ctx, paymentReq)
+    if err != nil {
+        return fmt.Errorf("payment gateway error: %w", err)
+    }
+
+    span.SetAttributes(
+        attribute.String("transaction.id", result.TransactionID),
+        attribute.String("gateway.response_code", result.ResponseCode),
+    )
+
+    return nil
+}
+```
+
+### Alert Configuration
+
+**Intelligent Alerting Strategy:**
+
+```yaml
+# DataDog Monitor Configuration
+monitors:
+  - name: "High Error Rate - Payment Service"
+    type: metric
+    query: "avg(last_5m):sum:trace.express.request.errors{service:payment-service} / sum:trace.express.request.hits{service:payment-service} > 0.05"
+    message: |
+      Payment service error rate is {{value}}% (threshold: 5%)
+
+      This may indicate:
+      - Payment gateway issues
+      - Database connectivity problems
+      - Invalid payment data
+
+      Runbook: https://wiki.company.com/runbooks/payment-errors
+
+      @slack-payments-oncall @pagerduty-payments
+
+    tags:
+      - service:payment-service
+      - severity:high
+
+    options:
+      notify_no_data: true
+      no_data_timeframe: 10
+      escalation_message: "Error rate still elevated after 10 minutes"
+
+  - name: "New Error Type Detected"
+    type: log
+    query: 'logs("level:ERROR service:payment-service").rollup("count").by("error.fingerprint").last("5m") > 0'
+    message: |
+      New error type detected in payment service: {{error.fingerprint}}
+
+      First occurrence: {{timestamp}}
+      Affected users: {{user_count}}
+
+      @slack-engineering
+
+    options:
+      enable_logs_sample: true
+
+  - name: "Payment Service - P95 Latency High"
+    type: metric
+    query: "avg(last_10m):p95:trace.express.request.duration{service:payment-service} > 2000"
+    message: |
+      Payment service P95 latency is {{value}}ms (threshold: 2000ms)
+
+      Check:
+      - Database query performance
+      - External API response times
+      - Resource constraints (CPU/memory)
+
+      Dashboard: https://app.datadoghq.com/dashboard/payment-service
+
+      @slack-payments-team
+```
+
+## Production Incident Response
+
+### Incident Response Workflow
+
+**Phase 1: Detection and Triage (0-5 minutes)**
+
+1. Acknowledge the alert/incident
+2. Check incident severity and user impact
+3. Assign incident commander
+4. Create incident channel (#incident-2025-10-11-payment-errors)
+5. Update status page if customer-facing
+
+**Phase 2: Investigation (5-30 minutes)**
+
+1. Gather observability data:
+   - Error rates from Sentry/DataDog
+   - Traces showing failed requests
+   - Logs around the incident start time
+   - Metrics showing resource usage, latency, throughput
+2. Correlate with recent changes:
+   - Recent deployments (check CI/CD pipeline)
+   - Configuration changes
+   - Infrastructure changes
+   - External dependencies status
+3. Form initial hypothesis about root cause
+4. Document findings in incident log
+
+**Phase 3: Mitigation (Immediate)**
+
+1. Implement immediate fix based on hypothesis:
+   - Rollback recent deployment
+   - Scale up resources
+   - Disable problematic feature (feature flag)
+   - Failover to backup system
+   - Apply hotfix
+2. Verify mitigation worked (error rate decreases)
+3. Monitor for 15-30 minutes to ensure stability
+
+**Phase 4: Recovery and Validation**
+
+1. Verify all systems operational
+2. Check data consistency
+3. Process queued/failed requests
+4. Update status page: incident resolved
+5. Notify stakeholders
+
+**Phase 5: Post-Incident Review**
+
+1. Schedule postmortem within 48 hours
+2. Create detailed timeline of events
+3. Identify root cause (may differ from initial hypothesis)
+4. Document contributing factors
+5. Create action items for:
+   - Preventing similar incidents
+   - Improving detection time
+   - Improving mitigation time
+   - Improving communication
+
+### Incident Investigation Tools
+
+**Query Patterns for Common Incidents:**
+
+```
+# Find all errors for a specific time window (Elasticsearch)
+GET /logs-*/_search
+{
+  "query": {
+    "bool": {
+      "must": [
+        { "term": { "level": "ERROR" }},
+        { "term": { "service": "payment-service" }},
+        { "range": { "timestamp": {
+          "gte": "2025-10-11T14:00:00Z",
+          "lte": "2025-10-11T14:30:00Z"
+        }}}
+      ]
+    }
+  },
+  "sort": [{ "timestamp": "asc" }],
+  "size": 1000
+}
+
+# Find correlation between errors and deployments (DataDog)
+# Use deployment tracking to overlay deployment markers on error graphs
+# Query: sum:trace.express.request.errors{service:payment-service} by {version}
+
+# Identify affected users (Sentry)
+# Navigate to issue → User Impact tab
+# Shows: total users affected, new vs returning, geographic distribution
+
+# Trace specific failed request (OpenTelemetry/Jaeger)
+# Search by trace_id or correlation_id
+# Visualize full request path across services
+# Identify which service/span failed
+```
+
+### Communication Templates
+
+**Initial Incident Notification:**
+
+```
+🚨 INCIDENT: Payment Processing Errors
+
+Severity: High
+Status: Investigating
+Started: 2025-10-11 14:23 UTC
+Incident Commander: @jane.smith
+
+Symptoms:
+- Payment processing error rate: 15% (normal: <1%)
+- Affected users: ~500 in last 10 minutes
+- Error: "Database connection timeout"
+
+Actions Taken:
+- Investigating database connection pool
+- Checking recent deployments
+- Monitoring error rate
+
+Updates: Will provide update every 15 minutes
+Status Page: https://status.company.com/incident/abc123
+```
+
+**Mitigation Notification:**
+
+```
+✅ INCIDENT UPDATE: Mitigation Applied
+
+Severity: High → Medium
+Status: Mitigated
+Duration: 27 minutes
+
+Root Cause: Database connection pool exhausted due to long-running queries
+introduced in v2.3.1 deployment at 14:00 UTC
+
+Mitigation: Rolled back to v2.3.0
+
+Current Status:
+- Error rate: 0.5% (back to normal)
+- All systems operational
+- Processing backlog of queued payments
+
+Next Steps:
+- Monitor for 30 minutes
+- Fix query performance issue
+- Deploy fixed version with testing
+- Schedule postmortem
+```
+
+## Error Analysis Deliverables
+
+For each error analysis, provide:
+
+1. **Error Summary**: What happened, when, impact scope
+2. **Root Cause**: The fundamental reason the error occurred
+3. **Evidence**: Stack traces, logs, metrics supporting the diagnosis
+4. **Immediate Fix**: Code changes to resolve the issue
+5. **Testing Strategy**: How to verify the fix works
+6. **Preventive Measures**: How to prevent similar errors in the future
+7. **Monitoring Recommendations**: What to monitor/alert on going forward
+8. **Runbook**: Step-by-step guide for handling similar incidents
+
+Prioritize actionable recommendations that improve system reliability and reduce MTTR (Mean Time To Resolution) for future incidents.