blockmine 1.21.0 → 1.23.0

package/.claude/skills/observability-engineering/SKILL.md

@@ -0,0 +1,101 @@
+# Observability Engineering
+
+Deep dive into distributed tracing, APM tools, OpenTelemetry, logs aggregation, and observability best practices.
+
+## Purpose
+
+Enable teams to gain deep visibility into distributed systems with comprehensive observability practices beyond basic monitoring.
+
+## When to Use This Skill
+
+Automatically activates when working on:
+- Distributed tracing (Jaeger, Tempo, Zipkin)
+- OpenTelemetry instrumentation
+- APM tools (DataDog, New Relic, Dynatrace)
+- Logs aggregation (ELK, Loki)
+- Correlation strategies (trace IDs, unified observability)
+- Service mesh observability
+- Observability cost optimization
+
+## Quick Start Checklist
+
+- [ ] Instrument application with OpenTelemetry
+- [ ] Set up distributed tracing backend
+- [ ] Configure logs aggregation
+- [ ] Implement correlation (trace IDs across logs/metrics/traces)
+- [ ] Set up APM dashboards
+- [ ] Configure sampling strategies
+- [ ] Monitor observability costs
+- [ ] Create SLO-based alerts
+
+## Core Concepts
+
+### Three Pillars of Observability
+
+```
+Logs:    What happened (events)
+Metrics: How much (counters, gauges)
+Traces:  Where time was spent (distributed request flow)
+```
+
+### OpenTelemetry
+
+```typescript
+import { trace, context } from '@opentelemetry/api';
+
+const tracer = trace.getTracer('my-service');
+
+async function handleRequest(req, res) {
+  const span = tracer.startSpan('handleRequest');
+
+  try {
+    const user = await fetchUser(req.userId);
+    span.setAttribute('user.id', user.id);
+
+    const result = await processData(user);
+    res.json(result);
+  } finally {
+    span.end();
+  }
+}
+```
+
+## Resource Files
+
+### Tracing & APM
+- **[distributed-tracing.md](resources/distributed-tracing.md)** - Jaeger, Tempo, trace propagation, span design
+- **[opentelemetry.md](resources/opentelemetry.md)** - OTEL SDK, auto-instrumentation, collectors, exporters
+- **[apm-tools.md](resources/apm-tools.md)** - DataDog, New Relic, Dynatrace comparison
+
+### Logs & Correlation
+- **[logs-aggregation.md](resources/logs-aggregation.md)** - ELK Stack, Loki, structured logging
+- **[correlation-strategies.md](resources/correlation-strategies.md)** - Trace IDs, unified observability
+
+### Cost & Best Practices
+- **[observability-cost-optimization.md](resources/observability-cost-optimization.md)** - Sampling, retention, cost strategies
+
+## Best Practices
+
+✅ Use OpenTelemetry (vendor-neutral)
+✅ Implement structured logging
+✅ Correlate logs/metrics/traces (trace IDs)
+✅ Sample intelligently (head-based, tail-based)
+✅ Monitor observability costs
+✅ Create dashboards from traces
+✅ Use service mesh for automatic instrumentation
+
+## Integration Points
+
+- **sre**: Monitoring and alerting
+- **devsecops**: Security monitoring
+- **platform-engineering**: Service mesh observability
+- **backend-dev-guidelines**: Application instrumentation
+
+---
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+🎯 SKILL ACTIVATED: observability-engineering
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+
+**Total Resources:** 6 focused guides on observability engineering
+**Focus:** Distributed tracing, OpenTelemetry, APM, cost optimization

package/.claude/skills/observability-engineering/resources/apm-tools.md

@@ -0,0 +1,97 @@
+# APM Tools
+
+Comparison and guide to Application Performance Monitoring tools: DataDog, New Relic, Dynatrace.
+
+## DataDog
+
+```typescript
+// DataDog APM
+import tracer from 'dd-trace';
+tracer.init({
+  service: 'my-app',
+  env: 'production',
+  version: '1.0.0',
+});
+
+// Auto-instruments Express, Prisma, Redis, etc.
+```
+
+**Pros:**
+- ✅ Easy setup (auto-instrumentation)
+- ✅ Unified logs/metrics/traces
+- ✅ Great dashboards
+- ✅ APM + infrastructure monitoring
+
+**Cons:**
+- ❌ Expensive at scale
+- ❌ Vendor lock-in
+
+## New Relic
+
+```javascript
+// newrelic.js
+exports.config = {
+  app_name: ['My App'],
+  license_key: process.env.NEW_RELIC_LICENSE_KEY,
+  distributed_tracing: {
+    enabled: true,
+  },
+};
+
+// Load before app
+require('newrelic');
+const app = require('./app');
+```
+
+**Pros:**
+- ✅ Powerful query language (NRQL)
+- ✅ Good for complex architectures
+- ✅ Real-time alerting
+
+**Cons:**
+- ❌ Steeper learning curve
+- ❌ Can be expensive
+
+## Dynatrace
+
+```
+Dynatrace OneAgent (auto-deploys)
+- No code changes required
+- Full-stack monitoring
+```
+
+**Pros:**
+- ✅ AI-powered root cause analysis
+- ✅ Zero-code instrumentation
+- ✅ Enterprise features
+
+**Cons:**
+- ❌ Most expensive
+- ❌ Overkill for small teams
+
+## Cost Comparison
+
+```
+Small team (5 services, 1M spans/month):
+- DataDog: ~$500/month
+- New Relic: ~$400/month
+- Dynatrace: ~$800/month
+
+Large team (50 services, 100M spans/month):
+- DataDog: ~$5,000/month
+- New Relic: ~$4,000/month
+- Dynatrace: ~$10,000/month
+```
+
+## Open Source Alternative
+
+```
+Jaeger + Prometheus + Grafana = $0
+(but requires self-hosting and maintenance)
+```
+
+---
+
+**Related Resources:**
+- observability-cost-optimization.md - Cost management
+- opentelemetry.md - Vendor-neutral instrumentation

package/.claude/skills/observability-engineering/resources/correlation-strategies.md

@@ -0,0 +1,87 @@
+# Correlation Strategies
+
+Guide to correlating logs, metrics, and traces for unified observability.
+
+## Trace ID Propagation
+
+```typescript
+import { trace, context } from '@opentelemetry/api';
+import { logger } from './logger';
+
+app.use((req, res, next) => {
+  const span = trace.getActiveSpan();
+  const traceId = span?.spanContext().traceId;
+
+  // Attach trace ID to logger context
+  req.logger = logger.child({ traceId });
+
+  next();
+});
+
+app.get('/users/:id', async (req, res) => {
+  req.logger.info('Fetching user', { userId: req.params.id });
+  // Log includes traceId automatically
+
+  const user = await getUser(req.params.id);
+  res.json(user);
+});
+```
+
+## Unified Observability
+
+```
+Request arrives:
+1. Generate trace ID
+2. Create span (trace)
+3. Emit metrics (counter, histogram)
+4. Log with trace ID
+
+All three pillars linked by trace ID:
+- Logs: {"traceId": "abc123", "message": "User fetched"}
+- Metrics: user_requests_total{trace_id="abc123"}
+- Traces: Trace abc123 shows spans
+```
+
+## Correlation in Practice
+
+```typescript
+// 1. Start span (tracing)
+const span = tracer.startSpan('handleRequest');
+const traceId = span.spanContext().traceId;
+
+// 2. Log with trace ID (logs)
+logger.info('Processing request', { traceId, userId: req.userId });
+
+// 3. Increment metric (metrics)
+requestCounter.add(1, { traceId, endpoint: '/users' });
+
+// 4. All linked by traceId
+span.end();
+```
+
+## Querying Across Pillars
+
+```
+# Find all logs for a trace
+traceId:"abc123"
+
+# Find traces with errors
+status:error AND service:"user-service"
+
+# Find metrics for specific trace
+trace_id="abc123" AND metric="http_request_duration"
+```
+
+## Best Practices
+
+✅ Always propagate trace IDs
+✅ Include trace IDs in all logs
+✅ Use trace IDs in metric labels (when cardinality allows)
+✅ Link dashboards (click log → view trace)
+✅ Standardize field names (traceId, spanId)
+
+---
+
+**Related Resources:**
+- distributed-tracing.md - Trace propagation
+- logs-aggregation.md - Structured logging with trace IDs

package/.claude/skills/observability-engineering/resources/distributed-tracing.md

@@ -0,0 +1,98 @@
+# Distributed Tracing
+
+Guide to implementing distributed tracing with Jaeger, Tempo, and other tools.
+
+## What is Distributed Tracing?
+
+Tracks requests as they flow through microservices:
+
+```
+User Request
+  ↓
+API Gateway (span 1) ──→ 50ms
+  ↓
+User Service (span 2) ──→ 20ms
+  ↓
+Database (span 3) ──────→ 80ms
+Total: 150ms
+
+Trace shows:
+- Total request time: 150ms
+- Database was bottleneck: 80ms (53%)
+```
+
+## Jaeger Setup
+
+```yaml
+# docker-compose.yml
+version: '3'
+services:
+  jaeger:
+    image: jaegertracing/all-in-one:latest
+    ports:
+      - "16686:16686"  # UI
+      - "14268:14268"  # HTTP collector
+      - "6831:6831/udp"  # Agent
+```
+
+## Instrumentation
+
+```typescript
+import { trace, context, SpanStatusCode } from '@opentelemetry/api';
+
+const tracer = trace.getTracer('user-service');
+
+async function getUser(userId: string) {
+  const span = tracer.startSpan('getUser');
+  span.setAttribute('user.id', userId);
+
+  try {
+    const user = await db.users.findUnique({ where: { id: userId } });
+    span.setStatus({ code: SpanStatusCode.OK });
+    return user;
+  } catch (error) {
+    span.setStatus({
+      code: SpanStatusCode.ERROR,
+      message: error.message
+    });
+    span.recordException(error);
+    throw error;
+  } finally {
+    span.end();
+  }
+}
+```
+
+## Trace Propagation
+
+```typescript
+// Service A creates trace
+const span = tracer.startSpan('callServiceB');
+const headers = {};
+
+// Inject trace context into headers
+propagation.inject(context.active(), headers);
+
+// Call Service B with trace context
+await fetch('http://service-b/api', { headers });
+
+span.end();
+
+// Service B continues the trace
+const extractedContext = propagation.extract(context.active(), req.headers);
+const span = tracer.startSpan('handleRequest', {}, extractedContext);
+```
+
+## Best Practices
+
+✅ Trace critical paths
+✅ Add meaningful attributes
+✅ Sample high-volume endpoints
+✅ Keep span names consistent
+✅ Record errors in spans
+
+---
+
+**Related Resources:**
+- opentelemetry.md - OTEL implementation
+- correlation-strategies.md - Trace ID usage

package/.claude/skills/observability-engineering/resources/logs-aggregation.md

@@ -0,0 +1,118 @@
+# Logs Aggregation
+
+Guide to ELK Stack, Loki, and structured logging for centralized log management.
+
+## ELK Stack (Elasticsearch, Logstash, Kibana)
+
+```yaml
+# docker-compose.yml
+version: '3'
+services:
+  elasticsearch:
+    image: elasticsearch:8.11.0
+    environment:
+      - discovery.type=single-node
+    ports:
+      - "9200:9200"
+
+  logstash:
+    image: logstash:8.11.0
+    volumes:
+      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
+
+  kibana:
+    image: kibana:8.11.0
+    ports:
+      - "5601:5601"
+```
+
+## Structured Logging
+
+```typescript
+import winston from 'winston';
+
+const logger = winston.createLogger({
+  format: winston.format.json(),
+  transports: [
+    new winston.transports.Console(),
+  ],
+});
+
+logger.info('User logged in', {
+  userId: 'user-123',
+  ipAddress: req.ip,
+  traceId: span.spanContext().traceId,  // Correlation
+  timestamp: new Date().toISOString(),
+});
+
+// Output:
+// {"level":"info","message":"User logged in","userId":"user-123","traceId":"abc123",...}
+```
+
+## Grafana Loki
+
+```yaml
+# loki-config.yaml
+auth_enabled: false
+
+server:
+  http_listen_port: 3100
+
+ingester:
+  lifecycler:
+    ring:
+      kvstore:
+        store: inmemory
+      replication_factor: 1
+
+schema_config:
+  configs:
+    - from: 2020-10-24
+      store: boltdb-shipper
+      object_store: s3
+      schema: v11
+```
+
+```typescript
+// Ship logs to Loki
+import Pino from 'pino';
+
+const logger = Pino({
+  transport: {
+    target: 'pino-loki',
+    options: {
+      batching: true,
+      interval: 5,
+      host: 'http://loki:3100',
+      labels: {
+        service: 'my-app',
+        env: 'production',
+      },
+    },
+  },
+});
+```
+
+## Log Levels
+
+```typescript
+logger.error('Payment failed', { orderId: '123', error: err.message });
+logger.warn('High memory usage', { memoryMB: 850 });
+logger.info('User registered', { userId: 'user-456' });
+logger.debug('Cache miss', { key: 'user:123' });
+```
+
+## Best Practices
+
+✅ Use structured logging (JSON)
+✅ Include trace IDs for correlation
+✅ Log at appropriate levels
+✅ Avoid logging sensitive data (PII, passwords)
+✅ Set retention policies (30-90 days)
+✅ Index important fields
+
+---
+
+**Related Resources:**
+- correlation-strategies.md - Linking logs with traces
+- observability-cost-optimization.md - Log retention costs

package/.claude/skills/observability-engineering/resources/observability-cost-optimization.md

@@ -0,0 +1,141 @@
+# Observability Cost Optimization
+
+Guide to managing observability costs through sampling, retention policies, and efficient practices.
+
+## Sampling Strategies
+
+### Head-Based Sampling
+```typescript
+// Sample 10% of traces
+const sampler = new TraceIdRatioBasedSampler(0.1);
+
+const sdk = new NodeSDK({
+  sampler,
+  // ...
+});
+
+// Pros: Simple, predictable cost
+// Cons: May miss rare errors
+```
+
+### Tail-Based Sampling
+```typescript
+// Sample based on outcome (keep all errors, 1% of success)
+const tailSampler = {
+  shouldSample: (span) => {
+    if (span.status.code === SpanStatusCode.ERROR) {
+      return true;  // Keep all errors
+    }
+    if (span.attributes['http.status_code'] >= 500) {
+      return true;  // Keep server errors
+    }
+    return Math.random() < 0.01;  // 1% of successful requests
+  },
+};
+
+// Pros: Keeps important traces, lower cost
+// Cons: More complex
+```
+
+## Retention Policies
+
+```
+Hot storage (recent, fast):
+- Traces: 7 days
+- Logs: 14 days
+- Metrics: 30 days
+
+Warm storage (archived, slower):
+- Traces: 30 days
+- Logs: 90 days
+- Metrics: 1 year
+
+Cold storage (compliance):
+- Logs: 7 years (compressed)
+```
+
+## Cost Breakdown
+
+```
+Typical observability costs (10 services, 1M RPM):
+
+DataDog:
+- APM: $2,000/month (100M spans)
+- Logs: $1,500/month (500 GB)
+- Metrics: $500/month
+- Total: $4,000/month
+
+Self-Hosted (Jaeger + Loki + Prometheus):
+- Infrastructure: $800/month
+- Maintenance: $3,000/month (0.3 engineer)
+- Total: $3,800/month
+
+Savings with optimization:
+- 90% sampling: $400 APM (vs $2,000)
+- 30-day retention: $500 logs (vs $1,500)
+- Total: $1,400/month (65% reduction)
+```
+
+## Optimization Strategies
+
+### 1. Sample Aggressively
+```
+High-volume endpoints: 1% sample rate
+Low-volume endpoints: 100% sample rate
+Errors: Always keep
+```
+
+### 2. Reduce Log Volume
+```typescript
+// Don't log every request
+if (req.path !== '/health' && req.path !== '/metrics') {
+  logger.info('Request', { method: req.method, path: req.path });
+}
+
+// Sample logs
+if (Math.random() < 0.1) {  // 10% of logs
+  logger.debug('Debug info', { ...details });
+}
+```
+
+### 3. Optimize Metric Cardinality
+```typescript
+// ❌ Bad: Unbounded cardinality
+requestCounter.add(1, { userId: req.userId });  // Millions of users
+
+// ✅ Good: Bounded cardinality
+requestCounter.add(1, { endpoint: req.route });  // ~100 endpoints
+```
+
+### 4. Use Cheaper Storage
+```
+S3 Glacier for old logs: $0.004/GB (vs $0.023/GB Standard)
+83% cheaper for infrequently accessed data
+```
+
+## Monitoring Observability Costs
+
+```typescript
+// Track observability data volume
+observabilityCostGauge.set(dailySpanCount * 0.0001);  // $0.0001 per span
+
+// Alert on cost spikes
+if (dailyObservabilityCost > threshold) {
+  alert('Observability costs exceeded budget');
+}
+```
+
+## Best Practices
+
+✅ Sample high-volume endpoints
+✅ Always keep errors
+✅ Set aggressive retention policies
+✅ Reduce metric cardinality
+✅ Use cheaper storage tiers
+✅ Monitor observability costs
+
+---
+
+**Related Resources:**
+- distributed-tracing.md - Sampling implementation
+- logs-aggregation.md - Log retention