cfsa-antigravity 1.0.0

package/template/.agent/skill-library/stack/observability/distributed-tracing/SKILL.md

@@ -0,0 +1,140 @@
+---
+name: distributed-tracing
+description: |
+  Implement distributed tracing across microservices with Jaeger, Zipkin, and W3C TraceContext. Use when: setting up trace propagation, configuring Jaeger or Zipkin, understanding span relationships, choosing sampling strategies, or correlating traces with logs and metrics.
+version: 1.0.0
+---
+
+# Distributed Tracing
+
+**Status**: Production Ready
+**Last Updated**: 2026-02-24
+
+---
+
+## Description
+
+Distributed tracing tracks requests as they flow through multiple services, providing end-to-end visibility into latency, errors, and dependencies. It is essential for debugging and performance optimization in microservice architectures.
+
+## When to Use
+
+- Setting up trace propagation (W3C TraceContext, B3)
+- Configuring Jaeger for trace collection and visualization
+- Configuring Zipkin for trace collection and visualization
+- Understanding span relationships (parent/child, follows-from)
+- Choosing between head-based and tail-based sampling
+- Planning trace storage and retention policies
+- Correlating traces with logs and metrics
+
+---
+
+## Key Concepts
+
+### Trace Propagation Formats
+
+| Format | Standard | Use Case |
+|--------|----------|----------|
+| **W3C TraceContext** | W3C Recommendation | Default for OpenTelemetry, broadly supported |
+| **B3** | Zipkin | Legacy Zipkin ecosystems, broad language support |
+| **Jaeger** | Jaeger-specific | Jaeger-native environments (being deprecated in favor of W3C) |
+
+### Span Relationships
+
+| Relationship | Meaning |
+|---|---|
+| **ChildOf** | The parent span depends on the child span's result |
+| **FollowsFrom** | The parent span does not depend on the child (fire-and-forget) |
+
+### Sampling Strategies
+
+| Strategy | Description | Trade-off |
+|----------|-------------|-----------|
+| **Head-based** | Decision made at trace start | Simple, but may miss interesting traces |
+| **Tail-based** | Decision made after trace completes | Captures errors/slow traces, but requires buffering |
+| **Adaptive** | Adjusts rate based on traffic volume | Balances cost and coverage |
+
+---
+
+## Best Practices
+
+1. **Use W3C TraceContext** — it is the industry standard and supported by all major tracing backends
+2. **Propagate context through all transport layers** — HTTP headers, message queue metadata, gRPC metadata
+3. **Set span names to describe the operation** — `POST /api/orders` not `handler`
+4. **Record errors on spans** — set span status to ERROR and call `recordException`
+5. **Use tail-based sampling for production** — capture all error and high-latency traces while sampling normal traffic
+6. **Correlate traces with logs** — inject `trace_id` and `span_id` into log records for cross-signal navigation
+7. **Set retention policies** — keep error traces longer than normal traces to support incident investigation
+8. **Monitor trace pipeline health** — alert on dropped spans, exporter failures, and collector queue depth
+
+---
+
+## Common Patterns
+
+### Jaeger Setup (Docker Compose)
+
+```yaml
+services:
+  jaeger:
+    image: jaegertracing/all-in-one:latest
+    ports:
+      - "16686:16686"   # Jaeger UI
+      - "4317:4317"     # OTLP gRPC
+      - "4318:4318"     # OTLP HTTP
+    environment:
+      COLLECTOR_OTLP_ENABLED: "true"
+```
+
+### Zipkin Setup (Docker Compose)
+
+```yaml
+services:
+  zipkin:
+    image: openzipkin/zipkin:latest
+    ports:
+      - "9411:9411"     # Zipkin UI + API
+```
+
+### W3C TraceContext Header
+
+```
+traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
+```
+
+Format: `version-traceId-spanId-traceFlags`
+
+### Correlating Traces with Logs
+
+```typescript
+import { trace, context } from "@opentelemetry/api";
+
+function getTraceContext() {
+  const span = trace.getActiveSpan();
+  if (!span) return {};
+  const ctx = span.spanContext();
+  return {
+    trace_id: ctx.traceId,
+    span_id: ctx.spanId,
+    trace_flags: ctx.traceFlags,
+  };
+}
+
+// Inject into structured logs
+logger.info({ ...getTraceContext(), message: "Order processed", orderId });
+```
+
+---
+
+## Anti-Patterns
+
+| Anti-Pattern | Why It Hurts | Correct Approach |
+|---|---|---|
+| Not propagating context across async boundaries | Broken traces, orphan spans | Use context-aware async utilities |
+| Sampling at 100% in production | Storage costs explode, performance impact | Use 5–20% head-based or tail-based sampling |
+| Generic span names | Cannot filter or search effectively | Use descriptive, route-based span names |
+| Ignoring span status on errors | Errors invisible in trace UI | Always set ERROR status and record exception |
+| No trace-log correlation | Cannot jump from trace to logs | Inject trace_id into all structured logs |
+| Unbounded trace retention | Storage grows without limit | Set retention policies (7–30 days typical) |
+
+---
+
+**Last verified**: 2026-02-24 | **Skill version**: 1.0.0

package/template/.agent/skill-library/stack/observability/logging-best-practices/SKILL.md

@@ -0,0 +1,168 @@
+---
+name: logging-best-practices
+description: |
+  Implement structured logging with JSON output, log levels, correlation IDs, and log aggregation. Use when: setting up Pino or Winston, configuring log levels (trace/debug/info/warn/error/fatal), adding request tracing, implementing sensitive data redaction, or integrating with ELK or Loki.
+version: 1.0.0
+---
+
+# Logging Best Practices
+
+**Status**: Production Ready
+**Last Updated**: 2026-02-24
+
+---
+
+## Description
+
+Production-grade logging requires structured output, consistent log levels, request correlation, and PII redaction. This skill covers best practices for Node.js logging with Pino and Winston, log aggregation with ELK and Loki, and performance-conscious logging patterns.
+
+## When to Use
+
+- Setting up JSON structured logging
+- Choosing and configuring log levels (trace/debug/info/warn/error/fatal)
+- Adding correlation IDs and request tracing
+- Configuring Pino for high-performance logging
+- Configuring Winston for flexible logging
+- Integrating with log aggregation (ELK stack, Grafana Loki)
+- Implementing sensitive data redaction
+- Optimizing async logging and log sampling for performance
+
+---
+
+## Key Concepts
+
+### Log Levels
+
+| Level | When to Use |
+|-------|-------------|
+| **trace** | Extremely detailed diagnostic info (function entry/exit) |
+| **debug** | Diagnostic info useful during development |
+| **info** | Normal operational events (request completed, job finished) |
+| **warn** | Unexpected but recoverable situations (deprecation, retry) |
+| **error** | Failures that need attention (unhandled exception, external service down) |
+| **fatal** | Process cannot continue (missing critical config, OOM) |
+
+### Structured Logging
+
+Always log as structured JSON objects, never string-interpolated messages:
+
+```typescript
+// Good — structured, searchable, parseable
+logger.info({ orderId, userId, total: 49.99 }, "Order placed");
+
+// Bad — string interpolation, not searchable
+logger.info(`Order ${orderId} placed by user ${userId} for $49.99`);
+```
+
+### Correlation IDs
+
+Every request should carry a unique correlation ID (often called `requestId` or `traceId`) that appears in all log entries for that request. This enables tracing a single request across all log lines.
+
+---
+
+## Best Practices
+
+1. **Use structured JSON output** — every log entry is a JSON object with consistent fields
+2. **Set log level via environment variable** — `LOG_LEVEL=info` in production, `LOG_LEVEL=debug` in development
+3. **Include correlation IDs** — generate a UUID per request and inject it into every log entry
+4. **Redact sensitive data** — never log passwords, tokens, PII, or full credit card numbers
+5. **Log at the right level** — `info` for business events, `error` for failures, `debug` for diagnostics
+6. **Use child loggers** — create child loggers with request-scoped context to avoid passing IDs everywhere
+7. **Avoid logging in hot loops** — log sampling or rate-limiting prevents performance degradation
+8. **Set up log rotation** — prevent disk exhaustion with size or time-based rotation
+
+---
+
+## Common Patterns
+
+### Pino Setup
+
+```typescript
+import pino from "pino";
+
+const logger = pino({
+  level: process.env.LOG_LEVEL || "info",
+  formatters: {
+    level(label) {
+      return { level: label };
+    },
+  },
+  redact: {
+    paths: ["req.headers.authorization", "req.headers.cookie", "body.password"],
+    censor: "[REDACTED]",
+  },
+  transport:
+    process.env.NODE_ENV !== "production"
+      ? { target: "pino-pretty", options: { colorize: true } }
+      : undefined,
+});
+
+export default logger;
+```
+
+### Winston Setup
+
+```typescript
+import winston from "winston";
+
+const logger = winston.createLogger({
+  level: process.env.LOG_LEVEL || "info",
+  format: winston.format.combine(
+    winston.format.timestamp(),
+    winston.format.errors({ stack: true }),
+    winston.format.json()
+  ),
+  defaultMeta: { service: "my-service" },
+  transports: [new winston.transports.Console()],
+});
+
+export default logger;
+```
+
+### Request Correlation Middleware
+
+```typescript
+import { randomUUID } from "node:crypto";
+
+function correlationMiddleware(req, res, next) {
+  const requestId = req.headers["x-request-id"] || randomUUID();
+  req.log = logger.child({ requestId });
+  res.setHeader("x-request-id", requestId);
+  next();
+}
+```
+
+### Sensitive Data Redaction
+
+```typescript
+const redactPaths = [
+  "password",
+  "token",
+  "authorization",
+  "cookie",
+  "ssn",
+  "creditCard",
+  "*.password",
+  "*.token",
+];
+
+const logger = pino({ redact: { paths: redactPaths, censor: "[REDACTED]" } });
+```
+
+---
+
+## Anti-Patterns
+
+| Anti-Pattern | Why It Hurts | Correct Approach |
+|---|---|---|
+| `console.log` in production | No structure, no levels, no redaction | Use a structured logger (Pino, Winston) |
+| Logging full request/response bodies | PII leaks, huge log volume | Log only essential fields, redact sensitive data |
+| String-interpolated messages | Not searchable, not parseable | Use structured key-value pairs |
+| No correlation IDs | Cannot trace a request across log lines | Generate and propagate requestId |
+| `debug` level in production | Enormous log volume, cost explosion | Use `info` in production, configure via env var |
+| Synchronous file writes | Blocks event loop, degrades performance | Use async transports or stdout piping |
+| No log rotation | Disk fills up, service crashes | Use logrotate or size-based rotation |
+
+---
+
+**Last verified**: 2026-02-24 | **Skill version**: 1.0.0

package/template/.agent/skill-library/stack/observability/opentelemetry/SKILL.md

@@ -0,0 +1,164 @@
+---
+name: opentelemetry
+description: |
+  Set up OpenTelemetry for distributed tracing, metrics, and structured logging with trace correlation. Use when: configuring OpenTelemetry SDK for Node.js/Python/Go/Rust, setting up OTLP exporters, integrating with Jaeger or Zipkin, implementing auto or manual instrumentation, context propagation, or choosing sampling strategies.
+version: 1.0.0
+---
+
+# OpenTelemetry
+
+**Status**: Production Ready
+**Last Updated**: 2026-02-24
+
+---
+
+## Description
+
+OpenTelemetry (OTel) is the vendor-neutral observability framework for generating, collecting, and exporting telemetry data — traces, metrics, and logs. It provides a single set of APIs, SDKs, and tooling across languages.
+
+## When to Use
+
+- Setting up distributed tracing with spans and traces
+- Collecting application metrics (counters, histograms, gauges)
+- Implementing structured logging with trace correlation
+- Configuring the OTel SDK for Node.js, Python, Go, or Rust
+- Setting up exporters (OTLP, Jaeger, Zipkin)
+- Deciding between auto and manual instrumentation
+- Implementing context propagation across service boundaries
+- Choosing head-based vs tail-based sampling strategies
+
+---
+
+## Key Concepts
+
+### Signals
+
+| Signal | Purpose | Examples |
+|--------|---------|----------|
+| **Traces** | Request flow across services | HTTP requests, database queries, RPC calls |
+| **Metrics** | Quantitative measurements over time | Request count, latency histogram, CPU gauge |
+| **Logs** | Discrete event records with trace context | Structured JSON logs correlated to spans |
+
+### Core Components
+
+- **API** — Stable, vendor-neutral interface for instrumentation
+- **SDK** — Configurable implementation of the API (sampling, export, processing)
+- **Exporters** — Send telemetry to backends (OTLP, Jaeger, Zipkin, Prometheus)
+- **Collector** — Vendor-agnostic proxy for receiving, processing, and exporting telemetry
+- **Instrumentation Libraries** — Auto-instrument common frameworks and libraries
+
+### Context Propagation
+
+- W3C TraceContext (default, recommended)
+- B3 (Zipkin compatibility)
+- Baggage for cross-service key-value pairs
+
+### Sampling Strategies
+
+| Strategy | When to Use |
+|----------|-------------|
+| **AlwaysOn** | Development, low-traffic services |
+| **AlwaysOff** | Disable tracing entirely |
+| **TraceIdRatioBased** | Production — sample a percentage of traces |
+| **ParentBased** | Respect upstream sampling decisions |
+| **Tail-based (Collector)** | Sample based on completed trace attributes (errors, latency) |
+
+---
+
+## Best Practices
+
+1. **Start with auto-instrumentation** — instrument frameworks and libraries automatically, add manual spans only where needed
+2. **Use OTLP as the export protocol** — it is the native OTel protocol and supports all three signals
+3. **Propagate context** — ensure W3C TraceContext headers flow across all service boundaries
+4. **Set meaningful span names** — use `HTTP GET /api/users` not `span-1`
+5. **Add semantic attributes** — follow OpenTelemetry semantic conventions for consistent metadata
+6. **Sample wisely in production** — 5–20% trace sampling is typical; keep error sampling at 100%
+7. **Use the Collector** — deploy the OTel Collector as a sidecar or gateway to decouple instrumentation from export
+8. **Correlate logs with traces** — inject `trace_id` and `span_id` into structured log records
+
+---
+
+## Common Patterns
+
+### Node.js SDK Setup
+
+```typescript
+import { NodeSDK } from "@opentelemetry/sdk-node";
+import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
+import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
+import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
+import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
+
+const sdk = new NodeSDK({
+  serviceName: "my-service",
+  traceExporter: new OTLPTraceExporter({ url: "http://localhost:4318/v1/traces" }),
+  metricReader: new PeriodicExportingMetricReader({
+    exporter: new OTLPMetricExporter({ url: "http://localhost:4318/v1/metrics" }),
+  }),
+  instrumentations: [getNodeAutoInstrumentations()],
+});
+
+sdk.start();
+```
+
+### Custom Span
+
+```typescript
+import { trace } from "@opentelemetry/api";
+
+const tracer = trace.getTracer("my-service");
+
+async function processOrder(orderId: string) {
+  return tracer.startActiveSpan("process-order", async (span) => {
+    span.setAttribute("order.id", orderId);
+    try {
+      const result = await doWork(orderId);
+      span.setStatus({ code: 1 }); // OK
+      return result;
+    } catch (error) {
+      span.setStatus({ code: 2, message: String(error) }); // ERROR
+      span.recordException(error as Error);
+      throw error;
+    } finally {
+      span.end();
+    }
+  });
+}
+```
+
+### Custom Metrics
+
+```typescript
+import { metrics } from "@opentelemetry/api";
+
+const meter = metrics.getMeter("my-service");
+
+const requestCounter = meter.createCounter("http.requests.total", {
+  description: "Total HTTP requests",
+});
+
+const latencyHistogram = meter.createHistogram("http.request.duration_ms", {
+  description: "HTTP request latency in milliseconds",
+});
+
+// Usage
+requestCounter.add(1, { method: "GET", route: "/api/users" });
+latencyHistogram.record(42, { method: "GET", route: "/api/users" });
+```
+
+---
+
+## Anti-Patterns
+
+| Anti-Pattern | Why It Hurts | Correct Approach |
+|---|---|---|
+| No sampling in production | Enormous data volume, high cost | Use ratio-based or tail-based sampling |
+| Ignoring context propagation | Broken traces across services | Ensure W3C headers propagate everywhere |
+| Too many custom spans | Noisy traces, performance overhead | Instrument meaningful operations only |
+| Hardcoded exporter URLs | Inflexible deployments | Use environment variables (`OTEL_EXPORTER_OTLP_ENDPOINT`) |
+| Skipping semantic conventions | Inconsistent attribute names | Follow OTel semantic conventions |
+| Not ending spans | Memory leaks, orphan spans | Always call `span.end()` in a `finally` block |
+
+---
+
+**Last verified**: 2026-02-24 | **Skill version**: 1.0.0