cfsa-antigravity 3.0.2 → 3.1.0

package/template/.codex/skill-library/stack/observability/datadog/SKILL.md

@@ -0,0 +1,179 @@
+---
+name: datadog
+description: |
+  Set up Datadog for APM, log management, metrics, dashboards, and alerting. Use when: configuring Datadog APM and trace collection, setting up log parsing and management, creating custom metrics, building dashboards and monitors, configuring alerting and SLOs, or integrating dd-trace libraries.
+version: 1.0.0
+---
+
+# Datadog
+
+**Status**: Production Ready
+**Last Updated**: 2026-02-24
+
+---
+
+## Description
+
+Datadog is a full-stack observability platform providing APM (distributed tracing), log management, infrastructure monitoring, metrics, dashboards, and alerting. This skill covers agent configuration, dd-trace library integration, log correlation with traces, and best practices for dashboards and SLOs.
+
+## When to Use
+
+- Setting up Datadog APM and trace collection
+- Configuring log management and log parsing
+- Creating custom metrics and dashboards
+- Building monitors and configuring alerting
+- Defining SLOs (Service Level Objectives)
+- Setting up infrastructure monitoring
+- Configuring the Datadog Agent
+- Integrating dd-trace libraries (Node.js, Python, Go, Ruby, Java)
+- Correlating logs with traces
+
+---
+
+## Key Concepts
+
+### Datadog Agent
+
+The Datadog Agent runs on your hosts or as a sidecar container. It collects metrics, traces, and logs and forwards them to the Datadog backend.
+
+### APM (Application Performance Monitoring)
+
+dd-trace libraries auto-instrument your application to capture distributed traces, including HTTP requests, database queries, cache operations, and message queue interactions.
+
+### Log Management
+
+Datadog ingests, indexes, and archives logs. Logs can be enriched with trace IDs for cross-signal correlation. Pipelines parse unstructured logs into structured fields.
+
+### Metrics
+
+Datadog supports custom metrics (counters, gauges, histograms, distributions) alongside infrastructure metrics. Metrics power dashboards, monitors, and SLOs.
+
+### Monitors and SLOs
+
+Monitors alert on threshold breaches, anomalies, or missing data. SLOs track service reliability over time windows (e.g., 99.9% availability over 30 days).
+
+---
+
+## Best Practices
+
+1. **Use unified tagging** — apply consistent tags (`service`, `env`, `version`) across traces, logs, and metrics
+2. **Enable log-trace correlation** — inject `dd.trace_id` and `dd.span_id` into log records
+3. **Set appropriate sampling rates** — use Datadog's Ingestion Controls to manage trace volume and costs
+4. **Build SLOs from meaningful SLIs** — measure what users experience (latency, error rate), not internal metrics
+5. **Use dashboards for context, monitors for alerts** — dashboards for exploration, monitors for automated detection
+6. **Tag custom metrics meaningfully** — avoid high-cardinality tags that explode metric volume
+7. **Archive logs for compliance** — use Datadog Log Archives to S3/GCS for long-term retention at lower cost
+8. **Configure the Agent via environment variables** — `DD_ENV`, `DD_SERVICE`, `DD_VERSION`, `DD_AGENT_HOST`
+
+---
+
+## Common Patterns
+
+### Node.js dd-trace Setup
+
+```typescript
+// instrument.ts — import at very top of entry point
+import tracer from "dd-trace";
+
+tracer.init({
+  service: "my-service",
+  env: process.env.DD_ENV || "development",
+  version: process.env.DD_VERSION || "1.0.0",
+  logInjection: true, // Inject trace IDs into logs
+  runtimeMetrics: true,
+  profiling: true,
+});
+
+export default tracer;
+```
+
+### Python dd-trace Setup
+
+```python
+from ddtrace import tracer, patch_all
+
+patch_all()  # Auto-instrument all supported libraries
+tracer.configure(
+    hostname=os.environ.get("DD_AGENT_HOST", "localhost"),
+    port=8126,
+)
+```
+
+### Log-Trace Correlation (Node.js + Pino)
+
+```typescript
+import pino from "pino";
+import tracer from "dd-trace";
+
+const logger = pino({
+  mixin() {
+    const span = tracer.scope().active();
+    if (!span) return {};
+    return {
+      dd: {
+        trace_id: span.context().toTraceId(),
+        span_id: span.context().toSpanId(),
+        service: "my-service",
+        env: process.env.DD_ENV,
+        version: process.env.DD_VERSION,
+      },
+    };
+  },
+});
+```
+
+### Custom Metrics
+
+```typescript
+import { StatsD } from "hot-shots";
+
+const dogstatsd = new StatsD({
+  host: process.env.DD_AGENT_HOST || "localhost",
+  port: 8125,
+  prefix: "myapp.",
+  globalTags: { env: process.env.DD_ENV, service: "my-service" },
+});
+
+// Usage
+dogstatsd.increment("orders.placed", 1, { payment_method: "card" });
+dogstatsd.histogram("order.processing_time_ms", 142, { status: "success" });
+dogstatsd.gauge("queue.depth", 47);
+```
+
+### Datadog Agent (Docker Compose)
+
+```yaml
+services:
+  datadog-agent:
+    image: gcr.io/datadoghq/agent:latest
+    environment:
+      DD_API_KEY: ${DD_API_KEY}
+      DD_SITE: "datadoghq.com"
+      DD_APM_ENABLED: "true"
+      DD_LOGS_ENABLED: "true"
+      DD_APM_NON_LOCAL_TRAFFIC: "true"
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /proc/:/host/proc/:ro
+      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
+    ports:
+      - "8126:8126"  # APM traces
+      - "8125:8125/udp"  # DogStatsD
+```
+
+---
+
+## Anti-Patterns
+
+| Anti-Pattern | Why It Hurts | Correct Approach |
+|---|---|---|
+| No unified tagging | Cannot correlate traces, logs, and metrics | Use `service`, `env`, `version` everywhere |
+| High-cardinality custom metric tags | Metric volume explodes, costs skyrocket | Use bounded tag values (status codes, not user IDs) |
+| Alerting on every error | Alert fatigue, team ignores alerts | Alert on error rate spikes, anomalies, SLO burns |
+| No log-trace correlation | Cannot navigate from logs to traces | Enable `logInjection` and standardize log format |
+| Ingesting all traces at 100% | Expensive, unnecessary for most services | Use Ingestion Controls and sampling rules |
+| Missing `DD_VERSION` tag | Cannot track regressions across deploys | Set version from CI/CD build number |
+
+---
+
+**Last verified**: 2026-02-24 | **Skill version**: 1.0.0

package/template/.codex/skill-library/stack/observability/distributed-tracing/SKILL.md

@@ -0,0 +1,140 @@
+---
+name: distributed-tracing
+description: |
+  Implement distributed tracing across microservices with Jaeger, Zipkin, and W3C TraceContext. Use when: setting up trace propagation, configuring Jaeger or Zipkin, understanding span relationships, choosing sampling strategies, or correlating traces with logs and metrics.
+version: 1.0.0
+---
+
+# Distributed Tracing
+
+**Status**: Production Ready
+**Last Updated**: 2026-02-24
+
+---
+
+## Description
+
+Distributed tracing tracks requests as they flow through multiple services, providing end-to-end visibility into latency, errors, and dependencies. It is essential for debugging and performance optimization in microservice architectures.
+
+## When to Use
+
+- Setting up trace propagation (W3C TraceContext, B3)
+- Configuring Jaeger for trace collection and visualization
+- Configuring Zipkin for trace collection and visualization
+- Understanding span relationships (parent/child, follows-from)
+- Choosing between head-based and tail-based sampling
+- Planning trace storage and retention policies
+- Correlating traces with logs and metrics
+
+---
+
+## Key Concepts
+
+### Trace Propagation Formats
+
+| Format | Standard | Use Case |
+|--------|----------|----------|
+| **W3C TraceContext** | W3C Recommendation | Default for OpenTelemetry, broadly supported |
+| **B3** | Zipkin | Legacy Zipkin ecosystems, broad language support |
+| **Jaeger** | Jaeger-specific | Jaeger-native environments (being deprecated in favor of W3C) |
+
+### Span Relationships
+
+| Relationship | Meaning |
+|---|---|
+| **ChildOf** | The parent span depends on the child span's result |
+| **FollowsFrom** | The parent span does not depend on the child (fire-and-forget) |
+
+### Sampling Strategies
+
+| Strategy | Description | Trade-off |
+|----------|-------------|-----------|
+| **Head-based** | Decision made at trace start | Simple, but may miss interesting traces |
+| **Tail-based** | Decision made after trace completes | Captures errors/slow traces, but requires buffering |
+| **Adaptive** | Adjusts rate based on traffic volume | Balances cost and coverage |
+
+---
+
+## Best Practices
+
+1. **Use W3C TraceContext** — it is the industry standard and supported by all major tracing backends
+2. **Propagate context through all transport layers** — HTTP headers, message queue metadata, gRPC metadata
+3. **Set span names to describe the operation** — `POST /api/orders` not `handler`
+4. **Record errors on spans** — set span status to ERROR and call `recordException`
+5. **Use tail-based sampling for production** — capture all error and high-latency traces while sampling normal traffic
+6. **Correlate traces with logs** — inject `trace_id` and `span_id` into log records for cross-signal navigation
+7. **Set retention policies** — keep error traces longer than normal traces to support incident investigation
+8. **Monitor trace pipeline health** — alert on dropped spans, exporter failures, and collector queue depth
+
+---
+
+## Common Patterns
+
+### Jaeger Setup (Docker Compose)
+
+```yaml
+services:
+  jaeger:
+    image: jaegertracing/all-in-one:latest
+    ports:
+      - "16686:16686"   # Jaeger UI
+      - "4317:4317"     # OTLP gRPC
+      - "4318:4318"     # OTLP HTTP
+    environment:
+      COLLECTOR_OTLP_ENABLED: "true"
+```
+
+### Zipkin Setup (Docker Compose)
+
+```yaml
+services:
+  zipkin:
+    image: openzipkin/zipkin:latest
+    ports:
+      - "9411:9411"     # Zipkin UI + API
+```
+
+### W3C TraceContext Header
+
+```
+traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
+```
+
+Format: `version-traceId-spanId-traceFlags`
+
+### Correlating Traces with Logs
+
+```typescript
+import { trace, context } from "@opentelemetry/api";
+
+function getTraceContext() {
+  const span = trace.getActiveSpan();
+  if (!span) return {};
+  const ctx = span.spanContext();
+  return {
+    trace_id: ctx.traceId,
+    span_id: ctx.spanId,
+    trace_flags: ctx.traceFlags,
+  };
+}
+
+// Inject into structured logs
+logger.info({ ...getTraceContext(), message: "Order processed", orderId });
+```
+
+---
+
+## Anti-Patterns
+
+| Anti-Pattern | Why It Hurts | Correct Approach |
+|---|---|---|
+| Not propagating context across async boundaries | Broken traces, orphan spans | Use context-aware async utilities |
+| Sampling at 100% in production | Storage costs explode, performance impact | Use 5–20% head-based or tail-based sampling |
+| Generic span names | Cannot filter or search effectively | Use descriptive, route-based span names |
+| Ignoring span status on errors | Errors invisible in trace UI | Always set ERROR status and record exception |
+| No trace-log correlation | Cannot jump from trace to logs | Inject trace_id into all structured logs |
+| Unbounded trace retention | Storage grows without limit | Set retention policies (7–30 days typical) |
+
+---
+
+**Last verified**: 2026-02-24 | **Skill version**: 1.0.0

package/template/.codex/skill-library/stack/observability/opentelemetry/SKILL.md

@@ -0,0 +1,164 @@
+---
+name: opentelemetry
+description: |
+  Set up OpenTelemetry for distributed tracing, metrics, and structured logging with trace correlation. Use when: configuring OpenTelemetry SDK for Node.js/Python/Go/Rust, setting up OTLP exporters, integrating with Jaeger or Zipkin, implementing auto or manual instrumentation, context propagation, or choosing sampling strategies.
+version: 1.0.0
+---
+
+# OpenTelemetry
+
+**Status**: Production Ready
+**Last Updated**: 2026-02-24
+
+---
+
+## Description
+
+OpenTelemetry (OTel) is the vendor-neutral observability framework for generating, collecting, and exporting telemetry data — traces, metrics, and logs. It provides a single set of APIs, SDKs, and tooling across languages.
+
+## When to Use
+
+- Setting up distributed tracing with spans and traces
+- Collecting application metrics (counters, histograms, gauges)
+- Implementing structured logging with trace correlation
+- Configuring the OTel SDK for Node.js, Python, Go, or Rust
+- Setting up exporters (OTLP, Jaeger, Zipkin)
+- Deciding between auto and manual instrumentation
+- Implementing context propagation across service boundaries
+- Choosing head-based vs tail-based sampling strategies
+
+---
+
+## Key Concepts
+
+### Signals
+
+| Signal | Purpose | Examples |
+|--------|---------|----------|
+| **Traces** | Request flow across services | HTTP requests, database queries, RPC calls |
+| **Metrics** | Quantitative measurements over time | Request count, latency histogram, CPU gauge |
+| **Logs** | Discrete event records with trace context | Structured JSON logs correlated to spans |
+
+### Core Components
+
+- **API** — Stable, vendor-neutral interface for instrumentation
+- **SDK** — Configurable implementation of the API (sampling, export, processing)
+- **Exporters** — Send telemetry to backends (OTLP, Jaeger, Zipkin, Prometheus)
+- **Collector** — Vendor-agnostic proxy for receiving, processing, and exporting telemetry
+- **Instrumentation Libraries** — Auto-instrument common frameworks and libraries
+
+### Context Propagation
+
+- W3C TraceContext (default, recommended)
+- B3 (Zipkin compatibility)
+- Baggage for cross-service key-value pairs
+
+### Sampling Strategies
+
+| Strategy | When to Use |
+|----------|-------------|
+| **AlwaysOn** | Development, low-traffic services |
+| **AlwaysOff** | Disable tracing entirely |
+| **TraceIdRatioBased** | Production — sample a percentage of traces |
+| **ParentBased** | Respect upstream sampling decisions |
+| **Tail-based (Collector)** | Sample based on completed trace attributes (errors, latency) |
+
+---
+
+## Best Practices
+
+1. **Start with auto-instrumentation** — instrument frameworks and libraries automatically, add manual spans only where needed
+2. **Use OTLP as the export protocol** — it is the native OTel protocol and supports all three signals
+3. **Propagate context** — ensure W3C TraceContext headers flow across all service boundaries
+4. **Set meaningful span names** — use `HTTP GET /api/users` not `span-1`
+5. **Add semantic attributes** — follow OpenTelemetry semantic conventions for consistent metadata
+6. **Sample wisely in production** — 5–20% trace sampling is typical; keep error sampling at 100%
+7. **Use the Collector** — deploy the OTel Collector as a sidecar or gateway to decouple instrumentation from export
+8. **Correlate logs with traces** — inject `trace_id` and `span_id` into structured log records
+
+---
+
+## Common Patterns
+
+### Node.js SDK Setup
+
+```typescript
+import { NodeSDK } from "@opentelemetry/sdk-node";
+import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
+import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
+import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
+import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
+
+const sdk = new NodeSDK({
+  serviceName: "my-service",
+  traceExporter: new OTLPTraceExporter({ url: "http://localhost:4318/v1/traces" }),
+  metricReader: new PeriodicExportingMetricReader({
+    exporter: new OTLPMetricExporter({ url: "http://localhost:4318/v1/metrics" }),
+  }),
+  instrumentations: [getNodeAutoInstrumentations()],
+});
+
+sdk.start();
+```
+
+### Custom Span
+
+```typescript
+import { trace } from "@opentelemetry/api";
+
+const tracer = trace.getTracer("my-service");
+
+async function processOrder(orderId: string) {
+  return tracer.startActiveSpan("process-order", async (span) => {
+    span.setAttribute("order.id", orderId);
+    try {
+      const result = await doWork(orderId);
+      span.setStatus({ code: 1 }); // OK
+      return result;
+    } catch (error) {
+      span.setStatus({ code: 2, message: String(error) }); // ERROR
+      span.recordException(error as Error);
+      throw error;
+    } finally {
+      span.end();
+    }
+  });
+}
+```
+
+### Custom Metrics
+
+```typescript
+import { metrics } from "@opentelemetry/api";
+
+const meter = metrics.getMeter("my-service");
+
+const requestCounter = meter.createCounter("http.requests.total", {
+  description: "Total HTTP requests",
+});
+
+const latencyHistogram = meter.createHistogram("http.request.duration_ms", {
+  description: "HTTP request latency in milliseconds",
+});
+
+// Usage
+requestCounter.add(1, { method: "GET", route: "/api/users" });
+latencyHistogram.record(42, { method: "GET", route: "/api/users" });
+```
+
+---
+
+## Anti-Patterns
+
+| Anti-Pattern | Why It Hurts | Correct Approach |
+|---|---|---|
+| No sampling in production | Enormous data volume, high cost | Use ratio-based or tail-based sampling |
+| Ignoring context propagation | Broken traces across services | Ensure W3C headers propagate everywhere |
+| Too many custom spans | Noisy traces, performance overhead | Instrument meaningful operations only |
+| Hardcoded exporter URLs | Inflexible deployments | Use environment variables (`OTEL_EXPORTER_OTLP_ENDPOINT`) |
+| Skipping semantic conventions | Inconsistent attribute names | Follow OTel semantic conventions |
+| Not ending spans | Memory leaks, orphan spans | Always call `span.end()` in a `finally` block |
+
+---
+
+**Last verified**: 2026-02-24 | **Skill version**: 1.0.0