ecip-observability-stack 1.0.0

package/docs/M08-Observability-Design.md

@@ -0,0 +1,639 @@
+# M08 — Observability Stack
+## Module Design Document
+
+> **Document ID:** ECIP-M08-MDD · **Revision:** 1.1 · **Date:** March 2026
+> **Status:** Design Complete — Ready for Implementation
+> **Supplements:** ECIP-TDD-001 §4.8, ECIP-TDD-002 §9
+> **Team:** Platform / Infra · **Classification:** Confidential — Internal Engineering Use Only
+
+---
+
+## Table of Contents
+
+- [§0 — Architect's Note](#0--architects-note)
+- [§1 — Module Overview](#1--module-overview)
+  - [Responsibilities](#responsibilities)
+  - [Non-Goals](#non-goals)
+- [§2 — Folder Structure](#2--folder-structure)
+  - [Top-Level Layout](#top-level-layout)
+  - [Key File Explanations](#key-file-explanations)
+- [§3 — Four Pillars: Detailed Design](#3--four-pillars-detailed-design)
+  - [Pillar 1: Distributed Tracing](#pillar-1-distributed-tracing)
+  - [Pillar 2: Metrics Collection](#pillar-2-metrics-collection)
+  - [Pillar 3: Structured Log Aggregation](#pillar-3-structured-log-aggregation)
+  - [Pillar 4: Alerting](#pillar-4-alerting)
+- [§4 — Security Event Pipeline](#4--security-event-pipeline)
+- [§5 — SDK Integration Contract](#5--sdk-integration-contract)
+- [§6 — Task Breakdown](#6--task-breakdown)
+- [§7 — Module Dependencies](#7--module-dependencies)
+- [§8 — Testing Plan](#8--testing-plan)
+- [§9 — Open Design Decisions](#9--open-design-decisions)
+- [§10 — Risk Register](#10--risk-register)
+- [§11 — Architect's Review Notes](#11--architects-review-notes)
+
+---
+
+## §0 — Architect's Note
+
+M08 is not like the other 7 modules. It has no business logic. It owns no domain data. It ships no user-facing features. Its entire job is to make every other module's behaviour observable in production — and that makes it the most important thing to get right before anyone else writes a single line of production code.
+
+**Three rules govern this module:**
+
+1. **M08 ships in Week 1, before any other module integration.** The SDK guide and shared logging library are Day 1 deliverables. A module that starts without them will create permanent blind spots that are expensive to retrofit.
+
+2. **Silent failure is the primary failure mode of observability infrastructure.** Instrumentation appearing to work while producing no data is far worse than outright failure — it creates false confidence. Every design decision in this document is oriented around making failures loud.
+
+3. **M08 must not become a performance bottleneck.** Observability infrastructure that adds >5ms to the hot query path (M01 → M04 → M03) is worse than no observability. The DaemonSet topology, async span export, and tail-based sampling rules are all chosen for this reason.
+
+> **Architect's sign-off:** This design supersedes the TDD-001 §4.8 task list. The task list was a starting point; this document is the implementable specification. Teams should reference this document, not TDD-001, for M08 implementation details.
+
+---
+
+## §1 — Module Overview
+
+| Field | Value |
+|---|---|
+| **Module** | M08 — Observability Stack |
+| **Type** | Cross-Cutting Infrastructure |
+| **Team** | Platform / Infra |
+| **Stack** | OTel Collector · Grafana Tempo · Prometheus · Grafana · Elasticsearch |
+| **Timeline** | Week 1 → Week 28 (continuous; SDK guide Week 1, dashboards iterative) |
+| **Deployment** | Kubernetes (DaemonSet for collector, Deployments for backends) |
+| **Consumes** | Nothing — M08 has no runtime dependency on any other ECIP module |
+| **Consumed by** | All modules (M01–M07) via `@ecip/observability` SDK |
+
+**Purpose:** Distributed tracing, metrics collection, structured log aggregation, and alerting across all 7 ECIP modules via OpenTelemetry auto-instrumentation. Minimal per-module code changes are required — the shared library handles all boilerplate.
+
+### Responsibilities
+
+- **Distributed tracing** — End-to-end W3C TraceContext traces from M01 through all downstream services, stored in Grafana Tempo with 14-day retention.
+- **Metrics collection** — Standard Kubernetes metrics plus a curated catalog of application-level histograms, counters, and gauges scraped by Prometheus.
+- **Structured log aggregation** — A mandatory JSON log schema enforced at compile time via the shared logging library. All services emit structured logs; unstructured `console.log` is a PR-review violation.
+- **Alerting** — SLA-backed Prometheus alert rules routed to PagerDuty and Slack. Every alert has a corresponding runbook.
+- **Security event logging** — Auth failures and RBAC denials routed to a dedicated Elasticsearch index for SIEM use. Entirely separate pipeline from general application logs (NFR-SEC-007).
+- **Grafana dashboards** — Eight pre-built dashboards delivered as dashboard-as-code JSON, auto-provisioned via Grafana sidecar.
+- **SDK and integration guide** — Published packages (`@ecip/observability` for Node.js, `ecip-observability` for Python) plus a runbook targeting 30-minute integration time per team.
+
+### Non-Goals
+
+- **SIEM analysis and alert rules** — Security team operates the Elasticsearch SIEM layer. M08 writes the raw events; the security team owns all query logic above that.
+- **Business-level metrics** (e.g., repos indexed per org, daily active users) — Owned by individual product modules. M08 provides the emission infrastructure only.
+- **Log retention beyond 14 days for traces, 30 days for application logs** — Retention policy is a platform decision; storage provisioning is an infra decision. Neither is M08's call.
+- **APM continuous profiling** (e.g., Pyroscope) — Out of scope for V1. Re-evaluate at Week 22 performance review.
+- **Real user monitoring / browser RUM** — ECIP is a backend-only platform in V1.
+- **On-call schedule management** — M08 wires alerts to PagerDuty; on-call rotation configuration is a team ops concern.
+
+---
+
+## §2 — Folder Structure
+
+### Top-Level Layout
+
+```
+ecip-observability/                        # M08 root — Platform team repo
+│
+├── collector/                             # OTel Collector configuration
+│   ├── otel-collector-config.yaml         # OTLP receiver, processors, exporters
+│   ├── otel-collector-daemonset.yaml      # K8s DaemonSet manifest
+│   └── sampling-config.yaml              # Tail sampling rules (5% default / 100% on error)
+│
+├── dashboards/                            # Grafana dashboard-as-code (JSON)
+│   ├── query-latency.json                 # p50/p95/p99 per mode (lsp/vector/hybrid)
+│   ├── analysis-throughput.json           # Events processed / backlog / Kafka consumer lag
+│   ├── cache-performance.json             # Hit rate by cache_type and repo
+│   ├── lsp-daemon-health.json             # Daemon status, restart rate, OOM events
+│   ├── mcp-call-graph.json                # MCP fan-out topology, latency per target_repo
+│   ├── event-bus-dlq.json                 # DLQ depth, DLQ age, retry counts
+│   ├── cross-repo-fanout.json             # Fan-out depth distribution, cycle warnings
+│   ├── security-events.json               # Auth failures, RBAC denials (from Elasticsearch)
+│   └── _provisioning/
+│       └── grafana-dashboards.yaml        # Grafana sidecar auto-provision config
+│
+├── alerts/                                # Prometheus alerting rules
+│   ├── sla-latency.yaml                   # query_duration_ms p95 > 1500ms
+│   ├── analysis-backlog.yaml              # event backlog > 1000 events
+│   ├── lsp-daemon.yaml                    # restart_rate > 2/hour
+│   ├── dlq-depth.yaml                     # DLQ depth > 100
+│   ├── mcp-latency.yaml                   # mcp_call_duration_ms p95 > 800ms
+│   ├── cache-degradation.yaml             # cache_hit_rate < 60%
+│   └── security-anomaly.yaml              # Auth failure burst > 10 in 5 min
+│
+├── logging-lib/                           # Shared structured logging library (published package)
+│   ├── nodejs/                            # For M01, M04, M05, M07 (TypeScript services)
+│   │   ├── src/
+│   │   │   ├── logger.ts                  # Pino wrapper — mandatory fields enforced at type level
+│   │   │   ├── tracer.ts                  # OTel SDK init + trace/span helpers
+│   │   │   ├── security-events.ts         # emitAuthFailure() / emitRbacDenial() helpers
+│   │   │   └── middleware.ts              # Express/Fastify: injects trace_id into request context
+│   │   ├── package.json                   # Published as @ecip/observability
+│   │   └── tests/
+│   │       ├── logger.test.ts
+│   │       └── tracer.test.ts
+│   │
+│   └── python/                            # For M02 (Analysis Engine — Python components)
+│       ├── src/
+│       │   ├── logger.py                  # structlog wrapper with mandatory fields
+│       │   ├── tracer.py                  # OTel Python SDK init + @traced decorator
+│       │   └── security_events.py
+│       ├── pyproject.toml                 # Published as ecip-observability
+│       └── tests/
+│           └── test_logger.py
+│
+├── prometheus/                            # Prometheus deployment
+│   ├── prometheus-values.yaml             # Helm values for kube-prometheus-stack
+│   ├── scrape-configs.yaml                # Per-service scrape targets (updated as modules ship)
+│   └── recording-rules.yaml              # Pre-computed rate/ratio metrics for dashboard perf
+│
+├── tempo/                                 # Grafana Tempo trace storage
+│   ├── tempo-values.yaml                  # Helm values: S3 backend, 14-day retention
+│   └── tempo-datasource.yaml              # Grafana datasource provisioning
+│
+├── elasticsearch/                         # SIEM security event index (NFR-SEC-007)
+│   ├── index-template.json                # Security event schema + field mappings
+│   ├── ilm-policy.json                    # ILM: 90-day hot → 1-year cold → delete
+│   └── kibana-space.yaml                  # Security team Kibana space config
+│
+├── helm/                                  # Umbrella Helm chart — deploys entire M08 stack
+│   ├── Chart.yaml
+│   ├── values.yaml                        # Default values (dev/staging)
+│   ├── values.prod.yaml                   # Production overrides
+│   └── templates/
+│       ├── otel-collector.yaml
+│       ├── prometheus.yaml
+│       ├── grafana.yaml
+│       ├── tempo.yaml
+│       ├── elasticsearch.yaml
+│       └── configmaps.yaml                # Alert rule + dashboard provisioning refs
+│
+├── runbooks/                              # Operations runbooks and SDK integration guide
+│   ├── SDK-INTEGRATION.md                 # M08-T04: 30-min setup guide for all teams
+│   ├── alert-response/
+│   │   ├── LSP_DAEMON_RESTART.md
+│   │   ├── HIGH_QUERY_LATENCY.md
+│   │   ├── DLQ_DEPTH_EXCEEDED.md
+│   │   ├── ANALYSIS_BACKLOG.md
+│   │   └── SECURITY_ANOMALY.md
+│   └── dashboard-guide.md
+│
+└── tests/                                 # M08 validation tests
+    ├── metric-label-validation.test.ts    # Assert all required labels present on emission
+    ├── alert-threshold-config.test.ts     # Parse + validate all alert YAML files
+    ├── log-schema-validation.test.ts      # JSON log shape + compile-time field enforcement
+    └── otel-pipeline-integration.test.ts  # Testcontainers: Collector → Tempo end-to-end
+```
+
+### Key File Explanations
+
+**`collector/otel-collector-config.yaml`** — The most operationally critical file in M08. It defines three separate pipelines: `traces` (to Tempo), `metrics` (to Prometheus), and `logs` (to Elasticsearch, security events only). An error in this file silently drops all observability data with no visible failure to services. This file must be version-controlled and change-reviewed with the same rigour as application code.
+
+**`logging-lib/nodejs/src/logger.ts`** — The single most consumed M08 artifact. Every Node.js service imports this. The mandatory fields (`trace_id`, `span_id`, `repo`, `branch`, `user_id`, `module`) are enforced at the TypeScript type level — missing any of them is a compilation failure. This is intentional. The alternative is runtime validation which is too easily bypassed.
+
+**`alerts/sla-latency.yaml`** — Directly governs whether NFR-AVL-001 (99.9% query path availability) breaches are detected. The PromQL expression must use `histogram_quantile(0.95, ...)` not an average — averaging latency hides tail problems. This is a common mistake and worth explicit attention during review.
+
+**`runbooks/SDK-INTEGRATION.md`** — The highest-leverage document M08 produces. Every other module's observability quality is directly proportional to how well teams follow this guide. It must be ready by end of Week 1, before any other module writes production code. Treat it as a P0 deliverable, not documentation.
+
+---
+
+## §3 — Four Pillars: Detailed Design
+
+### Pillar 1: Distributed Tracing
+
+**Stack:** OTel Collector (DaemonSet) → Grafana Tempo (S3-backed) → Grafana UI
+
+Every external request is injected with a W3C `traceparent` header at M01. The trace propagates automatically through every downstream gRPC and HTTP call via the OTel SDK's auto-instrumentation. Engineers debugging a slow query can jump from the Grafana latency dashboard to a complete flamegraph-style trace in Tempo in one click, with the trace ID correlating directly to the log entry that fired the alert.
+
+**Topology decision: DaemonSet, not central Deployment**
+
+The OTel Collector runs as a Kubernetes DaemonSet — one collector pod per node, receiving spans from all pods on the same node via localhost. This is a deliberate architectural choice:
+
+- A central Deployment collector is a single point of failure; a DaemonSet crash only affects pods on that node.
+- Localhost communication avoids network latency on the critical span export path.
+- Horizontal scale is automatic as nodes are added to the cluster.
+
+The tradeoff is that each collector pod must be configured identically, which is managed entirely by the Helm chart. No manual per-node configuration is ever required.
+
+**Sampling strategy**
+
+Tail-based sampling is configured at the collector, not the SDK. This means sampling decisions are made after the full trace is assembled, enabling intelligent rules:
+
+| Rule | Rate | Condition |
+|---|---|---|
+| `errors-always-sample` | 100% | Any span with status `ERROR` |
+| `slow-queries-sample` | 100% | Any trace with total latency > 1000ms |
+| `default` | 5% | All other traces |
+
+Head-based sampling (sampling at trace start) is explicitly rejected because it would drop most error traces before they complete. Tail-based sampling with a 10-second decision window ensures errors are always captured.
+
+**Trace propagation flow**
+
+```
+Client Request
+    │
+    ▼
+M01 (API Gateway)        → inject traceparent header
+    │                    → start root span: "gateway.route"
+    ▼
+M04 (Query Service)      → child span: "query.intent_classification"
+    │                    → child span: "query.context_fusion"
+    │                    → child span: "query.filter_authorized_repos" [gRPC to M06]
+    ├──────────────────────────────────────────────────────────────┐
+    ▼                                                              ▼
+M03 (Knowledge Store)                                     M06 (Registry)
+child span: "knowledge.vector_search"                     child span: "registry.check_access"
+child span: "knowledge.symbol_lookup"                     child span: "registry.filter_repos"
+    │
+    ▼ (if cross-repo, depth ≤ 2)
+M05 (MCP Server) × N    → child span: "mcp.tool_call" [per target_repo, in parallel]
+    │
+    ▼
+All spans → OTel Collector (localhost:4317) → Grafana Tempo (S3)
+```
+
+### Pillar 2: Metrics Collection
+
+**Stack:** OTel SDK (emit) → OTel Collector → Prometheus → Grafana
+
+All services expose a `/metrics` endpoint scraped by Prometheus. The scrape interval is 15 seconds for application metrics and 30 seconds for infrastructure metrics. Recording rules in `prometheus/recording-rules.yaml` pre-compute expensive quantile and rate calculations so Grafana dashboard queries are fast even at scale.
+
+**Metrics Catalog**
+
+This is the authoritative list of application-level metrics M08 defines. All other modules must instrument exactly these metrics with exactly these label sets. Additional metrics may be added by modules but must be reviewed by the Platform team before shipping to prevent high-cardinality label explosions.
+
+| Metric Name | Type | Source | Labels | Alert Threshold |
+|---|---|---|---|---|
+| `query_duration_ms` | Histogram | M04 | `repo`, `mode` (lsp/vector/hybrid), `cached` | p95 > 1500ms → CRITICAL |
+| `analysis_duration_ms` | Histogram | M02 | `repo`, `branch_type` (trunk/pr), `language` | p95 > 120000ms → WARN |
+| `cache_hit_rate` | Gauge | M03, M04 | `cache_type` (symbol/query), `repo` | < 60% → WARN |
+| `lsp_daemon_restarts_total` | Counter | M02 | `repo`, `language` | > 2/hour → CRITICAL |
+| `mcp_call_duration_ms` | Histogram | M04 | `target_repo`, `tool_name` | p95 > 800ms → WARN |
+| `event_bus_dlq_depth` | Gauge | M07 | `topic` | > 100 → CRITICAL |
+| `cross_repo_fanout_count` | Histogram | M04 | `depth`, `repo` | — (dashboard only) |
+| `rbac_denial_total` | Counter | M01, M06 | `resource`, `action` | > 10 in 5min → SECURITY WARN |
+| `auth_failure_total` | Counter | M01 | `reason` (expired/invalid/missing) | > 10 in 5min → SECURITY WARN |
+| `grpc_request_duration_ms` | Histogram | M01, M04, M06 | `service`, `method`, `status_code` | p95 > 500ms → WARN |
+| `knowledge_store_write_duration_ms` | Histogram | M03 | `store_type` (redis/pgvector), `namespace` | p95 > 200ms → WARN |
+| `hnsw_rebuild_duration_ms` | Histogram | M03 | `repo` | — (TDD-002; dashboard only) |
+| `filter_authorized_repos_duration_ms` | Histogram | M06 | `org` | p95 > 20ms → WARN (NFR-SEC-011) |
+| `embedding_migration_progress` | Gauge | M02, M03 | `repo`, `phase` (backfill/shadow/cutover) | — (TDD-002; dashboard only) |
+
+> **Label cardinality rule:** `user_id` is explicitly prohibited as a Prometheus label. User-scoped metrics belong in Elasticsearch (security events) or application logs. A single high-cardinality label like `user_id` can cause Prometheus OOM on a busy cluster. This rule is enforced by the metric label validation test.
+
+### Pillar 3: Structured Log Aggregation
+
+**Stack:** `@ecip/observability` logger → stdout (JSON) → Fluentd/Fluent Bit → central log store
+
+All services must use the M08-published logging library. Raw `console.log`, `print()`, or any unstructured log emission is a contract violation, caught at PR review, and treated as a build failure in CI.
+
+**Mandatory JSON log schema**
+
+```json
+{
+  "timestamp": "2026-03-01T10:00:00.123Z",   // ISO 8601, always UTC
+  "level":     "info",                        // trace | debug | info | warn | error | fatal
+  "module":    "M04",                         // which ECIP module emitted this
+  "trace_id":  "d9f3c1a8-2e7b-44cc-...",     // W3C TraceContext traceId
+  "span_id":   "f4a912b3...",                // active span ID at emission time
+  "repo":      "acme-corp/auth-service",      // {org}/{repo} — required
+  "branch":    "main",                        // branch context of the operation
+  "user_id":   "u_8f3a1c",                   // hashed user identifier — no raw PII
+  "env":       "production",
+  "msg":       "Query resolved via hybrid mode",
+  // ... arbitrary structured fields follow
+  "query_type": "hybrid",
+  "duration_ms": 43,
+  "cached": false
+}
+```
+
+`trace_id`, `span_id`, `repo`, `branch`, `user_id`, and `module` are mandatory at the TypeScript type level. Missing them is a compilation failure. The Python wrapper raises `MissingObservabilityContext` at import time if the OTel SDK was not initialized.
+
+**What goes in logs vs. metrics vs. traces**
+
+This is a common source of confusion and must be explicit:
+
+- **Metrics** — Aggregatable numeric values: latency histograms, counters, rates. If you need to know "how many times did X happen", it's a metric.
+- **Logs** — Discrete events with rich contextual detail. If you need to know "what were the exact parameters when X failed", it's a log.
+- **Traces** — Causal chains across service boundaries. If you need to know "which downstream call made this request slow", it's a trace.
+
+Never duplicate data across all three. A query duration belongs in a histogram metric and in the trace span duration — not also in a log line.
+
+### Pillar 4: Alerting
+
+**Stack:** Prometheus → Alertmanager → PagerDuty / Slack
+
+All alert rules live in `alerts/`. Every alert must have: a corresponding entry in the metrics catalog, a PromQL expression reviewed by the Platform team, a severity level, a routing target, and a runbook document in `runbooks/alert-response/`.
+
+**Alert rules**
+
+| Alert Name | PromQL Condition | For | Severity | Routes To | Runbook |
+|---|---|---|---|---|---|
+| `QueryLatencySLABreach` | `histogram_quantile(0.95, query_duration_ms) > 1500` | 5min | CRITICAL | PagerDuty + Slack #alerts | `HIGH_QUERY_LATENCY.md` |
+| `AnalysisBacklogCritical` | `sum(kafka_consumergroup_lag{group=~"ecip-analysis.*"}) > 1000` | 10min | CRITICAL | PagerDuty + Slack #alerts | `ANALYSIS_BACKLOG.md` |
+| `LSPDaemonRestartRate` | `rate(lsp_daemon_restarts_total[1h]) > 2` | 0min | CRITICAL | PagerDuty + Slack #alerts | `LSP_DAEMON_RESTART.md` |
+| `DLQDepthExceeded` | `event_bus_dlq_depth > 100` | 5min | CRITICAL | PagerDuty + Slack #alerts | `DLQ_DEPTH_EXCEEDED.md` |
+| `MCPCallLatencyWarn` | `histogram_quantile(0.95, mcp_call_duration_ms) > 800` | 10min | WARNING | Slack #alerts-warn | `HIGH_QUERY_LATENCY.md` |
+| `CacheHitRateDegraded` | `cache_hit_rate < 0.60` | 15min | WARNING | Slack #alerts-warn | `HIGH_QUERY_LATENCY.md` |
+| `FilterAuthorizedReposLatency` | `histogram_quantile(0.95, filter_authorized_repos_duration_ms) > 20` | 5min | WARNING | Slack #alerts-warn | `HIGH_QUERY_LATENCY.md` |
+| `SecurityAuthBurst` | `increase(auth_failure_total[5m]) > 10` | 0min | CRITICAL | PagerDuty + Slack #security | `SECURITY_ANOMALY.md` |
+| `SecurityRBACDenialBurst` | `increase(rbac_denial_total[5m]) > 10` | 0min | WARNING | Slack #security | `SECURITY_ANOMALY.md` |
+
+> **Architect's note on `LSPDaemonRestartRate`:** The `for: 0min` duration (fires immediately) is intentional. An LSP daemon crash is always a significant event; there is no grace period. The circuit breaker in M04 handles the graceful degradation; the alert gets a human involved in parallel.
+
+---
+
+## §4 — Security Event Pipeline
+
+Security events are a distinct category from application logs. They are governed by NFR-SEC-007 and require a separate OTel pipeline that routes exclusively to Elasticsearch, never to the general log store.
+
+**What constitutes a security event**
+
+- JWT validation failure at M01 (expired, invalid signature, missing)
+- RBAC denial at M01 or M06 (user lacks required role for resource)
+- Service-to-service authentication failure (mTLS or service token rejection)
+- Any access attempt to a repo not in the user's authorized set (detected by M06's `FilterAuthorizedRepos`)
+
+**Security event schema**
+
+```json
+{
+  "@timestamp":      "2026-03-01T10:00:00.123Z",
+  "event.kind":      "event",
+  "event.category":  "authentication",             // or "authorization"
+  "event.type":      "denied",
+  "event.outcome":   "failure",
+  "trace.id":        "d9f3c1a8-2e7b-44cc-...",    // correlates to Tempo trace
+  "user.id":         "u_8f3a1c",                  // hashed — no raw PII
+  "source.ip":       "10.0.14.22",
+  "resource":        "acme-corp/auth-service",
+  "action":          "read",
+  "reason":          "rbac_insufficient_role",     // or "jwt_expired" | "jwt_invalid"
+  "module":          "M01"
+}
+```
+
+**Pipeline separation**
+
+Security events must never leak into general application logs and must never be emitted via the general `logger` object. The `emitSecurityEvent()` helper in `security-events.ts` is the only permitted emission path. It uses a separate OTel logger provider configured to route exclusively to the security Elasticsearch pipeline. This separation is enforced architecturally — it cannot be accidentally bypassed by a developer using the standard logger.
+
+**Elasticsearch index lifecycle**
+
+```
+90 days → hot tier (SSD, fast query)
+90–365 days → warm tier (standard storage)
+365 days → cold tier (object storage)
+> 365 days → delete
+```
+
+The security team owns all Kibana queries, dashboards, and SIEM detection rules above the index layer. M08 owns only the index template, ILM policy, and raw event schema.
+
+---
+
+## §5 — SDK Integration Contract
+
+This section defines what M08 delivers to other module teams and what those teams are contractually required to do with it. This is the interface contract between M08 and every other module.
+
+### What M08 delivers (by end of Week 1)
+
+- **`@ecip/observability`** — npm package (Node.js / TypeScript). Published to the internal registry.
+- **`ecip-observability`** — Python package. Published to the internal PyPI.
+- **`runbooks/SDK-INTEGRATION.md`** — Step-by-step integration guide. Target: 30-minute setup for any team.
+- **OTel Collector endpoint** — Available at `http://otel-collector.monitoring:4317` (gRPC) and `:4318` (HTTP) in all namespaces by Week 1.
+
+### What every module team must do
+
+**Node.js / TypeScript services (M01, M04, M05, M07)**
+
+```typescript
+// Step 1: Install
+// npm install @ecip/observability @opentelemetry/sdk-node
+
+// Step 2: src/instrument.ts — import BEFORE all other imports
+import { initTracer } from '@ecip/observability';
+
+initTracer({
+  serviceName: 'ecip-query-service',
+  otlpEndpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT, // injected by Helm
+});
+
+// Step 3: Use the logger in every handler
+import { createLogger } from '@ecip/observability';
+
+const log = createLogger({ repo, branch, user_id, module: 'M04' });
+log.info({ duration_ms: 43, cached: false }, 'Query resolved');
+
+// Step 4: Emit security events via dedicated helper (never via log)
+import { emitRbacDenial } from '@ecip/observability';
+
+emitRbacDenial({ userId, resource: repoId, action: 'read', reason: 'rbac_insufficient_role' });
+```
+
+**Python services (M02)**
+
+```python
+# Step 1: pip install ecip-observability opentelemetry-sdk
+
+# Step 2: Initialize at process entry (before any other imports)
+from ecip_observability import init_tracer, get_logger
+
+init_tracer(service_name="ecip-analysis-engine")
+
+# Step 3: Use the logger
+log = get_logger(repo=repo, branch=branch, user_id=user_id, module="M02")
+log.info("Analysis complete", duration_ms=14200, files_indexed=47)
+
+# Step 4: Decorate functions for automatic span creation
+from ecip_observability import traced
+
+@traced(name="lsp.symbol_extraction")
+def extract_symbols(file_path: str) -> list:
+    ...  # span automatically started/ended; exceptions auto-captured
+```
+
+**Helm value injection (Infra team responsibility)**
+
+All OTel configuration is injected by the Helm chart. Module teams do not hardcode endpoints.
+
+```yaml
+env:
+  - name: OTEL_EXPORTER_OTLP_ENDPOINT
+    value: "http://otel-collector.monitoring:4318"
+  - name: OTEL_SERVICE_NAME
+    valueFrom:
+      fieldRef:
+        fieldPath: metadata.labels['app.kubernetes.io/name']
+  - name: OTEL_RESOURCE_ATTRIBUTES
+    value: "ecip.module=M04,deployment.environment=production"
+```
+
+### Integration enforcement (CI gate)
+
+A module PR that does not satisfy all of the following is blocked from merging:
+
+1. `@ecip/observability` or `ecip-observability` is in the dependency list.
+2. `initTracer()` is called before the service starts accepting requests (verified by startup integration test).
+3. At least one log line per request handler uses `createLogger()` with all mandatory fields.
+4. No `console.log` or `print()` statements exist in production code paths (ESLint / Ruff rules).
+
+---
+
+## §6 — Task Breakdown
+
+| # | Task | Est. | Owner | Week | Priority | Notes |
+|---|---|---|---|---|---|---|
+| M08-T01 | Deploy OTel Collector as DaemonSet; configure OTLP receiver (gRPC :4317 + HTTP :4318) | 2d | Infra | W1 | **P0** | Must be online before any other module begins instrumentation. Hard blocker for all teams. |
+| M08-T02 | Deploy Grafana Tempo; S3 backend; 14-day retention; microservices mode | 2d | Infra | W1 | **P0** | Grafana Tempo over Jaeger — native Grafana integration, S3 cost model better at scale. |
+| M08-T03 | Deploy Prometheus + Grafana; scrape configs; recording rules | 2d | Infra | W1 | **P0** | `kube-prometheus-stack` Helm chart. Add per-module scrape configs as modules come online. |
+| M08-T04 | Publish `@ecip/observability` + `ecip-observability`; write SDK-INTEGRATION.md | 2d | Senior Backend | W1 | **P0** | Highest-leverage M08 deliverable. Treat as feature, not documentation. |
+| M08-T05 | Build 8 Grafana dashboards as JSON; wire via sidecar provisioning | 4d | Backend | W2–W3 | P1 | Ship `query-latency.json` first. Others follow as modules come online. |
+| M08-T06 | Configure all alert rules; wire PagerDuty + Slack; write runbook for each alert | 2d | Backend | W2 | **P0** | `QueryLatencySLABreach` and `LSPDaemonRestartRate` must be live before CP1 (Week 4). |
+| M08-T07 | Implement logging library: mandatory fields, security event helpers, middleware | 1d | Senior Backend | W1 | **P0** | Delivered as part of M08-T04 package. Compile-time field enforcement is non-negotiable. |
+| M08-T08 | Elasticsearch security event index; ILM policy; schema; Kibana space | 2d | Security + Backend | W3–W4 | P1 | NFR-SEC-007. Security team owns Kibana; M08 owns index schema only. |
+| M08-T09 | Add TDD-002 metrics: HNSW rebuild, embedding migration progress, `FilterAuthorizedRepos` latency | 1d | Backend | W5–W6 | P2 | Extend existing dashboards once M06's new RPCs are wired. |
+| M08-T10 | Write CI gate enforcement: ESLint/Ruff rules, startup integration check, label validation | 1d | Backend | W2 | P1 | Without this, SDK adoption will drift. CI enforcement is the only reliable mechanism. |
+
+**Total estimated effort:** ~17 days (Platform team; 2 engineers)
+
+---
+
+## §7 — Module Dependencies
+
+M08 has no runtime dependency on any other ECIP module. All arrows point inward — every other module depends on M08.
+
+| Module | Depends on M08 for | M08 Artifact | Required by Week |
+|---|---|---|---|
+| M01 — API Gateway | Trace injection, auth failure logging, request logging | `@ecip/observability`, Collector endpoint | **Week 1** |
+| M03 — Knowledge Store | Write latency metrics, cache hit rate | `@ecip/observability`, Prometheus scrape config | **Week 1** |
+| M07 — Event Bus | DLQ depth metrics, event processing latency | `@ecip/observability`, Prometheus scrape config | **Week 1** |
+| M06 — Registry | RBAC denial events, gRPC duration metrics, `FilterAuthorizedRepos` latency | `@ecip/observability`, security event schema | Week 5 |
+| M02 — Analysis Engine | Span wrapping around LSP calls, analysis duration metrics | `ecip-observability` (Python), Collector endpoint | Week 5 |
+| M04 — Query Service | Query latency metrics, circuit breaker state logging, MCP call spans | `@ecip/observability`, Collector endpoint | Week 11 |
+| M05 — MCP Server | Tool call duration metrics, RBAC denial logging | `@ecip/observability` | Week 17 |
+
+M01, M03, and M07 all start in Week 1 or Week 2. M08's core infrastructure (Collector, Tempo, Prometheus, SDK) must be ready before any of them write production code. This is the hardest constraint in the M08 timeline.
+
+---
+
+## §8 — Testing Plan
+
+### Unit Tests (coverage target: 80%)
+
+| Test File | What It Validates |
+|---|---|
+| `metric-label-validation.test.ts` | All metric emissions include the exact label sets defined in the catalog. Uses a mock Prometheus registry to capture emitted metrics and validates label presence and types. |
+| `alert-threshold-config.test.ts` | All YAML files in `alerts/` parse without error; all PromQL expressions are syntactically valid (`promtool check rules`); all referenced metrics exist in the catalog; all runbook file paths resolve on disk. |
+| `log-schema-validation.test.ts` | JSON log output from `createLogger()` matches the mandatory field schema. Compile-time test: TypeScript build fails when mandatory fields are omitted (validated by compiling a deliberately broken test file and asserting non-zero exit code). |
+| `security-events.test.ts` | `emitAuthFailure()` and `emitRbacDenial()` produce correct ECS-formatted events. No `user_id` raw values in output (only hashed). Events route to the correct OTel logger provider (not the general logger). |
+
+### Integration Tests (Testcontainers)
+
+**`otel-pipeline-integration.test.ts`** — The most important test in M08.
+
+Setup: Testcontainers spins up an OTel Collector container and a Grafana Tempo container. The `@ecip/observability` SDK is initialized pointing at the collector.
+
+Assertions:
+1. Emit 5 spans across a simulated async service boundary.
+2. All 5 spans are retrievable from Tempo via HTTP API within 10 seconds.
+3. `traceparent` header is correctly propagated across the async boundary (child spans have correct parent ID).
+4. An error span (forced `status: ERROR`) is 100% sampled (present in Tempo despite < 5% default rate).
+5. A normal span in a 5%-sampled-only run may or may not be present — test does not assert its presence (this validates the sampling rule is not broken by asserting only error capture).
+
+**Alert rule CI validation:**
+```bash
+# Run in CI on every PR touching alerts/
+promtool check rules alerts/*.yaml
+```
+
+**Dashboard JSON lint:**
+```bash
+# Run in CI on every PR touching dashboards/
+grafana-dashboard-lint dashboards/*.json
+```
+
+> **Why the Testcontainers test is non-negotiable:** Unit tests cannot catch a misconfigured collector pipeline, an mTLS cert error on the Tempo exporter, or a sampling rule that silently drops all traces. The integration test is the only meaningful correctness signal for the observability pipeline itself. It must run in CI on every PR, not just nightly.
+
+### What M08 does not test
+
+M08 does not test that other modules have correctly instrumented themselves. That responsibility belongs to each module's own test suite. M08's CI gate (§5) enforces minimum instrumentation standards; module teams own the quality of their own spans and log lines.
+
+---
+
+## §9 — Open Design Decisions
+
+These decisions are not yet finalised and must be resolved before the relevant tasks begin.
+
+**OD-01: Log aggregation backend for general application logs**
+
+The current design routes security events to Elasticsearch but leaves the general application log destination unspecified. Options are: (a) Elasticsearch with a separate index, (b) Loki (Grafana-native, lower cost), (c) CloudWatch / GCP Logging if running on a managed cloud. This decision affects the Helm chart, the Fluent Bit / Fluentd sidecar configuration, and potentially the SDK (if the log backend requires a specific OTel exporter). Must be decided before M08-T01 is complete.
+
+**OD-02: Collector resource limits in production**
+
+The DaemonSet manifest specifies `limit_mib: 512` for the collector memory limiter. This was estimated for a 200-node cluster at expected trace volume with 5% sampling. It has not been load-tested. At Week 8, after M01, M03, and M07 are generating real traces, the actual memory profile must be measured and the limit updated. If volume exceeds estimates, the sampling rate is the first lever to pull (reduce from 5% to 2%), not the memory limit.
+
+**OD-03: Dashboard ownership after Week 28**
+
+The 8 dashboards in `dashboards/` are built and maintained by the Platform team during the build phase. Post-Week 28, who owns them? Module teams adding new metrics need to update the dashboards. Without a clear ownership model, dashboards will drift from the actual metrics being emitted. Recommendation: each module team owns the panels in the dashboards that relate to their metrics; Platform team owns the infrastructure-level panels (Collector health, Prometheus performance, Tempo storage).
+
+---
+
+## §10 — Risk Register
+
+| Risk | Likelihood | Impact | Mitigation |
+|---|---|---|---|
+| Module teams skip SDK integration — observability blind spots in production | **High** | **High** | CI gate (M08-T10) blocks merges without instrumentation. SDK guide ready Week 1. Make it easier to integrate than to skip. |
+| OTel Collector config error silently drops all trace data | **Medium** | **High** | Pipeline integration test (Testcontainers) catches this in CI. Config changes to `otel-collector-config.yaml` require Platform team review. Collector exposes its own health metrics at `:13133/healthz`. |
+| High-cardinality Prometheus labels cause OOM | **Medium** | **High** | `user_id` prohibited as Prometheus label (enforced by label validation test). New metrics require Platform team review before shipping. Prometheus has a 10M series limit alert configured. |
+| Tempo storage costs blow out with 14-day retention | **Medium** | **Low** | Tail-based sampling (5% healthy) reduces volume ~20x vs. 100% sampling. Review storage costs at Week 8 with real traffic data. Reduce sampling to 2% if needed before touching retention. |
+| Security event pipeline mixes with general logs | **Low** | **High** | Architecturally separated: dedicated OTel logger provider, dedicated Collector logs pipeline, dedicated Elasticsearch index. `emitSecurityEvent()` is the only permitted path. Enforced by security-events unit test. |
+| M08 infrastructure is itself unobservable | **Low** | **Medium** | OTel Collector exposes Prometheus metrics at `:8888`. Grafana has a collector health panel. Collector logs are shipped to the same log store as other services. The `monitoring` namespace has its own Prometheus alert for collector pod restarts. |
+
+---
+
+## §11 — Architect's Review Notes
+
+These are design issues identified during the architectural review of TDD-001's M08 task list. They are not hypothetical concerns — they reflect gaps in the original specification that would have caused real problems during implementation.
+
+### Review finding 1: The task list is not a design
+
+TDD-001 §4.8 lists 8 tasks and a metrics table. That is a starting point, not an implementable specification. Missing from the original:
+
+- No definition of the OTel Collector topology (DaemonSet vs. Deployment) and the rationale for it.
+- No sampling strategy. The default OTel SDK samples 100% of traces, which is disastrously expensive at production volume.
+- No specification of what "structured logging" means — just "deliver as logging library wrapper."
+- No enforcement mechanism. A guide with no enforcement is a suggestion.
+- No security event pipeline design. NFR-SEC-007 requires it, but TDD-001 only lists it as a task with no schema, no routing, no pipeline separation.
+
+This document addresses all of the above.
+
+### Review finding 2: Log aggregation destination is unspecified
+
+TDD-001 mentions Elasticsearch for security events but says nothing about where general application logs go. This is a gap. The OTel Collector's `logs` pipeline needs an exporter target. Leaving it unspecified means the Infra team will make the decision independently, potentially choosing a backend that is incompatible with the Grafana Tempo + Grafana stack. This is captured as OD-01 above and must be resolved before M08-T01 closes.
+
+### Review finding 3: `FilterAuthorizedRepos` latency alert was missing
+
+TDD-002 adds `FilterAuthorizedRepos` as a new RPC to M06 with an explicit p95 < 20ms SLA (NFR-SEC-011). TDD-001's M08 task list predates TDD-002 and therefore has no alert for this SLA. The metrics catalog and alert rules in this document add `filter_authorized_repos_duration_ms` and the `FilterAuthorizedReposLatency` alert to cover it. This is not captured in M08-T09 (which only mentions dashboard additions) — the alert rule is a separate deliverable and must be added to M08-T06's scope.
+
+### Review finding 4: Dashboard delivery sequence matters
+
+TDD-001 specifies all dashboards as a single 4-day task. In practice, a dashboard for `query_duration_ms` is useless before M04 exists, and a dashboard for `mcp_call_duration_ms` is useless before M05 exists. Delivering all 8 dashboards by Week 3 means 5 of them display empty panels for months.
+
+The correct approach is to ship dashboards in module-dependency order:
+
+1. **Week 2** — `lsp-daemon-health.json`, `event-bus-dlq.json` (M02, M07 start early)
+2. **Week 3** — `cache-performance.json`, `analysis-throughput.json` (M03 foundation)
+3. **Week 11** — `query-latency.json`, `mcp-call-graph.json` (M04 comes online)
+4. **Week 17** — `cross-repo-fanout.json` (M05 comes online)
+5. **Week 4** — `security-events.json` (tied to M08-T08, not module timelines)
+
+This does not change the total effort; it changes when effort is spent and ensures dashboards are immediately useful when delivered.
+
+### Review finding 5: The CI gate is load-bearing, not optional
+
+The SDK integration guide says "all teams should use `@ecip/observability`." Without a CI gate, this will not happen. Engineers under deadline pressure skip instrumentation. Two months into the build, when the first production incident occurs and traces are missing for 3 of 7 modules, the cost of retrofitting is high and the debugging is impossible.
+
+M08-T10 (CI gate enforcement) is listed as P1 in the task breakdown. Architecturally it should be P0, delivering alongside M08-T04 in Week 1. The CI gate is the enforcement mechanism that makes the SDK guide meaningful. Without it, the SDK guide is optional documentation.
+
+---
+
+*ECIP-M08-MDD · Observability Stack · Module Design Document · Rev 1.1 · March 2026*
+*Confidential — Internal Engineering Use Only · Platform Team*
+*This document supersedes ECIP-TDD-001 §4.8 for implementation purposes.*