ecip-observability-stack 1.0.0

package/docs/PROGRESS.md

@@ -0,0 +1,375 @@
+# ECIP M08 — Observability Stack: Progress Report
+
+> **Module:** M08 — Observability Stack  
+> **Design Authority:** `docs/M08-Observability-Design.md` (Rev 1.1)  
+> **Last Updated:** 2025-07-17  
+> **Previous Update:** 2025-07-07  
+
+---
+
+## Summary
+
+| Category           | Count  |
+|--------------------|--------|
+| Tasks Done         | 10 / 10 |
+| Tasks Partial      | 0 / 10 |
+| Tasks Pending      | 0 / 10 |
+| Gaps Found         | 12 |
+| Gaps Resolved      | 8 |
+| Gaps Remaining     | 4 |
+| Enhancements Done  | 6 / 10 |
+| Total Files        | 83 |
+
+### Changes Since Last Update (2025-07-07 → 2025-07-17)
+
+Architect review items implemented:
+- **M08-T10 CI gate** — Fully implemented (ESLint plugin, Ruff config, contract checker, GitHub Actions workflow)
+- **GAP-01** — `tests/security-events.test.ts` created (ECS format, PII hashing, stderr isolation)
+- **GAP-03** — `scripts/lint-dashboards.js` created (8 validation checks)
+- **GAP-06** — Elasticsearch URL templated in Helm
+- **GAP-07** — Grafana admin credentials externalized to K8s Secret
+- **GAP-11** — OOM panel PromQL fixed (both dashboard + alert rule)
+- **GAP-12 / OD-01** — Loki chosen as log backend; full integration (Helm dependency, OTel exporter, Grafana datasource, Tempo tracesToLogs)
+
+New files added: 9 | Modified files: 8 | File count: 73 → 83
+
+---
+
+## Task-Level Progress (§6 Task Breakdown)
+
+### M08-T01 — OTel Collector DaemonSet (P0, Week 1) — ✅ DONE
+
+**Deliverables created:**
+- `collector/otel-collector-config.yaml` — Full OTLP pipeline config with **4 pipelines** (traces→Tempo, metrics→Prometheus, logs/security→Elasticsearch, logs→Loki)
+- `collector/otel-collector-daemonset.yaml` — K8s DaemonSet + Service + ServiceAccount + RBAC
+- `collector/sampling-config.yaml` — Tail-based sampling with 5 policies (errors 100%, slow 100%, security 100%, LSP 20%, default 5%)
+- `helm/templates/otel-collector.yaml` — Helm template (DaemonSet + Service + SA + ClusterRole/Binding)
+- `helm/templates/configmaps.yaml` — ConfigMap for collector config
+
+**Status:** All design-specified artifacts exist. Pipeline supports OTLP gRPC (:4317) and HTTP (:4318). DaemonSet topology matches the design rationale. Memory limiter, batch processor, k8sattributes, and tail-sampling all configured. **Loki exporter added** — general application logs now routed to Loki (OD-01 resolved).
+
+---
+
+### M08-T02 — Grafana Tempo Deployment (P0, Week 1) — ✅ DONE
+
+**Deliverables created:**
+- `tempo/tempo-values.yaml` — S3 backend, 14-day retention, microservices mode (ingester/compactor/distributor replicas)
+- `tempo/tempo-datasource.yaml` — Grafana provisioning with tracesToMetrics, tracesToLogs→**Loki**, nodeGraph, serviceMap
+- `helm/templates/tempo.yaml` — Tempo datasource ConfigMap for Grafana sidecar
+- `helm/values.prod.yaml` — Production overrides with S3 bucket, HA replica counts
+
+**Status:** Complete. Matches design spec: S3 backend, 14d retention, microservices mode, Grafana-native integration. **tracesToLogs now points to Loki** (previously referenced Elasticsearch).
+
+---
+
+### M08-T03 — Prometheus + Grafana Deployment (P0, Week 1) — ✅ DONE
+
+**Deliverables created:**
+- `prometheus/prometheus-values.yaml` — kube-prometheus-stack Helm values with 30d retention, 100Gi PVC, Alertmanager with PagerDuty + Slack routing
+- `prometheus/scrape-configs.yaml` — Per-module K8s service discovery targets (M01–M07 + OTel Collector)
+- `prometheus/recording-rules.yaml` — Pre-computed quantiles and rates (query p50/p95/p99, analysis p50/p95, MCP p95, gRPC p95, KS write p95, throughput rates, error rates, auth/RBAC rates)
+- `helm/templates/prometheus.yaml` — Alert rules ConfigMap, recording rules ConfigMap, scrape configs Secret
+- `helm/Chart.yaml` — Umbrella chart with kube-prometheus-stack, tempo-distributed, elasticsearch, **loki** dependencies
+- `helm/values.yaml` — Dev/staging defaults (now includes loki config, Grafana secret ref, ES host/port/protocol)
+- `helm/values.prod.yaml` — Production overrides (now includes Loki S3 production config, Grafana admin secret)
+- `dashboards/_provisioning/grafana-dashboards.yaml` — Sidecar provisioning config
+- `helm/templates/grafana-secret.yaml` — Grafana admin credentials K8s Secret template
+- `helm/templates/loki.yaml` — Loki datasource provisioning for Grafana (with TraceID→Tempo correlation)
+
+**Status:** Complete. All scrape targets match the 7 modules + collector. Recording rules cover the primary metrics. Alertmanager routing configured for PagerDuty (critical) and Slack (warning/security). **Grafana admin credentials** now externalized to K8s Secret (GAP-07 resolved). **Loki** added as Helm dependency and Grafana datasource (OD-01 resolved).
+
+---
+
+### M08-T04 — Publish SDK Packages + Integration Guide (P0, Week 1) — ✅ DONE
+
+**Deliverables created:**
+
+**Node.js (`@ecip/observability`):**
+- `logging-lib/nodejs/src/logger.ts` — Pino-based structured logger with compile-time `ECIPLoggerContext` enforcement (repo, branch, user_id, module)
+- `logging-lib/nodejs/src/tracer.ts` — OTel SDK init, `withSpan()`, `getTracer()`, `injectTraceContext()`, `extractTraceContext()`
+- `logging-lib/nodejs/src/security-events.ts` — `emitAuthFailure()`, `emitRbacDenial()` with ECS schema, PII hashing, buffered stderr emission
+- `logging-lib/nodejs/src/middleware.ts` — Express middleware + Fastify plugin for trace context injection
+- `logging-lib/nodejs/src/index.ts` — Barrel exports
+- `logging-lib/nodejs/package.json` — All OTel + Pino dependencies
+- `logging-lib/nodejs/tsconfig.json`
+- `logging-lib/nodejs/tests/logger.test.ts` — Logger factory tests
+- `logging-lib/nodejs/tests/tracer.test.ts` — Tracer initialization and span tests
+
+**Python (`ecip-observability`):**
+- `logging-lib/python/src/logger.py` — structlog-based logger with `MissingObservabilityContext` validation
+- `logging-lib/python/src/tracer.py` — OTel SDK init, `@traced` decorator (sync + async), gRPC auto-instrumentation
+- `logging-lib/python/src/security_events.py` — `emit_auth_failure()`, `emit_rbac_denial()` with ECS schema, PII hashing
+- `logging-lib/python/src/__init__.py` — Package re-exports
+- `logging-lib/python/pyproject.toml` — Dependencies, Ruff + mypy config
+- `logging-lib/python/tests/test_logger.py` — Logger and security event tests
+
+**Integration Guide:**
+- `runbooks/SDK-INTEGRATION.md` — Step-by-step setup guide
+
+**Status:** All SDK packages and integration guide created. Both Node.js and Python libraries implement the full contract from §5.
+
+---
+
+### M08-T05 — 8 Grafana Dashboards (P1, Week 2–3) — ✅ DONE
+
+**Dashboards created (all as provisioned JSON):**
+
+| # | File | Title | Panels | Datasources | Template Vars |
+|---|------|-------|--------|-------------|---------------|
+| 1 | `query-latency.json` | Query Latency | 8 | Prometheus | `$repo` |
+| 2 | `analysis-throughput.json` | Analysis Throughput | 9 | Prometheus | `$repo` |
+| 3 | `cache-performance.json` | Cache Performance | 7 | Prometheus | `$repo` |
+| 4 | `lsp-daemon-health.json` | LSP Daemon Health | 7 | Prometheus | — |
+| 5 | `mcp-call-graph.json` | MCP Call Graph | 6 | Prometheus + Tempo | `$target_repo` |
+| 6 | `event-bus-dlq.json` | Event Bus DLQ | 8 | Prometheus | — |
+| 7 | `cross-repo-fanout.json` | Cross-Repo Fan-out | 5 | Prometheus | — |
+| 8 | `security-events.json` | Security Events | 7 | Prometheus + Elasticsearch | — |
+
+**Status:** All 8 dashboards implemented with appropriate thresholds, template variables, and multi-datasource support. Grafana sidecar provisioning configured.
+
+---
+
+### M08-T06 — Alert Rules + PagerDuty/Slack + Runbooks (P0, Week 2) — ✅ DONE
+
+**Alert rules created (7 files, 17 alerts total):**
+
+| Design-Required Alert | Implemented | File | Match |
+|---|---|---|---|
+| `QueryLatencySLABreach` | ✅ | `sla-latency.yaml` | Exact |
+| `AnalysisBacklogCritical` | ✅ | `analysis-backlog.yaml` | Metric name differs (see gaps) |
+| `LSPDaemonRestartRate` | ✅ | `lsp-daemon.yaml` | Exact |
+| `LSPDaemonOOMKill` (bonus) | ✅ | `lsp-daemon.yaml` | Fixed — uses `kube_pod_container_status_last_terminated_reason` |
+| `DLQDepthExceeded` | ✅ | `dlq-depth.yaml` | Exact |
+| `MCPCallLatencyWarn` | ✅ | `mcp-latency.yaml` | Exact |
+| `CacheHitRateDegraded` | ✅ | `cache-degradation.yaml` | Exact |
+| `FilterAuthorizedReposLatency` | ✅ | `sla-latency.yaml` | Exact |
+| `SecurityAuthBurst` | ✅ | `security-anomaly.yaml` | Exact |
+| `SecurityRBACDenialBurst` | ✅ | `security-anomaly.yaml` | Exact |
+
+**Bonus alerts (8 additional, beyond design spec):**
+- `GRPCRequestLatencyHigh` (sla-latency) — Generic gRPC catch-all
+- `AnalysisBacklogWarning` (analysis-backlog) — Early warning tier
+- `LSPDaemonOOMKill` (lsp-daemon) — OOM-specific detection (**PromQL fixed** — now uses `kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}`)
+- `DLQDepthWarning` (dlq-depth) — Early warning tier
+- `DLQMessageAgeHigh` (dlq-depth) — Stale message detection
+- `MCPCallErrorRateHigh` (mcp-latency) — Error rate detection
+- `KnowledgeStoreWriteLatencyHigh` (cache-degradation) — Write path latency
+- `ServiceAuthFailure` (security-anomaly) — mTLS rejection detection
+
+**Runbooks created:**
+- `runbooks/SDK-INTEGRATION.md`
+- `runbooks/dashboard-guide.md`
+- `runbooks/alert-response/LSP_DAEMON_RESTART.md`
+- `runbooks/alert-response/HIGH_QUERY_LATENCY.md`
+- `runbooks/alert-response/DLQ_DEPTH_EXCEEDED.md`
+- `runbooks/alert-response/ANALYSIS_BACKLOG.md`
+- `runbooks/alert-response/SECURITY_ANOMALY.md`
+
+**Status:** All 9 design-required alerts implemented. All 5 alert-response runbooks created. PagerDuty + Slack routing configured in `prometheus/prometheus-values.yaml`.
+
+---
+
+### M08-T07 — Logging Library (P0, Week 1) — ✅ DONE
+
+Delivered as part of M08-T04. See above for full artifact list.
+
+**Key compliance points:**
+- ✅ Mandatory fields enforced at creation time (both Node.js and Python)
+- ✅ `ECIPModule` type restricts to M01–M08
+- ✅ Trace context auto-injected via Pino mixin (Node.js) and structlog processor (Python)
+- ✅ Security events use dedicated emission path (separate from general logger)
+- ✅ PII hashing on user IDs (SHA-256, `u_` prefix)
+- ✅ Express middleware + Fastify plugin for automatic trace context propagation
+
+---
+
+### M08-T08 — Elasticsearch Security Pipeline (P1, Week 3–4) — ✅ DONE
+
+**Deliverables created:**
+- `elasticsearch/index-template.json` — ECS-compatible schema for `ecip-security-events-*` indices
+- `elasticsearch/ilm-policy.json` — Lifecycle: 90d hot → 1y warm → cold → delete at 2y
+- `elasticsearch/kibana-space.yaml` — Security team Kibana space + index pattern ConfigMap
+- `helm/templates/elasticsearch.yaml` — ConfigMap + post-install/upgrade Job for ILM/template/alias setup (**ES URL now templated** via `Values.elasticsearch.{protocol,host,port}`)
+
+**Status:** Complete. Schema matches the ECS format from security event helpers. ILM lifecycle matches design spec. Kibana space scoped with disabled non-security features. **Elasticsearch URL is no longer hardcoded** — fully configurable through Helm values (GAP-06 resolved).
+
+---
+
+### M08-T09 — TDD-002 Metrics (P2, Week 5–6) — ✅ DONE
+
+**Design requirements:**
+- HNSW rebuild duration metric in dashboards
+- Embedding migration progress metric in dashboards
+- `FilterAuthorizedRepos` latency metric + alert
+
+**Current state:**
+- ✅ `FilterAuthorizedReposLatency` alert — Implemented in `alerts/sla-latency.yaml`
+- ✅ `filter_authorized_repos_duration_ms` in recording rules
+- ✅ `hnsw_rebuild_duration_ms` panel — Present in `cache-performance.json` ("HNSW Rebuild Duration")
+- ✅ `embedding_migration_progress` panel — Present in `analysis-throughput.json` ("Embedding Migration Progress" gauge)
+
+**Status:** All three TDD-002 metrics are represented in dashboards and/or alerts. Validated with real M06 scrape targets configured.
+
+---
+
+### M08-T10 — CI Gate Enforcement (P1, Week 2) — ✅ DONE
+
+**Design requirements (§5 Integration Enforcement):**
+1. ESLint rule: ban `console.log` in production code paths (Node.js modules)
+2. Ruff rule: ban `print()` in production code paths (Python modules)
+3. CI check: `@ecip/observability` or `ecip-observability` in dependency list
+4. CI check: `initTracer()` called before server startup
+5. CI check: `createLogger()` used with mandatory fields in handlers
+6. `promtool check rules` on alert YAML files
+7. `grafana-dashboard-lint` on dashboard JSON files
+
+**Deliverables created:**
+- `ci/eslint-plugin-ecip/index.js` — Custom ESLint plugin with 3 rules: `no-console-log` (bans `console.*` in prod), `require-observability-import` (entry files must import `@ecip/observability`), `require-init-tracer`
+- `ci/eslint-plugin-ecip/package.json` — Plugin package metadata (`eslint-plugin-ecip` v1.0.0, peerDependencies: eslint >= 8.0.0)
+- `ci/ruff-shared.toml` — Shared Ruff config for Python modules (targets Python 3.11, enables `T20` flake8-print to ban `print()`, exempts test files/scripts)
+- `ci/check-observability-contract.js` — Node CLI tool validating module-level M08 contract (SDK dependency check, `initTracer()` call, `createLogger()` usage, absence of `console.log`/`print()`)
+- `ci/github-actions-observability-gate.yaml` — GitHub Actions workflow with 5 jobs: contract-check (per-module matrix), eslint-ecip (Node modules), ruff-check (Python modules), alert-validation (promtool), dashboard-lint
+- `scripts/lint-dashboards.js` — Dashboard JSON linter checking: valid JSON, required fields (title/uid/panels), schemaVersion ≥ 30, UID uniqueness, panel structure, datasource references, template variables
+
+**Current state:**
+- ✅ ESLint plugin with `no-console-log`, `require-observability-import`, `require-init-tracer` rules
+- ✅ Ruff config banning `print()` across Python modules
+- ✅ Contract checker validates SDK dependency + `initTracer()` + `createLogger()`
+- ✅ GitHub Actions workflow with 5 jobs, triggers on PRs touching module `src/` and observability config
+- ✅ `scripts/lint-dashboards.js` with 8 validation checks
+- ✅ `package.json` has `lint:dashboards` and `lint:alerts` scripts pointing to real implementations
+
+**Status:** Fully implemented. All 7 design requirements addressed. The CI gate is now load-bearing as specified in §11 finding 5.
+
+---
+
+## Gaps Identified
+
+### Resolved Gaps (8 of 12)
+
+| Gap | Description | Resolution |
+|-----|-------------|------------|
+| GAP-01 | Missing `security-events.test.ts` | ✅ `tests/security-events.test.ts` created — validates `emitAuthFailure()` and `emitRbacDenial()` for ECS format, PII hashing (no raw user_id), trace context, stderr routing, and logger provider isolation |
+| GAP-02 | No CI Gate Implementation | ✅ Fully implemented — ESLint plugin (3 rules), Ruff config (T20), contract checker, GitHub Actions workflow (5 jobs), dashboard linter |
+| GAP-03 | Missing `scripts/lint-dashboards.js` | ✅ Created with 8 validation checks (JSON validity, required fields, schemaVersion, UID uniqueness, panel structure, datasource refs, template vars) |
+| GAP-06 | Elasticsearch URL Hardcoded in Helm Job | ✅ `helm/templates/elasticsearch.yaml` now uses `Values.elasticsearch.{protocol,host,port}` — values defined in `helm/values.yaml` |
+| GAP-05 | `AnalysisBacklogCritical` metric name mismatch | ✅ Design doc updated: `event_bus_backlog_events` → `sum(kafka_consumergroup_lag{group=~"ecip-analysis.*"})`. Implementation was already correct. |
+| GAP-07 | Grafana Admin Password Hardcoded | ✅ `helm/templates/grafana-secret.yaml` created. `values.yaml` references `existingSecret: ecip-grafana-admin`, `values.prod.yaml` references `ecip-grafana-admin-prod` |
+| GAP-11 | LSP Daemon OOM Dashboard Panel Query Issue | ✅ Both `dashboards/lsp-daemon-health.json` and `alerts/lsp-daemon.yaml` updated to use `kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}` |
+| GAP-12 | OD-01 Log Aggregation Backend Unresolved | ✅ Loki chosen — Helm dependency added (v5.47.0), OTel collector has `loki` exporter + `logs` pipeline, Grafana datasource provisioned via `helm/templates/loki.yaml`, Tempo tracesToLogs→Loki |
+
+---
+
+### Remaining Gaps (4 of 12)
+
+### GAP-04: Compile-Time Enforcement Test Incomplete (Severity: Medium)
+
+`tests/log-schema-validation.test.ts` checks that `logger.ts` defines `ECIPLoggerContext` with mandatory fields, but does NOT compile a deliberately broken test file and assert a non-zero exit code — as specified in the design doc §8. The current test only reads source file text.
+
+### ~~GAP-05: `AnalysisBacklogCritical` Metric Name Mismatch~~ — ✅ RESOLVED
+
+The design doc originally specified `event_bus_backlog_events > 1000` but the implementation uses `sum(kafka_consumergroup_lag{group=~"ecip-analysis.*"}) > 1000` — the real metric exported by kafka-exporter. **Design doc updated** to reflect the correct metric. No silent divergence remains.
+
+### GAP-08: No PodDisruptionBudgets (Severity: Low)
+
+No PDB defined for the OTel Collector DaemonSet or any other workload template. Could result in observability gaps during cluster maintenance.
+
+### GAP-09: No NetworkPolicy Templates (Severity: Low)
+
+No NetworkPolicy resources in the Helm chart to restrict observability stack traffic within the monitoring namespace.
+
+### GAP-10: `chaos/redis-node-failure.sh` Quality Issue (Severity: Medium)
+
+The file contains ~195 lines of corrupted/reversed content at the top (lines appear in reverse order, e.g., `#!/bin/bash#!/usr/bin/env bash`), followed by a clean implementation starting around line 196. The garbled first section is non-functional and needs to be removed.
+
+---
+
+## Enhancements
+
+### Completed Enhancements (6 of 10)
+
+| # | Enhancement | Status | Resolution |
+|---|------------|--------|------------|
+| ENH-01 | Add `security-events.test.ts` | ✅ Done | `tests/security-events.test.ts` — ECS format, PII hashing, stderr isolation, logger provider routing |
+| ENH-02 | Implement CI Gate (M08-T10) | ✅ Done | ESLint plugin, Ruff config, contract checker, GitHub Actions workflow, dashboard linter |
+| ENH-03 | Create `scripts/lint-dashboards.js` | ✅ Done | 8 validation checks (JSON, required fields, schemaVersion, UIDs, panels, datasources, vars) |
+| ENH-05 | Template Elasticsearch URL in Helm | ✅ Done | `Values.elasticsearch.{protocol,host,port}` with defaults |
+| ENH-06 | Externalize Grafana Admin Credentials | ✅ Done | K8s Secret template + `existingSecret` in both dev and prod values |
+| ENH-10 | Fix OOM Panel PromQL | ✅ Done | Dashboard + alert both use `kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}` |
+
+### Remaining Enhancements (4 of 10)
+
+### ENH-04: Strengthen Compile-Time Test
+Add a test case that attempts `tsc --noEmit` on a deliberately broken TypeScript file missing mandatory `ECIPLoggerContext` fields, asserting a non-zero exit code. (Related to GAP-04)
+
+### ENH-07: Add PodDisruptionBudgets
+Add PDB templates for OTel Collector, Tempo ingester/distributor, and Prometheus to protect against observability blackouts during node maintenance. (Related to GAP-08)
+
+### ENH-08: Fix `chaos/redis-node-failure.sh`
+Remove the ~195 lines of corrupted/reversed content at the top of the chaos script. Retain only the clean implementation starting around line 196. (Related to GAP-10)
+
+### ENH-09: Add Template Variables to Remaining Dashboards
+Add `$repo` or `$module` template variables to LSP Daemon Health, Event Bus DLQ, Cross-Repo Fan-out, and Security Events dashboards for consistent per-repo filtering.
+
+---
+
+## Open Design Decisions
+
+| ID | Decision | Impact | Status |
+|----|----------|--------|--------|
+| OD-01 | General log aggregation backend (Elasticsearch / Loki / Cloud) | Affects Helm chart, Fluent Bit config, SDK log exporter | ✅ **Resolved** — Loki chosen. Helm dependency added (v5.47.0), OTel collector exports to Loki, Grafana datasource provisioned, Tempo tracesToLogs→Loki |
+| OD-02 | Collector resource limits (512Mi estimated, needs load testing) | May require sampling rate reduction (5% → 2%) | **Unresolved** — revisit at Week 8 with real traffic |
+| OD-03 | Dashboard ownership post-Week 28 | Module teams vs. Platform team for panel maintenance | **Unresolved** — process decision needed |
+
+---
+
+## File Inventory (83 files)
+
+| Directory | Files | Description |
+|-----------|-------|-------------|
+| `collector/` | 3 | OTel Collector config, DaemonSet manifest, sampling config |
+| `dashboards/` | 9 | 8 Grafana JSON dashboards + provisioning config |
+| `alerts/` | 7 | Alert rule YAML files (17 alert rules total) |
+| `logging-lib/nodejs/` | 9 | `@ecip/observability` npm package (src + tests) |
+| `logging-lib/python/` | 6 | `ecip-observability` Python package (src + tests) |
+| `prometheus/` | 3 | Helm values, scrape configs, recording rules |
+| `tempo/` | 2 | Helm values, Grafana datasource provisioning |
+| `elasticsearch/` | 3 | Index template, ILM policy, Kibana space config |
+| `helm/` | 12 | Umbrella chart + **8 templates** + 2 values files *(+3 new: grafana-secret.yaml, loki.yaml, +loki dependency in Chart.yaml)* |
+| `ci/` | 5 | **NEW** — ESLint plugin (2 files), Ruff config, contract checker, GitHub Actions workflow |
+| `scripts/` | 1 | **NEW** — `lint-dashboards.js` dashboard JSON linter |
+| `runbooks/` | 7 | SDK integration guide, dashboard guide, 5 alert-response runbooks |
+| `tests/` | 5 | Metric validation, alert config, log schema, OTel pipeline integration, **security events** *(+1 new)* |
+| `chaos/` | 3 | LSP kill, Redis failure, Kafka broker restart |
+| Root | 4 | `package.json`, `tsconfig.json`, `vitest.config.ts`, `vitest.integration.config.ts` |
+| Docs | 5 | `CLAUDE.md`, `README.md`, `docs/M08-Observability-Design.md`, `docs/module-documentation.md`, `docs/PROGRESS.md` |
+| **Total** | **83** | *(+10 files since last update)* |
+
+---
+
+## Risk Status (from §10)
+
+| Risk | Current Mitigation Status |
+|------|--------------------------|
+| Module teams skip SDK integration | ✅ **Fully mitigated** — SDK and guide exist, CI gate (M08-T10) fully implemented with GitHub Actions workflow enforcing SDK dependency, `initTracer()`, `createLogger()`, and `console.log`/`print()` bans. |
+| Collector config error drops traces | ✅ **Mitigated** — `otel-pipeline-integration.test.ts` with Testcontainers catches pipeline misconfigurations in CI. |
+| High-cardinality Prometheus labels | ✅ **Mitigated** — `metric-label-validation.test.ts` prohibits `user_id`, `trace_id`, `sha` as labels. |
+| Tempo storage costs | ✅ **Mitigated** — Tail-based sampling at 5% default configured. Review at Week 8 with live data. |
+| Security pipeline mixes with general logs | ✅ **Mitigated** — Separate OTel logger provider, dedicated Collector pipeline (`logs/security` → ES), general logs → Loki. |
+| M08 is itself unobservable | ✅ **Mitigated** — Collector exposes metrics at :8888, self-scrape configured. |
+| General application logs have no backend | ✅ **Mitigated** — Loki deployed as log aggregation backend (OD-01 resolved). OTel collector `logs` pipeline exports to Loki. |
+
+---
+
+## Recommended Next Actions (Priority Order)
+
+1. **ENH-08** — Fix `chaos/redis-node-failure.sh` (remove ~195 lines of corrupted content) — **Quick win, 5 min**
+2. **ENH-04 / GAP-04** — Strengthen compile-time enforcement test (`tsc --noEmit` on broken file) — **Medium, 30 min**
+3. **ENH-09** — Add template variables to remaining dashboards — **Medium, 1 hr**
+4. **ENH-07 / GAP-08** — Add PodDisruptionBudgets to Helm templates — **Medium, 30 min**
+5. **GAP-09** — Add NetworkPolicy templates — **Low priority, nice-to-have**
+6. ~~**GAP-05**~~ — ✅ Resolved — Design doc updated to match implementation
+
+---
+
+*Generated from code audit against M08-Observability-Design.md Rev 1.1*

package/docs/module-documentation.md

@@ -0,0 +1,64 @@
+# M08 — Observability Stack: Module Documentation
+
+**Document ID:** ECIP-M08-DOC · **Status:** Starter Draft
+
+---
+
+## Overview
+
+The Observability Stack provides the monitoring, tracing, and alerting infrastructure for the entire ECIP platform. It is a pure infrastructure module — it does not contain application business logic.
+
+All other modules instrument themselves using the OpenTelemetry SDK. M08 collects, stores, and visualizes that telemetry.
+
+---
+
+## Instrumentation Strategy
+
+**Auto-instrumentation first.** The OTel auto-instrumentation libraries handle HTTP, gRPC, Kafka, and database spans automatically for Node.js, Go, and Rust. Modules do not write custom span code unless they need custom attributes.
+
+**Custom metrics** use the OTel Metrics API (`meter.createHistogram`, `meter.createCounter`) — never the Prometheus client directly.
+
+---
+
+## Required Dashboards
+
+Each module has a corresponding Grafana dashboard. All dashboard JSON definitions live in `dashboards/` and are version-controlled here.
+
+**Platform SLA Dashboard** is the top-level view and must show:
+- End-to-end query latency p50/p95/p99
+- Analysis pipeline throughput and error rate  
+- Uptime per module
+- Active alerts
+
+---
+
+## Alert Runbook Index
+
+| Alert | Severity | Runbook |
+|-------|----------|---------|
+| `QueryLatencyHigh` | Critical | [runbooks/query-latency-high.md](./runbooks/query-latency-high.md) |
+| `AnalysisBacklogGrowing` | Warning | [runbooks/analysis-backlog.md](./runbooks/analysis-backlog.md) |
+| `KafkaDLQDepthHigh` | Warning | [runbooks/dlq-depth.md](./runbooks/dlq-depth.md) |
+| `MCPServerDown` | Critical | [runbooks/mcp-server-down.md](./runbooks/mcp-server-down.md) |
+| `CacheHitRateLow` | Warning | [runbooks/cache-hit-rate.md](./runbooks/cache-hit-rate.md) |
+
+---
+
+## Production Readiness Checklist
+
+Before each production release:
+- [ ] All SLA dashboards show green for 48h in staging
+- [ ] Chaos tests passed (LSP daemon kill, Redis failure, Kafka restart)
+- [ ] 200 CCU load test completed: p95 < 1.5s
+- [ ] All critical alerts have runbooks
+- [ ] On-call rotation configured in PagerDuty
+
+---
+
+## Known Gaps / TODOs
+
+- [ ] Grafana dashboards not yet created (just schemas defined)
+- [ ] Loki vs ELK log aggregation decision pending
+- [ ] Chaos test scripts not yet written
+- [ ] PagerDuty integration not yet configured
+- [ ] DR test procedure not yet documented

package/elasticsearch/ilm-policy.json

@@ -0,0 +1,57 @@
+{
+  "policy": {
+    "description": "ECIP security event ILM policy: 90d hot → 1y warm → cold → delete",
+    "phases": {
+      "hot": {
+        "min_age": "0ms",
+        "actions": {
+          "rollover": {
+            "max_primary_shard_size": "50gb",
+            "max_age": "7d"
+          },
+          "set_priority": {
+            "priority": 100
+          }
+        }
+      },
+      "warm": {
+        "min_age": "90d",
+        "actions": {
+          "shrink": {
+            "number_of_shards": 1
+          },
+          "forcemerge": {
+            "max_num_segments": 1
+          },
+          "set_priority": {
+            "priority": 50
+          },
+          "allocate": {
+            "require": {
+              "data": "warm"
+            }
+          }
+        }
+      },
+      "cold": {
+        "min_age": "365d",
+        "actions": {
+          "set_priority": {
+            "priority": 0
+          },
+          "allocate": {
+            "require": {
+              "data": "cold"
+            }
+          }
+        }
+      },
+      "delete": {
+        "min_age": "730d",
+        "actions": {
+          "delete": {}
+        }
+      }
+    }
+  }
+}

package/elasticsearch/index-template.json

@@ -0,0 +1,62 @@
+{
+  "index_patterns": ["ecip-security-events-*"],
+  "template": {
+    "settings": {
+      "number_of_shards": 2,
+      "number_of_replicas": 1,
+      "index.lifecycle.name": "ecip-security-events-ilm",
+      "index.lifecycle.rollover_alias": "ecip-security-events"
+    },
+    "mappings": {
+      "properties": {
+        "@timestamp": {
+          "type": "date"
+        },
+        "event.kind": {
+          "type": "keyword"
+        },
+        "event.category": {
+          "type": "keyword"
+        },
+        "event.type": {
+          "type": "keyword"
+        },
+        "event.outcome": {
+          "type": "keyword"
+        },
+        "trace.id": {
+          "type": "keyword"
+        },
+        "user.id": {
+          "type": "keyword"
+        },
+        "source.ip": {
+          "type": "ip"
+        },
+        "resource": {
+          "type": "keyword"
+        },
+        "action": {
+          "type": "keyword"
+        },
+        "reason": {
+          "type": "keyword"
+        },
+        "module": {
+          "type": "keyword"
+        },
+        "metadata": {
+          "type": "object",
+          "enabled": true,
+          "dynamic": true
+        }
+      }
+    }
+  },
+  "priority": 200,
+  "composed_of": [],
+  "version": 1,
+  "_meta": {
+    "description": "ECIP security event index template (NFR-SEC-007). Governs auth failures, RBAC denials, and service auth events routed from the OTel Collector security pipeline."
+  }
+}

package/elasticsearch/kibana-space.yaml

@@ -0,0 +1,53 @@
+# =============================================================================
+# ECIP M08 — Kibana Space Configuration (Security Team)
+# =============================================================================
+# The security team owns this space. M08 provides the index;
+# the security team manages all Kibana queries/dashboards/SIEM rules.
+# =============================================================================
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: kibana-security-space
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: elasticsearch
+    app.kubernetes.io/component: kibana-config
+    ecip.module: M08
+data:
+  security-space.json: |
+    {
+      "id": "ecip-security",
+      "name": "ECIP Security",
+      "description": "Security events from the ECIP platform — auth failures, RBAC denials, service auth events",
+      "color": "#DD0000",
+      "initials": "ES",
+      "disabledFeatures": [
+        "canvas",
+        "maps",
+        "ml",
+        "monitoring",
+        "apm",
+        "uptime",
+        "observability",
+        "fleet"
+      ]
+    }
+  index-pattern.json: |
+    {
+      "title": "ecip-security-events-*",
+      "timeFieldName": "@timestamp",
+      "fields": [
+        "@timestamp",
+        "event.kind",
+        "event.category",
+        "event.type",
+        "event.outcome",
+        "trace.id",
+        "user.id",
+        "source.ip",
+        "resource",
+        "action",
+        "reason",
+        "module"
+      ]
+    }

package/helm/Chart.yaml

@@ -0,0 +1,30 @@
+apiVersion: v2
+name: ecip-observability-stack
+description: ECIP M08 — Umbrella Helm chart deploying the full observability stack
+type: application
+version: 1.0.0
+appVersion: "1.0.0"
+maintainers:
+  - name: ECIP Platform Team
+    email: platform@ecip.internal
+
+dependencies:
+  - name: kube-prometheus-stack
+    version: "56.6.2"
+    repository: https://prometheus-community.github.io/helm-charts
+    condition: prometheus.enabled
+
+  - name: tempo-distributed
+    version: "1.8.0"
+    repository: https://grafana.github.io/helm-charts
+    condition: tempo.enabled
+
+  - name: elasticsearch
+    version: "8.5.1"
+    repository: https://helm.elastic.co
+    condition: elasticsearch.enabled
+
+  - name: loki
+    version: "5.47.0"
+    repository: https://grafana.github.io/helm-charts
+    condition: loki.enabled

package/helm/templates/configmaps.yaml

@@ -0,0 +1,25 @@
+{{- /*
+ECIP M08 — ConfigMaps for collector, dashboards, and alert provisioning
+*/ -}}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: otel-collector-config
+  namespace: {{ .Values.namespace | default "monitoring" }}
+  labels:
+    app.kubernetes.io/name: otel-collector
+    app: ecip
+data:
+  otel-collector-config.yaml: |-
+    {{ .Files.Get "collector/otel-collector-config.yaml" | nindent 4 }}
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboard-provisioning
+  namespace: {{ .Values.namespace | default "monitoring" }}
+  labels:
+    app: ecip
+data:
+  grafana-dashboards.yaml: |-
+    {{ .Files.Get "dashboards/_provisioning/grafana-dashboards.yaml" | nindent 4 }}