ecip-observability-stack 1.0.0

package/runbooks/alert-response/ANALYSIS_BACKLOG.md

@@ -0,0 +1,128 @@
+# Alert Runbook: Analysis Backlog
+
+> **Alert Name:** `AnalysisBacklogCritical` / `AnalysisBacklogWarning`  
+> **Severity:** Critical / Warning  
+> **Module:** M02 — Analysis Engine
+
+---
+
+## Alert Definitions
+
+```yaml
+# Warning
+expr: ecip_analysis_backlog_size{module="M02"} > 500
+for: 15m
+
+# Critical
+expr: ecip_analysis_backlog_size{module="M02"} > 1000
+for: 10m
+```
+
+---
+
+## Impact
+
+- Code analysis jobs are queuing faster than they can be processed
+- Repository updates are delayed
+- Users see stale code intelligence results
+- If backlog grows indefinitely: eventual disk/memory pressure
+
+---
+
+## Triage Steps
+
+### 1. Check Analysis Throughput Dashboard
+
+Open **Grafana → ECIP → Analysis Throughput**. Look for:
+- Current backlog size and trend
+- Analysis throughput (analyses per minute) — has it dropped?
+- Error rate — are analyses failing?
+- Active analyses — are workers busy or idle?
+
+### 2. Check LSP Daemon Health
+
+Analysis throughput depends on healthy LSP daemons. Open **Grafana → ECIP → LSP Daemon Health**.
+- If daemon restarts are elevated: see [LSP_DAEMON_RESTART runbook](./LSP_DAEMON_RESTART.md)
+- If OOM kills are occurring: daemons are crashing before completing work
+
+### 3. Check Event Bus
+
+Is the backlog caused by a flood of events?
+
+```bash
+# Check Kafka lag for analysis consumer group
+kubectl exec -n ecip kafka-0 -- kafka-consumer-groups.sh \
+  --bootstrap-server localhost:9092 \
+  --group ecip-analysis-consumers \
+  --describe
+```
+
+### 4. Check for Large Repository Events
+
+```bash
+kubectl logs -n ecip deployment/ecip-analysis-engine --tail=200 \
+  | grep -E "files_to_analyze|large_repo|backlog"
+```
+
+A single large repository push can flood the queue.
+
+---
+
+## Common Causes & Fixes
+
+### LSP Daemon Failures
+
+**Symptoms:** `LSPDaemonRestartRate` also firing, analysis throughput dropped.
+
+**Fix:**
+1. Resolve LSP daemon issues first (see [LSP_DAEMON_RESTART.md](./LSP_DAEMON_RESTART.md))
+2. Backlog will drain naturally once daemons stabilize
+
+### Large Repository Push
+
+**Symptoms:** Sudden backlog spike correlating with a specific large repo event.
+
+**Fix:**
+1. This may be normal — a large monorepo push creates many analysis jobs
+2. Monitor: backlog should drain at normal throughput rate
+3. If one repo is blocking everything: consider priority queuing
+
+### Insufficient Worker Capacity
+
+**Symptoms:** Workers are all busy, throughput is at capacity, but event rate exceeds processing rate.
+
+**Fix:**
+1. Scale workers: `kubectl scale deployment -n ecip ecip-analysis-engine --replicas=N`
+2. Check HPA: is it already at max replicas?
+3. If at max: increase HPA maxReplicas or add nodes
+
+### Downstream Failure (Knowledge Store)
+
+**Symptoms:** Analysis completes but writing results to M03 fails, causing retries.
+
+**Fix:**
+1. Check M03 Knowledge Store health
+2. Check write latency on Cache Performance dashboard
+3. Fix M03 issue → retries succeed → backlog drains
+
+---
+
+## Escalation
+
+- **Warning (> 500, 15min):** M02 on-call investigates
+- **Critical (> 1000, 10min):** Page M02 on-call via PagerDuty
+- **If backlog growing > 100/min:** Consider pausing non-critical analysis (PR branches)
+- **If correlated with LSP issues:** Also engage Platform team
+
+---
+
+## Related Alerts
+
+- `LSPDaemonRestartRate` — Often the root cause
+- `LSPDaemonOOMKill` — Memory pressure variant
+- `DLQDepthExceeded` — If event processing is part of the failure chain
+- `KnowledgeStoreWriteLatencyHigh` — Downstream bottleneck
+
+---
+
+*Last updated: March 2026 · Platform Team*

package/runbooks/alert-response/DLQ_DEPTH_EXCEEDED.md

@@ -0,0 +1,150 @@
+# Alert Runbook: DLQ Depth Exceeded
+
+> **Alert Name:** `DLQDepthExceeded` (critical) / `DLQDepthWarning` (warning)  
+> **Severity:** Critical / Warning  
+> **Module:** M07 — Event Bus
+
+---
+
+## Alert Definitions
+
+```yaml
+# Warning
+expr: ecip_dlq_depth{module="M07"} > 10
+for: 10m
+
+# Critical
+expr: ecip_dlq_depth{module="M07"} > 100
+for: 5m
+```
+
+Also related:
+```yaml
+# DLQ Message Age
+expr: ecip_dlq_oldest_message_age_seconds{module="M07"} > 3600
+for: 5m
+```
+
+---
+
+## Impact
+
+- Failed events are not being processed
+- Repositories may not be indexed or updated
+- Stale data in M03 Knowledge Store
+- Downstream: queries return outdated results
+
+---
+
+## Triage Steps
+
+### 1. Check Event Bus DLQ Dashboard
+
+Open **Grafana → ECIP → Event Bus DLQ**. Look for:
+- Current DLQ depth and trend
+- Oldest message age (> 1hr is concerning)
+- DLQ ingestion rate (how fast are messages arriving?)
+- Processing lag for main topics
+
+### 2. Check Kafka Consumer Lag
+
+```bash
+kubectl exec -n ecip kafka-0 -- kafka-consumer-groups.sh \
+  --bootstrap-server localhost:9092 \
+  --group ecip-event-processors \
+  --describe
+```
+
+### 3. Inspect DLQ Messages
+
+```bash
+kubectl exec -n ecip kafka-0 -- kafka-console-consumer.sh \
+  --bootstrap-server localhost:9092 \
+  --topic ecip.dlq \
+  --from-beginning \
+  --max-messages 5 \
+  --property print.headers=true
+```
+
+Look at the `error` header to determine why messages were sent to DLQ.
+
+### 4. Check Event Bus Service Logs
+
+```bash
+kubectl logs -n ecip deployment/ecip-event-bus --tail=200 \
+  | grep -E "error|dlq|dead.letter|retry.exhausted"
+```
+
+---
+
+## Common Causes & Fixes
+
+### Schema Validation Failure
+
+**Symptoms:** DLQ messages have `error: schema_validation_failed` header.
+
+**Fix:**
+1. Check if a recent schema change was deployed without updating consumers
+2. Verify Schema Registry has the correct version
+3. If schema was updated: update consumers and replay DLQ messages
+
+### Downstream Service Unavailable
+
+**Symptoms:** DLQ messages have `error: downstream_timeout` or `connection_refused`.
+
+**Fix:**
+1. Check the target service health (M02 Analysis Engine, M03 Knowledge Store)
+2. If the service is down: fix the service, then replay DLQ
+3. If the service is overloaded: scale it up, then replay
+
+### Poison Messages
+
+**Symptoms:** Same message(s) repeatedly failing, growing DLQ depth slowly.
+
+**Fix:**
+1. Inspect the failing messages for malformed data
+2. If a specific repository event is broken, manually acknowledge and skip it
+3. File a bug for the source that produced the malformed event
+
+### Kafka Broker Issues
+
+**Symptoms:** Multiple consumer groups affected, not just ECIP.
+
+**Fix:**
+1. Check Kafka broker health: `kubectl get pods -n kafka`
+2. Check broker disk usage and partition balance
+3. Escalate to Infrastructure/Kafka team
+
+---
+
+## Replaying DLQ Messages
+
+After fixing the root cause, replay DLQ messages:
+
+```bash
+# Using the M07 replay tool
+kubectl exec -n ecip deployment/ecip-event-bus -- \
+  node scripts/replay-dlq.js --topic ecip.dlq --batch-size 50
+```
+
+Monitor the DLQ depth dashboard to confirm depth decreasing.
+
+---
+
+## Escalation
+
+- **Warning (> 10 messages):** Investigate within 30 minutes
+- **Critical (> 100 messages):** Page M07 on-call
+- **If age > 4hr:** Escalate to P1 — data staleness affecting users
+- **If Kafka infrastructure issue:** Page Infrastructure on-call
+
+---
+
+## Related Alerts
+
+- `DLQMessageAgeHigh` — Oldest message > 1 hour
+- `AnalysisBacklogCritical` — May correlate if M02 events failing
+
+---
+
+*Last updated: March 2026 · Platform Team*

package/runbooks/alert-response/HIGH_QUERY_LATENCY.md

@@ -0,0 +1,134 @@
+# Alert Runbook: High Query Latency
+
+> **Alert Name:** `QueryLatencySLABreach`  
+> **Severity:** Critical  
+> **Module:** M04 — Query Service  
+> **SLA:** p95 < 2000ms
+
+---
+
+## Alert Definition
+
+```yaml
+expr: >
+  histogram_quantile(0.95,
+    sum(rate(ecip_query_duration_ms_bucket{module="M04"}[5m])) by (le)
+  ) > 1500
+for: 5m
+```
+
+Fires when p95 query latency exceeds 1500ms for 5 consecutive minutes. Threshold is set at 1500ms (below the 2000ms SLA) to allow time for investigation before SLA breach.
+
+---
+
+## Impact
+
+- Users experience slow code intelligence responses
+- IDE integrations may timeout
+- SLA breach if not resolved within the `for` window
+
+---
+
+## Triage Steps
+
+### 1. Check the Query Latency Dashboard
+
+Open **Grafana → ECIP → Query Latency**. Look for:
+- Is p95 above 1500ms? Which percentile is affected?
+- Is this a gradual trend or a sudden spike?
+- Is the request rate higher than normal? (capacity issue vs. performance regression)
+
+### 2. Check Downstream Dependencies
+
+Query latency is often caused by slow dependencies:
+
+| Dependency | Check | Dashboard |
+|---|---|---|
+| M03 Knowledge Store | Cache hit rate, write latency | Cache Performance |
+| M05 MCP Server | MCP call latency | MCP Call Graph |
+| M06 Registry | RBAC check latency | (Prometheus direct) |
+
+```promql
+# Check cache hit rate
+ecip_cache_hit_rate{module="M03"}
+
+# Check MCP call latency
+histogram_quantile(0.95, sum(rate(ecip_mcp_call_duration_ms_bucket[5m])) by (le))
+
+# Check FilterAuthorizedRepos latency
+histogram_quantile(0.95, sum(rate(ecip_filter_authorized_repos_duration_ms_bucket[5m])) by (le))
+```
+
+### 3. Check Traces in Tempo
+
+1. Open **Grafana → Explore → Tempo**
+2. Search for traces with `service.name = ecip-query-service` and duration > 1500ms
+3. Look at the span waterfall — which span is the bottleneck?
+
+### 4. Check Pod Resources
+
+```bash
+kubectl top pods -n ecip -l app=ecip-query-service
+```
+
+If CPU or memory is near limits, this is a capacity issue.
+
+---
+
+## Common Causes & Fixes
+
+### Cache Miss Storm
+
+**Symptoms:** Cache hit rate drops below 85%, query latency rises proportionally.
+
+**Fix:**
+1. Check if cache was recently flushed (deployment, Redis restart)
+2. Monitor cache warm-up — latency should recover as cache repopulates
+3. If persistent: check for cache eviction due to memory pressure
+
+### MCP Fan-out Explosion
+
+**Symptoms:** Cross-repo queries with high fan-out depth (> 5 repos).
+
+**Fix:**
+1. Check `cross-repo-fanout` dashboard for depth distribution
+2. If specific queries cause excessive fan-out, investigate query patterns
+3. Consider circuit breaker tuning in M04
+
+### FilterAuthorizedRepos Slow
+
+**Symptoms:** RBAC filtering taking > 20ms (NFR-SEC-011 SLA breach).
+
+**Fix:**
+1. Check M06 Registry Service health
+2. Verify RBAC cache is populated
+3. Escalate to M06 team if their latency is the bottleneck
+
+### Capacity Issue
+
+**Symptoms:** Request rate is significantly higher than baseline, CPU/memory near limits.
+
+**Fix:**
+1. Scale horizontally: `kubectl scale deployment -n ecip ecip-query-service --replicas=N`
+2. Review HPA configuration for M04
+
+---
+
+## Escalation
+
+- **Critical:** Pages M04 on-call via PagerDuty automatically
+- **If caused by M03 cache:** Escalate to M03 team
+- **If caused by M06 RBAC:** Escalate to M06 team
+- **If capacity issue:** Engage Platform team for scaling
+
+---
+
+## Related Alerts
+
+- `CacheHitRateDegraded` — Often precedes query latency spikes
+- `FilterAuthorizedReposLatency` — RBAC check SLA
+- `MCPCallLatencyWarn` — MCP tool call slowness
+
+---
+
+*Last updated: March 2026 · Platform Team*

package/runbooks/alert-response/LSP_DAEMON_RESTART.md

@@ -0,0 +1,118 @@
+# Alert Runbook: LSP Daemon Restart Rate
+
+> **Alert Name:** `LSPDaemonRestartRate`  
+> **Severity:** Warning  
+> **Module:** M02 — Analysis Engine  
+> **for:** 0m (fires immediately — M02-specific tuning)
+
+---
+
+## Alert Definition
+
+```yaml
+expr: rate(ecip_lsp_daemon_restarts_total{module="M02"}[5m]) > 3
+for: 0m
+```
+
+Fires when LSP daemon restarts exceed 3 per 5-minute window. The `for: 0m` is intentional — LSP daemon instability cascades immediately to analysis failures and should be investigated without delay.
+
+---
+
+## Impact
+
+- Active code analyses may fail or produce incomplete results
+- Analysis backlog grows while daemons restart
+- Downstream: M04 queries may return stale data if re-indexing is blocked
+
+---
+
+## Triage Steps
+
+### 1. Check the LSP Daemon Health Dashboard
+
+Open **Grafana → ECIP → LSP Daemon Health**. Look for:
+- Restart rate spike pattern (steady increase vs. sudden burst)
+- OOM kill correlation (if OOM kills are rising, go to §OOM below)
+- Memory trend before restarts
+
+### 2. Check Pod Events
+
+```bash
+kubectl get events -n ecip --field-selector involvedObject.name=ecip-analysis-engine \
+  --sort-by='.lastTimestamp' | head -20
+```
+
+Look for:
+- `OOMKilled` — memory limit exceeded
+- `CrashLoopBackOff` — repeated crash-restart cycles
+- `Unhealthy` — failed health checks
+
+### 3. Check Pod Logs
+
+```bash
+kubectl logs -n ecip deployment/ecip-analysis-engine -c lsp-daemon --tail=200 \
+  | grep -E "error|fatal|panic|OOM"
+```
+
+### 4. Check Recent Deployments
+
+```bash
+kubectl rollout history -n ecip deployment/ecip-analysis-engine
+```
+
+If a recent deploy correlates with the restart spike → consider rollback.
+
+---
+
+## Common Causes & Fixes
+
+### OOM Kill
+
+**Symptoms:** `LSPDaemonOOMKill` alert also firing, `OOMKilled` in pod events.
+
+**Fix:**
+1. Check if a specific file type is causing excessive memory usage
+2. If memory usage is generally trending up, increase the memory limit:
+   ```bash
+   kubectl set resources -n ecip deployment/ecip-analysis-engine \
+     -c lsp-daemon --limits=memory=2Gi
+   ```
+3. Long-term: file an issue for the M02 team to investigate the memory leak
+
+### Large File Processing
+
+**Symptoms:** Restarts correlate with analysis of specific large repositories.
+
+**Fix:**
+1. Check analysis queue for unusually large repos
+2. Consider adding file-size limits in M02 configuration
+3. Check if `max_file_size_bytes` is properly set in M02 config
+
+### Language Server Crash
+
+**Symptoms:** Clean exit codes (no OOM), restarts correlate with specific file types.
+
+**Fix:**
+1. Check which language server is crashing (TypeScript, Python, Go, etc.)
+2. Check for known issues in the language server version
+3. Update the language server image if a fix is available
+
+---
+
+## Escalation
+
+- **If restarts > 10/5min:** Page the M02 on-call engineer
+- **If associated with `AnalysisBacklogCritical`:** Escalate to P1
+- **If OOM and no obvious fix:** Engage Platform team for memory profiling
+
+---
+
+## Related Alerts
+
+- `LSPDaemonOOMKill` — Fires on OOM kill events specifically
+- `AnalysisBacklogCritical` — May fire as a downstream effect
+- `AnalysisBacklogWarning` — Early warning of backlog buildup
+
+---
+
+*Last updated: March 2026 · Platform Team*

package/runbooks/alert-response/SECURITY_ANOMALY.md

@@ -0,0 +1,160 @@
+# Alert Runbook: Security Anomaly
+
+> **Alert Names:** `SecurityAuthBurst`, `SecurityRBACDenialBurst`, `ServiceAuthFailure`  
+> **Severity:** Critical  
+> **Module:** All (Security cross-cutting)  
+> **Notification:** Slack #ecip-security + PagerDuty
+
+---
+
+## Alert Definitions
+
+```yaml
+# Auth burst — possible brute force
+SecurityAuthBurst:
+  expr: sum(rate(ecip_auth_failures_total[5m])) > 10
+  for: 0m  # Fires immediately
+
+# RBAC denial burst — possible privilege escalation attempt
+SecurityRBACDenialBurst:
+  expr: sum(rate(ecip_rbac_denials_total[5m])) > 20
+  for: 0m
+
+# Service auth failure — inter-service mTLS/token issues
+ServiceAuthFailure:
+  expr: rate(ecip_service_auth_failures_total[5m]) > 0
+  for: 2m
+```
+
+All security alerts fire to the `#ecip-security` Slack channel **and** PagerDuty.
+
+---
+
+## Impact
+
+- **Auth burst:** Possible credential stuffing or brute-force attack
+- **RBAC denial burst:** Possible unauthorized access attempt or misconfigured permissions
+- **Service auth failure:** Inter-service communication broken (mTLS cert expired, token invalid)
+
+---
+
+## Triage Steps
+
+### 1. Check Security Events Dashboard
+
+Open **Grafana → ECIP → Security Events**. This dashboard queries **Elasticsearch** (not Prometheus).
+
+Look for:
+- Auth failure rate and source IP distribution
+- RBAC denial patterns — same user? same resource? same action?
+- Geographic anomalies in auth failures
+
+### 2. Query Elasticsearch Directly
+
+```bash
+# Recent auth failures
+curl -s "http://elasticsearch.monitoring:9200/ecip-security-events/_search" \
+  -H "Content-Type: application/json" -d '{
+  "query": { "bool": { "must": [
+    { "term": { "event.type": "authentication_failure" }},
+    { "range": { "@timestamp": { "gte": "now-30m" }}}
+  ]}},
+  "sort": [{ "@timestamp": "desc" }],
+  "size": 20
+}'
+
+# Recent RBAC denials
+curl -s "http://elasticsearch.monitoring:9200/ecip-security-events/_search" \
+  -H "Content-Type: application/json" -d '{
+  "query": { "bool": { "must": [
+    { "term": { "event.type": "rbac_denial" }},
+    { "range": { "@timestamp": { "gte": "now-30m" }}}
+  ]}},
+  "sort": [{ "@timestamp": "desc" }],
+  "size": 20
+}'
+```
+
+### 3. Identify Patterns
+
+For auth failures:
+- **Same source IP:** Possible brute force → consider IP block
+- **Same user (hashed):** Possible credential stuffing → force password reset
+- **Distributed IPs, same user:** Possible botnet → escalate to security team
+
+For RBAC denials:
+- **Same user, various resources:** User recently lost permissions → check with access management
+- **Various users, same resource:** Resource permissions recently changed → check recent RBAC config changes
+- **Single user, escalating actions:** Possible privilege escalation → escalate immediately
+
+---
+
+## Response Actions
+
+### SecurityAuthBurst
+
+1. **If concentrated from single IP:**
+   ```bash
+   # Check source IP in recent events
+   # If malicious, block at WAF/API Gateway level
+   ```
+2. **If distributed attack:** Engage security team, consider rate limit reduction at M01
+
+3. **If false positive (e.g., misconfigured CI/CD):**
+   - Identify the automated system
+   - Fix its authentication configuration
+   - Verify auth failures stop
+
+### SecurityRBACDenialBurst
+
+1. **Check recent permission changes:**
+   - Query M06 Registry Service audit log
+   - If permissions were recently revoked: expected behavior, adjust alert threshold if too sensitive
+
+2. **If no recent changes:** Possible unauthorized access attempt
+   - Identify the user(s) involved
+   - Check if the denied actions are suspicious (e.g., accessing repos they've never accessed before)
+   - Engage security team if pattern suggests enumeration
+
+### ServiceAuthFailure
+
+1. **Check certificate expiry:**
+   ```bash
+   kubectl get secrets -n ecip -l type=tls -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.cert-manager\.io/expiry}{"\n"}{end}'
+   ```
+
+2. **Check service account tokens:**
+   ```bash
+   kubectl get serviceaccounts -n ecip
+   ```
+
+3. **If cert expired:** Renew immediately, this blocks all inter-service communication
+
+---
+
+## Escalation
+
+- **All security alerts:** Notify security team immediately in `#ecip-security`
+- **Auth burst > 50/5min:** Page security on-call
+- **RBAC burst with escalation pattern:** P1 security incident
+- **Service auth failure:** Page Platform on-call (infrastructure issue)
+- **Any confirmed attack:** Follow the organization's incident response procedure
+
+---
+
+## Important Notes
+
+- **User IDs in security events are SHA-256 hashed.** You cannot directly identify users from Elasticsearch data. Correlation requires access to the identity provider.
+- **Security events use a dedicated pipeline** (OTel Collector → Elasticsearch). They do NOT flow through the general logs pipeline.
+- **Never disable these alerts** without security team approval, even during maintenance windows.
+
+---
+
+## Related Alerts
+
+- `GRPCRequestLatencyHigh` — May correlate if auth checks are causing slowdowns
+- `QueryLatencySLABreach` — May be downstream effect of auth issues
+
+---
+
+*Last updated: March 2026 · Platform Team · Security Team*