@senad-d/observme 0.1.0

package/ObservMe-Production-Docs/10-testing-release-operations.md

@@ -0,0 +1,319 @@
+# ObservMe Testing, Release, and Operations
+
+## 1. Test Strategy
+
+ObservMe must be tested at five levels:
+
+1. Unit tests
+2. Contract tests against Pi event payloads
+3. Integration tests with OTEL Collector
+4. Backend tests with Grafana stack
+5. Chaos and failure tests
+
+## 2. Unit Tests
+
+Required areas:
+
+- Config loading and precedence
+- Redaction patterns
+- Hashing stability
+- Truncation behavior
+- Attribute mapping
+- Metric label cardinality checks
+- Span registry eviction
+- Agent lineage ID generation and propagation
+- Safe handler error isolation
+
+## 3. Event Mapping Tests
+
+Use JSON fixtures for Pi session entries and extension event payloads:
+
+```text
+test/fixtures/session-user-message.json
+test/fixtures/session-assistant-usage.json
+test/fixtures/session-assistant-usage-reasoning-cache.json
+test/fixtures/tool-result-error.json
+test/fixtures/bash-execution.json
+test/fixtures/compaction.json
+test/fixtures/branch-summary.json
+test/fixtures/events/session-start.json
+test/fixtures/events/agent-start-end.json
+test/fixtures/events/turn-start-end.json
+test/fixtures/events/message-end-assistant.json
+test/fixtures/events/tool-execution-start-end.json
+test/fixtures/events/session-compact.json
+test/fixtures/events/session-tree.json
+test/fixtures/events/subagent-spawn.json
+test/fixtures/events/agent-wait-join.json
+test/fixtures/events/orphan-agent.json
+```
+
+Assertions:
+
+- Correct span name or documented GenAI-compatible provider span name
+- Correct `pi.*`, `observme.*`, and applicable `gen_ai.*` attributes
+- No forbidden metric labels, including workflow IDs, session IDs, or agent IDs
+- Agent-run spans are parented under session spans, and turn spans under agent-run spans when available
+- Subagent spawn telemetry carries workflow/parent/root/depth attributes and trace context when configured
+- Agent wait/join telemetry exposes child status and critical-path timing without high-cardinality metric labels
+- Orphan-agent and trace-context propagation failure metrics/logs are emitted when lineage is missing or malformed
+- Optional content absent by default
+- Redacted content when capture enabled
+
+## 4. Redaction Tests
+
+Test cases must include:
+
+- AWS keys
+- GitHub tokens
+- bearer tokens
+- OpenAI-like keys
+- Anthropic-like keys
+- Slack tokens
+- password assignments
+- private key blocks
+- environment variable dumps
+- filesystem paths
+- URL credentials
+
+Example assertion:
+
+```text
+Input:  Authorization: <Bearer abcdefghijklmnopqrstuvwxyz123456>
+Output: Authorization: [REDACTED:bearer:<hash>]
+```
+
+## 5. Collector Integration Tests
+
+Run Collector with debug exporter:
+
+```bash
+docker run --rm \
+  -p 4317:4317 \
+  -p 4318:4318 \
+  -v "$PWD/test/collector-debug.yaml:/etc/otelcol/config.yaml" \
+  otel/opentelemetry-collector:latest
+```
+
+The core Collector image is sufficient for this debug config. Test production configs with the exact target distribution, for example Collector Contrib or Grafana Alloy when using `prometheusremotewrite`, `tail_sampling`, or other contrib-only components.
+
+Test:
+
+- Traces arrive
+- Metrics arrive
+- Logs arrive
+- Attributes are present
+- Content capture defaults are respected
+
+## 6. Grafana Stack Integration Tests
+
+Use Docker Compose with:
+
+- Collector
+- Tempo
+- Loki
+- Prometheus or Mimir
+- Grafana
+
+Tests:
+
+- Tempo trace query by trace id
+- Tempo or Loki query by `pi.agent.id` / `pi.agent.parent_id` for lineage
+- Loki log query by session id
+- Prometheus metric query for token totals
+- Grafana dashboard import validates
+
+## 7. Failure Tests
+
+### Collector Down
+
+Expected:
+
+- Pi continues.
+- ObservMe increments drop/export error metrics if possible.
+- No unhandled exception.
+
+### Collector Slow
+
+Expected:
+
+- Export timeout triggers.
+- Handler latency remains below budget.
+
+### Subagent Without Propagated Context
+
+Expected:
+
+- Child agent still starts and exports telemetry.
+- Child telemetry includes `pi.workflow.id`, `pi.agent.parent_id`, and `pi.agent.root_id` if provided by environment.
+- If no parent context is available, child is marked as root or orphan according to config.
+- `observme_trace_context_propagation_failures_total` or `observme_orphan_agents_total` increments as appropriate.
+- No high-cardinality workflow IDs or agent IDs are emitted as metric labels.
+
+### Runaway Fan-Out or Depth
+
+Expected:
+
+- Pi continues unless a separate policy extension blocks execution.
+- Fan-out/depth metrics update (`observme_agent_fanout_count`, `observme_agent_tree_depth`).
+- Alerts can fire from dashboard/alert rules.
+- No high-cardinality workflow/parent/child IDs are emitted as metric labels.
+
+### Queue Full
+
+Expected:
+
+- Telemetry drops.
+- Drop counter increments.
+- Memory remains bounded.
+
+### Redaction Exception
+
+Expected:
+
+- Field is dropped.
+- Redaction failure metric increments.
+- No raw value exported.
+
+## 8. Performance Tests
+
+Synthetic workload:
+
+```text
+100 sessions
+1,000 turns/session
+5 tool calls/turn
+2 LLM calls/turn
+1 subagent spawn every 20 turns
+```
+
+Required measurements:
+
+- handler duration p50/p95/p99
+- memory growth
+- dropped telemetry count
+- export batch sizes
+- CPU overhead
+
+Targets:
+
+```text
+handler p95 < 10ms
+handler p99 < 25ms
+steady memory < configured limit
+no unbounded maps
+```
+
+## 9. Cardinality Tests
+
+Ensure metrics do not include:
+
+- workflow id
+- session id
+- agent id
+- parent agent id
+- child agent id
+- agent run id
+- spawn id
+- spawn tool call id
+- trace id
+- span id
+- entry id
+- raw path
+- raw command
+- raw prompt
+- raw error
+
+Reject or sanitize dynamically generated labels.
+
+## 10. Release Process
+
+1. Update version.
+2. Run unit tests.
+3. Run integration tests.
+4. Run redaction corpus tests.
+5. Run dashboard JSON validation.
+6. Run Collector config validation.
+7. Generate changelog.
+8. Publish package/artifact.
+9. Tag release.
+
+## 11. Compatibility Matrix
+
+Maintain a tested matrix:
+
+```text
+Pi version
+Node.js version
+OpenTelemetry JS package set
+Collector distribution and version
+Tempo version
+Loki version
+Prometheus/Mimir version
+Grafana version
+```
+
+## 12. Operational Metrics for ObservMe Itself
+
+ObservMe must emit self-observability metrics:
+
+```text
+observme_events_observed_total
+observme_handler_errors_total
+observme_handler_duration_ms
+observme_telemetry_dropped_total
+observme_export_errors_total
+observme_redaction_failures_total
+observme_active_spans
+observme_active_agents
+observme_workflows_started_total
+observme_workflows_completed_total
+observme_workflow_errors_total
+observme_workflow_duration_ms
+observme_agent_runs_total
+observme_subagents_spawned_total
+observme_subagent_spawn_failures_total
+observme_orphan_agents_total
+observme_trace_context_propagation_failures_total
+observme_child_agent_failures_total
+observme_agent_fanout_count
+observme_agent_tree_depth
+observme_agent_tree_width
+observme_agent_wait_duration_ms
+observme_agent_join_duration_ms
+```
+
+## 13. Export Health operational contract
+
+The Export Health dashboard is considered operationally useful when it proves both healthy liveness and induced failure visibility without changing a user's working local setup.
+
+Healthy local validation:
+
+- Keep the existing trusted-project setup and configured local OTLP endpoint; do not change `/obs status`, `/obs health`, Grafana auth/profile, or local debug capture settings to satisfy this contract.
+- Generate representative Pi handler activity and verify `observme_events_observed_total` increases.
+- Verify failure-only stat panels display `0` when no matching series exists for telemetry drops, redaction failures, export failures, or handler failures.
+- Treat empty Loki tables for `telemetry.dropped`, `redaction.failed`, `export.failed`, and `trace_context.propagation_failed` as healthy when no matching failure was induced.
+
+Induced failure validation:
+
+- Queue-full or bounded-registry eviction increments `observme_telemetry_dropped_total` with bounded `reason` labels and emits a `telemetry.dropped` log.
+- Redaction exceptions increment `observme_redaction_failures_total` with bounded `operation`/`error_class` labels and emit a `redaction.failed` log without raw content.
+- Export failures increment `observme_export_errors_total` with bounded `reason`/`error_class` labels and emit existing `export.failed` logs.
+- Subagent trace-context propagation failures increment the documented counter and emit `trace_context.propagation_failed` logs.
+- Active span pressure is visible through `observme_active_spans` by bounded `operation`; normal completion, eviction, and shutdown must not leave negative or leaked gauge values.
+
+Label validation for Export Health metrics is limited to `operation`, `reason`, `error_class`, and `status` where needed. Session IDs, workflow IDs, agent IDs, trace/span IDs, entry IDs, raw prompts, raw paths, raw commands, and raw error messages remain span/log attributes or are omitted; they are never metric labels.
+
+## 14. Support Procedure
+
+When troubleshooting:
+
+1. Run `/obs status`.
+2. Run `/obs health`.
+3. Run `/obs agents` when troubleshooting workflow, parent/subagent lineage, fan-out, depth, or orphan agents.
+4. Check Collector health endpoint.
+5. Check Collector logs.
+6. Query `observme_export_errors_total`.
+7. Query Loki for `event_name="export.failed"` (or equivalent normalized structured metadata).
+8. Temporarily enable debug exporter in Collector.
+9. Never enable raw content capture in production without approval.

package/ObservMe-Production-Docs/11-deployment-runbooks.md

@@ -0,0 +1,203 @@
+# ObservMe Deployment Runbooks
+
+## 1. Local Debug Runbook
+
+### Start Collector
+
+```bash
+cat > collector-debug.yaml <<'YAML'
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+exporters:
+  debug:
+    verbosity: detailed
+processors:
+  batch: {}
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug]
+    metrics:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug]
+    logs:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug]
+YAML
+
+docker run --rm -p 4317:4317 -p 4318:4318 \
+  -v "$PWD/collector-debug.yaml:/etc/otelcol/config.yaml" \
+  otel/opentelemetry-collector:latest
+```
+
+This debug config uses only core Collector components. Production configs that use `prometheusremotewrite`, `tail_sampling`, or other contrib components must run a distribution that includes those components.
+
+### Configure ObservMe
+
+```yaml
+otlp:
+  endpoint: http://localhost:4318   # ObservMe base endpoint; SDK exporters use /v1/traces, /v1/metrics, /v1/logs
+  protocol: http/protobuf
+traces:
+  enabled: true
+metrics:
+  enabled: true
+logs:
+  enabled: true
+capture:
+  prompts: false
+  responses: false
+```
+
+### Validate
+
+Run Pi with ObservMe loaded and execute:
+
+```text
+/obs status
+/obs health
+```
+
+## 2. Docker Compose LGTM Runbook
+
+Services:
+
+- otel-collector
+- tempo
+- loki
+- prometheus
+- grafana
+
+Recommended ports:
+
+```text
+4317 collector OTLP gRPC
+4318 collector OTLP HTTP
+3000 grafana
+3100 loki
+3200 tempo
+9090 prometheus
+```
+
+## 3. CI Agent Runbook
+
+CI agents are short lived. Use one of:
+
+### Option A: Direct to central Collector
+
+```text
+Pi CI job -> central collector
+```
+
+Simplest. Requires network availability and credentials.
+
+### Option B: Sidecar Collector
+
+```text
+Pi CI job -> sidecar collector -> central collector
+```
+
+Best for high-volume jobs because Pi hands off telemetry locally.
+
+Shutdown steps:
+
+1. Parent Pi passes trace context, `OBSERVME_WORKFLOW_ID`, and `OBSERVME_PARENT_AGENT_ID`/`OBSERVME_ROOT_AGENT_ID` to any child Pi processes it launches.
+2. Pi exits.
+3. ObservMe flushes with timeout.
+4. Sidecar collector drains with timeout.
+5. CI job ends.
+
+## 4. Production Configuration Checklist
+
+- [ ] OTLP endpoint uses TLS or private network.
+- [ ] Authentication configured.
+- [ ] Prompts disabled by default.
+- [ ] Responses disabled by default.
+- [ ] Tool arguments disabled by default.
+- [ ] Redaction enabled.
+- [ ] Metric labels audited for cardinality.
+- [ ] Agent IDs, session IDs, trace IDs, and spawn IDs are not metric labels.
+- [ ] Workflow/subagent lineage propagation is enabled where subagents are used.
+- [ ] Workflow/agent-tree alerts are tuned for expected max depth, fan-out, active-agent count, and cost budget.
+- [ ] Collector distribution contains every configured component.
+- [ ] Collector memory limiter enabled.
+- [ ] Collector batch processor enabled.
+- [ ] Loki structured metadata enabled when required by the deployed Loki version.
+- [ ] Prometheus OTLP receiver enabled with `--web.enable-otlp-receiver` if sending directly to Prometheus.
+- [ ] Backend retention configured.
+- [ ] Dashboards imported.
+- [ ] Alerts enabled.
+- [ ] `/obs health` succeeds.
+- [ ] `/obs agents` shows expected workflow/root/subagent lineage, depth, fan-out, active-child count, and orphan count when subagents are enabled.
+
+## 5. Common Incidents
+
+### No Traces in Tempo
+
+Check:
+
+1. `/obs status` traces enabled
+2. Collector receives traces
+3. Collector trace pipeline exports to Tempo
+4. Tempo OTLP endpoint reachable
+5. Grafana Tempo datasource configured
+
+### No Logs in Loki
+
+Check:
+
+1. Logs enabled in ObservMe
+2. Collector logs pipeline configured
+3. Loki accepts the OTLP endpoint (`/otlp` for Collector `otlphttp` exporter)
+4. Structured metadata enabled if required
+5. Queries use normalized attribute names such as `event_name` and `pi_session_id`
+6. Label cardinality within limits
+
+### No Metrics in Prometheus
+
+Check:
+
+1. Metrics enabled in ObservMe
+2. Collector metrics pipeline configured
+3. Prometheus OTLP receiver enabled (`--web.enable-otlp-receiver`) and exporter endpoint set to base `/api/v1/otlp` or explicit signal URL `/api/v1/otlp/v1/metrics`, or scrape endpoint configured
+4. Metric names visible
+5. Resource attributes not unexpectedly dropped
+
+### Subagents Not Linked to Parent
+
+Check:
+
+1. Parent runtime creates `pi.agent.spawn` spans.
+2. Child process receives `traceparent`/`tracestate` or parent trace/span fallback attributes.
+3. Child process receives `OBSERVME_WORKFLOW_ID`, `OBSERVME_PARENT_AGENT_ID`, `OBSERVME_ROOT_AGENT_ID`, and `OBSERVME_AGENT_DEPTH`.
+4. Queries use normalized attribute names such as `pi_workflow_id`, `pi_agent_id`, `pi_agent_parent_id`, and `pi_agent_root_id`.
+5. Collector/Loki configuration has not promoted high-cardinality workflow IDs or agent IDs to index labels unless explicitly accepted.
+
+### High Telemetry Drops
+
+Actions:
+
+1. Increase collector capacity.
+2. Increase SDK queue within memory budget.
+3. Reduce content capture.
+4. Add sampling.
+5. Increase batch size.
+
+### Sensitive Data Found
+
+Actions:
+
+1. Disable capture immediately.
+2. Rotate exposed secrets.
+3. Reduce backend retention or delete data according to backend process.
+4. Add redaction test case.
+5. Release hotfix.