@senad-d/observme 0.1.0

package/ObservMe-Production-Docs/05-otel-pipeline-and-collector.md

@@ -0,0 +1,355 @@
+# ObservMe OTEL Pipeline and Collector Design
+
+## 1. Export Strategy
+
+ObservMe emits telemetry through OpenTelemetry SDKs using OTLP.
+
+Recommended protocol:
+
+```text
+OTLP/HTTP protobuf to Collector base endpoint http://collector:4318
+```
+
+Also supported:
+
+```text
+OTLP/gRPC to Collector endpoint collector:4317
+```
+
+## 2. Why Collector First
+
+A Collector decouples ObservMe from backend-specific topology. It allows batching, retrying, filtering, redaction, sampling, routing, authentication, and backend replacement without changing the Pi extension.
+
+## 3. SDK Exporter Defaults
+
+```yaml
+otlp:
+  protocol: http/protobuf
+  endpoint: http://localhost:4318     # base endpoint; append /v1/{signal} for explicit JS exporter URLs
+  signalEndpoints:
+    traces: http://localhost:4318/v1/traces
+    metrics: http://localhost:4318/v1/metrics
+    logs: http://localhost:4318/v1/logs
+  timeoutMs: 3000
+  headers: {}
+
+traces:
+  enabled: true
+  sampler: parentbased_traceidratio
+  sampleRatio: 1.0
+  batch:
+    maxQueueSize: 2048
+    maxExportBatchSize: 512
+    scheduledDelayMillis: 1000
+    exportTimeoutMillis: 3000
+
+metrics:
+  enabled: true
+  exportIntervalMillis: 15000
+  exportTimeoutMillis: 3000
+
+logs:
+  enabled: true
+  batch:
+    maxQueueSize: 2048
+    maxExportBatchSize: 512
+    scheduledDelayMillis: 1000
+```
+
+The OpenTelemetry JS OTLP HTTP exporters default to signal-specific paths such as `http://localhost:4318/v1/traces`, `http://localhost:4318/v1/metrics`, and `/v1/logs`. Treat `otlp.endpoint` as a base endpoint in ObservMe config, but pass signal-specific URLs to SDK exporters that require an explicit `url`.
+
+## 4. Trace Context and Workflow/Agent-Lineage Propagation
+
+When ObservMe launches or wraps a subagent process, the parent runtime should propagate both standard trace context and ObservMe-specific lineage:
+
+```text
+traceparent
+tracestate
+OBSERVME_WORKFLOW_ID
+OBSERVME_AGENT_ID              # optional explicit child id, normally omitted so child generates one
+OBSERVME_PARENT_AGENT_ID
+OBSERVME_ROOT_AGENT_ID
+OBSERVME_PARENT_SESSION_ID
+OBSERVME_PARENT_TRACE_ID
+OBSERVME_PARENT_SPAN_ID
+OBSERVME_AGENT_DEPTH
+OBSERVME_SPAWN_ID
+```
+
+Preferred behavior is W3C trace-context continuation: the child `pi.session` span becomes part of the same trace as the parent `pi.agent.spawn` span. If the child cannot continue the parent trace, ObservMe should start a new trace and record a span link or structured log with the parent trace/span IDs, `pi.workflow.id`, and `pi.agent.parent_id`.
+
+These identifiers are high cardinality. Keep them on resource/span/log attributes only; do not promote them to Collector-generated metric labels or Loki index labels unless an operator explicitly accepts the cardinality cost.
+
+## 5. Minimal Debug Collector
+
+Use this for local validation.
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+
+processors:
+  batch: {}
+
+exporters:
+  debug:
+    verbosity: detailed
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug]
+    metrics:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug]
+    logs:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug]
+```
+
+## 6. Production Collector for Grafana Stack
+
+This configuration routes traces to Tempo, logs to Loki, and metrics to a Prometheus-compatible remote write endpoint such as Mimir. Use a Collector distribution that contains every configured component. The minimal debug example works with the core Collector, while `prometheusremotewrite`, `probabilistic_sampler`, and `tail_sampling` are commonly deployed from the Collector Contrib distribution or a vendor distribution such as Grafana Alloy.
+
+```yaml
+extensions:
+  health_check:
+    endpoint: 0.0.0.0:13133
+
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+
+processors:
+  memory_limiter:
+    check_interval: 1s
+    limit_mib: 1024
+    spike_limit_mib: 256
+
+  batch:
+    timeout: 2s
+    send_batch_size: 1024
+    send_batch_max_size: 2048
+
+  resource/observme:
+    attributes:
+      - key: telemetry.source
+        value: observme
+        action: upsert
+      - key: service.name
+        value: observme-pi-extension
+        action: insert
+      - key: service.namespace
+        value: pi
+        action: upsert
+
+  resource/drop_high_cardinality_metric_attrs:
+    attributes:
+      - key: pi.workflow.id
+        action: delete
+      - key: pi.workflow.root_agent_id
+        action: delete
+      - key: pi.agent.id
+        action: delete
+      - key: pi.agent.parent_id
+        action: delete
+      - key: pi.agent.root_id
+        action: delete
+      - key: pi.agent.run.id
+        action: delete
+      - key: pi.agent.spawn.id
+        action: delete
+      - key: pi.agent.spawn.tool_call_id
+        action: delete
+      - key: pi.agent.child.id
+        action: delete
+      - key: pi.session.id
+        action: delete
+
+  # Drops accidental content attributes from logs. Intentional LLM content capture
+  # is emitted as already-redacted log bodies and opt-in span attributes.
+  attributes/drop_content_attributes:
+    actions:
+      - key: gen_ai.input.messages
+        action: delete
+      - key: gen_ai.output.messages
+        action: delete
+      - key: pi.llm.prompt.redacted
+        action: delete
+      - key: pi.llm.response.redacted
+        action: delete
+      - key: pi.llm.thinking.redacted
+        action: delete
+      - key: pi.tool.arguments.redacted
+        action: delete
+      - key: pi.tool.result.redacted
+        action: delete
+
+exporters:
+  otlp/tempo:
+    endpoint: tempo:4317
+    tls:
+      insecure: true
+    sending_queue:
+      enabled: true
+    retry_on_failure:
+      enabled: true
+
+  otlphttp/loki:
+    endpoint: http://loki:3100/otlp
+    sending_queue:
+      enabled: true
+    retry_on_failure:
+      enabled: true
+
+  prometheusremotewrite/mimir:
+    endpoint: http://mimir:9009/api/v1/push
+    resource_to_telemetry_conversion:
+      enabled: true
+    sending_queue:
+      enabled: true
+    retry_on_failure:
+      enabled: true
+
+  debug:
+    verbosity: basic
+
+service:
+  extensions: [health_check]
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [memory_limiter, resource/observme, batch]
+      exporters: [otlp/tempo]
+
+    metrics:
+      receivers: [otlp]
+      processors: [memory_limiter, resource/observme, resource/drop_high_cardinality_metric_attrs, batch]
+      exporters: [prometheusremotewrite/mimir]
+
+    logs:
+      receivers: [otlp]
+      processors: [memory_limiter, resource/observme, attributes/drop_content_attributes, batch]
+      exporters: [otlphttp/loki]
+```
+
+The Collector attribute processor is defense in depth for log attributes only. It must not be the only redaction layer, and it does not sanitize arbitrary log bodies; ObservMe must redact or drop sensitive content before export. The traces pipeline intentionally keeps `pi.llm.prompt.redacted`, `pi.llm.response.redacted`, and `pi.llm.thinking.redacted` so Tempo can display redacted content after explicit capture is enabled. Old telemetry that an earlier Collector dropped cannot be recovered; generate new LLM events after updating the Collector.
+
+Keep high-cardinality lineage attributes on traces/logs, but remove them from the metrics pipeline unless the organization explicitly accepts the cardinality. The bundled local Loki pipeline promotes `pi.agent.id` and `pi.agent.run.id` as log labels so the LLM Conversations dashboard can filter captured chat content by agent and run. Prometheus resource-to-telemetry conversion is safe only after the metrics pipeline drops `pi.workflow.id`, `pi.agent.id`, `pi.session.id`, trace IDs, and spawn IDs; the remaining generated `service.instance.id`/`observme.instance.id` labels keep concurrent ObservMe metric streams distinct so session counters can be summed accurately.
+
+## 7. Direct-to-Backend Development Mode
+
+Allowed only for development:
+
+```text
+ObservMe -> Tempo OTLP HTTP for traces
+ObservMe -> Loki OTLP HTTP for logs
+ObservMe -> Prometheus OTLP HTTP receiver for metrics only
+```
+
+Not recommended for production because every Pi agent would need backend credentials and backend-specific routing. If direct metric export is used, configure SDK views/exporter resource handling so `pi.workflow.*`, `pi.agent.*`, `pi.session.id`, trace IDs, and spawn IDs are not emitted as Prometheus labels.
+
+## 8. Sampling Strategy
+
+Default for developer and CI agents:
+
+```yaml
+traces:
+  sampleRatio: 1.0
+```
+
+Production high-volume environments can sample traces in the Collector:
+
+```yaml
+processors:
+  probabilistic_sampler:
+    sampling_percentage: 25
+```
+
+For AI agent observability, tail sampling can be valuable:
+
+- Always keep errors.
+- Always keep expensive turns.
+- Always keep tool failures.
+- Sample successful low-cost turns.
+
+Example policy concept:
+
+```yaml
+processors:
+  tail_sampling:
+    decision_wait: 10s
+    policies:
+      - name: keep-errors
+        type: status_code
+        status_code:
+          status_codes: [ERROR]
+      - name: keep-expensive
+        type: numeric_attribute
+        numeric_attribute:
+          key: pi.llm.cost.total_usd
+          min_value: 0.10
+      - name: sample-rest
+        type: probabilistic
+        probabilistic:
+          sampling_percentage: 20
+```
+
+## 9. Collector Security
+
+Production collectors must:
+
+- Require TLS or run inside trusted network boundaries.
+- Authenticate external agents.
+- Restrict OTLP ingestion ports.
+- Apply rate limits at ingress.
+- Avoid exposing collector debug endpoints publicly.
+- Use separate tenants or headers for environments.
+
+## 10. Multi-Tenant Routing
+
+Use resource attributes or headers:
+
+```text
+observme.tenant.id
+deployment.environment.name
+observme.environment
+pi.project.name
+pi.agent.role
+pi.agent.depth
+```
+
+Collector routing can send tenants to separate backends or tenants within Mimir/Loki/Tempo.
+
+## 11. Backend Notes
+
+### Tempo
+
+Use Tempo for traces. Send via OTLP gRPC or HTTP. Prefer Collector -> Tempo.
+
+### Loki
+
+Use Loki OTLP ingestion for logs. Ensure structured metadata support is enabled for Loki versions/configurations that require it.
+
+### Prometheus / Mimir
+
+For Prometheus-only setups, choose one:
+
+1. Collector exposes Prometheus scrape endpoint.
+2. Prometheus enables the OTLP HTTP receiver with `--web.enable-otlp-receiver`; send SDK metrics to base endpoint `http://prometheus:9090/api/v1/otlp` when the exporter appends `/v1/metrics`, or to explicit URL `http://prometheus:9090/api/v1/otlp/v1/metrics` when configuring a signal-specific exporter URL.
+3. Collector remote writes to Mimir, Thanos, Cortex, or compatible backend.
+
+For production scale, Mimir or compatible remote write backend is preferred.

package/ObservMe-Production-Docs/06-security-privacy-redaction.md

@@ -0,0 +1,294 @@
+# ObservMe Security, Privacy, and Redaction Specification
+
+## 1. Security Philosophy
+
+ObservMe observes AI agents that may process source code, credentials, logs, customer data, incident data, infrastructure names, filesystem paths, and proprietary prompts. Therefore, ObservMe must be privacy-preserving by default.
+
+## 2. Default Capture Policy
+
+Default configuration:
+
+```yaml
+capture:
+  prompts: false
+  responses: false
+  thinking: false
+  toolArguments: false
+  toolResults: false
+  bashCommands: false
+  bashOutput: false
+  filePaths: false
+```
+
+Failed-tool output is content, not default operational metadata. It may appear in the Tools dashboard only after `capture.toolResults` is explicitly enabled. The value must pass the shared capture policy before being emitted as the body of a dedicated `tool.error.captured` log; capture-disabled and fail-closed redaction paths emit no body. Broad session-log queries must exclude `event.category="tool_content"`.
+
+Default allowed metadata:
+
+```text
+lengths
+hashes
+counts
+durations
+status
+model/provider
+token usage
+cost
+tool names
+error class
+```
+
+## 3. Data Classification
+
+### Safe by Default
+
+- Provider name
+- Model name
+- Token counts
+- Cost numbers
+- Stop reason
+- Tool name
+- Boolean success/failure
+- Redacted error class
+- Duration
+- Generated agent role/depth metadata
+
+### Sensitive
+
+- User prompts
+- Assistant responses
+- Thinking/reasoning content
+- Tool arguments
+- Tool results
+- Bash commands
+- Bash output
+- Full file paths
+- Git remote URLs
+- Environment variables
+- Stack traces
+- Raw parent process command lines or environment variables used to launch subagents
+
+### Highly Sensitive
+
+- API keys
+- Tokens
+- Passwords
+- SSH keys
+- Private certificates
+- Personal data
+- Customer data
+- Proprietary code snippets
+
+## 4. Redaction Pipeline
+
+Every optional content field must pass through:
+
+```text
+raw value
+  -> size guard
+  -> secret detector
+  -> PII detector if enabled
+  -> path scrubber
+  -> custom regex redactors
+  -> truncation
+  -> hashing
+  -> export
+```
+
+## 5. Secret Detection Patterns
+
+Minimum built-in patterns:
+
+```text
+AWS access key id:           (A3T[A-Z0-9]|AKIA|ASIA)[0-9A-Z]{16}
+Generic bearer token:        (?i)bearer\s+[a-z0-9._\-]{20,}
+GitHub token:                (gh[pousr]_[A-Za-z0-9_]{36,}|github_pat_[A-Za-z0-9_]{22,255})
+OpenAI-like key:             sk-[A-Za-z0-9_-]{20,}
+Anthropic-like key:          sk-ant-[A-Za-z0-9_-]{20,}
+Slack token:                 xox[baprs]-[A-Za-z0-9-]{10,}
+Private key block:           -----BEGIN [A-Z ]*PRIVATE KEY-----
+Password assignment:         (?i)(password|passwd|pwd)\s*[:=]\s*[^\s]+
+API key assignment:          (?i)(api[_-]?key|token|secret|client[_-]?secret)\s*[:=]\s*[^\s]+
+URL credentials:             [a-z][a-z0-9+.-]*://[^\s:/?#]+:[^\s@/]+@
+```
+
+These patterns are a minimum safety net, not a complete secret scanner. Keep them covered by tests and allow organizations to add custom regexes for proprietary credential formats.
+
+Matched values are replaced with:
+
+```text
+[REDACTED:<type>:<sha256-prefix>]
+```
+
+## 6. Path Redaction
+
+Path recognition covers standalone and embedded POSIX absolute paths, Windows drive paths, and UNC paths. It must use the matching POSIX or Windows basename/dirname semantics and must not classify normal URLs or harmless slash-separated prose as filesystem paths.
+
+Representative sensitive inputs:
+
+```text
+/home/alice/projects/customer-x/app.ts
+/workspace/project/file.ts
+/etc/hosts
+C:\Users\alice\secret.txt
+\\server\share\secret.txt
+```
+
+Options:
+
+```yaml
+privacy:
+  pathMode: hash        # hash|basename|full|drop
+```
+
+`hash`, `basename`, and `drop` must remove every recognized raw absolute path. Hashes are deterministic only within the configured tenant salt and fail closed when salt resolution fails. `drop` omits a standalone path and uses a bounded placeholder for an embedded path. `full` is the only explicit mode that preserves raw path text; it is reached only after the relevant content/path capture opt-in, and all other redaction stages still apply. URL credential redaction runs before path handling so the remaining URL is not malformed.
+
+## 7. Hashing
+
+Hashes are used for correlation without storing raw data.
+
+Recommended minimum:
+
+```text
+sha256(tenant_salt + "\0" + normalized_value)
+```
+
+Preferred when the salt is secret:
+
+```text
+hmac_sha256(tenant_salt, normalized_value)
+```
+
+Use a tenant-specific salt to prevent cross-tenant dictionary attacks. The salt itself is a secret and must come from environment or secure runtime configuration.
+
+## 8. Workflow and Agent Identity Privacy
+
+Workflow, agent, and subagent lineage should use generated IDs or salted hashes only:
+
+```text
+pi.workflow.id
+pi.workflow.root_agent_id
+pi.agent.id
+pi.agent.parent_id
+pi.agent.root_id
+pi.agent.spawn.id
+```
+
+Do not derive these IDs directly from raw cwd, username, prompt text, file path, shell command, PID, container name, or hostname. Treat parent-process command lines and inherited environment variables as sensitive; if they are captured for debugging, they must pass through the redaction pipeline first.
+
+Workflow IDs and agent IDs are high-cardinality operational identifiers. They may be exported as resource/span/log attributes for drill-down, but they must not be metric labels by default.
+
+## 9. Content Size Limits
+
+Defaults:
+
+```yaml
+limits:
+  maxPromptChars: 12000
+  maxResponseChars: 12000
+  maxToolArgumentChars: 8000
+  maxToolResultChars: 16000
+  maxBashOutputChars: 16000
+  maxLogBodyChars: 32000
+```
+
+If content exceeds limit:
+
+- Truncate
+- Add attribute `observme.truncated=true`
+- Add original length attribute
+
+## 10. Opt-in LLM Content Export
+
+Prompt, response, and thinking bodies are not exported by default. They become visible in Tempo and Loki only when the corresponding capture flag is enabled, redaction remains enabled, and a tenant hash salt is available for the session:
+
+```text
+OBSERVME_CAPTURE_PROMPTS=true
+OBSERVME_CAPTURE_RESPONSES=true
+OBSERVME_CAPTURE_THINKING=true
+OBSERVME_REDACTION_ENABLED=true
+OBSERVME_HASH_SALT=<project-specific-random-salt>
+```
+
+Set `OBSERVME_ALLOW_UNSAFE_CAPTURE=true` only with `OBSERVME_REDACTION_ENABLED=false` for intentionally raw local debugging. When redacted capture is enabled, ObservMe writes the already-redacted value to the LLM span attribute (`pi.llm.prompt.redacted`, `pi.llm.response.redacted`, or `pi.llm.thinking.redacted`) and emits one correlated Loki log whose body is the same redacted value.
+
+Example:
+
+```json
+{
+  "event.name": "llm.prompt.captured",
+  "event.category": "llm_content",
+  "pi.session.id": "...",
+  "pi.turn.id": "...",
+  "pi.llm.content.kind": "prompt",
+  "trace_id": "...",
+  "span_id": "...",
+  "body": "<redacted prompt>"
+}
+```
+
+The Collector cannot recover content dropped by an older configuration; generate new LLM events after updating the Collector and dashboards. Do not use raw chat text in Loki or Tempo query strings.
+
+## 11. Authentication
+
+ObservMe supports OTLP headers:
+
+```yaml
+otlp:
+  headers:
+    Authorization: "Bearer ${OBSERVME_OTLP_TOKEN}"
+```
+
+Secrets must be read from environment variables or secure runtime config, not hardcoded in extension source.
+
+## 12. TLS
+
+Production default:
+
+```yaml
+otlp:
+  tls:
+    enabled: true
+    insecureSkipVerify: false
+```
+
+Development may use insecure local endpoints only when explicitly configured.
+
+## 13. Tenant Isolation
+
+Set tenant and environment as resource attributes:
+
+```yaml
+resource:
+  attributes:
+    observme.tenant.id: platform
+    deployment.environment.name: production
+```
+
+Never derive tenant solely from raw cwd or username.
+
+## 14. Auditability
+
+ObservMe should emit security-relevant logs:
+
+- capture settings at startup
+- agent-lineage propagation enabled/disabled
+- redaction enabled/disabled
+- redaction failures
+- exporter authentication failures
+- rejected unsafe config
+
+Do not log secrets while reporting security failures.
+
+## 15. Safe Config Validation
+
+Reject config if:
+
+- capture is enabled but redaction is disabled, unless `allowUnsafeCapture: true`
+- OTLP endpoint is plain HTTP in production, unless `allowInsecureTransport: true`
+- metric labels include forbidden high-cardinality fields such as workflow IDs, session IDs, or agent IDs
+- propagated workflow or agent lineage values are malformed, too long, or contain unsafe characters
+- queue sizes exceed configured memory limits
+
+## 16. Compliance Posture
+
+ObservMe does not claim compliance by itself. It provides controls needed for organizations to implement compliant telemetry pipelines. Retention, access control, deletion, and legal policy are backend responsibilities.