@senad-d/observme 0.1.0

package/ObservMe-Production-Docs/02-reference-architecture.md

@@ -0,0 +1,306 @@
+# ObservMe Reference Architecture
+
+## 1. High-Level Architecture
+
+```text
+┌──────────────────────────────────────────────────────────────┐
+│                          Pi Agent                            │
+│                                                              │
+│  ┌────────────────────────────────────────────────────────┐  │
+│  │                ObservMe Extension                      │  │
+│  │                                                        │  │
+│  │  Event Capture  -> Semantic Mapper -> OTEL Emitters    │  │
+│  │       │                 │               │              │  │
+│  │       │                 │               ├─ Traces      │  │
+│  │       │                 │               ├─ Metrics     │  │
+│  │       │                 │               └─ Logs        │  │
+│  │       │                 │                              │  │
+│  │       └──── Query Commands /obs ───────────────────────┘  │
+│  └────────────────────────────────────────────────────────┘  │
+└──────────────────────────────┬───────────────────────────────┘
+                               │ OTLP HTTP/protobuf (4318) or gRPC (4317)
+                               ▼
+┌──────────────────────────────────────────────────────────────┐
+│                    OpenTelemetry Collector                   │
+│                                                              │
+│  Receivers -> Processors -> Exporters                        │
+│                                                              │
+│  processors: memory_limiter, batch, attributes/filter, sampling │
+└──────────────┬────────────────┬────────────────┬─────────────┘
+               │                │                │
+               ▼                ▼                ▼
+            Tempo             Loki          Prometheus/Mimir
+            Traces            Logs             Metrics
+               └────────────────┬────────────────┘
+                                ▼
+                             Grafana
+```
+
+## 2. Component Responsibilities
+
+### 2.1 ObservMe Pi Extension
+
+Responsibilities:
+
+- Subscribe to Pi lifecycle events.
+- Read Pi event/session metadata from event payloads and session APIs.
+- Maintain in-memory span context and workflow/agent lineage context.
+- Create OTEL traces, metrics, and logs.
+- Redact sensitive values before export.
+- Provide `/obs` commands for health, links, and downstream queries.
+
+Not responsible for:
+
+- Long-term telemetry storage
+- Heavy aggregation
+- Dashboard rendering engine
+- Backend-specific data retention
+- Collector-side sampling
+- Durable agent registry storage
+
+### 2.2 Agent and Subagent Correlation
+
+Each ObservMe runtime has an extension instance identity, a workflow identity, and a logical agent identity:
+
+```text
+observme.instance.id    # unique ObservMe telemetry session/process startup
+pi.workflow.id          # generated root workflow/execution id, high-cardinality trace/log attribute
+pi.workflow.root_agent_id
+pi.agent.id             # logical Pi agent runtime
+pi.agent.parent_id      # parent agent when spawned as a subagent
+pi.agent.root_id        # root of the agent tree
+pi.agent.depth          # bounded integer depth
+```
+
+When a Pi tool or extension starts another Pi agent, ObservMe should create a `pi.agent.spawn` span under the current turn/tool span and pass correlation to the child through environment variables and W3C trace context. If the child receives `traceparent`, its `pi.session` span can continue the parent trace. If it cannot continue the trace, it must still record `pi.agent.parent_id`, `pi.agent.root_id`, and a span link or log event so Grafana can reconstruct lineage.
+
+`pi.workflow.id`, `pi.workflow.root_agent_id`, `pi.agent.id`, `pi.agent.parent_id`, and `pi.agent.spawn.id` are high-cardinality identifiers. They belong in resource/span/log attributes and trace links, not Prometheus labels.
+
+For orchestrator workloads, ObservMe treats the root agent and every descendant as one logical workflow tree. The root agent creates `pi.workflow.id`; descendants inherit it through environment propagation. Aggregate metrics describe the tree using low-cardinality dimensions such as `agent_role`, `agent_capability`, `subagent_depth`, `spawn_type`, `spawn_reason`, and `status`, while per-workflow drill-down uses Tempo/Loki attributes.
+
+### 2.3 OpenTelemetry Collector
+
+Responsibilities:
+
+- Receive OTLP from ObservMe.
+- Batch telemetry and retry when exporter `sending_queue`/`retry_on_failure` settings are enabled.
+- Apply memory limits.
+- Apply central redaction/filtering when needed.
+- Route traces to Tempo.
+- Route logs to Loki.
+- Route metrics to Prometheus/Mimir.
+
+Recommended processors:
+
+```yaml
+processors:
+  memory_limiter:
+    check_interval: 1s
+    limit_mib: 512
+    spike_limit_mib: 128
+  batch:
+    timeout: 2s
+    send_batch_size: 1024
+    send_batch_max_size: 2048
+```
+
+Optional processors such as `probabilistic_sampler` and `tail_sampling` are useful at scale. Verify they are included in the Collector distribution you deploy; several advanced processors/exporters are in the Collector Contrib distribution rather than the minimal core distribution.
+
+### 2.4 Tempo
+
+Stores traces. Trace granularity is session -> agent run -> turn -> operation.
+
+Tempo is best used for:
+
+- Session timeline
+- Root cause analysis
+- Latency breakdown
+- Tool/LLM correlation
+- Trace-to-log and trace-to-metric linking
+
+### 2.5 Loki
+
+Stores structured logs. Loki is best used for:
+
+- Redacted prompt/response capture if enabled
+- Tool arguments/results if enabled
+- Error bodies
+- Session audit event stream
+- Searching historical sessions without local Pi logs
+
+### 2.6 Prometheus / Mimir
+
+Stores metrics. Used for:
+
+- Cost dashboards
+- Token dashboards
+- Latency dashboards
+- Alerting
+- SLO monitoring
+- Aggregation across agent roles/depths, projects, users, models, providers
+
+## 3. Data Flow
+
+### 3.1 Session Start
+
+```text
+Pi emits session_start
+ObservMe creates root span pi.session
+ObservMe emits log event session.started
+ObservMe increments session counter
+```
+
+### 3.2 Agent Run and Turn Start
+
+```text
+Pi emits agent_start
+ObservMe creates child span pi.agent.run for the user prompt lifecycle
+Pi emits turn_start for each LLM/tool iteration
+ObservMe creates child span pi.turn under the active pi.agent.run
+ObservMe records current model, cwd hash, session id, agent id, agent run id, turn id
+```
+
+### 3.3 LLM Request
+
+```text
+Pi emits before_provider_request before the provider payload is sent
+ObservMe creates a GenAI client span (`pi.llm.request` or official `chat <model>` naming)
+Attributes: provider, model, api, estimated input, settings
+Pi emits message_update/message_end for the assistant response
+ObservMe ends the span and updates usage/cost metrics from the finalized assistant message
+```
+
+### 3.4 Tool Call
+
+```text
+Pi emits tool_execution_start and tool_call before execution
+ObservMe creates pi.tool.call span
+Pi emits tool_result/tool_execution_end and later a toolResult message
+ObservMe closes the span from tool result/execution-end data
+Metrics update success/failure and latency
+```
+
+### 3.5 Subagent Spawn and Agent-Tree Flow
+
+```text
+A parent tool/extension decides to start another Pi agent
+ObservMe creates pi.agent.spawn span with spawn id, workflow id, parent agent id, parent session id, tool call id
+ObservMe propagates traceparent/tracestate, OBSERVME_WORKFLOW_ID, and OBSERVME_AGENT_* environment variables to the child
+Child ObservMe runtime starts pi.session with pi.workflow.id, pi.agent.parent_id, and pi.agent.root_id
+If the parent waits for the child, ObservMe records pi.agent.wait and/or pi.agent.join spans/events
+Metrics increment subagent spawn, fan-out, depth, active-agent, wait/join, orphan, and propagation-failure counters/histograms; per-agent drill-down uses traces/logs, not metric labels
+```
+
+### 3.6 Compaction
+
+```text
+Pi emits session_compact with compactionEntry
+ObservMe emits pi.compaction span/event/log
+Metrics increment compaction counters and tokens_before histogram
+```
+
+### 3.7 Branching
+
+```text
+Pi emits session_tree after tree navigation; summaryEntry is present when a branch summary was written
+ObservMe emits pi.branch span/event/log
+Metrics increment branch counters
+```
+
+### 3.8 Shutdown
+
+```text
+Pi emits session_shutdown
+ObservMe closes root span
+Flushes OTEL SDK with timeout
+No infinite blocking allowed
+```
+
+## 4. Trace Shape
+
+```text
+pi.session
+├── pi.agent.run 1
+│   ├── pi.turn 1
+│   │   ├── pi.llm.request
+│   │   ├── pi.tool.call bash
+│   │   ├── pi.tool.call read
+│   │   └── event: llm.request.completed
+│   └── pi.turn 2
+│       ├── pi.llm.request
+│       ├── pi.tool.call subagent
+│       ├── pi.agent.spawn
+│       ├── pi.agent.wait
+│       └── pi.agent.join
+├── pi.compaction
+└── pi.branch
+```
+
+## 5. Multi-Agent Operational Signals
+
+For orchestrator workloads, operators should watch these signals closely:
+
+| Signal | Why it matters | Primary telemetry |
+|---|---|---|
+| Fan-out per parent operation | Detects runaway delegation and cost amplification | `observme_agent_fanout_count`, `pi.agent.child.count` |
+| Tree depth and width | Detects unexpectedly deep/large agent trees | `observme_agent_tree_depth`, `observme_agent_tree_width`, `subagent_depth` |
+| Active agents | Detects stuck or leaked child agents | `observme_active_agents`, `pi.agent.children.active` |
+| Orphan agents | Detects broken lineage propagation | `observme_orphan_agents_total`, `agent.orphaned` logs |
+| Trace-context propagation failures | Detects fragmented traces | `observme_trace_context_propagation_failures_total`, span links/log fallback |
+| Wait/join latency | Shows critical path and slow child results | `pi.agent.wait`, `pi.agent.join`, `observme_agent_wait_duration_ms`, `observme_agent_join_duration_ms` |
+| Child-agent failures and recovery | Separates child failure from parent workflow failure | `observme_child_agent_failures_total`, `observme_parent_recovered_from_child_failure_total` |
+| Cost by role/depth | Shows cost amplification without workflow IDs as labels | existing token/cost metrics labeled by `agent_role`, `agent_capability`, `subagent_depth`, provider, and model |
+
+Per-workflow and per-agent drill-down should use Tempo/Loki attributes such as `pi.workflow.id`, `pi.agent.id`, `pi.agent.parent_id`, and trace/span IDs. These identifiers must not become Prometheus labels by default.
+
+## 6. Deployment Topologies
+
+### 5.1 Local Development
+
+```text
+Pi -> localhost:4318 -> Collector -> debug exporter
+```
+
+Used for extension testing.
+
+### 5.2 Developer Workstation with Shared Collector
+
+```text
+Pi -> regional collector DNS -> Grafana stack
+```
+
+Recommended when developers run Pi locally but telemetry must be centralized.
+
+### 5.3 Ephemeral CI Agent
+
+```text
+CI Pi container -> sidecar collector -> central collector gateway -> Grafana stack
+```
+
+Sidecar collector provides fast local handoff and retries while the job is alive.
+
+### 5.4 Kubernetes
+
+```text
+Pi pod -> daemonset collector -> gateway collector -> Tempo/Loki/Mimir
+```
+
+Recommended for scale and centralized policy.
+
+## 7. Failure Modes
+
+| Failure | Expected Behavior |
+|---|---|
+| Collector unavailable | Drop after bounded retry; Pi continues |
+| Backend unavailable | Collector buffers/retries if configured; Pi unaffected |
+| Redaction failure | Drop sensitive field and emit redaction error metric |
+| Span context lost | Emit orphan span with session id and agent id attributes |
+| Subagent trace context not propagated | Emit child trace with `pi.agent.parent_id`, `pi.agent.root_id`, `pi.workflow.id`, and span/log link metadata; increment propagation failure counter |
+| Orphan child agent | Mark as root or orphan according to config, emit `agent.orphaned` log, increment orphan counter |
+| Runaway agent fan-out/depth | Continue exporting fail-open but emit high fan-out/depth metrics and alerts; never block Pi by default |
+| Queue full | Drop telemetry and increment drop counter |
+| Shutdown timeout | Abort flush after configured timeout |
+
+## 8. Extension Boundary
+
+ObservMe should not parse the full session file continuously during normal operation if Pi events provide the data. It may read session state on startup or command execution to reconstruct current session context, but runtime observability should be event-driven.

package/ObservMe-Production-Docs/03-pi-event-and-session-model.md

@@ -0,0 +1,266 @@
+# Pi Event and Session Model for ObservMe
+
+## 1. Purpose
+
+This document defines how ObservMe interprets Pi events and session files. The Pi session file is not ObservMe storage. It is a canonical Pi execution record that ObservMe may use as an event source, correlation source, and recovery source.
+
+## 2. Pi Session Entries Relevant to ObservMe
+
+ObservMe must understand these entry types:
+
+- `session`
+- `message`
+- `model_change`
+- `thinking_level_change`
+- `compaction`
+- `branch_summary`
+- `custom`
+- `custom_message`
+- `label`
+- `session_info`
+
+## 3. Message Roles and Entry Types Relevant to ObservMe
+
+For real-time capture, ObservMe should prefer Pi extension events (`session_start`, `agent_start`, `agent_end`, `turn_start`, `before_provider_request`, `after_provider_response`, `message_*`, `tool_execution_*`, `tool_call`, `tool_result`, `user_bash`, `model_select`, `thinking_level_select`, `session_compact`, `session_tree`, and `session_shutdown`). Session entries are still authoritative for startup recovery, explicit backfill, `/obs session`, and cross-checking event payloads.
+
+### `user`
+
+Used to detect user turns and optionally capture redacted prompt content.
+
+Default telemetry:
+
+- role
+- timestamp
+- content length
+- content hash
+- image count if multimodal
+
+Optional telemetry:
+
+- prompt text
+- image metadata
+
+### `assistant`
+
+Used to capture provider/model/usage/cost and tool calls.
+
+Default telemetry:
+
+- provider
+- model
+- api
+- stopReason
+- usage.input
+- usage.output
+- usage.cacheRead
+- usage.cacheWrite
+- usage.cacheWrite1h if reported by the provider
+- usage.reasoning if reported by the provider
+- usage.cost.total
+- responseModel and responseId when available
+- diagnostics count when present
+- tool call count
+- error class if present
+
+Optional telemetry:
+
+- response text
+- thinking content
+- tool call arguments
+
+### `toolResult`
+
+Used to close tool spans and emit tool metrics.
+
+Default telemetry:
+
+- toolCallId
+- toolName
+- isError
+- content length
+- content hash
+
+Optional telemetry:
+
+- tool result body
+
+### `bashExecution`
+
+Used for command execution observability.
+
+Default telemetry:
+
+- command fingerprint
+- exitCode
+- cancelled
+- truncated
+- duration if derivable from event timestamps
+- output length
+- fullOutputPath presence
+
+Optional telemetry:
+
+- command string
+- output body
+
+### `custom` entry
+
+Extension state entry created through `pi.appendEntry()`. It does not participate in LLM context. ObservMe should append a custom entry only for explicitly enabled minimal correlation state; by default it should not append custom entries.
+
+### `custom_message` entry
+
+Extension-injected context entry created through `pi.sendMessage()`/`appendCustomMessageEntry()`. It participates in LLM context and can alter agent behavior. ObservMe must not use `custom_message` for telemetry bookkeeping.
+
+## 4. Session Tree Implications
+
+Pi sessions are tree-shaped via `id` and `parentId`. ObservMe telemetry must preserve this structure.
+
+Required attributes:
+
+```text
+pi.entry.id
+pi.entry.parent_id
+pi.entry.type
+pi.branch.path_hash
+pi.leaf.id
+```
+
+Branch events must include:
+
+```text
+pi.branch.from_id
+pi.branch.to_id
+pi.branch.common_ancestor_id        # if available
+pi.branch.summary.hash
+```
+
+## 5. Context and Compaction
+
+Compaction changes what the LLM sees, but it does not delete the historical session file. ObservMe should emit compaction telemetry from the `session_compact` event, which carries the saved `compactionEntry`, `reason`, `willRetry`, and whether the summary came from an extension.
+
+Required fields:
+
+```text
+pi.compaction.first_kept_entry_id
+pi.compaction.tokens_before
+pi.compaction.summary.hash
+pi.compaction.from_hook
+pi.compaction.reason
+pi.compaction.will_retry
+```
+
+Optional fields:
+
+```text
+pi.compaction.read_files_count
+pi.compaction.modified_files_count
+```
+
+## 6. Event-to-Span Mapping
+
+| Pi Source | ObservMe Operation |
+|---|---|
+| `session_start` | start `pi.session` span and initialize resource/session attributes |
+| `agent_start` / `agent_end` | start/end a `pi.agent.run` span for one user-prompt lifecycle; turns are children of this span |
+| `turn_start` / `turn_end` | start/end `pi.turn` span using `turnIndex` and finalized turn data |
+| `before_provider_request` | start GenAI client span and record provider request metadata available from payload/`ctx.model` |
+| `after_provider_response` | record provider HTTP status/headers that are safe to capture |
+| `message_update` | optionally record streaming progress such as first-token/chunk timing |
+| assistant `message_end` | record final model result, stop reason, usage, cost, and end GenAI span |
+| assistant toolCall block from finalized message | prepare correlation for tool spans when needed |
+| `tool_execution_start` | start `pi.tool.call` span before execution |
+| `tool_call` | validate/record mutable tool input metadata and blocking outcome |
+| `tool_result` / `tool_execution_end` | record result metadata and end `pi.tool.call` span |
+| parent tool/extension launches another Pi agent | emit `pi.agent.spawn` span and propagate trace/workflow/agent lineage to the child |
+| parent tool/extension waits for or receives child agent result | emit `pi.agent.wait` / `pi.agent.join` spans or events with child status and critical-path timing |
+| `toolResult` message | cross-check persisted tool result order and content hashes |
+| `user_bash` plus `bashExecution` session message | create or complete `pi.bash.execution` telemetry for `!`/`!!` commands |
+| `model_select` plus `model_change` entry | emit model-change log event and metric |
+| `thinking_level_select` plus `thinking_level_change` entry | emit thinking-level-change log event and metric |
+| `session_compact` | emit compaction span/log/metric from `compactionEntry` |
+| `session_tree` | emit branch/tree-navigation span/log/metric and include `summaryEntry` when present |
+| `session_shutdown` | end open spans and flush exporters within timeout |
+
+User-bash correlation is intentionally single-flight because Pi does not expose a durable call ID for `user_bash`. ObservMe starts the pending span at `user_bash`, retains only safe span/timing metadata, and completes it from the corresponding `bashExecution` record. Overlapping pre-events evict the ambiguous pending span and emit a bounded drop signal rather than risking a wrong match. An unmatched completion records duration only when it contains a valid start/completion timestamp pair; otherwise duration is omitted. Shutdown closes any pending user-bash span as incomplete. Raw commands and output remain subject to the shared opt-in content-capture policy.
+
+The root `pi.session` span is intentionally long-lived and remains open until `session_shutdown`. During an active session, Tempo searches may show ended child spans before the root span is exported; post-shutdown traces must include the canonical `pi.session` root with the session and workflow attributes.
+
+## 7. Correlation IDs
+
+ObservMe must maintain the following IDs:
+
+```text
+observme.instance.id       # unique per ObservMe telemetry session/process startup
+observme.export.session_id # same as pi.session.id if available
+pi.session.id
+pi.workflow.id             # generated workflow/root execution id, high-cardinality trace/log attribute
+pi.workflow.root_agent_id
+pi.agent.id                # logical Pi agent runtime
+pi.agent.parent_id         # parent agent when this is a subagent
+pi.agent.root_id           # root agent in the lineage tree
+pi.agent.run.id            # one user-prompt lifecycle from agent_start to agent_end
+pi.agent.spawn.id          # individual subagent spawn operation if known
+pi.entry.id
+pi.turn.id
+pi.tool_call.id
+trace_id
+span_id
+```
+
+## 8. Multi-Agent and Subagent Lineage
+
+Pi's documented extension events identify sessions, agent prompt lifecycles, turns, tools, and session-tree changes. They do not provide a built-in cross-process subagent lineage event. Therefore ObservMe must create and propagate lineage when an extension/tool starts another Pi agent.
+
+Required lineage behavior:
+
+1. Generate `pi.agent.id` when the ObservMe runtime starts unless a trusted parent supplied one for a resumed child context.
+2. Set `pi.agent.root_id = pi.agent.id` for root agents.
+3. Generate `pi.workflow.id` in the root agent unless a trusted parent supplied one, and preserve it across every descendant.
+4. For subagents, set `pi.agent.parent_id` to the parent `pi.agent.id`, preserve `pi.agent.root_id`, and increment `pi.agent.depth`.
+5. Create a `pi.agent.spawn` span around the parent operation that launches the child process.
+6. When the parent waits for or receives the child result, emit `pi.agent.wait` and/or `pi.agent.join` spans/events so critical-path latency is visible.
+7. Propagate W3C `traceparent`/`tracestate` plus ObservMe lineage environment variables such as `OBSERVME_WORKFLOW_ID`, `OBSERVME_PARENT_AGENT_ID`, `OBSERVME_ROOT_AGENT_ID`, and `OBSERVME_PARENT_SESSION_ID`.
+8. In the child, continue the trace when trace context is available; otherwise start a new trace and attach a span link or log event with parent trace/span metadata.
+9. If a child starts with malformed or missing parent lineage, classify it as root or orphan according to config and emit an orphan log/metric.
+
+Only lineage supplied in the child Pi process environment by an ObservMe-aware launcher is eligible for automatic continuation. Project-local `.env` values are configuration inputs and must not establish parent provenance. A propagated child candidate must carry a complete validated workflow/parent/root/depth/spawn envelope and, when trace propagation is enabled, a valid W3C `traceparent`; `tracestate` and duplicate parent trace/span metadata must also validate and agree when present. Partial, malformed, oversized, or stale envelopes fail open to a new root/orphan trace without exporting inherited raw values. Trusted lineage without usable W3C continuation starts a new trace with a validated parent span link when trace/span metadata is available, otherwise it emits the bounded `trace_context.propagation_failed` fallback signal.
+
+Workflow IDs and agent IDs must be generated identifiers or salted hashes. Never derive them directly from raw cwd, username, prompt text, or command line. They are high-cardinality and must not be metric labels.
+
+## 9. Turn Identification
+
+Pi extension events provide `turnIndex`, not a durable session-file turn id. ObservMe should derive a stable-in-process turn id from `pi.session.id`, `pi.agent.run.id`, active branch/leaf context, and `turnIndex`. If a future Pi API provides a durable turn id, ObservMe must prefer it.
+
+Derived format:
+
+```text
+agent-run-000001-turn-000001
+agent-run-000001-turn-000002
+```
+
+The derived id must be scoped by session id and agent run id and must not be used as a metric label.
+
+## 10. Recovery Behavior
+
+On extension startup in an existing session:
+
+1. Read current session header if available.
+2. Set `pi.session.id`, `pi.workflow.id`, `pi.agent.id`, and resource attributes.
+3. Reconstruct optional workflow/agent lineage only from trusted environment variables or an explicitly enabled minimal custom correlation entry.
+4. Do not replay old telemetry by default.
+5. Optionally support `replayOnStart: true` for backfill, but it must be off by default.
+6. If replay is enabled, mark spans/logs with `observme.replayed=true`.
+
+## 11. Local Session Reading Rules
+
+Allowed:
+
+- Read header on startup.
+- Read current branch on `/obs session`.
+- Read entries for explicit `/obs backfill` command.
+
+Not allowed by default:
+
+- Continuous tailing of session file when lifecycle events are sufficient.
+- Storing parsed session data locally.
+- Appending telemetry summaries into Pi context.