@senad-d/observme 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +57 -0
- package/CHANGELOG.md +52 -0
- package/LICENSE +21 -0
- package/ObservMe-Production-Docs/00-README.md +79 -0
- package/ObservMe-Production-Docs/01-requirements-and-scope.md +207 -0
- package/ObservMe-Production-Docs/02-reference-architecture.md +306 -0
- package/ObservMe-Production-Docs/03-pi-event-and-session-model.md +266 -0
- package/ObservMe-Production-Docs/04-telemetry-semantic-conventions.md +722 -0
- package/ObservMe-Production-Docs/05-otel-pipeline-and-collector.md +355 -0
- package/ObservMe-Production-Docs/06-security-privacy-redaction.md +294 -0
- package/ObservMe-Production-Docs/07-extension-implementation-blueprint.md +447 -0
- package/ObservMe-Production-Docs/08-query-grafana-integration.md +276 -0
- package/ObservMe-Production-Docs/09-dashboards-alerts-slos.md +640 -0
- package/ObservMe-Production-Docs/10-testing-release-operations.md +319 -0
- package/ObservMe-Production-Docs/11-deployment-runbooks.md +203 -0
- package/ObservMe-Production-Docs/12-configuration-reference.md +337 -0
- package/ObservMe-Production-Docs/13-source-notes.md +33 -0
- package/ObservMe-Production-Docs/pi-session-format.md +427 -0
- package/README.md +356 -0
- package/SECURITY.md +45 -0
- package/dashboards/observme-agent-node-graphs.json +250 -0
- package/dashboards/observme-agents.json +1880 -0
- package/dashboards/observme-alerts.yaml +113 -0
- package/dashboards/observme-branches-compactions.json +1042 -0
- package/dashboards/observme-cost.json +1254 -0
- package/dashboards/observme-errors.json +1659 -0
- package/dashboards/observme-export-health.json +1802 -0
- package/dashboards/observme-latency.json +1494 -0
- package/dashboards/observme-llm-conversations.json +730 -0
- package/dashboards/observme-logs-llm.json +644 -0
- package/dashboards/observme-models.json +933 -0
- package/dashboards/observme-overview.json +2129 -0
- package/dashboards/observme-slo-health.json +737 -0
- package/dashboards/observme-slos.yaml +56 -0
- package/dashboards/observme-tools.json +902 -0
- package/dashboards/observme-trace-journey.json +1793 -0
- package/docs/STRUCTURE.md +49 -0
- package/docs/agent-subagent-observability-requirements.md +997 -0
- package/docs/compatibility-matrix.md +39 -0
- package/docs/configuration-tui-design-standard.md +767 -0
- package/docs/configuration.md +38 -0
- package/docs/review-validation.md +115 -0
- package/docs/validation-flow.md +117 -0
- package/examples/collector.yaml +123 -0
- package/examples/observme.yaml +131 -0
- package/img/demo.gif +0 -0
- package/img/icon.svg +47 -0
- package/package.json +103 -0
- package/src/commands/obs-agents-runtime.ts +150 -0
- package/src/commands/obs-agents.ts +491 -0
- package/src/commands/obs-args.ts +63 -0
- package/src/commands/obs-backfill.ts +1334 -0
- package/src/commands/obs-command-support.ts +43 -0
- package/src/commands/obs-cost.ts +228 -0
- package/src/commands/obs-diagnostics.ts +22 -0
- package/src/commands/obs-errors.ts +156 -0
- package/src/commands/obs-health.ts +301 -0
- package/src/commands/obs-link.ts +90 -0
- package/src/commands/obs-logs.ts +194 -0
- package/src/commands/obs-loki-summary.ts +184 -0
- package/src/commands/obs-session.ts +259 -0
- package/src/commands/obs-status.ts +359 -0
- package/src/commands/obs-tools.ts +274 -0
- package/src/commands/obs-trace.ts +411 -0
- package/src/commands/obs.ts +211 -0
- package/src/config/bootstrap-project-config.ts +300 -0
- package/src/config/defaults.ts +143 -0
- package/src/config/load-config.ts +631 -0
- package/src/config/project-paths.ts +61 -0
- package/src/config/schema.ts +405 -0
- package/src/config/validate.ts +456 -0
- package/src/constants.ts +4 -0
- package/src/diagnostics/sanitize.ts +6 -0
- package/src/extension.ts +38 -0
- package/src/otel/logs.ts +160 -0
- package/src/otel/metrics.ts +165 -0
- package/src/otel/otlp-endpoint.ts +10 -0
- package/src/otel/sdk.ts +114 -0
- package/src/otel/shutdown.ts +102 -0
- package/src/otel/traces.ts +166 -0
- package/src/pi/agent-lineage.ts +378 -0
- package/src/pi/agent-tree-tracker.ts +258 -0
- package/src/pi/event-handlers/agent-turn.ts +155 -0
- package/src/pi/event-handlers/lifecycle.ts +642 -0
- package/src/pi/event-handlers/llm.ts +115 -0
- package/src/pi/event-handlers/session-events.ts +159 -0
- package/src/pi/event-handlers/tool-bash.ts +275 -0
- package/src/pi/handler-internals.ts +2154 -0
- package/src/pi/handler-runtime.ts +633 -0
- package/src/pi/handler-types.ts +261 -0
- package/src/pi/handlers.ts +75 -0
- package/src/pi/subagent-spawn.ts +975 -0
- package/src/pi/subagent-types.ts +29 -0
- package/src/privacy/content-capture.ts +104 -0
- package/src/privacy/hash.ts +93 -0
- package/src/privacy/redact.ts +619 -0
- package/src/privacy/secret-patterns.ts +185 -0
- package/src/privacy/truncate.ts +69 -0
- package/src/query/grafana-readiness.ts +164 -0
- package/src/query/grafana-transport.ts +481 -0
- package/src/query/grafana.ts +371 -0
- package/src/query/loki.ts +332 -0
- package/src/query/prometheus.ts +388 -0
- package/src/query/tempo.ts +332 -0
- package/src/safety/sensitive-input.ts +208 -0
- package/src/semconv/attributes.ts +279 -0
- package/src/semconv/metrics.ts +146 -0
- package/src/semconv/spans.ts +19 -0
- package/src/semconv/values.ts +13 -0
- package/src/util/bounded-map.ts +97 -0
- package/tsconfig.json +15 -0
package/README.md
ADDED
|
@@ -0,0 +1,356 @@
|
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img alt="ObservMe icon" src="img/icon.svg" width="128">
|
|
3
|
+
</p>
|
|
4
|
+
|
|
5
|
+
<p align="center">
|
|
6
|
+
<a href="https://pi.dev"><img alt="pi package" src="https://img.shields.io/badge/pi-package-6f42c1?style=flat-square" /></a>
|
|
7
|
+
<a href="https://www.npmjs.com/package/@senad-d/observme"><img alt="npm" src="https://img.shields.io/npm/v/%40senad-d%2Fobservme?style=flat-square" /></a>
|
|
8
|
+
<a href="LICENSE"><img alt="license" src="https://img.shields.io/badge/license-MIT-blue?style=flat-square" /></a>
|
|
9
|
+
<a href="https://sonarcloud.io/summary/new_code?id=senad-d_ObservMe"><img alt="Quality Gate Status" src="https://sonarcloud.io/api/project_badges/measure?project=senad-d_ObservMe&metric=alert_status" /></a>
|
|
10
|
+
</p>
|
|
11
|
+
|
|
12
|
+
<p align="center">
|
|
13
|
+
OpenTelemetry observability for <a href="https://pi.dev">pi</a> agent sessions.
|
|
14
|
+
<br />Turns Pi session, turn, tool, LLM, bash, branch, compaction, model, thinking, and agent-lineage events into OTLP traces, metrics, and logs.
|
|
15
|
+
</p>
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
ObservMe maps Pi lifecycle and session events to OpenTelemetry traces, metrics, and logs, then exports them via OTLP to an OpenTelemetry Collector. It uses `pi.*`, `observme.*`, and applicable official `gen_ai.*` attributes. ObservMe is **privacy-preserving by default**: prompts, responses, thinking content, tool arguments/results, bash commands/output, and file paths are not captured unless explicitly enabled and redacted.
|
|
20
|
+
|
|
21
|
+
<table align="center">
|
|
22
|
+
<tr>
|
|
23
|
+
<th>ObservMe</th>
|
|
24
|
+
</tr>
|
|
25
|
+
<tr>
|
|
26
|
+
<td align="center">
|
|
27
|
+
<img src="img/demo.gif" alt="ObservMe banner" title="ObservMe" width="760">
|
|
28
|
+
</td>
|
|
29
|
+
</tr>
|
|
30
|
+
</table>
|
|
31
|
+
|
|
32
|
+
- **OTLP-first:** emits OpenTelemetry traces, metrics, and logs; no durable local telemetry database.
|
|
33
|
+
- **Fail open:** if the Collector or backend is unreachable, Pi keeps running — telemetry is dropped, never blocking.
|
|
34
|
+
- **Privacy by default:** optional content capture starts disabled and passes through the redaction pipeline when enabled.
|
|
35
|
+
- **Agent lineage aware:** propagates parent/child/root agent identity and W3C trace context across subagent process boundaries without adding high-cardinality identifiers to metric labels.
|
|
36
|
+
- **Pi-native:** usable as a Pi package, project-local install, git install, or local source checkout.
|
|
37
|
+
|
|
38
|
+
> **Security:** Pi packages run with your full system permissions. ObservMe reads Pi session/event data through Pi's extension API, does not execute shell commands itself, and sends telemetry only to configured OTLP endpoints. Read [`SECURITY.md`](SECURITY.md).
|
|
39
|
+
|
|
40
|
+
## Table of Contents
|
|
41
|
+
|
|
42
|
+
- [Implementation Status](#implementation-status)
|
|
43
|
+
- [Quick Start](#quick-start)
|
|
44
|
+
- [Installation](#installation)
|
|
45
|
+
- [Commands](#commands)
|
|
46
|
+
- [Architecture](#architecture)
|
|
47
|
+
- [Configuration and Privacy](#configuration-and-privacy)
|
|
48
|
+
- [Safety Model](#safety-model)
|
|
49
|
+
- [Dashboards and Examples](#dashboards-and-examples)
|
|
50
|
+
- [Documentation Set](#documentation-set)
|
|
51
|
+
- [Development](#development)
|
|
52
|
+
- [Publishing](#publishing)
|
|
53
|
+
- [License](#license)
|
|
54
|
+
|
|
55
|
+
---
|
|
56
|
+
|
|
57
|
+
## Implementation Status
|
|
58
|
+
|
|
59
|
+
This checkout implements the ObservMe MVP scope from the packaged `ObservMe-Production-Docs/00-README.md` design overview:
|
|
60
|
+
|
|
61
|
+
- Extension load checks and `/obs health` backend checks.
|
|
62
|
+
- Session, workflow, agent-run, turn, LLM, tool, bash, subagent-spawn, wait/join, compaction, branch, model-change, and thinking-level telemetry.
|
|
63
|
+
- Multi-agent tree metrics for active agents, depth, fan-out, orphan agents, trace-context propagation failures, child failures, and workflow duration.
|
|
64
|
+
- Session-scoped OTLP trace, metric, and log exporters with bounded queues, timeouts, and shutdown flushing.
|
|
65
|
+
- Configurable capture controls, redaction, path scrubbing, hashing, truncation, and high-cardinality metric-label guards.
|
|
66
|
+
- Grafana dashboard JSON, alert rules, SLO definitions, Collector examples, and compatibility matrix artifacts.
|
|
67
|
+
- Unit, contract, integration, chaos/failure, performance, package, and smoke-test coverage in the validation pipeline.
|
|
68
|
+
|
|
69
|
+
The repository version is currently `0.1.0`. Run the validation commands below before publishing or deploying a release build.
|
|
70
|
+
|
|
71
|
+
## Quick Start
|
|
72
|
+
|
|
73
|
+
Use this checklist to confirm telemetry end to end. You need a supported Pi installation; see the [compatibility matrix](docs/compatibility-matrix.md). If you already have an OTLP Collector and Grafana stack, skip the first step.
|
|
74
|
+
|
|
75
|
+
### 1) Start the included local stack
|
|
76
|
+
|
|
77
|
+
The local stack is available in a repository checkout, not in the npm package:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
git clone https://github.com/senad-d/ObservMe.git
|
|
81
|
+
cd ObservMe/observability-stack
|
|
82
|
+
cp .env.example .env
|
|
83
|
+
mkdir -p secrets
|
|
84
|
+
if [ ! -s secrets/grafana_admin_password ]; then
|
|
85
|
+
openssl rand -hex 24 > secrets/grafana_admin_password
|
|
86
|
+
fi
|
|
87
|
+
chmod 600 secrets/grafana_admin_password
|
|
88
|
+
docker compose up -d
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
### 2) Install the extension
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
cd /path/to/your/project
|
|
95
|
+
pi install npm:@senad-d/observme
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
For local-stack credentials or other environment variables, copy `.env.example` from an ObservMe checkout to the project as `.env`. Keep secrets out of `.pi/observme.yaml`.
|
|
99
|
+
|
|
100
|
+
### 3) Start Pi and check connectivity
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
pi
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Inside Pi, run:
|
|
107
|
+
|
|
108
|
+
```text
|
|
109
|
+
/obs status
|
|
110
|
+
/obs health
|
|
111
|
+
/obs session
|
|
112
|
+
/obs trace
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### 4) Generate and inspect telemetry
|
|
116
|
+
|
|
117
|
+
In the running Pi session, start a normal task, such as:
|
|
118
|
+
|
|
119
|
+
```text
|
|
120
|
+
Summarize this repository in one sentence
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
Open your configured Grafana URL, then open **ObservMe Overview** and confirm that the task produced traces, logs, and metrics. ObservMe never blocks Pi execution when the backend is unavailable.
|
|
124
|
+
|
|
125
|
+
On the first trusted `session_start` for a project, ObservMe creates `observme.yaml` under Pi's exported project config directory (`.pi/observme.yaml` in the standard distribution) when the file is missing. The starter keeps raw prompt, response, thinking, tool, bash, and path capture disabled by default; opt in only to the specific redacted capture fields you need.
|
|
126
|
+
|
|
127
|
+
### Run from a source checkout
|
|
128
|
+
|
|
129
|
+
```bash
|
|
130
|
+
git clone https://github.com/senad-d/ObservMe.git
|
|
131
|
+
cd ObservMe
|
|
132
|
+
npm ci
|
|
133
|
+
npm run validate
|
|
134
|
+
pi --no-extensions -e .
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
## Installation
|
|
138
|
+
|
|
139
|
+
| Scope | Command | Notes |
|
|
140
|
+
| --- | --- | --- |
|
|
141
|
+
| Global | `pi install npm:@senad-d/observme` | Loads in every trusted Pi project. |
|
|
142
|
+
| Project-local | `pi install npm:@senad-d/observme -l` | Writes to `.pi/settings.json` in the current project. |
|
|
143
|
+
| One run | `pi -e npm:@senad-d/observme` | Try without changing settings. |
|
|
144
|
+
| Git | `pi install git:senad-d/ObservMe` | Install from Git; use a tag or commit when you need a fixed version. |
|
|
145
|
+
| Local checkout | `pi --no-extensions -e .` | Develop or test this repository. |
|
|
146
|
+
|
|
147
|
+
## Commands
|
|
148
|
+
|
|
149
|
+
All commands are registered under `/obs`.
|
|
150
|
+
|
|
151
|
+
| Command | Description | Backend access |
|
|
152
|
+
| --- | --- | --- |
|
|
153
|
+
| `/obs status` | Shows local ObservMe enablement, OTLP endpoint, config source/trust status, Grafana query URL/readiness, signal enablement, capture flags, queue drops, and last export error. | Local state only |
|
|
154
|
+
| `/obs health` | Checks Collector, Grafana, and configured datasource reachability with bounded timeouts. | Collector/Grafana |
|
|
155
|
+
| `/obs session` | Shows current-session turn, LLM-call, tool-call, cost, and trace-link state from runtime counters. | Local state only |
|
|
156
|
+
| `/obs cost` | Queries Prometheus/Mimir for safe model/provider token and cost aggregates. | Prometheus/Mimir via Grafana |
|
|
157
|
+
| `/obs trace` | Returns a Grafana Tempo trace link for current, last-turn, or safe session-id scopes. | Grafana link construction |
|
|
158
|
+
| `/obs link` | Direct Grafana trace-link helper using the configured URL template. | Grafana link construction |
|
|
159
|
+
| `/obs tools` | Queries tool-call and tool-failure aggregates using only safe tool/error labels. | Prometheus/Mimir via Grafana |
|
|
160
|
+
| `/obs errors` | Queries Loki for recent failed/dropped/orphan event names and renders a capped summary. | Loki via Grafana |
|
|
161
|
+
| `/obs logs` | Queries Loki for the current session's normalized `pi_session_id` label and renders a capped summary. | Loki via Grafana |
|
|
162
|
+
| `/obs agents` | Shows current workflow/agent identity, lineage, fan-out/depth, active children, orphan status, and wait/join hints. | Local state plus safe Tempo/Prometheus drill-downs |
|
|
163
|
+
| `/obs backfill` | Optional historical replay for the current session with explicit confirmation, replay markers, redaction, export rate limits, and a bounded `--since` window up to 30d. | OTLP export when confirmed |
|
|
164
|
+
|
|
165
|
+
Query-backed commands enforce configured timeouts and result limits and reject raw prompt, command, path, or other sensitive query inputs.
|
|
166
|
+
|
|
167
|
+
Trace visibility note: `/obs trace` and `/obs session` can link to the current trace as soon as `session_start` creates it. During an active Pi session, Tempo may show ended child spans before the long-lived root `pi.session` span appears; the canonical root span is ended, flushed, and visible after `session_shutdown`.
|
|
168
|
+
|
|
169
|
+
## Architecture
|
|
170
|
+
|
|
171
|
+
```text
|
|
172
|
+
Pi Agent
|
|
173
|
+
└── ObservMe Pi Extension
|
|
174
|
+
├── OTEL Traces ──► OTEL Collector ──► Tempo
|
|
175
|
+
├── OTEL Metrics ──► OTEL Collector ──► Prometheus / Mimir
|
|
176
|
+
└── OTEL Logs ──► OTEL Collector ──► Loki
|
|
177
|
+
|
|
178
|
+
Grafana
|
|
179
|
+
├── Tempo datasource
|
|
180
|
+
├── Loki datasource
|
|
181
|
+
└── Prometheus/Mimir datasource
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
The extension factory in `src/extension.ts` only registers handlers and commands. OTEL SDK startup happens from `session_start`, and bounded flush/shutdown happens from `session_shutdown`, so importing or registering the extension does not open exporters, timers, or sockets.
|
|
185
|
+
|
|
186
|
+
See `ObservMe-Production-Docs/02-reference-architecture.md` for the full architecture. Implementation planning specs are repository-only and live at <https://github.com/senad-d/ObservMe/tree/main/specs>. A reference Docker Compose deployment of the Grafana/Tempo/Loki/Prometheus/Collector stack is also repository-only at <https://github.com/senad-d/ObservMe/tree/main/observability-stack> because it contains generated local credentials and state placeholders.
|
|
187
|
+
|
|
188
|
+
## Configuration and Privacy
|
|
189
|
+
|
|
190
|
+
ObservMe supports layered configuration with this precedence:
|
|
191
|
+
|
|
192
|
+
```text
|
|
193
|
+
defaults → global ~/.pi/agent/observme.yaml → trusted project config → trusted project .env → system environment variables → runtime options
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
The extension automatically creates `<project>/<CONFIG_DIR_NAME>/observme.yaml` (`<project>/.pi/observme.yaml` in the standard distribution) the first time Pi emits `session_start` in a trusted project and the file does not already exist. It uses Pi's exported config-directory name and serialized file-mutation queue. Pi emits that lifecycle for startup, `/reload`, new-session, resume, and fork flows; ObservMe attempts the same idempotent bootstrap for each trusted flow, skips untrusted projects or contexts without `ctx.cwd`, and never overwrites an existing file. Edit this file for project-specific setup: `otlp.endpoint` / `otlp.signalEndpoints` for your Collector, `resource.attributes` for service/project/tenant/environment identity, `capture` and `privacy` for content visibility, and `query.grafana` plus `query.links.traceUrlTemplate` for `/obs` Grafana commands. The generated starter keeps content-capture flags disabled, `privacy.redactionEnabled: true`, and `privacy.allowUnsafeCapture: false`; for local debugging, set only the needed `capture.*` flags to `true` and keep redaction enabled. Keep secrets out of YAML; use environment variables or a trusted project `.env` for values such as `OBSERVME_OTLP_TOKEN`, `OBSERVME_GRAFANA_TOKEN`, `OBSERVME_GRAFANA_PASSWORD`, and `OBSERVME_HASH_SALT`. See [`docs/configuration.md`](docs/configuration.md) for the quick configuration guide.
|
|
197
|
+
|
|
198
|
+
Factory-safe loading uses defaults/global/system-environment/runtime options only. Session-scoped loading can add trusted project config and a project-local `.env` when Pi marks the project trusted. `/obs status` reports the effective config source, whether project-local config was loaded, skipped because the project is untrusted, or missing, plus bounded rejection issue codes, the configured Grafana URL, and query-readiness status without rendering tokens or passwords. In untrusted projects, ObservMe does not read project-local config or `.env` files and uses safe defaults/global/system-environment layers instead. Invalid or unsafe configuration emits a bounded `config.rejected` diagnostic and falls back safely without exposing rejected values.
|
|
199
|
+
|
|
200
|
+
Cross-process agent lineage has a separate boundary: only the Pi process environment available to the shipped extension, or explicit runtime options for controlled embedders, is eligible for parent provenance. Project-local `.env` values configure ObservMe but cannot establish lineage. A child accepts only a complete validated workflow/parent/root/depth/spawn envelope and valid W3C context; malformed or stale envelopes fail open to a root/orphan fallback with sanitized telemetry.
|
|
201
|
+
|
|
202
|
+
Default capture policy:
|
|
203
|
+
|
|
204
|
+
```yaml
|
|
205
|
+
capture:
|
|
206
|
+
prompts: false
|
|
207
|
+
responses: false
|
|
208
|
+
thinking: false
|
|
209
|
+
toolArguments: false
|
|
210
|
+
toolResults: false
|
|
211
|
+
bashCommands: false
|
|
212
|
+
bashOutput: false
|
|
213
|
+
filePaths: false
|
|
214
|
+
privacy:
|
|
215
|
+
redactionEnabled: true
|
|
216
|
+
allowUnsafeCapture: false
|
|
217
|
+
allowInsecureTransport: false
|
|
218
|
+
workflow:
|
|
219
|
+
enabled: true
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
When optional content capture is enabled, live telemetry and `/obs backfill` use the same policy. With `privacy.redactionEnabled: true`, the redaction pipeline applies size guards, secret detection, optional PII detection, cross-platform absolute-path scrubbing (`hash`, `basename`, `full`, or `drop`), custom regex redactors, truncation metadata, and tenant-salted hashing before export. POSIX, Windows-drive, and UNC paths are scrubbed without treating normal URLs or harmless slash-separated prose as paths. Redaction failures drop content and emit `redaction.failed` diagnostics. With `privacy.redactionEnabled: false` and `privacy.allowUnsafeCapture: true`, captured content is exported raw but still truncated to the configured limit. Hash salts are read from secure environment/runtime config, not hardcoded.
|
|
223
|
+
|
|
224
|
+
To show failed-tool output such as GuardMe denial messages in the Tools dashboard, explicitly set `capture.toolResults: true` (or `OBSERVME_CAPTURE_TOOL_RESULTS=true`) while keeping redaction enabled and providing `OBSERVME_HASH_SALT`. New failed calls then emit a separate `tool.error.captured` content log; normal `tool.call.failed` operational logs remain content-free.
|
|
225
|
+
|
|
226
|
+
Metadata such as token counts, duration, status, model/provider, tool name, and agent role/depth is captured by default. High-cardinality identifiers (session IDs, workflow IDs, agent IDs, trace/span IDs, entry IDs) are allowed on spans/logs for drill-down but are blocked from Prometheus metric labels.
|
|
227
|
+
|
|
228
|
+
Grafana-backed query commands use the Grafana HTTP API, so browser login cookies are irrelevant. Configure either a Grafana service-account token (`OBSERVME_GRAFANA_TOKEN`) or local Basic auth (`OBSERVME_GRAFANA_USERNAME`/`OBSERVME_GRAFANA_PASSWORD`); token auth takes precedence and secrets are never rendered in command errors. You can supply these values as system environment variables before starting Pi, or copy the `.env.example` shipped with ObservMe to `.env` in a trusted project; system environment variables override `.env` values.
|
|
229
|
+
|
|
230
|
+
### Supported local-stack query profile
|
|
231
|
+
|
|
232
|
+
The repository-only local stack at <https://github.com/senad-d/ObservMe/tree/main/observability-stack> supports `/obs` query commands through authenticated Grafana behind nginx HTTPS at `https://observability.local`. The default Compose stack does not publish Grafana on `localhost:3000`; use that direct HTTP path only if you add your own override.
|
|
233
|
+
|
|
234
|
+
```yaml
|
|
235
|
+
query:
|
|
236
|
+
enabled: true
|
|
237
|
+
links:
|
|
238
|
+
traceUrlTemplate: https://observability.local/explore?left=...
|
|
239
|
+
grafana:
|
|
240
|
+
url: https://observability.local
|
|
241
|
+
token: ${OBSERVME_GRAFANA_TOKEN}
|
|
242
|
+
username: admin
|
|
243
|
+
password: ${OBSERVME_GRAFANA_PASSWORD}
|
|
244
|
+
datasourceUids:
|
|
245
|
+
tempo: tempo
|
|
246
|
+
loki: loki
|
|
247
|
+
prometheus: prometheus
|
|
248
|
+
tls:
|
|
249
|
+
insecureSkipVerify: true
|
|
250
|
+
transport:
|
|
251
|
+
preferIPv4: true
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
Create a Grafana service-account token in Grafana (Administration → Users and access → Service accounts → Add service account/token; Viewer is enough for read-only datasource queries) and export it as `OBSERVME_GRAFANA_TOKEN`, or for local-only Basic auth read the generated admin password from the repository-only local stack's secrets directory. If you prefer a project-local env file, run `cp .env.example .env`, fill either `OBSERVME_GRAFANA_TOKEN` or `OBSERVME_GRAFANA_PASSWORD`, then restart Pi from that project. The local Collector and Loki profile uses `service.name=observme-pi-extension`; Loki queries use normalized labels such as `service_name`, `pi_session_id`, `pi_agent_id`, `pi_agent_run_id`, `event_name`, and `event_category`. If data is visible in Grafana but `/obs` commands fail, run `/obs health` and check extension env loading, Grafana auth, datasource UIDs, TLS, and DNS details.
|
|
255
|
+
|
|
256
|
+
### Show LLM chat content in Grafana
|
|
257
|
+
|
|
258
|
+
LLM prompt, response, and thinking bodies are hidden by default. To display redacted chat content in Tempo span attributes and Loki log panels, restart Pi with explicit capture and a tenant hash salt:
|
|
259
|
+
|
|
260
|
+
```bash
|
|
261
|
+
export OBSERVME_CAPTURE_PROMPTS=true
|
|
262
|
+
export OBSERVME_CAPTURE_RESPONSES=true
|
|
263
|
+
export OBSERVME_CAPTURE_THINKING=true
|
|
264
|
+
export OBSERVME_REDACTION_ENABLED=true
|
|
265
|
+
export OBSERVME_HASH_SALT="$(openssl rand -hex 32)"
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
For intentionally raw local debugging only, set `OBSERVME_REDACTION_ENABLED=false` together with `OBSERVME_ALLOW_UNSAFE_CAPTURE=true`. Environment variables and trusted project `.env` values override `.pi/observme.yaml`, so remove stale overrides before relying on YAML privacy settings.
|
|
269
|
+
|
|
270
|
+
Only new LLM events emitted after these settings and the updated Collector are active can show content; older telemetry dropped by the Collector cannot be recovered. Open the **ObservMe LLM Conversations** dashboard for a dedicated opt-in chat timeline. Do not query Grafana with raw prompt or response text.
|
|
271
|
+
|
|
272
|
+
Full configuration schema: `ObservMe-Production-Docs/12-configuration-reference.md`. Full redaction/privacy design: `ObservMe-Production-Docs/06-security-privacy-redaction.md`.
|
|
273
|
+
|
|
274
|
+
## Safety Model
|
|
275
|
+
|
|
276
|
+
- ObservMe does not execute shell commands itself; it only observes tool/bash execution events emitted by Pi.
|
|
277
|
+
- ObservMe never blocks Pi agent execution when the observability backend is degraded or unreachable.
|
|
278
|
+
- ObservMe starts exporters only for a trusted session and shuts them down with bounded timeouts.
|
|
279
|
+
- ObservMe does not continuously tail the full Pi session file; startup recovery reads only minimal session/correlation state, and historical replay requires an explicit `/obs backfill` command.
|
|
280
|
+
- Optional content capture is disabled by default and must pass through the redaction pipeline before export when enabled.
|
|
281
|
+
- `/obs` query commands are read-only and are not imported by telemetry-emission code paths.
|
|
282
|
+
- Invalid or unsafe configuration falls back to safe defaults with bounded `config.rejected`, `/obs status`, and available Pi UI diagnostics; rejected values are never rendered. Intentionally unsafe capture emits a visible warning.
|
|
283
|
+
|
|
284
|
+
See [`SECURITY.md`](SECURITY.md) and `ObservMe-Production-Docs/06-security-privacy-redaction.md` for details.
|
|
285
|
+
|
|
286
|
+
## Dashboards and Examples
|
|
287
|
+
|
|
288
|
+
These assets are included in the npm package:
|
|
289
|
+
|
|
290
|
+
- Grafana dashboards: `dashboards/observme-*.json`, including Overview, SLO Health, Export Health, Trace Journey, Agents, Agent Node Graphs, Cost, Models, Latency, Tools, Errors, Logs and LLM I/O, LLM Conversations, and Branches/Compactions.
|
|
291
|
+
- Alert rules: `dashboards/observme-alerts.yaml`.
|
|
292
|
+
- SLO definitions: `dashboards/observme-slos.yaml`.
|
|
293
|
+
- Supported local-stack ObservMe config: `examples/observme.yaml`.
|
|
294
|
+
- Production Collector config with high-cardinality and content-drop processors: `examples/collector.yaml`.
|
|
295
|
+
- Compatibility matrix: `docs/compatibility-matrix.md`.
|
|
296
|
+
|
|
297
|
+
Open **ObservMe Overview** first for health/SLO chips, workload, cost, latency, agent-lineage status, and time-preserving links to focused dashboards. Use **Trace Journey** to follow a session/workflow/agent/run across Prometheus aggregates, Loki logs, and Tempo traces. Use **LLM Conversations** only for redacted opt-in content; raw prompt, response, command, path, and error-message values should never be placed in dashboard URLs or Prometheus labels. Empty failure tables normally mean no matching failures in the selected range, while optional content panels can be empty because capture is disabled by default.
|
|
298
|
+
|
|
299
|
+
The full dashboard map, standard variables, drill-down workflow, threshold colors, and zero-state semantics are documented in `ObservMe-Production-Docs/09-dashboards-alerts-slos.md`.
|
|
300
|
+
|
|
301
|
+
## Documentation Set
|
|
302
|
+
|
|
303
|
+
- [`docs/configuration.md`](docs/configuration.md): quick configuration guide.
|
|
304
|
+
- [`docs/compatibility-matrix.md`](docs/compatibility-matrix.md): supported Pi, Node.js, OpenTelemetry, and local-stack versions.
|
|
305
|
+
- [`docs/validation-flow.md`](docs/validation-flow.md): secret-safe Grafana and `/obs` troubleshooting flow.
|
|
306
|
+
- `ObservMe-Production-Docs/`: architecture, configuration, privacy, dashboards, and operational reference docs.
|
|
307
|
+
- [`SECURITY.md`](SECURITY.md): security reporting and package safety guidance.
|
|
308
|
+
|
|
309
|
+
## Development
|
|
310
|
+
|
|
311
|
+
```bash
|
|
312
|
+
npm ci
|
|
313
|
+
npm run validate
|
|
314
|
+
```
|
|
315
|
+
|
|
316
|
+
Useful checks:
|
|
317
|
+
|
|
318
|
+
```bash
|
|
319
|
+
npm run typecheck
|
|
320
|
+
npm run typecheck:test
|
|
321
|
+
npm run lint # TypeScript, test TypeScript, ESLint, formatting, and script syntax checks
|
|
322
|
+
npm run lint:fix # ESLint auto-fix pass
|
|
323
|
+
npm run format:check
|
|
324
|
+
npm run test
|
|
325
|
+
npm run test:integration:collector
|
|
326
|
+
npm run test:integration:grafana-stack
|
|
327
|
+
npm run check:pack
|
|
328
|
+
npm run smoke:pi-runtime
|
|
329
|
+
npm run validate:grafana-obs
|
|
330
|
+
pi --no-extensions -e .
|
|
331
|
+
```
|
|
332
|
+
|
|
333
|
+
`npm run smoke:pi-lifecycle` runs lifecycle handlers with traces, metrics, logs, and query integration disabled through an explicit offline test config. `npm run smoke:pi-runtime` launches a real Pi RPC process against a temporary trusted project, verifies `/obs` discovery plus `/obs status` and `/obs session` routing after `session_start`, and exercises a bounded `/obs cost` timeout against a local deterministic Grafana backend.
|
|
334
|
+
|
|
335
|
+
`npm run test:coverage` writes `coverage/node-test-coverage.txt` and SonarQube-readable `coverage/lcov.info` for the default non-Docker test suite; `coverage/` is git-ignored, and `rm -rf coverage` removes generated coverage artifacts after review. Docker integration coverage is opt-in with `OBSERVME_INCLUDE_INTEGRATION_COVERAGE=1 npm run test:coverage`.
|
|
336
|
+
|
|
337
|
+
End-to-end troubleshooting flow: [`docs/validation-flow.md`](docs/validation-flow.md) provides a deterministic, secret-safe checklist and script for the common user-facing case where Grafana has data but `/obs` commands are failing.
|
|
338
|
+
|
|
339
|
+
## Publishing
|
|
340
|
+
|
|
341
|
+
ObservMe publishes to npm as `@senad-d/observme`. Run from a clean working tree after validation and a `CHANGELOG.md` update.
|
|
342
|
+
|
|
343
|
+
```bash
|
|
344
|
+
npm login
|
|
345
|
+
npm whoami
|
|
346
|
+
npm run validate
|
|
347
|
+
npm run validate:grafana-obs # with the release-candidate stack and Grafana query env configured
|
|
348
|
+
npm version <version>
|
|
349
|
+
npm publish --access public
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
Push the release commit and tag after the package is published.
|
|
353
|
+
|
|
354
|
+
## License
|
|
355
|
+
|
|
356
|
+
MIT
|
package/SECURITY.md
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Security Policy
|
|
2
|
+
|
|
3
|
+
ObservMe is a Pi extension that instruments Pi agent sessions and exports OpenTelemetry traces, metrics, and logs to an external observability stack such as an OpenTelemetry Collector, Grafana Tempo, Loki, Prometheus, Mimir, or Grafana Cloud. This policy summarizes the implemented trust model; see `ObservMe-Production-Docs/06-security-privacy-redaction.md` for the full security, privacy, and redaction specification.
|
|
4
|
+
|
|
5
|
+
## Trust model
|
|
6
|
+
|
|
7
|
+
Pi packages and extensions run with the full local permissions of the user account that starts Pi. Review extension source before installing it, pin versions in sensitive environments, and install only from trusted sources.
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
pi install npm:@senad-d/observme@<version>
|
|
11
|
+
pi install git:senad-d/ObservMe@<tag>
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
ObservMe is implemented to:
|
|
15
|
+
|
|
16
|
+
- Never execute shell commands itself; it only observes tool/bash execution events emitted by Pi.
|
|
17
|
+
- Read Pi session/event data through Pi extension APIs, with no continuous full-session-file tailing.
|
|
18
|
+
- Start OTEL exporters only from `session_start` and stop them from `session_shutdown` with bounded flush/shutdown timeouts.
|
|
19
|
+
- Send telemetry only to configured OTLP endpoints, with an OpenTelemetry Collector recommended for production.
|
|
20
|
+
- Disable prompt, response, thinking, tool-argument, tool-result, bash-command, bash-output, and file-path capture by default.
|
|
21
|
+
- Redact secrets, optional PII, POSIX/Windows-drive/UNC absolute paths, URL credentials, custom configured patterns, and oversized content before optional content export; normal URLs and harmless slash-separated prose are not treated as filesystem paths. Custom regex redactors are bounded and reject unsupported risky constructs such as nested quantified groups, lookaround, named groups, and backreferences.
|
|
22
|
+
- Accept child-agent lineage only from a complete validated Pi process-environment envelope (or controlled runtime override), never from trusted project `.env`; malformed, partial, oversized, or stale values fail open without exposing inherited values.
|
|
23
|
+
- Keep session IDs, workflow IDs, agent IDs, trace/span IDs, raw paths, raw commands, raw prompts, and raw errors out of Prometheus metric labels.
|
|
24
|
+
- Fail open: Pi keeps running if the Collector, Grafana, Tempo, Loki, or Prometheus backend is unreachable.
|
|
25
|
+
- Require explicit user confirmation for `/obs backfill` historical replay and mark replayed telemetry with `observme.replayed=true`.
|
|
26
|
+
|
|
27
|
+
## Reporting vulnerabilities
|
|
28
|
+
|
|
29
|
+
Please report suspected security vulnerabilities privately by email: <senad.dizdarevic@proton.me>.
|
|
30
|
+
|
|
31
|
+
For non-sensitive issues, use the repository issue tracker:
|
|
32
|
+
|
|
33
|
+
<https://github.com/senad-d/ObservMe/issues>
|
|
34
|
+
|
|
35
|
+
Do not open public issues for security-sensitive reports that include exploit details, private repository contents, secrets, or credentials.
|
|
36
|
+
|
|
37
|
+
## Secure development checklist
|
|
38
|
+
|
|
39
|
+
- Do not commit secrets, tokens, local `.pi/` state, or generated artifacts.
|
|
40
|
+
- Document any file, shell, network, or credential access added by the extension.
|
|
41
|
+
- Avoid starting background resources in the extension factory; start them from `session_start` and stop them from `session_shutdown`.
|
|
42
|
+
- Keep package contents minimal with `npm run check:pack`.
|
|
43
|
+
- Use isolated smoke tests with `pi --no-extensions -e .`.
|
|
44
|
+
- Keep default telemetry capture settings privacy-preserving; changing any default capture flag to `true` requires explicit review.
|
|
45
|
+
- Run `npm run validate` before release and after security-sensitive changes.
|