@senad-d/observme 0.1.4 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +35 -0
- package/README.md +24 -8
- package/docs/agent-subagent-observability-requirements.md +4 -3
- package/docs/compatibility-matrix.md +14 -3
- package/docs/configuration.md +2 -2
- package/docs/extension-integration.md +20 -13
- package/docs/reference/03-pi-event-and-session-model.md +6 -6
- package/docs/reference/04-telemetry-semantic-conventions.md +1 -0
- package/docs/reference/06-security-privacy-redaction.md +13 -6
- package/docs/reference/08-query-grafana-integration.md +30 -7
- package/docs/reference/11-deployment-runbooks.md +21 -0
- package/docs/reference/12-configuration-reference.md +30 -4
- package/docs/review-validation.md +10 -0
- package/examples/integrations/subagent-runner.ts +8 -5
- package/examples/observme.yaml +2 -2
- package/package.json +12 -4
- package/src/commands/obs-agents.ts +17 -12
- package/src/commands/obs-backfill.ts +2 -2
- package/src/commands/obs-command-support.ts +2 -1
- package/src/commands/obs-cost.ts +15 -7
- package/src/commands/obs-health.ts +22 -2
- package/src/commands/obs-loki-summary.ts +4 -8
- package/src/commands/obs-session.ts +35 -28
- package/src/commands/obs-status.ts +56 -13
- package/src/commands/obs-tools.ts +19 -8
- package/src/commands/obs-trace.ts +9 -0
- package/src/commands/obs.ts +6 -1
- package/src/config/bootstrap-project-config.ts +49 -32
- package/src/config/defaults.ts +0 -2
- package/src/config/load-config.ts +270 -92
- package/src/config/project-paths.ts +318 -6
- package/src/config/query-limits.ts +10 -0
- package/src/config/schema.ts +9 -8
- package/src/config/transport-security.ts +107 -0
- package/src/config/validate.ts +68 -11
- package/src/extension.ts +2 -12
- package/src/integration.ts +6 -2
- package/src/otel/logs.ts +24 -13
- package/src/otel/metrics.ts +24 -13
- package/src/otel/otlp-endpoint.ts +41 -2
- package/src/otel/otlp-http-options.ts +11 -0
- package/src/otel/sdk.ts +155 -9
- package/src/otel/shutdown.ts +39 -11
- package/src/otel/traces.ts +34 -16
- package/src/pi/agent-tree-tracker.ts +14 -1
- package/src/pi/compatibility.ts +30 -0
- package/src/pi/event-handlers/agent-turn.ts +16 -9
- package/src/pi/event-handlers/lifecycle.ts +207 -75
- package/src/pi/event-handlers/llm.ts +17 -9
- package/src/pi/event-handlers/session-events.ts +53 -19
- package/src/pi/event-handlers/tool-bash.ts +21 -27
- package/src/pi/handler-internals.ts +16 -26
- package/src/pi/handler-runtime.ts +142 -55
- package/src/pi/handler-types.ts +30 -15
- package/src/pi/handlers.ts +12 -1
- package/src/pi/integration-api.ts +27 -15
- package/src/pi/session-correlation.ts +167 -0
- package/src/pi/subagent-spawn.ts +164 -31
- package/src/privacy/redact.ts +574 -68
- package/src/privacy/secret-patterns.ts +6 -1
- package/src/query/grafana-readiness.ts +18 -2
- package/src/query/grafana-transport.ts +150 -27
- package/src/query/grafana-url.ts +36 -0
- package/src/query/grafana.ts +4 -136
- package/src/query/loki.ts +2 -4
- package/src/query/prometheus.ts +8 -10
- package/src/query/tempo.ts +2 -4
- package/src/query/trace-link.ts +186 -0
- package/src/safety/display-bounds.ts +84 -0
- package/src/semconv/metrics.ts +1 -0
- package/src/semconv/values.ts +2 -0
package/CHANGELOG.md
CHANGED
|
@@ -21,6 +21,11 @@
|
|
|
21
21
|
|
|
22
22
|
### Changed
|
|
23
23
|
|
|
24
|
+
- Raised the release-tested Pi and exact compatibility CI target to 0.81.1 while retaining 0.80.5 as the earliest validated target.
|
|
25
|
+
- Made the 0.1.5 hardening patch self-contained with explicit review units, all imported production modules included, and unrelated specification history preserved.
|
|
26
|
+
- Established a typed Pi event registration contract, removed non-event legacy registrations, pinned the release-tested Pi API for validation, and added capability-based pre-registration diagnostics.
|
|
27
|
+
- Centralized embedded-credential Grafana URL diagnostics so configuration validation, readiness checks, URL construction, and transports share one safe failure class and operator guidance.
|
|
28
|
+
- Documented credential-free Grafana base URLs and the fail-closed canonical project-file boundary across configuration, security, query, and troubleshooting guidance.
|
|
24
29
|
- Completed production active-agent lease documentation, raw-query migration guidance, GitHub Actions/self-hosted clock and cleanup runbooks, missing/expired-lease troubleshooting, Collector restart semantics, and sanitized release-validation evidence.
|
|
25
30
|
- Reframed the Collector's five-minute Prometheus `metric_expiration` as exporter-wide stale-series/cardinality cleanup, longer than the default active-agent lease and independent of leased liveness.
|
|
26
31
|
- Migrated active-agent dashboard totals, bounded breakdowns, aggregate topology inputs, and stuck-high alerts to leased activity, with raw/expired-claim diagnostics and a deployment-tunable stale-claim alert.
|
|
@@ -38,6 +43,36 @@
|
|
|
38
43
|
|
|
39
44
|
### Fixed
|
|
40
45
|
|
|
46
|
+
- Removed Pi version gating from extension startup entirely; tested versions are evidence only, essential ExtensionAPI capabilities are checked before registration, and optional APIs remain feature-detected.
|
|
47
|
+
- Made missing project-path component reversal explicit.
|
|
48
|
+
- Applied one control-safe 64-row/8,192-character policy to every `/obs` notification and capped query result counts at configuration and runtime boundaries.
|
|
49
|
+
- Coupled canonical project config, `.env`, and starter-config I/O to identity-verified file handles so concurrent symlink and ancestor swaps fail closed without exposing external paths.
|
|
50
|
+
- Rejected credentials embedded in Grafana base URLs during config validation, query readiness, and transport preflight without exposing their values.
|
|
51
|
+
- Rejected active and retained child-agent identifier collisions before creating integration spans, tree state, metrics, or propagation envelopes.
|
|
52
|
+
- Retained ownership of timed-out OpenTelemetry shutdowns, observed late settlement safely, and deferred session replacement until prior exporter cleanup completes.
|
|
53
|
+
- Replaced custom-redaction regex heuristics with bounded structural validation that rejects nested, ambiguous-alternative, and overlapping sequential repetition while preserving safe disjoint quantified alternatives.
|
|
54
|
+
- Used concise word-character syntax for single-brace unresolved trace-link placeholders.
|
|
55
|
+
- Used concise word-character syntax for dollar-brace unresolved trace-link placeholders.
|
|
56
|
+
- Used concise word-character syntax for double-brace unresolved trace-link placeholders.
|
|
57
|
+
- Split unresolved trace-link placeholder validation into simple per-syntax patterns while preserving rejection behavior.
|
|
58
|
+
- Documented why best-effort Grafana stream cancellation failures are intentionally ignored.
|
|
59
|
+
- Removed redundant explicit `undefined` support from the optional lifecycle model context property.
|
|
60
|
+
- Simplified lifecycle recovery header selection by replacing a nested conditional expression with explicit branches.
|
|
61
|
+
- Removed redundant awaiting of Pi's synchronous lifecycle status update.
|
|
62
|
+
- Replaced deprecated proxy tracer construction with an isolated always-off provider for disabled and shutdown trace states.
|
|
63
|
+
- Validated OTLP endpoints as secret-free absolute HTTP(S) URLs and constructed signal exporter paths with deterministic URL pathname semantics.
|
|
64
|
+
- Implemented opt-in, versioned, active-branch correlation persistence with bounded validation and idempotent reload recovery, and removed unsupported automatic replay configuration and synthetic duplicate startup telemetry.
|
|
65
|
+
- Bound live telemetry, query commands, and backfill correlation to Pi's typed session manager, preserved identity across reload, adopted replacement-session identity, and refreshed active metadata on session rename.
|
|
66
|
+
- Bounded and sanitized backend-derived `/obs cost`, `/obs tools`, and `/obs agents` labels, rows, and notification output with visible Unicode-safe truncation.
|
|
67
|
+
- Unified `/obs session`, `/obs trace`, and `/obs link` on one validated trace-link builder with canonical placeholders, structured Grafana fallback URLs, and bounded configuration diagnostics.
|
|
68
|
+
- Made multi-signal OpenTelemetry startup transactional with bounded rollback, a terminal failed controller state, sanitized Pi diagnostics, and clean later-session recovery.
|
|
69
|
+
- Preserved malformed environment and file configuration as bounded rejection diagnostics, with source-specific `/obs status` reporting and strict trusted `.env` parsing.
|
|
70
|
+
- Enforced coherent terminal subagent transitions across the integration API, agent tree, spans, events, metrics, runtime state, waits, and joins.
|
|
71
|
+
- Enforced production acknowledgement for Grafana and OTLP certificate-verification bypasses, wired retained OTLP TLS behavior into every exporter, and exposed effective transport security in `/obs status` and `/obs health`.
|
|
72
|
+
- Kept live-session and backfill OpenTelemetry providers scoped to ObservMe instead of replacing process-global providers.
|
|
73
|
+
- Redacted complete supported PEM private-key blocks, including malformed/truncated input, across live and backfilled content capture.
|
|
74
|
+
- Bounded all Grafana health and datasource response bodies before JSON parsing across default, custom Node, and injected fetch transports.
|
|
75
|
+
- Enforced top-level ObservMe disablement across lifecycle startup, runtime state, integration availability, and all OpenTelemetry signal factories.
|
|
41
76
|
- Restored clean npm dependency resolution by pinning TypeScript to the latest release supported by `typescript-eslint`.
|
|
42
77
|
- Integrated active-agent lease activation and deactivation with clean shutdown, duplicate/reload replacement, resume, and failed-start cleanup so final flushes cannot renew stale activity.
|
|
43
78
|
- Fixed session, workflow, agent, turn, LLM, tool, Bash, histogram, active-span, failure/recovery, and session-count telemetry accuracy.
|
package/README.md
CHANGED
|
@@ -162,12 +162,28 @@ const started = observme?.startSubagent({
|
|
|
162
162
|
});
|
|
163
163
|
|
|
164
164
|
if (started?.ok) {
|
|
165
|
-
|
|
166
|
-
|
|
165
|
+
let child;
|
|
166
|
+
try {
|
|
167
|
+
child = await launchChildPi({ env: started.env });
|
|
168
|
+
} catch (error) {
|
|
169
|
+
observme.failSubagent(started.spawnId, {
|
|
170
|
+
childAgentId: started.childAgentId,
|
|
171
|
+
errorClass: error instanceof Error ? error.name : "launcher_error",
|
|
172
|
+
});
|
|
173
|
+
}
|
|
174
|
+
|
|
175
|
+
if (child) {
|
|
176
|
+
const result = await waitForChildPi(child);
|
|
177
|
+
observme.completeSubagent(started.spawnId, {
|
|
178
|
+
childAgentId: started.childAgentId,
|
|
179
|
+
childStatus: result.status,
|
|
180
|
+
outcome: result.status,
|
|
181
|
+
});
|
|
182
|
+
}
|
|
167
183
|
}
|
|
168
184
|
```
|
|
169
185
|
|
|
170
|
-
The discovery helper fails open when the event bus or a provider is malformed. API methods also reject unsafe/oversized requests
|
|
186
|
+
The discovery helper fails open when the event bus or a provider is malformed. API methods also reject unsafe/oversized requests, duplicate active lifecycle IDs, and active or retained child-placeholder collisions without replacing existing telemetry state. Handle every discriminated failure result locally and keep orchestration functional.
|
|
171
187
|
|
|
172
188
|
Use [`docs/extension-integration.md`](docs/extension-integration.md) for the complete lifecycle, validation limits, and failure contract. The shipped [`examples/integrations/subagent-runner.ts`](examples/integrations/subagent-runner.ts) wraps a generic child transport, while [`docs/agent-subagent-observability-requirements.md`](docs/agent-subagent-observability-requirements.md) covers the larger orchestration design.
|
|
173
189
|
|
|
@@ -268,7 +284,7 @@ All operational logs use short event bodies plus structured attributes. Correlat
|
|
|
268
284
|
| Configuration, session, and workflow | `config.rejected`<br>`session.started`<br>`session.shutdown`<br>`session.duplicate_start`<br>`workflow.started`<br>`workflow.completed`<br>`workflow.failed` | Configuration diagnostics and top-level lifecycle/outcome events. |
|
|
269
285
|
| Reserved session events | `session.named`<br>`session.error` | Public event names reserved for compatibility; current live handlers do not emit them. |
|
|
270
286
|
| Agent run | `agent.run.started`<br>`agent.run.completed`<br>`agent.run.failed` | Agent-run lifecycle and outcome. |
|
|
271
|
-
| Subagent lineage | `agent.spawn.started`<br>`agent.spawn.completed`<br>`agent.spawn.failed`<br>`agent.wait.started`<br>`agent.wait.completed`<br>`agent.join.started`<br>`agent.join.completed`<br>`agent.orphaned`<br>`trace_context.propagation_failed` | Parent/child lifecycle, lineage gaps, propagation failures, and join outcomes. |
|
|
287
|
+
| Subagent lineage | `agent.spawn.started`<br>`agent.spawn.completed`<br>`agent.spawn.failed`<br>`agent.spawn.cancelled`<br>`agent.wait.started`<br>`agent.wait.completed`<br>`agent.join.started`<br>`agent.join.completed`<br>`agent.orphaned`<br>`trace_context.propagation_failed` | Parent/child lifecycle, lineage gaps, propagation failures, and join outcomes. |
|
|
272
288
|
| Turn | `turn.started`<br>`turn.completed` | Turn lifecycle with run/turn correlation. |
|
|
273
289
|
| LLM lifecycle | `llm.request.started`<br>`llm.request.completed`<br>`llm.request.failed` | Content-free provider request lifecycle, usage, cost, stop reason, and bounded errors. |
|
|
274
290
|
| LLM content | `llm.prompt.captured`<br>`llm.response.captured`<br>`llm.thinking.captured` | Emitted only when the corresponding prompt, response, or thinking capture flag is enabled and redaction succeeds. |
|
|
@@ -287,7 +303,7 @@ ObservMe supports layered configuration with this precedence:
|
|
|
287
303
|
defaults → global ~/.pi/agent/observme.yaml → trusted project config → trusted project .env → system environment variables → runtime options
|
|
288
304
|
```
|
|
289
305
|
|
|
290
|
-
Factory-safe loading uses defaults/global/system-environment/runtime options only. Session-scoped loading can add trusted project config and a project-local `.env` when Pi marks the project trusted. `/obs status` reports the effective config source, whether project-local config was loaded, skipped because the project is untrusted, or
|
|
306
|
+
Factory-safe loading uses defaults/global/system-environment/runtime options only. Session-scoped loading can add trusted project config and a project-local `.env` when Pi marks the project trusted. These project files must remain inside the stable canonical project root: in-root symlinks are supported, while out-of-root, dangling, replaced, or unverifiable paths fail closed through identity-verified file I/O. `/obs status` reports the effective config source, whether project-local config was loaded, skipped because the project is untrusted, missing, or rejected, plus bounded rejection issue codes, effective OTLP/Grafana transport security, the configured Grafana URL, and query-readiness status without rendering tokens, passwords, canonical targets, or external paths. In untrusted projects, ObservMe does not read project-local config or `.env` files and uses safe defaults/global/system-environment layers instead. Invalid or unsafe configuration emits a bounded `config.rejected` diagnostic and falls back safely without exposing rejected values.
|
|
291
307
|
|
|
292
308
|
Cross-process agent lineage has a separate boundary: only the Pi process environment available to the shipped extension, or explicit runtime options for controlled embedders, is eligible for parent provenance. Project-local `.env` values configure ObservMe but cannot establish lineage. A child accepts only a complete validated workflow/parent/root/depth/spawn envelope and valid W3C context; malformed or stale envelopes fail open to a root/orphan fallback with sanitized telemetry.
|
|
293
309
|
|
|
@@ -317,7 +333,7 @@ To show failed-tool output such as GuardMe denial messages in the Tools dashboar
|
|
|
317
333
|
|
|
318
334
|
Metadata such as token counts, duration, status, model/provider, tool name, and agent role/depth is captured by default. High-cardinality identifiers (session IDs, workflow IDs, agent IDs, trace/span IDs, entry IDs) are allowed on spans/logs for drill-down but are blocked from Prometheus metric labels.
|
|
319
335
|
|
|
320
|
-
Grafana-backed query commands use the Grafana HTTP API, so browser login cookies are irrelevant. Configure either a Grafana service-account token (`OBSERVME_GRAFANA_TOKEN`) or local Basic auth (`OBSERVME_GRAFANA_USERNAME`/`OBSERVME_GRAFANA_PASSWORD`); token auth takes precedence and secrets are never rendered in command errors. You can supply these values as system environment variables before starting Pi, or copy the `.env.example` shipped with ObservMe to `.env` in a trusted project; system environment variables override `.env` values.
|
|
336
|
+
Grafana-backed query commands use the Grafana HTTP API, so browser login cookies are irrelevant. Keep `query.grafana.url` credential-free; a base URL containing a username or password is rejected before network I/O with a bounded diagnostic that does not render the URL or credentials. Configure either a Grafana service-account token (`OBSERVME_GRAFANA_TOKEN`) or local Basic auth (`OBSERVME_GRAFANA_USERNAME`/`OBSERVME_GRAFANA_PASSWORD`); token auth takes precedence and secrets are never rendered in command errors. You can supply these values as system environment variables before starting Pi, or copy the `.env.example` shipped with ObservMe to `.env` in a trusted project; system environment variables override `.env` values.
|
|
321
337
|
|
|
322
338
|
### Supported local-stack query profile
|
|
323
339
|
|
|
@@ -343,7 +359,7 @@ query:
|
|
|
343
359
|
preferIPv4: true
|
|
344
360
|
```
|
|
345
361
|
|
|
346
|
-
Create a Grafana service-account token in Grafana (Administration → Users and access → Service accounts → Add service account/token; Viewer is enough for read-only datasource queries) and export it as `OBSERVME_GRAFANA_TOKEN`, or for local-only Basic auth read the generated admin password from the repository-only local stack's secrets directory. If you prefer a project-local env file, run `cp .env.example .env`, fill either `OBSERVME_GRAFANA_TOKEN` or `OBSERVME_GRAFANA_PASSWORD`, then restart Pi from that project. The local Collector and Loki profile uses `service.name=observme-pi-extension`; Loki queries use normalized labels such as `service_name`, `pi_session_id`, `pi_agent_id`, `pi_agent_run_id`, `event_name`, and `event_category`. If data is visible in Grafana but `/obs` commands fail, run `/obs health` and check extension env loading, Grafana auth, datasource UIDs, TLS, and DNS details.
|
|
362
|
+
Create a Grafana service-account token in Grafana (Administration → Users and access → Service accounts → Add service account/token; Viewer is enough for read-only datasource queries) and export it as `OBSERVME_GRAFANA_TOKEN`, or for local-only Basic auth read the generated admin password from the repository-only local stack's secrets directory. Certificate verification stays enabled for HTTPS by default. Production rejects OTLP or Grafana certificate-verification bypass unless `privacy.allowInsecureTransport: true` explicitly acknowledges the risk; `otlp.tls.enabled` is not a supported setting because each endpoint URL scheme selects HTTP or HTTPS. If you prefer a project-local env file, run `cp .env.example .env`, fill either `OBSERVME_GRAFANA_TOKEN` or `OBSERVME_GRAFANA_PASSWORD`, then restart Pi from that project. The local Collector and Loki profile uses `service.name=observme-pi-extension`; Loki queries use normalized labels such as `service_name`, `pi_session_id`, `pi_agent_id`, `pi_agent_run_id`, `event_name`, and `event_category`. If data is visible in Grafana but `/obs` commands fail, run `/obs health` and check extension env loading, Grafana auth, datasource UIDs, TLS, and DNS details.
|
|
347
363
|
|
|
348
364
|
### Show LLM chat content in Grafana
|
|
349
365
|
|
|
@@ -368,7 +384,7 @@ Full configuration schema: `docs/reference/12-configuration-reference.md`. Full
|
|
|
368
384
|
- ObservMe does not execute shell commands itself; it only observes tool/bash execution events emitted by Pi.
|
|
369
385
|
- ObservMe never blocks Pi agent execution when the observability backend is degraded or unreachable.
|
|
370
386
|
- ObservMe starts exporters only for a trusted session and shuts them down with bounded timeouts.
|
|
371
|
-
- ObservMe does not continuously tail the full Pi session file
|
|
387
|
+
- ObservMe does not continuously tail the full Pi session file. Optional correlation persistence uses one bounded `observme.correlation` custom entry on the active Pi branch, never LLM context; historical replay is never automatic and requires an explicit, confirmed `/obs backfill` command.
|
|
372
388
|
- Optional content capture is disabled by default and must pass through the redaction pipeline before export when enabled.
|
|
373
389
|
- `/obs` query commands are read-only and are not imported by telemetry-emission code paths.
|
|
374
390
|
- Invalid or unsafe configuration falls back to safe defaults with bounded `config.rejected`, `/obs status`, and available Pi UI diagnostics; rejected values are never rendered. Intentionally unsafe capture emits a visible warning.
|
|
@@ -565,6 +565,7 @@ agent.run.failed
|
|
|
565
565
|
agent.spawn.started
|
|
566
566
|
agent.spawn.completed
|
|
567
567
|
agent.spawn.failed
|
|
568
|
+
agent.spawn.cancelled
|
|
568
569
|
agent.wait.started
|
|
569
570
|
agent.wait.completed
|
|
570
571
|
agent.join.started
|
|
@@ -833,9 +834,9 @@ Minimum adapter behavior:
|
|
|
833
834
|
1. Request the API with `requestObservMeIntegration(pi)`.
|
|
834
835
|
2. Before spawning child Pi, call `startSubagent` with spawn type/reason and safe command metadata.
|
|
835
836
|
3. Pass the returned `env` into `child_process.spawn`.
|
|
836
|
-
4. On launcher
|
|
837
|
-
5.
|
|
838
|
-
6.
|
|
837
|
+
4. On launcher error before the child runs, call `failSubagent`.
|
|
838
|
+
5. Around blocking waits, call `startWait` and `endWait`.
|
|
839
|
+
6. When the child reaches a terminal state, call `completeSubagent` once with matching `completed`, `failed`, or `cancelled` status/outcome fields.
|
|
839
840
|
7. When child output is collected, call `startJoin` and `endJoin` with child status.
|
|
840
841
|
8. Ensure the child command loads ObservMe as an extension/package.
|
|
841
842
|
9. Ensure the child ObservMe runtime receives the complete propagated parent context.
|
|
@@ -1,9 +1,15 @@
|
|
|
1
1
|
# ObservMe Compatibility Matrix
|
|
2
2
|
|
|
3
|
-
Last updated: 2026-07-
|
|
3
|
+
Last updated: 2026-07-22
|
|
4
4
|
|
|
5
5
|
This matrix records the ObservMe runtime and observability-stack versions exercised or pinned for the current release line. A component is marked **validated** only when the named command actually exercised it; configured CI targets remain distinct from completed local or workflow evidence.
|
|
6
6
|
|
|
7
|
+
## Pi version policy
|
|
8
|
+
|
|
9
|
+
ObservMe never uses Pi's reported version as a startup gate. The Pi-mandated core package peers remain in `peerDependencies` with `"*"`, while exact 0.80.5 and 0.81.1 targets record compatibility evidence in CI. Before registering anything, startup checks only the essential `ExtensionAPI` capabilities `on` and `registerCommand`. When those methods exist, ObservMe starts regardless of whether Pi's version is older, newer, prerelease, malformed, or unavailable. Optional APIs are feature-detected by the handlers that use them.
|
|
10
|
+
|
|
11
|
+
Pi 0.80.5 is the earliest tested version, not a runtime minimum: its exported extension contract includes `session_info_changed`, which ObservMe uses for live rename metadata. Pi's current extension overloads do not include `bashExecution`, `model_change`, or `thinking_level_change`. `bashExecution` is an `AgentMessage` role observed through `message_end`, while model and thinking changes are session entry types represented at runtime by `model_select` and `thinking_level_select`; ObservMe no longer registers those three non-event names. Tested versions provide release evidence only and never define an accepted or rejected runtime range.
|
|
12
|
+
|
|
7
13
|
## Validation record
|
|
8
14
|
|
|
9
15
|
| Date | Environment | Command | Coverage |
|
|
@@ -12,6 +18,11 @@ This matrix records the ObservMe runtime and observability-stack versions exerci
|
|
|
12
18
|
| 2026-07-07 | Local Docker 29.5.2 with Collector Contrib 0.104.0 | `npm run test:integration:collector` | Starts a local debug-exporter Collector container, exports representative ObservMe traces/metrics/logs, and asserts default-disabled content capture is absent. |
|
|
13
19
|
| 2026-07-07 | Local Docker 29.5.2 with the pinned Grafana stack | `npm run test:integration:grafana-stack` | Starts `observability-stack/` services, exports representative ObservMe telemetry, queries Tempo by trace ID and lineage attributes, queries Loki by session ID, queries Prometheus token totals, and validates Grafana dashboard provisioning imports. |
|
|
14
20
|
| Continuous integration target | GitHub Actions `ubuntu-latest`, Node.js 22.19.0 | `npm ci --ignore-scripts` then `npm run validate` | CI target from `.github/workflows/ci.yml`; record the workflow run URL here after each release validation. |
|
|
21
|
+
| Pi compatibility CI matrix (updated 2026-07-22) | GitHub Actions `ubuntu-latest`, Node.js 22.19.0; exact Pi 0.80.5 and 0.81.1 matrix targets | Exact Pi install, version assertion, then `npm run validate:pi-compatibility` | Compile-time event fixtures, capability diagnostics, package install, handler/lifecycle smoke, and a real Pi RPC runtime are exercised separately at the earliest-tested and release-tested versions. |
|
|
22
|
+
| 2026-07-14 | Local Node.js v26.0.0, npm 11.12.1; exact `@earendil-works/pi-coding-agent` and `@earendil-works/pi-ai` 0.80.5 | Exact version assertion then `npm run validate:pi-compatibility` | Passed typecheck, typed event contracts, capability diagnostics, packaged install, handler/lifecycle smoke, and real Pi RPC runtime at the earliest-tested version. |
|
|
23
|
+
| 2026-07-14 | Local Node.js v26.0.0, npm 11.12.1; exact `@earendil-works/pi-coding-agent` and `@earendil-works/pi-ai` 0.80.6 | Exact version assertion then `npm run validate:pi-compatibility` | Passed the same compatibility suite at the former release-tested version. |
|
|
24
|
+
| 2026-07-22 | Local Node.js v26.0.0, npm 11.12.1; exact `@earendil-works/pi-coding-agent` and `@earendil-works/pi-ai` 0.81.1 | Exact version assertion, typechecks, compatibility tests, packaged install, handler/lifecycle smoke, and real Pi RPC runtime | Passed the complete `npm run validate:pi-compatibility` suite at the current release-tested version. |
|
|
25
|
+
| 2026-07-22 | Local Node.js v26.0.0, npm 11.12.1; exact Pi 0.81.1 development dependencies | `npm run validate` | Passed source/test typechecks, ESLint, formatting, 618 unit/contract tests, package-content checks, packaged-install smoke, handler smoke, Pi lifecycle smoke, and real Pi runtime smoke. |
|
|
15
26
|
| Lease cancellation CI profile (configured 2026-07-14) | GitHub-hosted Linux `ubuntu-latest` (exact `ImageOS`/`ImageVersion` recorded in the Actions runner log); Node.js 22.19.0; Collector Contrib 0.104.0; Prometheus 2.53.1; Grafana 11.1.0 | `npm run test:active-agent-lease:contracts` then `npm run test:integration:active-agent-lease` | The bounded `lease-cancellation` job validates clean shutdown, `SIGTERM`, and cancellation-oriented `SIGKILL` convergence against the pinned Collector and Prometheus without secrets or a Grafana dependency. Grafana 11.1.0 is the recorded dashboard compatibility baseline validated separately by `npm run test:integration:grafana-stack`. `if: always()` cleanup is best effort; lease expiry is the asserted correctness mechanism. |
|
|
16
27
|
| 2026-07-14 | Local Node.js v26.0.0, npm 11.12.1 | `npm run validate` | Passed 493 unit/contract tests plus typecheck, ESLint, formatting, script, package-content, packaged-install, handler, Pi lifecycle, and Pi runtime checks for the lease release candidate. |
|
|
17
28
|
| 2026-07-14 | Local Docker 29.5.2, Compose 5.1.4, Collector Contrib 0.104.0 | `npm run test:integration:collector`; Compose config; pinned Collector `validate` | Passed representative OTLP export, Compose interpolation, and the shipped Collector config validation. |
|
|
@@ -25,7 +36,7 @@ Active-agent lease interpretation additionally requires producer and Prometheus
|
|
|
25
36
|
|
|
26
37
|
| Component | Version tested or pinned | Source of truth | Status | Evidence and notes |
|
|
27
38
|
| --- | --- | --- | --- | --- |
|
|
28
|
-
| Pi package API | `@earendil-works/pi-coding-agent`
|
|
39
|
+
| Pi package API | Runtime policy: required API capabilities only; version never gates startup. Earliest tested Pi: 0.80.5; release-tested `@earendil-works/pi-coding-agent` and `@earendil-works/pi-ai`: 0.81.1 | `package.json` `observmeCompatibility`, exact dev dependencies, `package-lock.json`, `.github/workflows/ci.yml` | Validated locally at earliest-tested 0.80.5 and release-tested 0.81.1; exact CI matrix configured | `npm run validate:pi-compatibility` typechecks exported event contracts, tests pre-registration capability diagnostics, installs the package, runs handler/lifecycle smoke, and launches the real Pi RPC runtime. Pi version values do not participate in startup acceptance. |
|
|
29
40
|
| Node.js | Local: 26.0.0; CI target/minimum: 22.19.0 | `node --version`, `package.json` `engines.node`, `.github/workflows/ci.yml` | Validated locally for 26.0.0; CI target pinned at 22.19.0 | Local validation ran on 26.0.0. CI is configured to validate on 22.19.0, which is also the minimum supported Node.js version. |
|
|
30
41
|
| OpenTelemetry JS package set | `@opentelemetry/api` 1.9.1; `@opentelemetry/api-logs` 0.220.0; OTLP proto exporters 0.220.0; `@opentelemetry/resources` 2.9.0; `@opentelemetry/sdk-logs` 0.220.0; `@opentelemetry/sdk-metrics` 2.9.0; `@opentelemetry/sdk-trace-base` 2.9.0; `@opentelemetry/sdk-trace-node` 2.9.0 | `package.json`, `package-lock.json` | Validated locally | `npm run validate` includes the OTEL lifecycle/exporter unit tests under `test/otel-*.test.mjs`. |
|
|
31
42
|
|
|
@@ -41,7 +52,7 @@ Active-agent lease interpretation additionally requires producer and Prometheus
|
|
|
41
52
|
|
|
42
53
|
## Update rules
|
|
43
54
|
|
|
44
|
-
- Update this file whenever `package-lock.json`, `.github/workflows/ci.yml`, or `observability-stack/docker-compose.yml` changes a tracked version.
|
|
55
|
+
- Update this file whenever `package.json` `observmeCompatibility`, `package-lock.json`, `.github/workflows/ci.yml`, or `observability-stack/docker-compose.yml` changes a tracked version.
|
|
45
56
|
- Add a new validation-record row whenever a release candidate is validated locally, in CI, or against the Docker Compose stack.
|
|
46
57
|
- Do not mark backend components as validated unless a command actually started or queried that component.
|
|
47
58
|
- Keep workflow IDs, session IDs, agent IDs, trace IDs, and other high-cardinality values out of this document.
|
package/docs/configuration.md
CHANGED
|
@@ -18,12 +18,12 @@ The standard Pi distribution currently resolves this to `.pi/observme.yaml`. Obs
|
|
|
18
18
|
|
|
19
19
|
Edit the project config file (`.pi/observme.yaml` in the standard distribution) where you run Pi:
|
|
20
20
|
|
|
21
|
-
- `otlp.endpoint` and `otlp.signalEndpoints` —
|
|
21
|
+
- `otlp.endpoint` and `otlp.signalEndpoints` — absolute HTTP(S) OpenTelemetry Collector URLs. Base endpoints may include an intentional path; ObservMe appends one `/v1/{signal}` suffix with URL pathname semantics. Signal-specific endpoints must already end in the matching signal path. Keep credentials in `otlp.headers`; endpoint userinfo, unresolved placeholders, queries, and fragments are rejected.
|
|
22
22
|
- `resource.attributes` — service name, project name, tenant, and deployment environment labels.
|
|
23
23
|
- `capture` — whether prompts, responses, thinking, tool data, bash data, and file paths are exported. For local debugging, set only the specific fields you need to `true`.
|
|
24
24
|
- `privacy` — redaction, unsafe-capture acknowledgement, insecure transport, hash salt env var, and path handling. Keep `redactionEnabled: true`; set `allowUnsafeCapture: true` only when you intentionally accept unredacted sensitive-content export from this trusted project. Live telemetry and `/obs backfill` use the same policy: disabled capture omits content, enabled redaction redacts then truncates, redaction failures drop content, and `redactionEnabled: false` with `allowUnsafeCapture: true` exports raw truncated content.
|
|
25
25
|
- `query.grafana` — Grafana URL, datasource UIDs, TLS, and IPv4 transport settings for `/obs` query commands.
|
|
26
|
-
- `query.links.traceUrlTemplate` — the Grafana Explore trace-link template used by `/obs trace
|
|
26
|
+
- `query.links.traceUrlTemplate` — the canonical Grafana Explore trace-link template used by `/obs session`, `/obs trace`, and `/obs link`; use `{traceId}`, `{{traceId}}`, `${traceId}`, or `%TRACE_ID%`, or keep the generated `...` structured fallback.
|
|
27
27
|
- `metrics.activeAgentLeaseDurationMillis` — how long the last exported active-agent lease remains valid. The default is `60000` ms, the supported range is `10000`–`300000` ms, and the value must be at least `(2 * metrics.exportIntervalMillis) + 5000` ms. `OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS` overrides YAML through the normal precedence rules.
|
|
28
28
|
|
|
29
29
|
The shipped dashboards and alerts combine a positive `observme_active_agents` lifecycle claim with a current lease; raw active-claim sums are diagnostic only after an ungraceful exit. Clean shutdown reaches zero after normal export/scrape propagation. Crash, `SIGKILL`, forced GitHub Actions cancellation, or runner loss converges within the lease plus up to 5 seconds of supported clock skew and one Prometheus scrape/evaluation interval, without restarting the Collector. Keep producer and Prometheus clocks synchronized within 5 seconds; GitHub-hosted runners meet this expectation, while self-hosted runners require reliable NTP or an equivalent time service.
|
|
@@ -58,9 +58,9 @@ Use this order for each child process:
|
|
|
58
58
|
1. Request the ObservMe integration API.
|
|
59
59
|
2. Call `startSubagent()` immediately before launching the child.
|
|
60
60
|
3. Pass the returned `env` as the child process environment without logging it.
|
|
61
|
-
4. Call `
|
|
62
|
-
5. Call `
|
|
63
|
-
6. Call `
|
|
61
|
+
4. Call `failSubagent()` only when the launcher fails before the child is running.
|
|
62
|
+
5. Call `startWait()`/`endWait()` around time spent waiting for child completion.
|
|
63
|
+
6. Call `completeSubagent()` once with the matching terminal `childStatus` and `outcome` (`completed`, `failed`, or `cancelled`).
|
|
64
64
|
7. Call `startJoin()`/`endJoin()` when collecting the terminal child status or result.
|
|
65
65
|
|
|
66
66
|
```typescript
|
|
@@ -78,12 +78,9 @@ if (!started?.ok) {
|
|
|
78
78
|
return;
|
|
79
79
|
}
|
|
80
80
|
|
|
81
|
+
let child;
|
|
81
82
|
try {
|
|
82
|
-
await launchChildPi({ env: started.env });
|
|
83
|
-
observme.completeSubagent(started.spawnId, {
|
|
84
|
-
childAgentId: started.childAgentId,
|
|
85
|
-
childStatus: "active",
|
|
86
|
-
});
|
|
83
|
+
child = await launchChildPi({ env: started.env });
|
|
87
84
|
} catch (error) {
|
|
88
85
|
observme.failSubagent(started.spawnId, {
|
|
89
86
|
childAgentId: started.childAgentId,
|
|
@@ -91,6 +88,13 @@ try {
|
|
|
91
88
|
});
|
|
92
89
|
throw error;
|
|
93
90
|
}
|
|
91
|
+
|
|
92
|
+
const result = await waitForChildPi(child);
|
|
93
|
+
observme.completeSubagent(started.spawnId, {
|
|
94
|
+
childAgentId: started.childAgentId,
|
|
95
|
+
childStatus: result.status,
|
|
96
|
+
outcome: result.status,
|
|
97
|
+
});
|
|
94
98
|
```
|
|
95
99
|
|
|
96
100
|
Do not put raw tasks, prompts, command lines, environment values, child output, or private paths in `errorClass`, `spawnReason`, or other bounded fields.
|
|
@@ -101,7 +105,7 @@ Do not put raw tasks, prompts, command lines, environment values, child output,
|
|
|
101
105
|
| --- | --- |
|
|
102
106
|
| `getContext()` | Read the current workflow, root/parent/current agent, role, depth, session, and trace identifiers for local orchestration correlation. These are high-cardinality values and must not become metric labels. |
|
|
103
107
|
| `startSubagent(options)` | Starts `pi.agent.spawn`, records spawn metrics/logs, creates bounded parent tree state, and returns a sanitized propagation environment. |
|
|
104
|
-
| `completeSubagent(spawnId, options)` | Ends
|
|
108
|
+
| `completeSubagent(spawnId, options)` | Ends the active child lifecycle with one coherent `completed`, `failed`, or `cancelled` status/outcome pair. |
|
|
105
109
|
| `failSubagent(spawnId, options)` | Ends a launcher failure and records bounded failure telemetry. |
|
|
106
110
|
| `startWait(options)` / `endWait(id, options)` | Measures time the parent is blocked on a child or dependency. |
|
|
107
111
|
| `startJoin(options)` / `endJoin(id, options)` | Measures result collection and records child failure propagation or confirmed parent recovery. |
|
|
@@ -111,16 +115,16 @@ Do not put raw tasks, prompts, command lines, environment values, child output,
|
|
|
111
115
|
| Option | Values and meaning |
|
|
112
116
|
| --- | --- |
|
|
113
117
|
| `spawnId` | Optional caller-generated safe ID; omit to let ObservMe generate it. |
|
|
114
|
-
| `childAgentId` | Optional bounded parent-side placeholder; this is not propagated as the child's real agent ID. |
|
|
118
|
+
| `childAgentId` | Optional bounded parent-side placeholder; this is not propagated as the child's real agent ID and must remain unique while its active or terminal node is retained. |
|
|
115
119
|
| `command` / `args` | Used only to create a salted command fingerprint when configured; raw values are not exported. Omit them if the launcher cannot safely provide them. |
|
|
116
120
|
| `spawnType` | `command`, `tool`, `extension`, or `unknown`. |
|
|
117
121
|
| `spawnReason` | `delegated_task`, `parallel_search`, `review`, `tool_wrapper`, or `unknown`. |
|
|
118
122
|
| `toolCallId` | Optional high-cardinality trace/log correlation when a tool initiated the spawn. |
|
|
119
123
|
| `env` | Base child environment. ObservMe removes stale lineage/W3C keys and returns the replacement environment. |
|
|
120
124
|
|
|
121
|
-
Runtime callers are validated even when JavaScript bypasses the TypeScript types. Caller-provided lifecycle identifiers must match `[A-Za-z0-9._:-]{1,128}`. Commands and individual arguments are capped at 4096 characters, argument lists at 256 items, and environment objects at 4096 entries. Durations must be finite, non-negative milliseconds. Invalid or duplicate active operations return a failure without replacing an existing span.
|
|
125
|
+
Runtime callers are validated even when JavaScript bypasses the TypeScript types. Caller-provided lifecycle identifiers must match `[A-Za-z0-9._:-]{1,128}`. Commands and individual arguments are capped at 4096 characters, argument lists at 256 items, and environment objects at 4096 entries. Durations must be finite, non-negative milliseconds. Invalid or duplicate active operations return a failure without replacing an existing span. Child placeholders, including generated placeholders, are collision-checked before span, tree, metric, or propagation state is created; do not reuse a terminal placeholder while its bounded tree node remains retained.
|
|
122
126
|
|
|
123
|
-
Completion and
|
|
127
|
+
Completion accepts only terminal child states (`completed`, `failed`, `cancelled`), and any supplied outcome must match. Wait/join methods use bounded child states (`starting`, `active`, `completed`, `failed`, `cancelled`, `orphaned`), join states (`waiting`, `completed`, `failed`, `cancelled`, `timeout`, `unknown`), and wait reasons (`dependency`, `rate_limit`, `child_running`, `unknown`). `failurePropagated=false` on a completed join confirms that the parent recovered from a failed child.
|
|
124
128
|
|
|
125
129
|
All mutation methods return a discriminated result. Handle these reasons without crashing Pi:
|
|
126
130
|
|
|
@@ -129,7 +133,10 @@ All mutation methods return a discriminated result. Handle these reasons without
|
|
|
129
133
|
| `session_unavailable` | ObservMe is loaded but no telemetry session is active. |
|
|
130
134
|
| `invalid_request` | An identifier, enum, duration, command/argument field, or environment shape is invalid or oversized. |
|
|
131
135
|
| `spawn_already_exists` / `wait_already_exists` / `join_already_exists` | The requested lifecycle identifier is already active. Generate a unique identifier or finish the active operation; do not overwrite it. |
|
|
136
|
+
| `child_agent_already_exists` | The requested or generated child placeholder belongs to an active spawn or retained tree node. Generate a unique child identifier; do not reuse terminal placeholders. |
|
|
132
137
|
| `spawn_not_found` / `wait_not_found` / `join_not_found` | The lifecycle handle is absent or has already ended. |
|
|
138
|
+
| `child_agent_mismatch` | The supplied child ID does not match the child stored for the active spawn. |
|
|
139
|
+
| `invalid_terminal_transition` | Terminal status/outcome fields contradict each other or would rewrite an existing terminal tree state. |
|
|
133
140
|
| `operation_failed` | ObservMe could not safely complete the operation. |
|
|
134
141
|
|
|
135
142
|
Do not retry a completed lifecycle handle blindly; repeated completion can otherwise hide an orchestration-state bug.
|
|
@@ -183,7 +190,7 @@ The integration API is transport-agnostic. A launcher can use a local subprocess
|
|
|
183
190
|
|
|
184
191
|
- passes the returned environment unchanged to the child Pi process;
|
|
185
192
|
- does not serialize the envelope or raw task into telemetry or captured logs;
|
|
186
|
-
- reports launcher
|
|
193
|
+
- reports launcher failure separately from child completion, failure, or cancellation;
|
|
187
194
|
- records wait and join around the transport's actual blocking and result-collection boundaries;
|
|
188
195
|
- guarantees the child loads a compatible ObservMe extension;
|
|
189
196
|
- cleans up temporary files, pipes, sessions, containers, or remote resources deterministically.
|
|
@@ -105,7 +105,7 @@ Optional telemetry:
|
|
|
105
105
|
|
|
106
106
|
### `custom` entry
|
|
107
107
|
|
|
108
|
-
Extension state entry created through `pi.appendEntry()`. It does not participate in LLM context. ObservMe
|
|
108
|
+
Extension state entry created through `pi.appendEntry()`. It does not participate in LLM context. When `agent.writeCorrelationEntry` is explicitly enabled, ObservMe appends at most one `observme.correlation` entry per active branch after telemetry startup succeeds. The versioned data contains only bounded workflow/agent lineage identifiers and depth. Startup scans `ctx.sessionManager.getBranch()` from newest to oldest, restores the latest valid entry, ignores corrupt entries and entries on abandoned branches, and skips an idempotent reload write. By default ObservMe does not append custom entries.
|
|
109
109
|
|
|
110
110
|
### `custom_message` entry
|
|
111
111
|
|
|
@@ -244,12 +244,12 @@ The derived id must be scoped by session id and agent run id and must not be use
|
|
|
244
244
|
|
|
245
245
|
On extension startup in an existing session:
|
|
246
246
|
|
|
247
|
-
1. Read current session header if available.
|
|
247
|
+
1. Read the current session header from `ctx.sessionManager` if available.
|
|
248
248
|
2. Set `pi.session.id`, `pi.workflow.id`, `pi.agent.id`, and resource attributes.
|
|
249
|
-
3.
|
|
250
|
-
4.
|
|
251
|
-
5.
|
|
252
|
-
6.
|
|
249
|
+
3. When `agent.writeCorrelationEntry` is enabled, reconstruct optional workflow/agent lineage only from the latest validated `observme.correlation` custom entry on `ctx.sessionManager.getBranch()`.
|
|
250
|
+
4. Ignore malformed entries and entries outside the active branch without exporting their values or changing LLM context.
|
|
251
|
+
5. After successful startup, append the minimal custom entry only when the active branch does not already contain the same validated correlation.
|
|
252
|
+
6. Never replay historical telemetry automatically. Historical export requires the explicit, confirmed `/obs backfill` command, which marks each emitted record with `observme.replayed=true`.
|
|
253
253
|
|
|
254
254
|
## 11. Local Session Reading Rules
|
|
255
255
|
|
|
@@ -105,7 +105,7 @@ GitHub token: (gh[pousr]_[A-Za-z0-9_]{36,}|github_pat_[A-Za-z0-9_
|
|
|
105
105
|
OpenAI-like key: sk-[A-Za-z0-9_-]{20,}
|
|
106
106
|
Anthropic-like key: sk-ant-[A-Za-z0-9_-]{20,}
|
|
107
107
|
Slack token: xox[baprs]-[A-Za-z0-9-]{10,}
|
|
108
|
-
Private key block:
|
|
108
|
+
Private key block: complete uppercase labels ending in PRIVATE KEY (including PKCS#8, encrypted, RSA, and EC)
|
|
109
109
|
Password assignment: (?i)(password|passwd|pwd)\s*[:=]\s*[^\s]+
|
|
110
110
|
API key assignment: (?i)(api[_-]?key|token|secret|client[_-]?secret)\s*[:=]\s*[^\s]+
|
|
111
111
|
URL credentials: [a-z][a-z0-9+.-]*://[^\s:/?#]+:[^\s@/]+@
|
|
@@ -113,6 +113,8 @@ URL credentials: [a-z][a-z0-9+.-]*://[^\s:/?#]+:[^\s@/]+@
|
|
|
113
113
|
|
|
114
114
|
These patterns are a minimum safety net, not a complete secret scanner. Keep them covered by tests and allow organizations to add custom regexes for proprietary credential formats.
|
|
115
115
|
|
|
116
|
+
Private-key matching consumes a `PRIVATE KEY` label with an optional, bounded uppercase prefix, its body, and a same-label `END` marker with a 1,000,000-character body scan bound. This covers PKCS#8 (`PRIVATE KEY`), `ENCRYPTED PRIVATE KEY`, `RSA PRIVATE KEY`, `EC PRIVATE KEY`, and other uppercase private-key labels without accepting public-key labels. The shared input-size guard drops larger captured values before secret matching. If a supported private-key marker has no matching footer because it is missing, truncated, or mislabeled, redaction fails closed by replacing everything from that marker through the end of the captured value. `PUBLIC KEY` and `RSA PUBLIC KEY` blocks are not private-key matches.
|
|
117
|
+
|
|
116
118
|
Matched values are replaced with:
|
|
117
119
|
|
|
118
120
|
```text
|
|
@@ -228,7 +230,7 @@ Example:
|
|
|
228
230
|
|
|
229
231
|
The Collector cannot recover content dropped by an older configuration; generate new LLM events after updating the Collector and dashboards. Do not use raw chat text in Loki or Tempo query strings.
|
|
230
232
|
|
|
231
|
-
## 11. Authentication
|
|
233
|
+
## 11. Authentication and Credential-Free Endpoint URLs
|
|
232
234
|
|
|
233
235
|
ObservMe supports OTLP headers:
|
|
234
236
|
|
|
@@ -238,7 +240,9 @@ otlp:
|
|
|
238
240
|
Authorization: "Bearer ${OBSERVME_OTLP_TOKEN}"
|
|
239
241
|
```
|
|
240
242
|
|
|
241
|
-
|
|
243
|
+
Grafana base URLs must be absolute HTTP(S) URLs without embedded username or password components. Configure Grafana authentication only through `query.grafana.token` or the complete `query.grafana.username` and `query.grafana.password` pair. A credential-bearing base URL is rejected during configuration validation and query readiness, before either Grafana transport can perform network I/O. Diagnostics report only the safe `embedded_credentials` failure class and dedicated setting names; they do not render the rejected URL or credential values.
|
|
244
|
+
|
|
245
|
+
Secrets must be read from environment variables or secure runtime config, not hardcoded in extension source or embedded in endpoint URLs.
|
|
242
246
|
|
|
243
247
|
## 12. TLS
|
|
244
248
|
|
|
@@ -246,12 +250,12 @@ Production default:
|
|
|
246
250
|
|
|
247
251
|
```yaml
|
|
248
252
|
otlp:
|
|
253
|
+
endpoint: https://otel-collector.example.com:4318
|
|
249
254
|
tls:
|
|
250
|
-
enabled: true
|
|
251
255
|
insecureSkipVerify: false
|
|
252
256
|
```
|
|
253
257
|
|
|
254
|
-
Development may use insecure
|
|
258
|
+
The endpoint URL scheme selects TLS. The OTLP exporters explicitly keep certificate verification enabled unless `otlp.tls.insecureSkipVerify: true` is configured. Grafana query transport follows the same rule through `query.grafana.tls.insecureSkipVerify`. Production rejects either verification bypass, and any plain HTTP endpoint, unless `privacy.allowInsecureTransport: true` records the explicit insecure-transport acknowledgement. Development may use local HTTP or self-signed certificates, but the insecure setting remains visible in `/obs status` and `/obs health`.
|
|
255
259
|
|
|
256
260
|
## 13. Tenant Isolation
|
|
257
261
|
|
|
@@ -281,10 +285,13 @@ Do not log secrets while reporting security failures.
|
|
|
281
285
|
|
|
282
286
|
## 15. Safe Config Validation
|
|
283
287
|
|
|
288
|
+
Trusted project config, project `.env`, and starter-config creation share one filesystem boundary. Their lexical paths and canonical targets must remain inside the stable canonical project root. Existing symlinks are supported only when they resolve to files or directories inside that root; out-of-root, dangling, replaced, or unverifiable paths fail closed. Reads and starter writes use identity-verified file handles and recheck containment so a concurrent file, ancestor, or root change cannot substitute an external target. A rejected project source is not loaded, starter creation does not overwrite an existing file, and diagnostics expose only a bounded failure class rather than canonical or external path details.
|
|
289
|
+
|
|
284
290
|
Reject config if:
|
|
285
291
|
|
|
286
292
|
- capture is enabled but redaction is disabled, unless `allowUnsafeCapture: true`
|
|
287
|
-
- OTLP endpoint is plain HTTP in production, unless `allowInsecureTransport: true`
|
|
293
|
+
- an OTLP or Grafana endpoint is plain HTTP in production, unless `allowInsecureTransport: true`
|
|
294
|
+
- OTLP or Grafana certificate verification is bypassed in production, unless `allowInsecureTransport: true`
|
|
288
295
|
- metric labels include forbidden high-cardinality fields such as workflow IDs, session IDs, or agent IDs
|
|
289
296
|
- propagated workflow or agent lineage values are malformed, too long, or contain unsafe characters
|
|
290
297
|
- queue sizes exceed configured memory limits
|
|
@@ -43,9 +43,10 @@ query:
|
|
|
43
43
|
|
|
44
44
|
Authentication and local transport behavior:
|
|
45
45
|
|
|
46
|
-
-
|
|
46
|
+
- Keep `query.grafana.url` credential-free: it must be an absolute HTTP(S) base URL with no embedded username or password component.
|
|
47
|
+
- Configure authentication only with `query.grafana.token` or the dedicated `query.grafana.username` and `query.grafana.password` settings. Prefer the token for Grafana service-account or bearer authentication.
|
|
47
48
|
- If the token is blank or an unresolved environment placeholder, ObservMe may use `query.grafana.username` plus `query.grafana.password` as a local-development Basic auth fallback.
|
|
48
|
-
- Query-backed commands fail fast before backend calls when Grafana auth is unresolved/missing/incomplete, `query.grafana.url` is invalid, or a required datasource UID is blank.
|
|
49
|
+
- Query-backed commands fail fast before backend calls when Grafana auth is unresolved/missing/incomplete, `query.grafana.url` is invalid or contains credentials, or a required datasource UID is blank. Credential-bearing URLs report the safe `embedded_credentials` class without rendering the URL or credential values.
|
|
49
50
|
- `401` and `403` responses are surfaced as Grafana authentication failures without printing token or password values.
|
|
50
51
|
- Browser login cookies are irrelevant because `/obs` commands call the Grafana API directly from the Pi process.
|
|
51
52
|
|
|
@@ -71,7 +72,7 @@ query:
|
|
|
71
72
|
preferIPv4: true
|
|
72
73
|
```
|
|
73
74
|
|
|
74
|
-
For the bundled `observability-stack/`, the supported command path is nginx HTTPS at `https://observability.local`; the default stack does not publish Grafana on `localhost:3000`. Create a Grafana service-account token with Viewer access for read-only datasource queries and export it as `OBSERVME_GRAFANA_TOKEN`, or export/set `OBSERVME_GRAFANA_PASSWORD` from `observability-stack/secrets/grafana_admin_password` for local Basic auth. Extension environment values can come from system environment variables before Pi starts, or from a trusted project `.env` copied from `.env.example`; system variables override `.env`. The local self-signed certificate requires `query.grafana.tls.insecureSkipVerify=true`, and `query.grafana.transport.preferIPv4=true` avoids DNS stalls when `observability.local` resolves to IPv6 first. Provisioned datasource UIDs are `tempo`, `loki`, and `prometheus`; Loki selectors use normalized labels such as `service_name`, `pi_session_id`, `event_name`, and `event_category`.
|
|
75
|
+
For the bundled `observability-stack/`, the supported command path is nginx HTTPS at `https://observability.local`; the default stack does not publish Grafana on `localhost:3000`. Create a Grafana service-account token with Viewer access for read-only datasource queries and export it as `OBSERVME_GRAFANA_TOKEN`, or export/set `OBSERVME_GRAFANA_PASSWORD` from `observability-stack/secrets/grafana_admin_password` for local Basic auth. Extension environment values can come from system environment variables before Pi starts, or from a trusted project `.env` copied from `.env.example`; system variables override `.env`. The local self-signed certificate requires `query.grafana.tls.insecureSkipVerify=true`, and `query.grafana.transport.preferIPv4=true` avoids DNS stalls when `observability.local` resolves to IPv6 first. Production configurations using the TLS bypass must also set `privacy.allowInsecureTransport=true` as an explicit acknowledgement; trusting the production CA remains preferred. Provisioned datasource UIDs are `tempo`, `loki`, and `prometheus`; Loki selectors use normalized labels such as `service_name`, `pi_session_id`, `event_name`, and `event_category`.
|
|
75
76
|
|
|
76
77
|
## 4. Commands
|
|
77
78
|
|
|
@@ -84,6 +85,9 @@ Output:
|
|
|
84
85
|
```text
|
|
85
86
|
ObservMe: enabled
|
|
86
87
|
OTLP endpoint: https://otel.example.com:4318
|
|
88
|
+
OTLP transport security: TLS certificate verification enabled
|
|
89
|
+
Grafana URL: https://grafana.example.com/
|
|
90
|
+
Grafana transport security: TLS certificate verification enabled
|
|
87
91
|
Traces: enabled
|
|
88
92
|
Metrics: enabled
|
|
89
93
|
Logs: enabled
|
|
@@ -97,6 +101,8 @@ Last export error: none
|
|
|
97
101
|
Checks Collector and Grafana reachability, and reports Grafana query auth/config readiness before datasource calls.
|
|
98
102
|
|
|
99
103
|
```text
|
|
104
|
+
Collector transport security: TLS certificate verification enabled
|
|
105
|
+
Grafana transport security: TLS certificate verification enabled
|
|
100
106
|
Collector: reachable
|
|
101
107
|
Grafana: reachable
|
|
102
108
|
Tempo datasource: ok
|
|
@@ -216,13 +222,22 @@ Only available if Loki receives `pi.session.id` as structured metadata or label.
|
|
|
216
222
|
|
|
217
223
|
## 5. Trace Links
|
|
218
224
|
|
|
219
|
-
|
|
225
|
+
`/obs session`, `/obs trace`, and `/obs link` use the same pure trace-link builder, so one effective config and trace ID always produce one canonical URL. Custom absolute HTTP(S) templates support these trace-ID placeholders:
|
|
220
226
|
|
|
221
|
-
|
|
222
|
-
|
|
227
|
+
- `{traceId}`
|
|
228
|
+
- `{{traceId}}` or `{{ traceId }}`
|
|
229
|
+
- `${traceId}`
|
|
230
|
+
- `%TRACE_ID%`
|
|
231
|
+
|
|
232
|
+
The matching optional Tempo datasource placeholders are `{tempoDatasourceUid}`, `{{tempoDatasourceUid}}` (whitespace is allowed), `${tempoDatasourceUid}`, and `%TEMPO_DATASOURCE_UID%`. Trace IDs and datasource UIDs are URL-encoded before substitution.
|
|
233
|
+
|
|
234
|
+
An empty template or a template containing `...` selects the structured Grafana Explore fallback. The fallback uses `query.grafana.url`, preserves any path prefix, and constructs the current `schemaVersion=1&panes=...` Tempo query with `query.grafana.datasourceUids.tempo`. This is the behavior used by the generated starter:
|
|
235
|
+
|
|
236
|
+
```yaml
|
|
237
|
+
traceUrlTemplate: https://observability.local/explore?left=...
|
|
223
238
|
```
|
|
224
239
|
|
|
225
|
-
|
|
240
|
+
Unsupported placeholders, non-HTTP(S) URLs, credential-bearing URLs, and malformed or oversized templates produce one bounded actionable diagnostic; commands do not silently omit a link or apply different substitutions.
|
|
226
241
|
|
|
227
242
|
## 6. Grafana Query Abstraction
|
|
228
243
|
|
|
@@ -258,6 +273,14 @@ query:
|
|
|
258
273
|
maxAgents: 20
|
|
259
274
|
```
|
|
260
275
|
|
|
276
|
+
Each query result-count setting accepts integers from 1 through 100. Runtime query clients clamp programmatically supplied values to the same ceiling before adding a backend request limit or selecting parsed results.
|
|
277
|
+
|
|
278
|
+
### Bounded command display
|
|
279
|
+
|
|
280
|
+
Prometheus label keys and values used for `/obs` display are normalized before they enter command snapshots. Control and line-separator characters become spaces, repeated whitespace collapses, and each label is limited to 96 UTF-16 code units. Truncated labels retain a valid Unicode prefix and end with `…`.
|
|
281
|
+
|
|
282
|
+
`/obs cost` renders at most 20 rows. `/obs tools` renders at most 20 call rows and 20 failure rows. Omitted rows are reported explicitly. `/obs agents` retains its 10-row recent-child display limit. Every final `/obs` notification is normalized to remove terminal controls other than intended line feeds and is limited to 64 rows and 8,192 UTF-16 code units. A notification that reaches either hard limit ends with `… output truncated`. These display bounds do not change backend queries, metric identity, or emitted telemetry.
|
|
283
|
+
|
|
261
284
|
## 8. Dependency Direction
|
|
262
285
|
|
|
263
286
|
ObservMe telemetry emission must not depend on Grafana query availability.
|
|
@@ -169,6 +169,27 @@ GitHub-hosted runners satisfy the supported clock expectation. A self-hosted run
|
|
|
169
169
|
|
|
170
170
|
## 5. Common Incidents
|
|
171
171
|
|
|
172
|
+
### Grafana-Backed Commands Reject the Base URL
|
|
173
|
+
|
|
174
|
+
If `/obs status`, `/obs health`, or a query-backed command reports `embedded_grafana_url_credentials` or `embedded_credentials`:
|
|
175
|
+
|
|
176
|
+
1. Configure `query.grafana.url` as an absolute HTTP(S) base URL without an embedded username or password component.
|
|
177
|
+
2. Put bearer authentication in `query.grafana.token`, or configure both `query.grafana.username` and `query.grafana.password`. Prefer environment-variable references rather than secret values in YAML.
|
|
178
|
+
3. Restart Pi, then rerun `/obs status` and `/obs health`.
|
|
179
|
+
|
|
180
|
+
ObservMe rejects this URL class before default-fetch or custom-transport network I/O. Diagnostics intentionally identify only the safe failure class and setting names, never the rejected URL or credentials.
|
|
181
|
+
|
|
182
|
+
### Project Config or `.env` Is Rejected
|
|
183
|
+
|
|
184
|
+
Project-local configuration is loaded only for a trusted project and only while its lexical and canonical paths remain inside the stable canonical project root.
|
|
185
|
+
|
|
186
|
+
1. Confirm Pi marks the project trusted.
|
|
187
|
+
2. Use a normal in-root config directory and `.env` file, or symlinks whose existing targets also remain inside the same project root.
|
|
188
|
+
3. Replace out-of-root, dangling, or changing symlinks and retry the session start. Do not bypass the containment check.
|
|
189
|
+
4. Review `/obs status` for `config_source_rejected`; bootstrap failures appear as a bounded warning.
|
|
190
|
+
|
|
191
|
+
ObservMe fails closed if it cannot verify the root, target, or opened-file identity. A rejected project config or `.env` is not read, starter creation never overwrites an existing target, and diagnostics omit canonical and external path details. Other accepted configuration layers continue to apply.
|
|
192
|
+
|
|
172
193
|
### No Traces in Tempo
|
|
173
194
|
|
|
174
195
|
Check:
|