@senad-d/observme 0.1.2 → 0.1.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +4 -0
- package/CHANGELOG.md +14 -0
- package/README.md +15 -4
- package/dashboards/observme-agent-node-graphs.json +5 -5
- package/dashboards/observme-agents.json +169 -4
- package/dashboards/observme-alerts.yaml +16 -3
- package/dashboards/observme-overview.json +6 -6
- package/dashboards/observme-trace-journey.json +4 -4
- package/docs/agent-subagent-observability-requirements.md +15 -7
- package/docs/compatibility-matrix.md +10 -2
- package/docs/configuration.md +5 -0
- package/docs/extension-integration.md +14 -2
- package/docs/reference/04-telemetry-semantic-conventions.md +38 -1
- package/docs/reference/05-otel-pipeline-and-collector.md +23 -10
- package/docs/reference/09-dashboards-alerts-slos.md +132 -3
- package/docs/reference/10-testing-release-operations.md +44 -14
- package/docs/reference/11-deployment-runbooks.md +61 -5
- package/docs/reference/12-configuration-reference.md +18 -1
- package/docs/review-validation.md +17 -0
- package/examples/README.md +7 -2
- package/examples/collector.yaml +8 -8
- package/examples/observme.yaml +1 -0
- package/package.json +3 -1
- package/src/config/bootstrap-project-config.ts +1 -0
- package/src/config/defaults.ts +1 -0
- package/src/config/load-config.ts +11 -0
- package/src/config/schema.ts +9 -0
- package/src/config/validate.ts +20 -1
- package/src/integration.ts +46 -4
- package/src/pi/active-agent-lease.ts +101 -0
- package/src/pi/event-handlers/lifecycle.ts +120 -46
- package/src/pi/handler-runtime.ts +21 -3
- package/src/pi/handler-types.ts +10 -2
- package/src/pi/integration-api.ts +184 -13
- package/src/semconv/metrics.ts +6 -0
package/docs/configuration.md
CHANGED
|
@@ -24,6 +24,9 @@ Edit the project config file (`.pi/observme.yaml` in the standard distribution)
|
|
|
24
24
|
- `privacy` — redaction, unsafe-capture acknowledgement, insecure transport, hash salt env var, and path handling. Keep `redactionEnabled: true`; set `allowUnsafeCapture: true` only when you intentionally accept unredacted sensitive-content export from this trusted project. Live telemetry and `/obs backfill` use the same policy: disabled capture omits content, enabled redaction redacts then truncates, redaction failures drop content, and `redactionEnabled: false` with `allowUnsafeCapture: true` exports raw truncated content.
|
|
25
25
|
- `query.grafana` — Grafana URL, datasource UIDs, TLS, and IPv4 transport settings for `/obs` query commands.
|
|
26
26
|
- `query.links.traceUrlTemplate` — the Grafana Explore trace-link template used by `/obs trace` and `/obs link`.
|
|
27
|
+
- `metrics.activeAgentLeaseDurationMillis` — how long the last exported active-agent lease remains valid. The default is `60000` ms, the supported range is `10000`–`300000` ms, and the value must be at least `(2 * metrics.exportIntervalMillis) + 5000` ms. `OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS` overrides YAML through the normal precedence rules.
|
|
28
|
+
|
|
29
|
+
The shipped dashboards and alerts combine a positive `observme_active_agents` lifecycle claim with a current lease; raw active-claim sums are diagnostic only after an ungraceful exit. Clean shutdown reaches zero after normal export/scrape propagation. Crash, `SIGKILL`, forced GitHub Actions cancellation, or runner loss converges within the lease plus up to 5 seconds of supported clock skew and one Prometheus scrape/evaluation interval, without restarting the Collector. Keep producer and Prometheus clocks synchronized within 5 seconds; GitHub-hosted runners meet this expectation, while self-hosted runners require reliable NTP or an equivalent time service.
|
|
27
30
|
|
|
28
31
|
Keep credentials out of YAML. Reference environment variables such as `OBSERVME_OTLP_TOKEN`, `OBSERVME_GRAFANA_TOKEN`, `OBSERVME_GRAFANA_PASSWORD`, and `OBSERVME_HASH_SALT`, then set those values in the shell or a trusted project `.env` file. When any content capture flag is enabled with `privacy.redactionEnabled: true`, `OBSERVME_HASH_SALT` must be set before the event occurs; otherwise content capture fails closed and emits `redaction.failed` diagnostics instead of conversation rows.
|
|
29
32
|
|
|
@@ -45,3 +48,5 @@ Invalid or unsafe merged configuration falls back to safe defaults. `/obs status
|
|
|
45
48
|
- [Example configurations](../examples/README.md)
|
|
46
49
|
- [Security, privacy, and redaction](reference/06-security-privacy-redaction.md)
|
|
47
50
|
- [Grafana and `/obs` validation flow](validation-flow.md)
|
|
51
|
+
- [Canonical active-agent queries and raw-query migration](reference/09-dashboards-alerts-slos.md#131-canonical-active-agent-promql)
|
|
52
|
+
- [Active-lease deployment and troubleshooting runbook](reference/11-deployment-runbooks.md)
|
|
@@ -35,7 +35,7 @@ const observme: ObservMeIntegrationApi | undefined = requestObservMeIntegration(
|
|
|
35
35
|
|
|
36
36
|
ObservMe registers no global object and does not require another extension to import its internal telemetry session. The event bus is the runtime boundary; the package subpath provides the stable constants, types, and request helper.
|
|
37
37
|
|
|
38
|
-
The API can be absent when ObservMe is not installed, disabled by package configuration, not loaded yet, or
|
|
38
|
+
The API can be absent when ObservMe is not installed, disabled by package configuration, not loaded yet, incompatible, or connected through a failing/malformed event-bus provider. A method returns `{ ok: false, reason: "session_unavailable" }` when ObservMe is loaded but no telemetry session is active. Orchestration must remain functional in both cases and may run the child without ObservMe correlation after reporting a bounded local warning.
|
|
39
39
|
|
|
40
40
|
A package that cannot take a runtime dependency can implement the same synchronous request directly. Keep this channel and version stable:
|
|
41
41
|
|
|
@@ -118,9 +118,21 @@ Do not put raw tasks, prompts, command lines, environment values, child output,
|
|
|
118
118
|
| `toolCallId` | Optional high-cardinality trace/log correlation when a tool initiated the spawn. |
|
|
119
119
|
| `env` | Base child environment. ObservMe removes stale lineage/W3C keys and returns the replacement environment. |
|
|
120
120
|
|
|
121
|
+
Runtime callers are validated even when JavaScript bypasses the TypeScript types. Caller-provided lifecycle identifiers must match `[A-Za-z0-9._:-]{1,128}`. Commands and individual arguments are capped at 4096 characters, argument lists at 256 items, and environment objects at 4096 entries. Durations must be finite, non-negative milliseconds. Invalid or duplicate active operations return a failure without replacing an existing span.
|
|
122
|
+
|
|
121
123
|
Completion and wait/join methods use bounded child states (`starting`, `active`, `completed`, `failed`, `cancelled`, `orphaned`), join states (`waiting`, `completed`, `failed`, `cancelled`, `timeout`, `unknown`), and wait reasons (`dependency`, `rate_limit`, `child_running`, `unknown`). `failurePropagated=false` on a completed join confirms that the parent recovered from a failed child.
|
|
122
124
|
|
|
123
|
-
All mutation methods return a discriminated result. Handle
|
|
125
|
+
All mutation methods return a discriminated result. Handle these reasons without crashing Pi:
|
|
126
|
+
|
|
127
|
+
| Reason | Meaning |
|
|
128
|
+
| --- | --- |
|
|
129
|
+
| `session_unavailable` | ObservMe is loaded but no telemetry session is active. |
|
|
130
|
+
| `invalid_request` | An identifier, enum, duration, command/argument field, or environment shape is invalid or oversized. |
|
|
131
|
+
| `spawn_already_exists` / `wait_already_exists` / `join_already_exists` | The requested lifecycle identifier is already active. Generate a unique identifier or finish the active operation; do not overwrite it. |
|
|
132
|
+
| `spawn_not_found` / `wait_not_found` / `join_not_found` | The lifecycle handle is absent or has already ended. |
|
|
133
|
+
| `operation_failed` | ObservMe could not safely complete the operation. |
|
|
134
|
+
|
|
135
|
+
Do not retry a completed lifecycle handle blindly; repeated completion can otherwise hide an orchestration-state bug.
|
|
124
136
|
|
|
125
137
|
## Propagation environment
|
|
126
138
|
|
|
@@ -503,9 +503,46 @@ observme_handler_duration_ms
|
|
|
503
503
|
|
|
504
504
|
```text
|
|
505
505
|
observme_active_spans
|
|
506
|
-
observme_active_agents
|
|
506
|
+
observme_active_agents # compatibility lifecycle claim
|
|
507
|
+
observme_agent_lease_expires_unixtime_seconds # asynchronous gauge, unit s
|
|
507
508
|
```
|
|
508
509
|
|
|
510
|
+
#### 12.4.1 Active-agent lease and clock contract
|
|
511
|
+
|
|
512
|
+
This section is the single source of truth for active-agent lease constants and semantics. Runtime configuration, metric metadata, PromQL, dashboards, alerts, examples, and tests must reference these values rather than define different local windows.
|
|
513
|
+
|
|
514
|
+
| Contract item | Required value |
|
|
515
|
+
|---|---|
|
|
516
|
+
| Lease metric | `observme_agent_lease_expires_unixtime_seconds` |
|
|
517
|
+
| OTel instrument and unit | Asynchronous gauge, unit `s` |
|
|
518
|
+
| Observation value | `(wallClockUnixMillis + metrics.activeAgentLeaseDurationMillis) / 1000`; fractional Unix seconds are valid |
|
|
519
|
+
| Configuration | `metrics.activeAgentLeaseDurationMillis`; `OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS` is its environment override |
|
|
520
|
+
| Default lease | `60000` ms (60 seconds) |
|
|
521
|
+
| Inclusive duration bounds | `10000` ms through `300000` ms (10 seconds through 5 minutes) |
|
|
522
|
+
| Export relationship | Lease duration must be at least `(2 * metrics.exportIntervalMillis) + 5000` ms |
|
|
523
|
+
| Supported wall-clock skew | Producer and Prometheus clocks must differ by no more than 5 seconds |
|
|
524
|
+
| Future-timestamp horizon | A lease later than `time() + 305` seconds is pathological and must not count active |
|
|
525
|
+
|
|
526
|
+
`observme_active_agents` remains the clean lifecycle claim. An agent is operationally active only when its claim is positive and the matching lease is finite, later than Prometheus `time()`, and no later than the future-timestamp horizon. Missing, expired, malformed, non-finite, or pathologically future leases fail closed. Lease renewal occurs once per SDK metric collection while the session is active, including otherwise idle sessions; it does not depend on turns, tool calls, or a background heartbeat timer.
|
|
527
|
+
|
|
528
|
+
On clean shutdown, deactivate the lease before the final metric force-flush and record the matching `observme_active_agents` decrement. This makes the leased result zero after normal export/scrape propagation without waiting for expiry. If shutdown telemetry cannot run, the last exported lease is allowed to expire. With synchronized clocks, an ungraceful exit converges to zero within the configured lease plus one Prometheus scrape/evaluation interval. Across the full supported positive-skew budget, the measurable worst-case bound is the configured lease plus 5 seconds plus one scrape/evaluation interval; for the defaults and a 15-second scrape interval, that is at most 80 seconds.
|
|
529
|
+
|
|
530
|
+
The only per-runtime Prometheus join and deduplication key is the Collector-normalized `observme_instance_id` resource label. ObservMe generates its UUID independently for each telemetry session and never derives it from prompts, responses, content, paths, commands, GitHub run IDs, sessions, workflows, or logical agents. `service.instance.id` may carry the same resource value for OTel interoperability, but active-agent PromQL joins on `observme_instance_id` and deduplicates repeated exporter/scrape representations with `max by (observme_instance_id)`. The lease observation itself may carry only existing bounded metric labels; the instance identity comes from resource-to-metric conversion and must not become a dashboard variable or legend.
|
|
531
|
+
|
|
532
|
+
Lifecycle and failure behavior is fixed as follows:
|
|
533
|
+
|
|
534
|
+
| Condition | Required leased-active behavior |
|
|
535
|
+
|---|---|
|
|
536
|
+
| Clean exit | Deactivate before final flush; the active claim becomes zero without waiting for lease expiry. |
|
|
537
|
+
| `SIGTERM` | Use the clean path when Pi delivers shutdown; otherwise use the same expiry behavior as an abrupt exit. |
|
|
538
|
+
| Crash, `SIGKILL`, cancelled GitHub job, or runner/power loss | No cleanup is required for correctness; the last lease expires within the documented convergence bound. |
|
|
539
|
+
| Duplicate session start, reload, or resume | Deactivate and flush the replaced session before activating its replacement. If replacement cleanup is interrupted, the old instance lease expires independently. |
|
|
540
|
+
| Startup failure or partial runtime construction | Never leave an activated observable callback; dispose any registered callback. |
|
|
541
|
+
| Collector or OTLP outage | Cached positive claims stop counting when their last delivered leases expire. A still-running producer becomes active again after export recovers and a fresh lease arrives. Backend unavailability itself remains an unknown/no-query state. |
|
|
542
|
+
| Clock skew or clock movement | A producer clock ahead can delay expiry and one behind can fail closed early. Clocks outside the supported skew are unsupported; non-finite observations are omitted and values beyond the future horizon do not count. GitHub-hosted runners satisfy the synchronized-clock expectation; self-hosted runners must use reliable time synchronization. |
|
|
543
|
+
|
|
544
|
+
A renewable lease is a bounded liveness signal, not proof that the operating-system process is reachable at the exact query instant. `observme_active_agents` remains emitted with its existing name and clean-lifecycle behavior for compatibility, so the change is additive under the semantic-convention versioning policy and does not require a major convention-version increment. Consumers that need crash-safe live counts must migrate from raw active-claim sums to the leased definition.
|
|
545
|
+
|
|
509
546
|
### 12.5 Lifecycle recording points
|
|
510
547
|
|
|
511
548
|
- Record `observme_tool_result_size_chars` once from the finalized `tool_execution_end` result, using the same bounded tool labels as tool-call metrics.
|
|
@@ -47,6 +47,7 @@ metrics:
|
|
|
47
47
|
enabled: true
|
|
48
48
|
exportIntervalMillis: 15000
|
|
49
49
|
exportTimeoutMillis: 3000
|
|
50
|
+
activeAgentLeaseDurationMillis: 60000
|
|
50
51
|
|
|
51
52
|
logs:
|
|
52
53
|
enabled: true
|
|
@@ -58,6 +59,8 @@ logs:
|
|
|
58
59
|
|
|
59
60
|
The OpenTelemetry JS OTLP HTTP exporters default to signal-specific paths such as `http://localhost:4318/v1/traces`, `http://localhost:4318/v1/metrics`, and `/v1/logs`. Treat `otlp.endpoint` as a base endpoint in ObservMe config, but pass signal-specific URLs to SDK exporters that require an explicit `url`.
|
|
60
61
|
|
|
62
|
+
The default active-agent lease is 60 seconds and renews on each 15-second metric collection. Supported lease values are 10 seconds through 5 minutes and must be at least twice the export interval plus 5 seconds. Keep the producer and Prometheus clocks within 5 seconds; otherwise a lease may fail closed early or remain valid longer than intended. The exact contract is frozen in [`04-telemetry-semantic-conventions.md` §12.4.1](04-telemetry-semantic-conventions.md#1241-active-agent-lease-and-clock-contract).
|
|
63
|
+
|
|
61
64
|
## 4. Trace Context and Workflow/Agent-Lineage Propagation
|
|
62
65
|
|
|
63
66
|
When ObservMe launches or wraps a subagent process, the parent runtime should propagate both standard trace context and ObservMe-specific lineage:
|
|
@@ -118,10 +121,10 @@ service:
|
|
|
118
121
|
|
|
119
122
|
## 6. Production Collector for Grafana Stack
|
|
120
123
|
|
|
121
|
-
This configuration routes traces to Tempo, logs to Loki, and metrics to a Prometheus
|
|
124
|
+
This configuration routes traces to Tempo, logs to Loki, and metrics to a Prometheus scrape endpoint. Use a Collector distribution that contains every configured component. The minimal debug example works with the core Collector, while `prometheus`, `prometheusremotewrite`, `probabilistic_sampler`, and `tail_sampling` are commonly deployed from the Collector Contrib distribution or a vendor distribution such as Grafana Alloy.
|
|
122
125
|
|
|
123
126
|
```yaml
|
|
124
|
-
# Production-oriented Collector reference for Tempo, Loki, and
|
|
127
|
+
# Production-oriented Collector reference for Tempo, Loki, and Prometheus.
|
|
125
128
|
# Verify that your Collector distribution contains every configured component,
|
|
126
129
|
# replace backend endpoints/security for your deployment, and never add raw
|
|
127
130
|
# content or high-cardinality execution identifiers to metric labels.
|
|
@@ -220,14 +223,14 @@ exporters:
|
|
|
220
223
|
retry_on_failure:
|
|
221
224
|
enabled: true
|
|
222
225
|
|
|
223
|
-
|
|
224
|
-
endpoint:
|
|
226
|
+
prometheus:
|
|
227
|
+
endpoint: 0.0.0.0:8889
|
|
228
|
+
# Exporter-wide stale-series/cardinality cleanup for gauges, counters, and
|
|
229
|
+
# histograms. Five minutes exceeds ObservMe's default one-minute active-agent
|
|
230
|
+
# lease; lease-aware PromQL, not expiration, determines liveness.
|
|
231
|
+
metric_expiration: 5m
|
|
225
232
|
resource_to_telemetry_conversion:
|
|
226
233
|
enabled: true
|
|
227
|
-
sending_queue:
|
|
228
|
-
enabled: true
|
|
229
|
-
retry_on_failure:
|
|
230
|
-
enabled: true
|
|
231
234
|
|
|
232
235
|
debug:
|
|
233
236
|
verbosity: basic
|
|
@@ -243,7 +246,7 @@ service:
|
|
|
243
246
|
metrics:
|
|
244
247
|
receivers: [otlp]
|
|
245
248
|
processors: [memory_limiter, resource/observme, resource/drop_high_cardinality_metric_attrs, batch]
|
|
246
|
-
exporters: [
|
|
249
|
+
exporters: [prometheus]
|
|
247
250
|
|
|
248
251
|
logs:
|
|
249
252
|
receivers: [otlp]
|
|
@@ -251,9 +254,19 @@ service:
|
|
|
251
254
|
exporters: [otlphttp/loki]
|
|
252
255
|
```
|
|
253
256
|
|
|
257
|
+
### 6.1 Prometheus exporter expiration policy
|
|
258
|
+
|
|
259
|
+
The shipped local and production scrape-exporter examples set `metric_expiration: 5m`. This exporter-level setting applies uniformly to every Prometheus-exported gauge, counter, and histogram; it cannot be scoped only to active-agent metrics. Its only purpose is stale-series and cardinality cleanup: it bounds the Collector memory used by abandoned metric streams and eventually lets Prometheus mark an expired series stale after the exporter stops exposing it. Lease-aware PromQL remains the active-agent correctness mechanism, even while the Collector still exposes a cached positive `observme_active_agents` value.
|
|
260
|
+
|
|
261
|
+
Five minutes comfortably exceeds the default 60-second active-agent lease and 15-second SDK metric export interval. If either interval is customized, keep `metric_expiration` longer than the lease and allow enough additional time for expected export delays and scrape gaps. A shorter value cleans abandoned streams sooner but can evict gauges, counters, and histograms during a temporary producer outage; a longer value consumes Collector memory and preserves abandoned cardinality for longer. It must never be shortened merely to make active-agent totals converge.
|
|
262
|
+
|
|
263
|
+
Apply a Collector configuration change through the deployment's supported reload mechanism or restart the Collector. A restart clears the Prometheus exporter's in-memory series cache, but neither reload nor restart deletes samples already stored in Prometheus; historical samples remain subject to Prometheus retention and deletion policy. Restart is never required merely to make an expired active-agent lease stop counting. `metric_expiration` belongs only to the metrics scrape exporter, so traces and logs are unaffected. Remote-write deployments do not use this scrape-exporter cache setting and must apply their backend's series-retention policy, while continuing to use the same lease-aware active-agent queries.
|
|
264
|
+
|
|
265
|
+
When migrating, deploy a version that emits `observme_agent_lease_expires_unixtime_seconds`, verify the Collector preserves the generated `observme_instance_id`, then replace every current-activity raw sum in dashboards, recording rules, and alerts with the canonical queries in [`09-dashboards-alerts-slos.md` §1.3.1](09-dashboards-alerts-slos.md#131-canonical-active-agent-promql). Keep raw and expired claims only as clearly labeled diagnostics. Missing leases intentionally fail closed, so do not switch a mixed-version fleet until the lease metric is visible for the producers that should count.
|
|
266
|
+
|
|
254
267
|
The Collector attribute processor is defense in depth for log attributes only. It must not be the only redaction layer, and it does not sanitize arbitrary log bodies; ObservMe must redact or drop sensitive content before export. The traces pipeline intentionally keeps `pi.llm.prompt.redacted`, `pi.llm.response.redacted`, and `pi.llm.thinking.redacted` so Tempo can display redacted content after explicit capture is enabled. Old telemetry that an earlier Collector dropped cannot be recovered; generate new LLM events after updating the Collector.
|
|
255
268
|
|
|
256
|
-
Keep high-cardinality lineage attributes on traces/logs, but remove them from the metrics pipeline unless the organization explicitly accepts the cardinality. The bundled local Loki pipeline promotes `pi.agent.id` and `pi.agent.run.id` as log labels so the LLM Conversations dashboard can filter captured chat content by agent and run. Prometheus resource-to-telemetry conversion is safe only after the metrics pipeline drops `pi.workflow.id`, `pi.agent.id`, `pi.session.id`, trace IDs, and spawn IDs
|
|
269
|
+
Keep high-cardinality lineage attributes on traces/logs, but remove them from the metrics pipeline unless the organization explicitly accepts the cardinality. The bundled local Loki pipeline promotes `pi.agent.id` and `pi.agent.run.id` as log labels so the LLM Conversations dashboard can filter captured chat content by agent and run. Prometheus resource-to-telemetry conversion is safe only after the metrics pipeline drops `pi.workflow.id`, `pi.agent.id`, `pi.session.id`, trace IDs, and spawn IDs. Preserve the generated `service.instance.id`/`observme.instance.id` resource identity in the metrics pipeline so lease-aware queries can join and deduplicate concurrent ObservMe metric streams without exposing workflow, session, logical-agent, trace, span, or job-run identifiers as labels.
|
|
257
270
|
|
|
258
271
|
## 7. Direct-to-Backend Development Mode
|
|
259
272
|
|
|
@@ -129,9 +129,126 @@ Zero-state interpretation checklist:
|
|
|
129
129
|
- Empty failure-only Loki tables mean no matching failures in the selected range, not that the log pipeline is broken.
|
|
130
130
|
- `or vector(0)` stats must say whether zero means healthy idle/no activity or a true zero failure ratio.
|
|
131
131
|
- Optional redacted content panels can be empty when capture is disabled or when the Collector dropped content attributes by policy.
|
|
132
|
-
- Active-session estimates are approximate across process restarts
|
|
132
|
+
- Active-session estimates are approximate across process restarts and counter resets. Raw `observme_active_agents` values are lifecycle claims, not authoritative liveness after an ungraceful exit.
|
|
133
133
|
- High-cardinality execution identifiers are allowed as Loki/Tempo filter variables only and must never be introduced into Prometheus queries or panel links from aggregate panels.
|
|
134
134
|
|
|
135
|
+
### 1.3 Active-agent leased-state contract
|
|
136
|
+
|
|
137
|
+
The authoritative metric, duration, export-interval relationship, clock-skew allowance, future-timestamp horizon, convergence target, and failure-mode behavior are defined once in [`04-telemetry-semantic-conventions.md` §12.4.1](04-telemetry-semantic-conventions.md#1241-active-agent-lease-and-clock-contract). Dashboards, alerts, examples, and tests must use that contract without introducing panel-local lease or skew windows.
|
|
138
|
+
|
|
139
|
+
Operational active-agent views combine a positive `observme_active_agents` lifecycle claim with a currently valid `observme_agent_lease_expires_unixtime_seconds` value on the generated `observme_instance_id` resource label. The instance label is an internal join/deduplication key only: it must not be exposed as a dashboard variable, group, legend, URL value, or topology identity. Duplicate scrape/exporter representations of one instance count once.
|
|
140
|
+
|
|
141
|
+
Interpret active-agent states as follows:
|
|
142
|
+
|
|
143
|
+
- positive claim plus a valid current lease: active;
|
|
144
|
+
- clean-shutdown claim of zero: inactive after normal export/scrape propagation, without waiting for lease expiry;
|
|
145
|
+
- positive claim plus an expired lease: inactive stale claim, available only as a diagnostic;
|
|
146
|
+
- missing, malformed, non-finite, or pathologically future lease: inactive (fail closed);
|
|
147
|
+
- no matching series: zero-safe healthy idle/no activity when the telemetry backend itself is reachable.
|
|
148
|
+
|
|
149
|
+
A crash, `SIGKILL`, cancelled job, or lost runner therefore converges to zero within the semantic-convention bound even if the Collector continues to expose the cached raw claim. A Collector/export outage can intentionally undercount after the last delivered lease expires; use Export Health and backend reachability to distinguish that state from process termination. Clean lifecycle handling and GitHub Actions `if: always()` cleanup remain useful latency optimizations, but neither is the correctness mechanism.
|
|
150
|
+
|
|
151
|
+
### 1.3.1 Canonical active-agent PromQL
|
|
152
|
+
|
|
153
|
+
Use the following total everywhere current live-agent state is required:
|
|
154
|
+
|
|
155
|
+
```promql
|
|
156
|
+
sum(
|
|
157
|
+
max by (observme_instance_id) (
|
|
158
|
+
(observme_active_agents > 0)
|
|
159
|
+
and on (observme_instance_id)
|
|
160
|
+
(
|
|
161
|
+
(observme_agent_lease_expires_unixtime_seconds > time())
|
|
162
|
+
and
|
|
163
|
+
(observme_agent_lease_expires_unixtime_seconds <= time() + 305)
|
|
164
|
+
)
|
|
165
|
+
)
|
|
166
|
+
) or vector(0)
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Both comparisons are required. The lower bound expires stale claims; the upper bound rejects leases beyond the frozen 305-second future horizon. `and on (observme_instance_id)` fails closed when the lease is absent or non-comparable, and `max by (observme_instance_id)` removes duplicate Collector/exporter/scrape replicas before the outer sum. The final `or vector(0)` makes no-series results render as zero; when Prometheus and Export Health are reachable, zero can mean healthy idle, clean shutdown, an expired claim, or a lease that failed closed.
|
|
170
|
+
|
|
171
|
+
For a bounded emitted dimension, retain that dimension in both aggregations. Grouped queries use `or on() vector(0)` so the unlabeled zero appears only when no labeled breakdown exists. Canonical role and environment breakdowns are:
|
|
172
|
+
|
|
173
|
+
```promql
|
|
174
|
+
sum by (agent_role) (
|
|
175
|
+
max by (observme_instance_id, agent_role) (
|
|
176
|
+
(observme_active_agents > 0)
|
|
177
|
+
and on (observme_instance_id)
|
|
178
|
+
(
|
|
179
|
+
(observme_agent_lease_expires_unixtime_seconds > time())
|
|
180
|
+
and
|
|
181
|
+
(observme_agent_lease_expires_unixtime_seconds <= time() + 305)
|
|
182
|
+
)
|
|
183
|
+
)
|
|
184
|
+
) or on() vector(0)
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
```promql
|
|
188
|
+
sum by (environment) (
|
|
189
|
+
max by (observme_instance_id, environment) (
|
|
190
|
+
(observme_active_agents > 0)
|
|
191
|
+
and on (observme_instance_id)
|
|
192
|
+
(
|
|
193
|
+
(observme_agent_lease_expires_unixtime_seconds > time())
|
|
194
|
+
and
|
|
195
|
+
(observme_agent_lease_expires_unixtime_seconds <= time() + 305)
|
|
196
|
+
)
|
|
197
|
+
)
|
|
198
|
+
) or on() vector(0)
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
The shipped active claim and lease currently emit `agent_role` and `environment`; they do not emit the metric label `subagent_depth`. Therefore current-active depth panels must not group by `subagent_depth`. For a compatible pipeline that actually emits the same bounded `subagent_depth` on the active claim, this is the supported depth shape:
|
|
202
|
+
|
|
203
|
+
```promql
|
|
204
|
+
sum by (subagent_depth) (
|
|
205
|
+
max by (observme_instance_id, subagent_depth) (
|
|
206
|
+
(observme_active_agents > 0)
|
|
207
|
+
and on (observme_instance_id)
|
|
208
|
+
(
|
|
209
|
+
(observme_agent_lease_expires_unixtime_seconds > time())
|
|
210
|
+
and
|
|
211
|
+
(observme_agent_lease_expires_unixtime_seconds <= time() + 305)
|
|
212
|
+
)
|
|
213
|
+
)
|
|
214
|
+
) or on() vector(0)
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
Do not substitute resource-derived or high-cardinality execution labels for these dimensions. Use the bounded spawn/tree metrics for depth views in the shipped dashboards until `subagent_depth` is emitted on the active signals themselves.
|
|
218
|
+
|
|
219
|
+
Raw positive lifecycle claims are diagnostic only:
|
|
220
|
+
|
|
221
|
+
```promql
|
|
222
|
+
sum(max by (observme_instance_id) (observme_active_agents > 0)) or vector(0)
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
Expired positive claims are diagnostic and never contribute to the live total:
|
|
226
|
+
|
|
227
|
+
```promql
|
|
228
|
+
sum(
|
|
229
|
+
max by (observme_instance_id) (
|
|
230
|
+
(observme_active_agents > 0)
|
|
231
|
+
and on (observme_instance_id)
|
|
232
|
+
(observme_agent_lease_expires_unixtime_seconds <= time())
|
|
233
|
+
)
|
|
234
|
+
) or vector(0)
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
Compare raw claims, expired claims, the canonical live total, and Export Health when diagnosing a producer crash or export gap. A raw-minus-live difference can also include a missing, malformed, or pathologically future lease, so it must not be labeled strictly as “expired.” Never expose `observme_instance_id` as a variable or legend in these diagnostics.
|
|
238
|
+
|
|
239
|
+
### 1.3.2 Migration from raw active-agent queries
|
|
240
|
+
|
|
241
|
+
Use this rollout order for custom dashboards, recording rules, alerts, and API consumers:
|
|
242
|
+
|
|
243
|
+
1. Deploy ObservMe producers that emit both `observme_active_agents` and `observme_agent_lease_expires_unixtime_seconds` with the same generated `observme_instance_id` resource label.
|
|
244
|
+
2. Confirm the lease metric is present and clocks are synchronized before changing consumers. A missing lease fails closed, so a mixed-version producer without the new metric will not appear in the leased total.
|
|
245
|
+
3. Replace raw `sum(observme_active_agents)`, raw role sums, and any equivalent positive-claim aggregation with the canonical total or bounded-dimension form above. Preserve the future-horizon check, instance join, deduplication, and zero-safe fallback together.
|
|
246
|
+
4. Keep raw and expired-claim queries only in panels or runbooks explicitly labeled as diagnostics. Do not use them for paging, capacity decisions, or current topology.
|
|
247
|
+
5. Keep Collector `metric_expiration` longer than the lease. Do not shorten expiration or schedule Collector restarts to correct liveness; an expired lease stops counting while the cached raw claim remains available for diagnosis.
|
|
248
|
+
6. During rollback, old raw consumers continue to see the compatibility lifecycle claim, but they regain the known ghost-agent behavior after ungraceful exits. Treat rollback as a temporary loss of crash-safe liveness semantics.
|
|
249
|
+
|
|
250
|
+
Validate the migrated consumer with live+valid, clean-shutdown, expired, missing, pathologically future, duplicate-replica, and no-series cases before production use.
|
|
251
|
+
|
|
135
252
|
## 2. Overview Dashboard
|
|
136
253
|
|
|
137
254
|
Purpose: act as the operator landing page. Users should be able to answer whether ObservMe is healthy, workload is normal, agents/subagents are healthy, spend or latency is high, and which focused dashboard to open next without scanning every raw metric series.
|
|
@@ -584,10 +701,22 @@ Severity: warning.
|
|
|
584
701
|
### Active Agents Stuck High
|
|
585
702
|
|
|
586
703
|
```promql
|
|
587
|
-
|
|
704
|
+
(
|
|
705
|
+
sum(
|
|
706
|
+
max by (observme_instance_id) (
|
|
707
|
+
(observme_active_agents > 0)
|
|
708
|
+
and on (observme_instance_id)
|
|
709
|
+
(
|
|
710
|
+
(observme_agent_lease_expires_unixtime_seconds > time())
|
|
711
|
+
and
|
|
712
|
+
(observme_agent_lease_expires_unixtime_seconds <= time() + 305)
|
|
713
|
+
)
|
|
714
|
+
)
|
|
715
|
+
) or vector(0)
|
|
716
|
+
) > 100
|
|
588
717
|
```
|
|
589
718
|
|
|
590
|
-
Severity depends on normal fleet size; tune per deployment.
|
|
719
|
+
Severity depends on normal fleet size; tune per deployment. This alert uses leased live agents, not cached raw claims.
|
|
591
720
|
|
|
592
721
|
## 12. SLOs
|
|
593
722
|
|
|
@@ -14,7 +14,9 @@ ObservMe must be tested at five levels:
|
|
|
14
14
|
|
|
15
15
|
Required areas:
|
|
16
16
|
|
|
17
|
-
- Config loading and
|
|
17
|
+
- Config loading, precedence, and active-agent lease/export-interval bounds
|
|
18
|
+
- Active-agent lease renewal, deactivation, disposal, clock movement, and clean/abrupt lifecycle behavior
|
|
19
|
+
- Canonical lease-aware PromQL, replica deduplication, and missing/expired/future lease cases
|
|
18
20
|
- Redaction patterns
|
|
19
21
|
- Hashing stability
|
|
20
22
|
- Truncation behavior
|
|
@@ -123,6 +125,29 @@ Tests:
|
|
|
123
125
|
- Prometheus metric query for token totals
|
|
124
126
|
- Grafana dashboard import validates
|
|
125
127
|
|
|
128
|
+
### 6.1 Active-agent lease integration
|
|
129
|
+
|
|
130
|
+
Run the focused cached-exporter test separately from the broad Grafana stack:
|
|
131
|
+
|
|
132
|
+
```bash
|
|
133
|
+
npm run test:integration:active-agent-lease
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
The test starts pinned Collector and Prometheus containers, then runs real Node telemetry producers with a test-safe 2-second export interval, 10-second lease, and 1-second Prometheus scrape interval. It validates clean shutdown and `SIGTERM` first, then kills a producer with `SIGKILL` and requires all of the following:
|
|
137
|
+
|
|
138
|
+
- the raw `observme_active_agents` claim remains positive across fresh Prometheus scrapes;
|
|
139
|
+
- the canonical lease-aware count reaches zero without a shutdown event;
|
|
140
|
+
- convergence stays within the documented 10-second lease, 5-second clock-skew allowance, and one 1-second scrape interval;
|
|
141
|
+
- the Collector container identity and zero restart count remain unchanged.
|
|
142
|
+
|
|
143
|
+
The test uses no credentials or content capture. Its parent process owns bounded waits and unconditional cleanup for child processes, containers, anonymous storage, and the dedicated Docker network after success, assertion failure, startup failure, or timeout. A one-minute test-only `metric_expiration` deliberately remains much longer than the lease so exporter caching reproduces the old raw-query failure; production cleanup guidance remains five minutes for the default one-minute lease.
|
|
144
|
+
|
|
145
|
+
### 6.2 GitHub Actions cancellation validation
|
|
146
|
+
|
|
147
|
+
The `lease-cancellation` job in `.github/workflows/ci.yml` runs lease unit/contract tests and the focused integration on `ubuntu-latest` with a bounded timeout. It simulates cancellation by sending `SIGKILL` to a producer so the workflow can assert convergence; cancelling the validation job itself could not produce evidence about its own result. An `if: always()` step removes labeled Docker resources when GitHub still schedules cleanup, but the test must reach zero through lease expiry before that step runs.
|
|
148
|
+
|
|
149
|
+
Production workflows should likewise prefer graceful Pi and sidecar shutdown and use `if: always()` cleanup where available. Force cancellation, runner loss, or power loss can bypass both, so neither cleanup nor a shutdown callback may be the active-count correctness mechanism. GitHub-hosted runners meet the synchronized-clock expectation; self-hosted CI must keep the producer and Prometheus clocks within the supported 5-second skew.
|
|
150
|
+
|
|
126
151
|
## 7. Failure Tests
|
|
127
152
|
|
|
128
153
|
### Collector Down
|
|
@@ -228,15 +253,16 @@ Reject or sanitize dynamically generated labels.
|
|
|
228
253
|
|
|
229
254
|
## 10. Release Process
|
|
230
255
|
|
|
231
|
-
1. Update version.
|
|
232
|
-
2. Run unit tests.
|
|
233
|
-
3. Run integration
|
|
234
|
-
4. Run
|
|
235
|
-
5. Run dashboard
|
|
236
|
-
6.
|
|
237
|
-
7.
|
|
238
|
-
8.
|
|
239
|
-
9.
|
|
256
|
+
1. Update the version and `CHANGELOG.md`.
|
|
257
|
+
2. Run `npm run validate` for source/test typechecks, ESLint, formatting, script checks, unit/contract/dashboard/alert tests, package-content checks, packaged-install smoke, handler smoke, Pi lifecycle smoke, and Pi runtime smoke.
|
|
258
|
+
3. Run `npm run test:integration:collector` against the pinned Collector distribution.
|
|
259
|
+
4. Run `npm run test:integration:active-agent-lease` and require clean shutdown, `SIGTERM`, and `SIGKILL` convergence without Collector restart while the raw claim remains cached.
|
|
260
|
+
5. Run `npm run test:integration:grafana-stack` for live backends and dashboard provisioning. Record any unrelated pre-existing blocker precisely; it must not replace focused lease evidence.
|
|
261
|
+
6. Validate Compose interpolation and the Collector config with the pinned distribution when Collector/example configuration changed.
|
|
262
|
+
7. Run `npm run pack:dry-run` and confirm the lease metric source, dashboards, alerts, examples, user/operator/reference docs, and packaged `observme-docs` skill are present. Do not publish during this check.
|
|
263
|
+
8. Record sanitized date, environment versions, pass/fail status, cleanup result, and any blocker in `docs/compatibility-matrix.md` and `docs/review-validation.md`; never record credentials, raw content, private paths, commands from interactive Pi, or full high-cardinality identifiers.
|
|
264
|
+
9. Publish the package/artifact only after lease-related failures are resolved.
|
|
265
|
+
10. Tag the release.
|
|
240
266
|
|
|
241
267
|
## 11. Compatibility Matrix
|
|
242
268
|
|
|
@@ -266,6 +292,7 @@ observme_export_errors_total
|
|
|
266
292
|
observme_redaction_failures_total
|
|
267
293
|
observme_active_spans
|
|
268
294
|
observme_active_agents
|
|
295
|
+
observme_agent_lease_expires_unixtime_seconds
|
|
269
296
|
observme_workflows_started_total
|
|
270
297
|
observme_workflows_completed_total
|
|
271
298
|
observme_workflow_errors_total
|
|
@@ -313,7 +340,10 @@ When troubleshooting:
|
|
|
313
340
|
3. Run `/obs agents` when troubleshooting workflow, parent/subagent lineage, fan-out, depth, or orphan agents.
|
|
314
341
|
4. Check Collector health endpoint.
|
|
315
342
|
5. Check Collector logs.
|
|
316
|
-
6.
|
|
317
|
-
7. Query
|
|
318
|
-
8.
|
|
319
|
-
9.
|
|
343
|
+
6. For active-agent incidents, compare the canonical leased count, raw positive claims, expired claims, and direct presence of `observme_agent_lease_expires_unixtime_seconds`; then verify producer/Prometheus clock skew, `observme_instance_id` preservation, and the configured lease/export relationship.
|
|
344
|
+
7. Query `observme_export_errors_total` and inspect Export Health before deciding that a missing or expired lease means the producer stopped.
|
|
345
|
+
8. Query Loki for `event_name="export.failed"` (or equivalent normalized structured metadata).
|
|
346
|
+
9. Temporarily enable the debug exporter in the Collector.
|
|
347
|
+
10. Never enable raw content capture in production without approval.
|
|
348
|
+
|
|
349
|
+
See [`11-deployment-runbooks.md` §5](11-deployment-runbooks.md#5-common-incidents) for expired-claim and missing-lease decision trees.
|
|
@@ -88,6 +88,26 @@ Recommended ports:
|
|
|
88
88
|
9090 prometheus
|
|
89
89
|
```
|
|
90
90
|
|
|
91
|
+
### Prometheus exporter cleanup policy
|
|
92
|
+
|
|
93
|
+
The bundled Collector exposes all metrics through one Prometheus exporter with `metric_expiration: 5m`. Treat that value only as stale-series/cardinality cleanup, not as active-agent liveness. It applies to every exported gauge, counter, and histogram. The leased active-agent queries determine liveness and converge after the 60-second default lease even while a raw positive claim remains cached.
|
|
94
|
+
|
|
95
|
+
When changing `metrics.activeAgentLeaseDurationMillis`, `metrics.exportIntervalMillis`, scrape timing, or expected outage tolerance:
|
|
96
|
+
|
|
97
|
+
1. Keep `metric_expiration` longer than the active-agent lease; five minutes is the recommendation for the default 60-second lease and 15-second export interval.
|
|
98
|
+
2. Leave enough margin for ordinary export delays and scrape gaps. Shorter cleanup lowers Collector memory/cardinality but can remove every metric type during a temporary producer outage; longer cleanup retains abandoned series longer.
|
|
99
|
+
3. Preserve generated `observme.instance.id`/`service.instance.id` resource identity for lease joins, while continuing to drop workflow, session, logical-agent, trace, span, spawn, and job-run identifiers from metric labels.
|
|
100
|
+
4. Validate the configuration with the deployed Collector distribution, then use its supported reload mechanism or restart it. The bundled Compose stack requires a Collector restart.
|
|
101
|
+
5. Expect a restart to clear the exporter's in-memory cache only. Prometheus history remains stored until its normal retention or deletion policy removes it. Traces and logs are unaffected.
|
|
102
|
+
|
|
103
|
+
For the pinned local distribution, validate before restart:
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
docker compose -f observability-stack/docker-compose.yml config
|
|
107
|
+
docker compose -f observability-stack/docker-compose.yml exec -T otel-collector \
|
|
108
|
+
/otelcol-contrib validate --config=/etc/otel/otel-collector.yaml
|
|
109
|
+
```
|
|
110
|
+
|
|
91
111
|
## 3. CI Agent Runbook
|
|
92
112
|
|
|
93
113
|
CI agents are short lived. Use one of:
|
|
@@ -108,13 +128,17 @@ Pi CI job -> sidecar collector -> central collector
|
|
|
108
128
|
|
|
109
129
|
Best for high-volume jobs because Pi hands off telemetry locally.
|
|
110
130
|
|
|
111
|
-
|
|
131
|
+
Preferred shutdown steps:
|
|
112
132
|
|
|
113
133
|
1. Parent Pi passes trace context, `OBSERVME_WORKFLOW_ID`, and `OBSERVME_PARENT_AGENT_ID`/`OBSERVME_ROOT_AGENT_ID` to any child Pi processes it launches.
|
|
114
|
-
2. Pi
|
|
115
|
-
3.
|
|
116
|
-
4.
|
|
117
|
-
5. CI job
|
|
134
|
+
2. Stop Pi and child producers gracefully so ObservMe can deactivate the lease, export the zero lifecycle claim, and flush within its timeout.
|
|
135
|
+
3. Drain the sidecar Collector with a timeout.
|
|
136
|
+
4. Put best-effort process, container, and network cleanup in an `if: always()` GitHub Actions step.
|
|
137
|
+
5. End the CI job.
|
|
138
|
+
|
|
139
|
+
Graceful shutdown and unconditional cleanup reduce latency and resource leakage, but forced cancellation, `SIGKILL`, runner loss, or power loss can skip both. The active count still converges from the last lease; never make a post-job step, Pi shutdown callback, or Collector restart the correctness mechanism. Test cancellation by killing a child producer from a still-running validation job, because a cancelled workflow cannot assert its own convergence.
|
|
140
|
+
|
|
141
|
+
GitHub-hosted runners satisfy the supported clock expectation. A self-hosted runner must keep its wall clock and the Prometheus clock synchronized within 5 seconds using reliable NTP or an equivalent managed time service. Monitor synchronization health and clock steps; a producer clock behind can fail closed early, while one ahead can delay expiry until the bounded future-horizon rule rejects it.
|
|
118
142
|
|
|
119
143
|
## 4. Production Configuration Checklist
|
|
120
144
|
|
|
@@ -133,6 +157,10 @@ Shutdown steps:
|
|
|
133
157
|
- [ ] Collector batch processor enabled.
|
|
134
158
|
- [ ] Loki structured metadata enabled when required by the deployed Loki version.
|
|
135
159
|
- [ ] Prometheus OTLP receiver enabled with `--web.enable-otlp-receiver` if sending directly to Prometheus.
|
|
160
|
+
- [ ] Prometheus scrape-exporter `metric_expiration` is longer than the active-agent lease and is treated only as exporter-wide cleanup.
|
|
161
|
+
- [ ] Lease-aware queries, rather than raw active claims or Collector expiration, determine active-agent liveness.
|
|
162
|
+
- [ ] Producer and Prometheus clocks are synchronized within 5 seconds; self-hosted runner time synchronization is monitored.
|
|
163
|
+
- [ ] GitHub Actions uses graceful shutdown and `if: always()` cleanup where possible without depending on either for active-count correctness.
|
|
136
164
|
- [ ] Backend retention configured.
|
|
137
165
|
- [ ] Dashboards imported.
|
|
138
166
|
- [ ] Alerts enabled.
|
|
@@ -172,6 +200,34 @@ Check:
|
|
|
172
200
|
4. Metric names visible
|
|
173
201
|
5. Resource attributes not unexpectedly dropped
|
|
174
202
|
|
|
203
|
+
### Active Count Is Correct but Raw Claims Remain
|
|
204
|
+
|
|
205
|
+
This is expected after an ungraceful producer exit. Check:
|
|
206
|
+
|
|
207
|
+
1. The dashboard uses the leased active-agent query, not `sum(observme_active_agents)`.
|
|
208
|
+
2. The expired-claims diagnostic shows the abandoned stream.
|
|
209
|
+
3. The configured lease has elapsed and producer/Prometheus clocks are synchronized within 5 seconds.
|
|
210
|
+
4. `metric_expiration` remains longer than the lease. Do not shorten it to force correctness; the exporter will remove the raw gauge, counter, and histogram series together after the cleanup window.
|
|
211
|
+
5. Collector memory/cardinality is within budget. If cleanup timing changes, validate and reload/restart the Collector; stored Prometheus history will remain.
|
|
212
|
+
|
|
213
|
+
No Collector restart is required. Restarting clears the exporter's in-memory cache and can hide the diagnostic raw claim, but it does not improve lease convergence or delete Prometheus history.
|
|
214
|
+
|
|
215
|
+
### Live Producer Is Missing from the Leased Count
|
|
216
|
+
|
|
217
|
+
A missing, expired, malformed, or pathologically future lease fails closed. Diagnose in this order:
|
|
218
|
+
|
|
219
|
+
1. Confirm Prometheus and the Export Health dashboard are reachable. If the backend is unavailable, zero is not a reliable health result.
|
|
220
|
+
2. Compare the canonical leased total, raw positive claims, and expired claims. Query `observme_agent_lease_expires_unixtime_seconds` directly only for diagnosis; do not expose `observme_instance_id` in a public legend or variable.
|
|
221
|
+
3. If the raw claim and lease are both missing, check `metrics.enabled`, `/obs status`, `/obs health`, OTLP reachability, `observme_export_errors_total`, Collector logs, and the metrics pipeline.
|
|
222
|
+
4. If the raw claim exists but the lease is missing, confirm every expected producer runs a lease-capable ObservMe version and that the Collector preserves the same generated `observme_instance_id` label on both metrics. During a mixed-version rollout, missing leases intentionally do not count.
|
|
223
|
+
5. If the lease is expired while the producer runs, verify metric collection/export is still occurring, then inspect Export Health for drops or exporter failures. Confirm `metrics.activeAgentLeaseDurationMillis >= (2 * metrics.exportIntervalMillis) + 5000`.
|
|
224
|
+
6. If the lease is rejected as future, compare producer and Prometheus clocks. Restore reliable synchronization within 5 seconds; do not widen panel-local PromQL or remove the `time() + 305` safety bound.
|
|
225
|
+
7. After export recovers, wait for the next metric collection and Prometheus scrape. The lease renews automatically while the session remains active; neither workload activity nor Collector restart is required.
|
|
226
|
+
|
|
227
|
+
### Expired Claims Are Missing from Diagnostics
|
|
228
|
+
|
|
229
|
+
The live total can still be correct. Check whether `metric_expiration` elapsed, the Collector restarted, or the deployment uses remote write rather than the scrape exporter. Any of those can remove the current raw series while historical Prometheus samples remain. Use backend history and Export Health for incident timing; do not infer that graceful shutdown occurred merely because the expired-claim panel is empty.
|
|
230
|
+
|
|
175
231
|
### Subagents Not Linked to Parent
|
|
176
232
|
|
|
177
233
|
Check:
|
|
@@ -64,6 +64,7 @@ observme:
|
|
|
64
64
|
enabled: true
|
|
65
65
|
exportIntervalMillis: 15000
|
|
66
66
|
exportTimeoutMillis: 3000
|
|
67
|
+
activeAgentLeaseDurationMillis: 60000
|
|
67
68
|
|
|
68
69
|
logs:
|
|
69
70
|
enabled: true
|
|
@@ -146,6 +147,7 @@ OBSERVME_OTLP_TRACES_ENDPOINT
|
|
|
146
147
|
OBSERVME_OTLP_METRICS_ENDPOINT
|
|
147
148
|
OBSERVME_OTLP_LOGS_ENDPOINT
|
|
148
149
|
OBSERVME_OTLP_TOKEN
|
|
150
|
+
OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS
|
|
149
151
|
OBSERVME_WORKFLOW_ID
|
|
150
152
|
OBSERVME_WORKFLOW_MAX_DEPTH_WARNING
|
|
151
153
|
OBSERVME_WORKFLOW_MAX_FANOUT_WARNING
|
|
@@ -184,6 +186,17 @@ OBSERVME_ALLOW_UNSAFE_CAPTURE
|
|
|
184
186
|
OBSERVME_ALLOW_INSECURE_TRANSPORT
|
|
185
187
|
```
|
|
186
188
|
|
|
189
|
+
### 2.1 Active-agent lease configuration
|
|
190
|
+
|
|
191
|
+
| Setting | Default | Supported values | Purpose |
|
|
192
|
+
| --- | --- | --- | --- |
|
|
193
|
+
| `metrics.activeAgentLeaseDurationMillis` | `60000` | Integer `10000`–`300000`, and at least `(2 * metrics.exportIntervalMillis) + 5000` | Validity window renewed on each metric collection while the session is active. |
|
|
194
|
+
| `OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS` | unset | Same as the YAML field | Environment override through the normal trusted `.env` / system environment precedence. |
|
|
195
|
+
|
|
196
|
+
The value controls failure-convergence latency, not a background timer. A clean shutdown deactivates the lease before final flush and reaches zero after normal export/scrape propagation. An ungraceful exit reaches zero within the configured lease plus up to 5 seconds of supported clock skew and one Prometheus scrape/evaluation interval. Missing or invalid leases fail closed. Keep producer and Prometheus clocks synchronized within 5 seconds; GitHub-hosted runners meet this requirement, while self-hosted runners need reliable time synchronization.
|
|
197
|
+
|
|
198
|
+
Do not tune this field by shortening Collector `metric_expiration`. The recommended five-minute exporter cleanup remains longer than the default lease and applies to all gauges, counters, and histograms. Changing Collector configuration may require reload or restart, but restart is not required for active-agent correctness.
|
|
199
|
+
|
|
187
200
|
## 3. Workflow and Agent Lineage Configuration
|
|
188
201
|
|
|
189
202
|
The `workflow:` and `agent:` config blocks control generated workflow identity, generated agent identity, and parent/child propagation. `pi.workflow.id`, `pi.agent.id`, `pi.agent.parent_id`, and related values are high-cardinality identifiers for traces/logs only; they must not be metric labels by default.
|
|
@@ -243,6 +256,9 @@ agent:
|
|
|
243
256
|
writeCorrelationEntry: false
|
|
244
257
|
workflow:
|
|
245
258
|
enabled: true
|
|
259
|
+
metrics:
|
|
260
|
+
exportIntervalMillis: 15000
|
|
261
|
+
activeAgentLeaseDurationMillis: 60000
|
|
246
262
|
```
|
|
247
263
|
|
|
248
264
|
## 6. Development Defaults
|
|
@@ -321,7 +337,8 @@ Reject config when:
|
|
|
321
337
|
- `allowUnsafeCapture=false` and `redactionEnabled=false` while any content capture is true.
|
|
322
338
|
- Production endpoint uses `http://` and `allowInsecureTransport` is not true.
|
|
323
339
|
- `otlp.protocol=http/protobuf` but a signal-specific exporter URL omits the required `/v1/traces`, `/v1/metrics`, or `/v1/logs` path.
|
|
324
|
-
-
|
|
340
|
+
- `metrics.activeAgentLeaseDurationMillis` is fractional, non-numeric, below `10000`, above `300000`, or less than `(2 * metrics.exportIntervalMillis) + 5000`.
|
|
341
|
+
- Metric labels include high-cardinality fields such as workflow IDs, session IDs, agent IDs, parent/child agent IDs, trace IDs, span IDs, entry IDs, spawn IDs, or spawn tool-call IDs. The generated `observme.instance.id` / `service.instance.id` remains a resource identity used by the Collector for the `observme_instance_id` lease join; it must not be configured as an execution-derived label.
|
|
325
342
|
- Project-local config is read while `ctx.isProjectTrusted()` is false.
|
|
326
343
|
- Propagated workflow or agent lineage values are malformed, too long, or contain unsafe characters.
|
|
327
344
|
- Queue sizes exceed configured memory guardrails.
|
|
@@ -55,6 +55,7 @@ Use this checklist when a review, release candidate, or local operator needs cov
|
|
|
55
55
|
| `npm run smoke:pi-runtime` | Yes when the `pi` CLI is installed in the test image | A working local `pi` executable, dependencies installed, and permission to spawn a local child process. | No credentials. It deletes inherited `OBSERVME_*` values and uses a deterministic loopback backend. | Creates a temporary project and extension files under the OS temp directory, then removes them; local loopback server only. | Real Pi RPC extension load, `/obs` discovery/routing, reload/new-session replacement, and credential-free event-shape smoke coverage. |
|
|
56
56
|
| `npm run smoke:packaged` | Usually release/local only | `npm` available and package dependencies resolvable for a temporary install. | May need npm registry/cache access to install package dependencies; no project credentials. | Runs `npm pack`, creates a temporary project under the OS temp directory, installs the tarball with `--ignore-scripts --no-audit --no-fund --package-lock=false`, and removes the temp directory. | Packaged install layout plus declared Pi extension and skill resources in the installed package. |
|
|
57
57
|
| `npm run test:integration:collector` | No | Docker daemon available; Collector image `OBSERVME_COLLECTOR_IMAGE` or `otel/opentelemetry-collector-contrib:0.104.0` pullable or cached. | Docker image pull may require network; no Grafana/model credentials. | Starts a temporary Collector container with loopback-published ports and stops it in test cleanup. | OTLP export to Collector debug pipeline for representative traces, metrics, and logs. |
|
|
58
|
+
| `npm run test:integration:active-agent-lease` | No | Docker daemon available; pinned Collector Contrib and Prometheus images pullable or cached; producer and Prometheus clocks within 5 seconds. | Docker image pull may require network; no project, Grafana, or model credentials. | Creates labeled Collector/Prometheus containers and a dedicated network plus local child producers; bounded `finally` cleanup removes all resources. | Clean shutdown, `SIGTERM`, and cancellation-oriented `SIGKILL`; cached raw claim remains while lease-aware activity reaches zero without Collector restart. |
|
|
58
59
|
| `npm run test:integration:grafana-stack` | No | Docker Compose available; `observability-stack/docker-compose.yml` and test overlay available; required Grafana/Tempo/Loki/Prometheus images pullable or cached. | Docker image pull may require network; no external Grafana credentials. Test-created local Grafana admin secret is synthetic. | Creates isolated Docker Compose project/networks/containers. May create `observability-stack/secrets/grafana_admin_password` when missing and removes the test-created file during cleanup; also creates temporary command-project files under the OS temp directory. If interrupted, run `docker compose` with the printed project name or remove stale `observme-grafana-it-*` resources. | Live local stack ingestion, datasource queries, `/obs` query commands, and dashboard provisioning imports. |
|
|
59
60
|
| `npm run validate:grafana-obs` | No | Running Pi session with ObservMe loaded, reachable Collector/Grafana stack, and recent telemetry. See `docs/validation-flow.md`. | Requires Grafana read credentials from environment variables (`OBSERVME_GRAFANA_TOKEN` or username/password), datasource UIDs, and `OBSERVME_VALIDATION_SESSION_ID`; may access the configured Grafana URL. | Read-only against the configured stack; no file writes expected. Keep terminal output sanitized and omit secrets. | Operator-facing validation that Grafana has ObservMe data and representative `/obs` commands work against the same stack. |
|
|
60
61
|
|
|
@@ -76,6 +77,22 @@ This record was produced for the final-pass task "Run and record bounded live Pi
|
|
|
76
77
|
|
|
77
78
|
Post-run cleanup check: `docker ps --filter name=observme-grafana-it --format '{{.Names}}'` and `docker network ls --filter name=observme-grafana-it --format '{{.Name}}'` returned no resources.
|
|
78
79
|
|
|
80
|
+
## Active-agent lease release validation — 2026-07-14
|
|
81
|
+
|
|
82
|
+
This sanitized record covers the production active-agent lease release candidate on local Node.js v26.0.0, npm 11.12.1, Docker 29.5.2, and Docker Compose 5.1.4. The checkout contains the lease implementation and documentation changes. No project, Grafana, model, or registry credentials were required or printed.
|
|
83
|
+
|
|
84
|
+
| Command | Result | Evidence and notes |
|
|
85
|
+
| --- | --- | --- |
|
|
86
|
+
| `npm run validate` | Pass | Source/test typechecks, ESLint, formatting, script checks, 493 unit/contract tests, package-content checks, packaged-install smoke, handler smoke, Pi lifecycle smoke, and Pi runtime smoke passed. |
|
|
87
|
+
| `npm run test:integration:collector` | Pass | The pinned Collector Contrib 0.104.0 debug pipeline received representative traces, metrics, and logs without default content capture. |
|
|
88
|
+
| `npm run test:integration:active-agent-lease` | Pass | Clean shutdown, `SIGTERM`, and cancellation-oriented `SIGKILL` passed against Collector Contrib 0.104.0 and Prometheus 2.53.1. The raw claim remained cached while lease-aware activity reached zero with the same Collector instance and no restart. |
|
|
89
|
+
| `npm run test:integration:grafana-stack` | Fail/unrelated pre-existing broad-stack blocker | The current run timed out when the Prometheus total-token query returned an empty vector. This token-ingestion/query path is unrelated to active-agent lease arithmetic, PromQL, or abrupt-termination convergence and does not invalidate the focused passing lease evidence. The earlier 2026-07-09 broad-stack run was already blocked later in its Tempo content assertion. |
|
|
90
|
+
| `docker compose -f observability-stack/docker-compose.yml config --quiet` | Pass | Compose interpolation validated without rendering secret values. |
|
|
91
|
+
| Pinned Collector Contrib 0.104.0 `validate --config=/etc/otel/otel-collector.yaml` | Pass | The shipped `observability-stack/config/otel/otel-collector.yaml` parsed successfully with the deployed distribution. |
|
|
92
|
+
| `npm run pack:dry-run` | Pass | The 124-file package contained the lease source/metric convention, dashboards, alerts, examples, target user/operator/reference docs, and packaged `observme-docs` skill. |
|
|
93
|
+
|
|
94
|
+
Post-run cleanup check found no containers or networks labeled for the active-agent lease integration and no `observme-grafana-it-*` containers or networks. Release remains blocked on the unrelated broad Grafana-stack validation unless the release owner accepts the explicitly allowed pre-existing blocker; all lease-specific release criteria passed.
|
|
95
|
+
|
|
79
96
|
## Review-closure evidence categories
|
|
80
97
|
|
|
81
98
|
- **Read-only/check-only** — commands that inspect, type-check, lint, audit, run tests, dry-run packaging, or smoke local deterministic fixtures without publishing or writing tracked files.
|