@senad-d/observme 0.1.3 → 0.1.4

package/docs/reference/04-telemetry-semantic-conventions.md

@@ -503,9 +503,46 @@ observme_handler_duration_ms
 
 ```text
 observme_active_spans
-observme_active_agents
+observme_active_agents                              # compatibility lifecycle claim
+observme_agent_lease_expires_unixtime_seconds      # asynchronous gauge, unit s
 ```
 
+#### 12.4.1 Active-agent lease and clock contract
+
+This section is the single source of truth for active-agent lease constants and semantics. Runtime configuration, metric metadata, PromQL, dashboards, alerts, examples, and tests must reference these values rather than define different local windows.
+
+| Contract item | Required value |
+|---|---|
+| Lease metric | `observme_agent_lease_expires_unixtime_seconds` |
+| OTel instrument and unit | Asynchronous gauge, unit `s` |
+| Observation value | `(wallClockUnixMillis + metrics.activeAgentLeaseDurationMillis) / 1000`; fractional Unix seconds are valid |
+| Configuration | `metrics.activeAgentLeaseDurationMillis`; `OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS` is its environment override |
+| Default lease | `60000` ms (60 seconds) |
+| Inclusive duration bounds | `10000` ms through `300000` ms (10 seconds through 5 minutes) |
+| Export relationship | Lease duration must be at least `(2 * metrics.exportIntervalMillis) + 5000` ms |
+| Supported wall-clock skew | Producer and Prometheus clocks must differ by no more than 5 seconds |
+| Future-timestamp horizon | A lease later than `time() + 305` seconds is pathological and must not count active |
+
+`observme_active_agents` remains the clean lifecycle claim. An agent is operationally active only when its claim is positive and the matching lease is finite, later than Prometheus `time()`, and no later than the future-timestamp horizon. Missing, expired, malformed, non-finite, or pathologically future leases fail closed. Lease renewal occurs once per SDK metric collection while the session is active, including otherwise idle sessions; it does not depend on turns, tool calls, or a background heartbeat timer.
+
+On clean shutdown, deactivate the lease before the final metric force-flush and record the matching `observme_active_agents` decrement. This makes the leased result zero after normal export/scrape propagation without waiting for expiry. If shutdown telemetry cannot run, the last exported lease is allowed to expire. With synchronized clocks, an ungraceful exit converges to zero within the configured lease plus one Prometheus scrape/evaluation interval. Across the full supported positive-skew budget, the measurable worst-case bound is the configured lease plus 5 seconds plus one scrape/evaluation interval; for the defaults and a 15-second scrape interval, that is at most 80 seconds.
+
+The only per-runtime Prometheus join and deduplication key is the Collector-normalized `observme_instance_id` resource label. ObservMe generates its UUID independently for each telemetry session and never derives it from prompts, responses, content, paths, commands, GitHub run IDs, sessions, workflows, or logical agents. `service.instance.id` may carry the same resource value for OTel interoperability, but active-agent PromQL joins on `observme_instance_id` and deduplicates repeated exporter/scrape representations with `max by (observme_instance_id)`. The lease observation itself may carry only existing bounded metric labels; the instance identity comes from resource-to-metric conversion and must not become a dashboard variable or legend.
+
+Lifecycle and failure behavior is fixed as follows:
+
+| Condition | Required leased-active behavior |
+|---|---|
+| Clean exit | Deactivate before final flush; the active claim becomes zero without waiting for lease expiry. |
+| `SIGTERM` | Use the clean path when Pi delivers shutdown; otherwise use the same expiry behavior as an abrupt exit. |
+| Crash, `SIGKILL`, cancelled GitHub job, or runner/power loss | No cleanup is required for correctness; the last lease expires within the documented convergence bound. |
+| Duplicate session start, reload, or resume | Deactivate and flush the replaced session before activating its replacement. If replacement cleanup is interrupted, the old instance lease expires independently. |
+| Startup failure or partial runtime construction | Never leave an activated observable callback; dispose any registered callback. |
+| Collector or OTLP outage | Cached positive claims stop counting when their last delivered leases expire. A still-running producer becomes active again after export recovers and a fresh lease arrives. Backend unavailability itself remains an unknown/no-query state. |
+| Clock skew or clock movement | A producer clock ahead can delay expiry and one behind can fail closed early. Clocks outside the supported skew are unsupported; non-finite observations are omitted and values beyond the future horizon do not count. GitHub-hosted runners satisfy the synchronized-clock expectation; self-hosted runners must use reliable time synchronization. |
+
+A renewable lease is a bounded liveness signal, not proof that the operating-system process is reachable at the exact query instant. `observme_active_agents` remains emitted with its existing name and clean-lifecycle behavior for compatibility, so the change is additive under the semantic-convention versioning policy and does not require a major convention-version increment. Consumers that need crash-safe live counts must migrate from raw active-claim sums to the leased definition.
+
 ### 12.5 Lifecycle recording points
 
 - Record `observme_tool_result_size_chars` once from the finalized `tool_execution_end` result, using the same bounded tool labels as tool-call metrics.

package/docs/reference/05-otel-pipeline-and-collector.md

@@ -47,6 +47,7 @@ metrics:
   enabled: true
   exportIntervalMillis: 15000
   exportTimeoutMillis: 3000
+  activeAgentLeaseDurationMillis: 60000
 
 logs:
   enabled: true
@@ -58,6 +59,8 @@ logs:
 
 The OpenTelemetry JS OTLP HTTP exporters default to signal-specific paths such as `http://localhost:4318/v1/traces`, `http://localhost:4318/v1/metrics`, and `/v1/logs`. Treat `otlp.endpoint` as a base endpoint in ObservMe config, but pass signal-specific URLs to SDK exporters that require an explicit `url`.
 
+The default active-agent lease is 60 seconds and renews on each 15-second metric collection. Supported lease values are 10 seconds through 5 minutes and must be at least twice the export interval plus 5 seconds. Keep the producer and Prometheus clocks within 5 seconds; otherwise a lease may fail closed early or remain valid longer than intended. The exact contract is frozen in [`04-telemetry-semantic-conventions.md` §12.4.1](04-telemetry-semantic-conventions.md#1241-active-agent-lease-and-clock-contract).
+
 ## 4. Trace Context and Workflow/Agent-Lineage Propagation
 
 When ObservMe launches or wraps a subagent process, the parent runtime should propagate both standard trace context and ObservMe-specific lineage:
@@ -118,10 +121,10 @@ service:
 
 ## 6. Production Collector for Grafana Stack
 
-This configuration routes traces to Tempo, logs to Loki, and metrics to a Prometheus-compatible remote write endpoint such as Mimir. Use a Collector distribution that contains every configured component. The minimal debug example works with the core Collector, while `prometheusremotewrite`, `probabilistic_sampler`, and `tail_sampling` are commonly deployed from the Collector Contrib distribution or a vendor distribution such as Grafana Alloy.
+This configuration routes traces to Tempo, logs to Loki, and metrics to a Prometheus scrape endpoint. Use a Collector distribution that contains every configured component. The minimal debug example works with the core Collector, while `prometheus`, `prometheusremotewrite`, `probabilistic_sampler`, and `tail_sampling` are commonly deployed from the Collector Contrib distribution or a vendor distribution such as Grafana Alloy.
 
 ```yaml
-# Production-oriented Collector reference for Tempo, Loki, and Mimir.
+# Production-oriented Collector reference for Tempo, Loki, and Prometheus.
 # Verify that your Collector distribution contains every configured component,
 # replace backend endpoints/security for your deployment, and never add raw
 # content or high-cardinality execution identifiers to metric labels.
@@ -220,14 +223,14 @@ exporters:
     retry_on_failure:
       enabled: true
 
-  prometheusremotewrite/mimir:
-    endpoint: http://mimir:9009/api/v1/push
+  prometheus:
+    endpoint: 0.0.0.0:8889
+    # Exporter-wide stale-series/cardinality cleanup for gauges, counters, and
+    # histograms. Five minutes exceeds ObservMe's default one-minute active-agent
+    # lease; lease-aware PromQL, not expiration, determines liveness.
+    metric_expiration: 5m
     resource_to_telemetry_conversion:
       enabled: true
-    sending_queue:
-      enabled: true
-    retry_on_failure:
-      enabled: true
 
   debug:
     verbosity: basic
@@ -243,7 +246,7 @@ service:
     metrics:
       receivers: [otlp]
       processors: [memory_limiter, resource/observme, resource/drop_high_cardinality_metric_attrs, batch]
-      exporters: [prometheusremotewrite/mimir]
+      exporters: [prometheus]
 
     logs:
       receivers: [otlp]
@@ -251,9 +254,19 @@ service:
       exporters: [otlphttp/loki]
 ```
 
+### 6.1 Prometheus exporter expiration policy
+
+The shipped local and production scrape-exporter examples set `metric_expiration: 5m`. This exporter-level setting applies uniformly to every Prometheus-exported gauge, counter, and histogram; it cannot be scoped only to active-agent metrics. Its only purpose is stale-series and cardinality cleanup: it bounds the Collector memory used by abandoned metric streams and eventually lets Prometheus mark an expired series stale after the exporter stops exposing it. Lease-aware PromQL remains the active-agent correctness mechanism, even while the Collector still exposes a cached positive `observme_active_agents` value.
+
+Five minutes comfortably exceeds the default 60-second active-agent lease and 15-second SDK metric export interval. If either interval is customized, keep `metric_expiration` longer than the lease and allow enough additional time for expected export delays and scrape gaps. A shorter value cleans abandoned streams sooner but can evict gauges, counters, and histograms during a temporary producer outage; a longer value consumes Collector memory and preserves abandoned cardinality for longer. It must never be shortened merely to make active-agent totals converge.
+
+Apply a Collector configuration change through the deployment's supported reload mechanism or restart the Collector. A restart clears the Prometheus exporter's in-memory series cache, but neither reload nor restart deletes samples already stored in Prometheus; historical samples remain subject to Prometheus retention and deletion policy. Restart is never required merely to make an expired active-agent lease stop counting. `metric_expiration` belongs only to the metrics scrape exporter, so traces and logs are unaffected. Remote-write deployments do not use this scrape-exporter cache setting and must apply their backend's series-retention policy, while continuing to use the same lease-aware active-agent queries.
+
+When migrating, deploy a version that emits `observme_agent_lease_expires_unixtime_seconds`, verify the Collector preserves the generated `observme_instance_id`, then replace every current-activity raw sum in dashboards, recording rules, and alerts with the canonical queries in [`09-dashboards-alerts-slos.md` §1.3.1](09-dashboards-alerts-slos.md#131-canonical-active-agent-promql). Keep raw and expired claims only as clearly labeled diagnostics. Missing leases intentionally fail closed, so do not switch a mixed-version fleet until the lease metric is visible for the producers that should count.
+
 The Collector attribute processor is defense in depth for log attributes only. It must not be the only redaction layer, and it does not sanitize arbitrary log bodies; ObservMe must redact or drop sensitive content before export. The traces pipeline intentionally keeps `pi.llm.prompt.redacted`, `pi.llm.response.redacted`, and `pi.llm.thinking.redacted` so Tempo can display redacted content after explicit capture is enabled. Old telemetry that an earlier Collector dropped cannot be recovered; generate new LLM events after updating the Collector.
 
-Keep high-cardinality lineage attributes on traces/logs, but remove them from the metrics pipeline unless the organization explicitly accepts the cardinality. The bundled local Loki pipeline promotes `pi.agent.id` and `pi.agent.run.id` as log labels so the LLM Conversations dashboard can filter captured chat content by agent and run. Prometheus resource-to-telemetry conversion is safe only after the metrics pipeline drops `pi.workflow.id`, `pi.agent.id`, `pi.session.id`, trace IDs, and spawn IDs; the remaining generated `service.instance.id`/`observme.instance.id` labels keep concurrent ObservMe metric streams distinct so session counters can be summed accurately.
+Keep high-cardinality lineage attributes on traces/logs, but remove them from the metrics pipeline unless the organization explicitly accepts the cardinality. The bundled local Loki pipeline promotes `pi.agent.id` and `pi.agent.run.id` as log labels so the LLM Conversations dashboard can filter captured chat content by agent and run. Prometheus resource-to-telemetry conversion is safe only after the metrics pipeline drops `pi.workflow.id`, `pi.agent.id`, `pi.session.id`, trace IDs, and spawn IDs. Preserve the generated `service.instance.id`/`observme.instance.id` resource identity in the metrics pipeline so lease-aware queries can join and deduplicate concurrent ObservMe metric streams without exposing workflow, session, logical-agent, trace, span, or job-run identifiers as labels.
 
 ## 7. Direct-to-Backend Development Mode
 

package/docs/reference/09-dashboards-alerts-slos.md

@@ -129,9 +129,126 @@ Zero-state interpretation checklist:
 - Empty failure-only Loki tables mean no matching failures in the selected range, not that the log pipeline is broken.
 - `or vector(0)` stats must say whether zero means healthy idle/no activity or a true zero failure ratio.
 - Optional redacted content panels can be empty when capture is disabled or when the Collector dropped content attributes by policy.
-- Active-session estimates are approximate across process restarts/counter resets; `observme_active_agents` is the authoritative active agent gauge.
+- Active-session estimates are approximate across process restarts and counter resets. Raw `observme_active_agents` values are lifecycle claims, not authoritative liveness after an ungraceful exit.
 - High-cardinality execution identifiers are allowed as Loki/Tempo filter variables only and must never be introduced into Prometheus queries or panel links from aggregate panels.
 
+### 1.3 Active-agent leased-state contract
+
+The authoritative metric, duration, export-interval relationship, clock-skew allowance, future-timestamp horizon, convergence target, and failure-mode behavior are defined once in [`04-telemetry-semantic-conventions.md` §12.4.1](04-telemetry-semantic-conventions.md#1241-active-agent-lease-and-clock-contract). Dashboards, alerts, examples, and tests must use that contract without introducing panel-local lease or skew windows.
+
+Operational active-agent views combine a positive `observme_active_agents` lifecycle claim with a currently valid `observme_agent_lease_expires_unixtime_seconds` value on the generated `observme_instance_id` resource label. The instance label is an internal join/deduplication key only: it must not be exposed as a dashboard variable, group, legend, URL value, or topology identity. Duplicate scrape/exporter representations of one instance count once.
+
+Interpret active-agent states as follows:
+
+- positive claim plus a valid current lease: active;
+- clean-shutdown claim of zero: inactive after normal export/scrape propagation, without waiting for lease expiry;
+- positive claim plus an expired lease: inactive stale claim, available only as a diagnostic;
+- missing, malformed, non-finite, or pathologically future lease: inactive (fail closed);
+- no matching series: zero-safe healthy idle/no activity when the telemetry backend itself is reachable.
+
+A crash, `SIGKILL`, cancelled job, or lost runner therefore converges to zero within the semantic-convention bound even if the Collector continues to expose the cached raw claim. A Collector/export outage can intentionally undercount after the last delivered lease expires; use Export Health and backend reachability to distinguish that state from process termination. Clean lifecycle handling and GitHub Actions `if: always()` cleanup remain useful latency optimizations, but neither is the correctness mechanism.
+
+### 1.3.1 Canonical active-agent PromQL
+
+Use the following total everywhere current live-agent state is required:
+
+```promql
+sum(
+  max by (observme_instance_id) (
+    (observme_active_agents > 0)
+    and on (observme_instance_id)
+    (
+      (observme_agent_lease_expires_unixtime_seconds > time())
+      and
+      (observme_agent_lease_expires_unixtime_seconds <= time() + 305)
+    )
+  )
+) or vector(0)
+```
+
+Both comparisons are required. The lower bound expires stale claims; the upper bound rejects leases beyond the frozen 305-second future horizon. `and on (observme_instance_id)` fails closed when the lease is absent or non-comparable, and `max by (observme_instance_id)` removes duplicate Collector/exporter/scrape replicas before the outer sum. The final `or vector(0)` makes no-series results render as zero; when Prometheus and Export Health are reachable, zero can mean healthy idle, clean shutdown, an expired claim, or a lease that failed closed.
+
+For a bounded emitted dimension, retain that dimension in both aggregations. Grouped queries use `or on() vector(0)` so the unlabeled zero appears only when no labeled breakdown exists. Canonical role and environment breakdowns are:
+
+```promql
+sum by (agent_role) (
+  max by (observme_instance_id, agent_role) (
+    (observme_active_agents > 0)
+    and on (observme_instance_id)
+    (
+      (observme_agent_lease_expires_unixtime_seconds > time())
+      and
+      (observme_agent_lease_expires_unixtime_seconds <= time() + 305)
+    )
+  )
+) or on() vector(0)
+```
+
+```promql
+sum by (environment) (
+  max by (observme_instance_id, environment) (
+    (observme_active_agents > 0)
+    and on (observme_instance_id)
+    (
+      (observme_agent_lease_expires_unixtime_seconds > time())
+      and
+      (observme_agent_lease_expires_unixtime_seconds <= time() + 305)
+    )
+  )
+) or on() vector(0)
+```
+
+The shipped active claim and lease currently emit `agent_role` and `environment`; they do not emit the metric label `subagent_depth`. Therefore current-active depth panels must not group by `subagent_depth`. For a compatible pipeline that actually emits the same bounded `subagent_depth` on the active claim, this is the supported depth shape:
+
+```promql
+sum by (subagent_depth) (
+  max by (observme_instance_id, subagent_depth) (
+    (observme_active_agents > 0)
+    and on (observme_instance_id)
+    (
+      (observme_agent_lease_expires_unixtime_seconds > time())
+      and
+      (observme_agent_lease_expires_unixtime_seconds <= time() + 305)
+    )
+  )
+) or on() vector(0)
+```
+
+Do not substitute resource-derived or high-cardinality execution labels for these dimensions. Use the bounded spawn/tree metrics for depth views in the shipped dashboards until `subagent_depth` is emitted on the active signals themselves.
+
+Raw positive lifecycle claims are diagnostic only:
+
+```promql
+sum(max by (observme_instance_id) (observme_active_agents > 0)) or vector(0)
+```
+
+Expired positive claims are diagnostic and never contribute to the live total:
+
+```promql
+sum(
+  max by (observme_instance_id) (
+    (observme_active_agents > 0)
+    and on (observme_instance_id)
+    (observme_agent_lease_expires_unixtime_seconds <= time())
+  )
+) or vector(0)
+```
+
+Compare raw claims, expired claims, the canonical live total, and Export Health when diagnosing a producer crash or export gap. A raw-minus-live difference can also include a missing, malformed, or pathologically future lease, so it must not be labeled strictly as “expired.” Never expose `observme_instance_id` as a variable or legend in these diagnostics.
+
+### 1.3.2 Migration from raw active-agent queries
+
+Use this rollout order for custom dashboards, recording rules, alerts, and API consumers:
+
+1. Deploy ObservMe producers that emit both `observme_active_agents` and `observme_agent_lease_expires_unixtime_seconds` with the same generated `observme_instance_id` resource label.
+2. Confirm the lease metric is present and clocks are synchronized before changing consumers. A missing lease fails closed, so a mixed-version producer without the new metric will not appear in the leased total.
+3. Replace raw `sum(observme_active_agents)`, raw role sums, and any equivalent positive-claim aggregation with the canonical total or bounded-dimension form above. Preserve the future-horizon check, instance join, deduplication, and zero-safe fallback together.
+4. Keep raw and expired-claim queries only in panels or runbooks explicitly labeled as diagnostics. Do not use them for paging, capacity decisions, or current topology.
+5. Keep Collector `metric_expiration` longer than the lease. Do not shorten expiration or schedule Collector restarts to correct liveness; an expired lease stops counting while the cached raw claim remains available for diagnosis.
+6. During rollback, old raw consumers continue to see the compatibility lifecycle claim, but they regain the known ghost-agent behavior after ungraceful exits. Treat rollback as a temporary loss of crash-safe liveness semantics.
+
+Validate the migrated consumer with live+valid, clean-shutdown, expired, missing, pathologically future, duplicate-replica, and no-series cases before production use.
+
 ## 2. Overview Dashboard
 
 Purpose: act as the operator landing page. Users should be able to answer whether ObservMe is healthy, workload is normal, agents/subagents are healthy, spend or latency is high, and which focused dashboard to open next without scanning every raw metric series.
@@ -584,10 +701,22 @@ Severity: warning.
 ### Active Agents Stuck High
 
 ```promql
-sum(observme_active_agents) > 100
+(
+  sum(
+    max by (observme_instance_id) (
+      (observme_active_agents > 0)
+      and on (observme_instance_id)
+      (
+        (observme_agent_lease_expires_unixtime_seconds > time())
+        and
+        (observme_agent_lease_expires_unixtime_seconds <= time() + 305)
+      )
+    )
+  ) or vector(0)
+) > 100
 ```
 
-Severity depends on normal fleet size; tune per deployment.
+Severity depends on normal fleet size; tune per deployment. This alert uses leased live agents, not cached raw claims.
 
 ## 12. SLOs
 

package/docs/reference/10-testing-release-operations.md

@@ -14,7 +14,9 @@ ObservMe must be tested at five levels:
 
 Required areas:
 
-- Config loading and precedence
+- Config loading, precedence, and active-agent lease/export-interval bounds
+- Active-agent lease renewal, deactivation, disposal, clock movement, and clean/abrupt lifecycle behavior
+- Canonical lease-aware PromQL, replica deduplication, and missing/expired/future lease cases
 - Redaction patterns
 - Hashing stability
 - Truncation behavior
@@ -123,6 +125,29 @@ Tests:
 - Prometheus metric query for token totals
 - Grafana dashboard import validates
 
+### 6.1 Active-agent lease integration
+
+Run the focused cached-exporter test separately from the broad Grafana stack:
+
+```bash
+npm run test:integration:active-agent-lease
+```
+
+The test starts pinned Collector and Prometheus containers, then runs real Node telemetry producers with a test-safe 2-second export interval, 10-second lease, and 1-second Prometheus scrape interval. It validates clean shutdown and `SIGTERM` first, then kills a producer with `SIGKILL` and requires all of the following:
+
+- the raw `observme_active_agents` claim remains positive across fresh Prometheus scrapes;
+- the canonical lease-aware count reaches zero without a shutdown event;
+- convergence stays within the documented 10-second lease, 5-second clock-skew allowance, and one 1-second scrape interval;
+- the Collector container identity and zero restart count remain unchanged.
+
+The test uses no credentials or content capture. Its parent process owns bounded waits and unconditional cleanup for child processes, containers, anonymous storage, and the dedicated Docker network after success, assertion failure, startup failure, or timeout. A one-minute test-only `metric_expiration` deliberately remains much longer than the lease so exporter caching reproduces the old raw-query failure; production cleanup guidance remains five minutes for the default one-minute lease.
+
+### 6.2 GitHub Actions cancellation validation
+
+The `lease-cancellation` job in `.github/workflows/ci.yml` runs lease unit/contract tests and the focused integration on `ubuntu-latest` with a bounded timeout. It simulates cancellation by sending `SIGKILL` to a producer so the workflow can assert convergence; cancelling the validation job itself could not produce evidence about its own result. An `if: always()` step removes labeled Docker resources when GitHub still schedules cleanup, but the test must reach zero through lease expiry before that step runs.
+
+Production workflows should likewise prefer graceful Pi and sidecar shutdown and use `if: always()` cleanup where available. Force cancellation, runner loss, or power loss can bypass both, so neither cleanup nor a shutdown callback may be the active-count correctness mechanism. GitHub-hosted runners meet the synchronized-clock expectation; self-hosted CI must keep the producer and Prometheus clocks within the supported 5-second skew.
+
 ## 7. Failure Tests
 
 ### Collector Down
@@ -228,15 +253,16 @@ Reject or sanitize dynamically generated labels.
 
 ## 10. Release Process
 
-1. Update version.
-2. Run unit tests.
-3. Run integration tests.
-4. Run redaction corpus tests.
-5. Run dashboard JSON validation.
-6. Run Collector config validation.
-7. Generate changelog.
-8. Publish package/artifact.
-9. Tag release.
+1. Update the version and `CHANGELOG.md`.
+2. Run `npm run validate` for source/test typechecks, ESLint, formatting, script checks, unit/contract/dashboard/alert tests, package-content checks, packaged-install smoke, handler smoke, Pi lifecycle smoke, and Pi runtime smoke.
+3. Run `npm run test:integration:collector` against the pinned Collector distribution.
+4. Run `npm run test:integration:active-agent-lease` and require clean shutdown, `SIGTERM`, and `SIGKILL` convergence without Collector restart while the raw claim remains cached.
+5. Run `npm run test:integration:grafana-stack` for live backends and dashboard provisioning. Record any unrelated pre-existing blocker precisely; it must not replace focused lease evidence.
+6. Validate Compose interpolation and the Collector config with the pinned distribution when Collector/example configuration changed.
+7. Run `npm run pack:dry-run` and confirm the lease metric source, dashboards, alerts, examples, user/operator/reference docs, and packaged `observme-docs` skill are present. Do not publish during this check.
+8. Record sanitized date, environment versions, pass/fail status, cleanup result, and any blocker in `docs/compatibility-matrix.md` and `docs/review-validation.md`; never record credentials, raw content, private paths, commands from interactive Pi, or full high-cardinality identifiers.
+9. Publish the package/artifact only after lease-related failures are resolved.
+10. Tag the release.
 
 ## 11. Compatibility Matrix
 
@@ -266,6 +292,7 @@ observme_export_errors_total
 observme_redaction_failures_total
 observme_active_spans
 observme_active_agents
+observme_agent_lease_expires_unixtime_seconds
 observme_workflows_started_total
 observme_workflows_completed_total
 observme_workflow_errors_total
@@ -313,7 +340,10 @@ When troubleshooting:
 3. Run `/obs agents` when troubleshooting workflow, parent/subagent lineage, fan-out, depth, or orphan agents.
 4. Check Collector health endpoint.
 5. Check Collector logs.
-6. Query `observme_export_errors_total`.
-7. Query Loki for `event_name="export.failed"` (or equivalent normalized structured metadata).
-8. Temporarily enable debug exporter in Collector.
-9. Never enable raw content capture in production without approval.
+6. For active-agent incidents, compare the canonical leased count, raw positive claims, expired claims, and direct presence of `observme_agent_lease_expires_unixtime_seconds`; then verify producer/Prometheus clock skew, `observme_instance_id` preservation, and the configured lease/export relationship.
+7. Query `observme_export_errors_total` and inspect Export Health before deciding that a missing or expired lease means the producer stopped.
+8. Query Loki for `event_name="export.failed"` (or equivalent normalized structured metadata).
+9. Temporarily enable the debug exporter in the Collector.
+10. Never enable raw content capture in production without approval.
+
+See [`11-deployment-runbooks.md` §5](11-deployment-runbooks.md#5-common-incidents) for expired-claim and missing-lease decision trees.

package/docs/reference/11-deployment-runbooks.md

@@ -88,6 +88,26 @@ Recommended ports:
 9090 prometheus
 ```
 
+### Prometheus exporter cleanup policy
+
+The bundled Collector exposes all metrics through one Prometheus exporter with `metric_expiration: 5m`. Treat that value only as stale-series/cardinality cleanup, not as active-agent liveness. It applies to every exported gauge, counter, and histogram. The leased active-agent queries determine liveness and converge after the 60-second default lease even while a raw positive claim remains cached.
+
+When changing `metrics.activeAgentLeaseDurationMillis`, `metrics.exportIntervalMillis`, scrape timing, or expected outage tolerance:
+
+1. Keep `metric_expiration` longer than the active-agent lease; five minutes is the recommendation for the default 60-second lease and 15-second export interval.
+2. Leave enough margin for ordinary export delays and scrape gaps. Shorter cleanup lowers Collector memory/cardinality but can remove every metric type during a temporary producer outage; longer cleanup retains abandoned series longer.
+3. Preserve generated `observme.instance.id`/`service.instance.id` resource identity for lease joins, while continuing to drop workflow, session, logical-agent, trace, span, spawn, and job-run identifiers from metric labels.
+4. Validate the configuration with the deployed Collector distribution, then use its supported reload mechanism or restart it. The bundled Compose stack requires a Collector restart.
+5. Expect a restart to clear the exporter's in-memory cache only. Prometheus history remains stored until its normal retention or deletion policy removes it. Traces and logs are unaffected.
+
+For the pinned local distribution, validate before restart:
+
+```bash
+docker compose -f observability-stack/docker-compose.yml config
+docker compose -f observability-stack/docker-compose.yml exec -T otel-collector \
+  /otelcol-contrib validate --config=/etc/otel/otel-collector.yaml
+```
+
 ## 3. CI Agent Runbook
 
 CI agents are short lived. Use one of:
@@ -108,13 +128,17 @@ Pi CI job -> sidecar collector -> central collector
 
 Best for high-volume jobs because Pi hands off telemetry locally.
 
-Shutdown steps:
+Preferred shutdown steps:
 
 1. Parent Pi passes trace context, `OBSERVME_WORKFLOW_ID`, and `OBSERVME_PARENT_AGENT_ID`/`OBSERVME_ROOT_AGENT_ID` to any child Pi processes it launches.
-2. Pi exits.
-3. ObservMe flushes with timeout.
-4. Sidecar collector drains with timeout.
-5. CI job ends.
+2. Stop Pi and child producers gracefully so ObservMe can deactivate the lease, export the zero lifecycle claim, and flush within its timeout.
+3. Drain the sidecar Collector with a timeout.
+4. Put best-effort process, container, and network cleanup in an `if: always()` GitHub Actions step.
+5. End the CI job.
+
+Graceful shutdown and unconditional cleanup reduce latency and resource leakage, but forced cancellation, `SIGKILL`, runner loss, or power loss can skip both. The active count still converges from the last lease; never make a post-job step, Pi shutdown callback, or Collector restart the correctness mechanism. Test cancellation by killing a child producer from a still-running validation job, because a cancelled workflow cannot assert its own convergence.
+
+GitHub-hosted runners satisfy the supported clock expectation. A self-hosted runner must keep its wall clock and the Prometheus clock synchronized within 5 seconds using reliable NTP or an equivalent managed time service. Monitor synchronization health and clock steps; a producer clock behind can fail closed early, while one ahead can delay expiry until the bounded future-horizon rule rejects it.
 
 ## 4. Production Configuration Checklist
 
@@ -133,6 +157,10 @@ Shutdown steps:
 - [ ] Collector batch processor enabled.
 - [ ] Loki structured metadata enabled when required by the deployed Loki version.
 - [ ] Prometheus OTLP receiver enabled with `--web.enable-otlp-receiver` if sending directly to Prometheus.
+- [ ] Prometheus scrape-exporter `metric_expiration` is longer than the active-agent lease and is treated only as exporter-wide cleanup.
+- [ ] Lease-aware queries, rather than raw active claims or Collector expiration, determine active-agent liveness.
+- [ ] Producer and Prometheus clocks are synchronized within 5 seconds; self-hosted runner time synchronization is monitored.
+- [ ] GitHub Actions uses graceful shutdown and `if: always()` cleanup where possible without depending on either for active-count correctness.
 - [ ] Backend retention configured.
 - [ ] Dashboards imported.
 - [ ] Alerts enabled.
@@ -172,6 +200,34 @@ Check:
 4. Metric names visible
 5. Resource attributes not unexpectedly dropped
 
+### Active Count Is Correct but Raw Claims Remain
+
+This is expected after an ungraceful producer exit. Check:
+
+1. The dashboard uses the leased active-agent query, not `sum(observme_active_agents)`.
+2. The expired-claims diagnostic shows the abandoned stream.
+3. The configured lease has elapsed and producer/Prometheus clocks are synchronized within 5 seconds.
+4. `metric_expiration` remains longer than the lease. Do not shorten it to force correctness; the exporter will remove the raw gauge, counter, and histogram series together after the cleanup window.
+5. Collector memory/cardinality is within budget. If cleanup timing changes, validate and reload/restart the Collector; stored Prometheus history will remain.
+
+No Collector restart is required. Restarting clears the exporter's in-memory cache and can hide the diagnostic raw claim, but it does not improve lease convergence or delete Prometheus history.
+
+### Live Producer Is Missing from the Leased Count
+
+A missing, expired, malformed, or pathologically future lease fails closed. Diagnose in this order:
+
+1. Confirm Prometheus and the Export Health dashboard are reachable. If the backend is unavailable, zero is not a reliable health result.
+2. Compare the canonical leased total, raw positive claims, and expired claims. Query `observme_agent_lease_expires_unixtime_seconds` directly only for diagnosis; do not expose `observme_instance_id` in a public legend or variable.
+3. If the raw claim and lease are both missing, check `metrics.enabled`, `/obs status`, `/obs health`, OTLP reachability, `observme_export_errors_total`, Collector logs, and the metrics pipeline.
+4. If the raw claim exists but the lease is missing, confirm every expected producer runs a lease-capable ObservMe version and that the Collector preserves the same generated `observme_instance_id` label on both metrics. During a mixed-version rollout, missing leases intentionally do not count.
+5. If the lease is expired while the producer runs, verify metric collection/export is still occurring, then inspect Export Health for drops or exporter failures. Confirm `metrics.activeAgentLeaseDurationMillis >= (2 * metrics.exportIntervalMillis) + 5000`.
+6. If the lease is rejected as future, compare producer and Prometheus clocks. Restore reliable synchronization within 5 seconds; do not widen panel-local PromQL or remove the `time() + 305` safety bound.
+7. After export recovers, wait for the next metric collection and Prometheus scrape. The lease renews automatically while the session remains active; neither workload activity nor Collector restart is required.
+
+### Expired Claims Are Missing from Diagnostics
+
+The live total can still be correct. Check whether `metric_expiration` elapsed, the Collector restarted, or the deployment uses remote write rather than the scrape exporter. Any of those can remove the current raw series while historical Prometheus samples remain. Use backend history and Export Health for incident timing; do not infer that graceful shutdown occurred merely because the expired-claim panel is empty.
+
 ### Subagents Not Linked to Parent
 
 Check:

package/docs/reference/12-configuration-reference.md

@@ -64,6 +64,7 @@ observme:
     enabled: true
     exportIntervalMillis: 15000
     exportTimeoutMillis: 3000
+    activeAgentLeaseDurationMillis: 60000
 
   logs:
     enabled: true
@@ -146,6 +147,7 @@ OBSERVME_OTLP_TRACES_ENDPOINT
 OBSERVME_OTLP_METRICS_ENDPOINT
 OBSERVME_OTLP_LOGS_ENDPOINT
 OBSERVME_OTLP_TOKEN
+OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS
 OBSERVME_WORKFLOW_ID
 OBSERVME_WORKFLOW_MAX_DEPTH_WARNING
 OBSERVME_WORKFLOW_MAX_FANOUT_WARNING
@@ -184,6 +186,17 @@ OBSERVME_ALLOW_UNSAFE_CAPTURE
 OBSERVME_ALLOW_INSECURE_TRANSPORT
 ```
 
+### 2.1 Active-agent lease configuration
+
+| Setting | Default | Supported values | Purpose |
+| --- | --- | --- | --- |
+| `metrics.activeAgentLeaseDurationMillis` | `60000` | Integer `10000`–`300000`, and at least `(2 * metrics.exportIntervalMillis) + 5000` | Validity window renewed on each metric collection while the session is active. |
+| `OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS` | unset | Same as the YAML field | Environment override through the normal trusted `.env` / system environment precedence. |
+
+The value controls failure-convergence latency, not a background timer. A clean shutdown deactivates the lease before final flush and reaches zero after normal export/scrape propagation. An ungraceful exit reaches zero within the configured lease plus up to 5 seconds of supported clock skew and one Prometheus scrape/evaluation interval. Missing or invalid leases fail closed. Keep producer and Prometheus clocks synchronized within 5 seconds; GitHub-hosted runners meet this requirement, while self-hosted runners need reliable time synchronization.
+
+Do not tune this field by shortening Collector `metric_expiration`. The recommended five-minute exporter cleanup remains longer than the default lease and applies to all gauges, counters, and histograms. Changing Collector configuration may require reload or restart, but restart is not required for active-agent correctness.
+
 ## 3. Workflow and Agent Lineage Configuration
 
 The `workflow:` and `agent:` config blocks control generated workflow identity, generated agent identity, and parent/child propagation. `pi.workflow.id`, `pi.agent.id`, `pi.agent.parent_id`, and related values are high-cardinality identifiers for traces/logs only; they must not be metric labels by default.
@@ -243,6 +256,9 @@ agent:
   writeCorrelationEntry: false
 workflow:
   enabled: true
+metrics:
+  exportIntervalMillis: 15000
+  activeAgentLeaseDurationMillis: 60000
 ```
 
 ## 6. Development Defaults
@@ -321,7 +337,8 @@ Reject config when:
 - `allowUnsafeCapture=false` and `redactionEnabled=false` while any content capture is true.
 - Production endpoint uses `http://` and `allowInsecureTransport` is not true.
 - `otlp.protocol=http/protobuf` but a signal-specific exporter URL omits the required `/v1/traces`, `/v1/metrics`, or `/v1/logs` path.
-- Metric labels include high-cardinality fields such as workflow IDs, session IDs, agent IDs, parent/child agent IDs, trace IDs, span IDs, entry IDs, spawn IDs, or spawn tool-call IDs.
+- `metrics.activeAgentLeaseDurationMillis` is fractional, non-numeric, below `10000`, above `300000`, or less than `(2 * metrics.exportIntervalMillis) + 5000`.
+- Metric labels include high-cardinality fields such as workflow IDs, session IDs, agent IDs, parent/child agent IDs, trace IDs, span IDs, entry IDs, spawn IDs, or spawn tool-call IDs. The generated `observme.instance.id` / `service.instance.id` remains a resource identity used by the Collector for the `observme_instance_id` lease join; it must not be configured as an execution-derived label.
 - Project-local config is read while `ctx.isProjectTrusted()` is false.
 - Propagated workflow or agent lineage values are malformed, too long, or contain unsafe characters.
 - Queue sizes exceed configured memory guardrails.

package/docs/review-validation.md

@@ -55,6 +55,7 @@ Use this checklist when a review, release candidate, or local operator needs cov
 | `npm run smoke:pi-runtime` | Yes when the `pi` CLI is installed in the test image | A working local `pi` executable, dependencies installed, and permission to spawn a local child process. | No credentials. It deletes inherited `OBSERVME_*` values and uses a deterministic loopback backend. | Creates a temporary project and extension files under the OS temp directory, then removes them; local loopback server only. | Real Pi RPC extension load, `/obs` discovery/routing, reload/new-session replacement, and credential-free event-shape smoke coverage. |
 | `npm run smoke:packaged` | Usually release/local only | `npm` available and package dependencies resolvable for a temporary install. | May need npm registry/cache access to install package dependencies; no project credentials. | Runs `npm pack`, creates a temporary project under the OS temp directory, installs the tarball with `--ignore-scripts --no-audit --no-fund --package-lock=false`, and removes the temp directory. | Packaged install layout plus declared Pi extension and skill resources in the installed package. |
 | `npm run test:integration:collector` | No | Docker daemon available; Collector image `OBSERVME_COLLECTOR_IMAGE` or `otel/opentelemetry-collector-contrib:0.104.0` pullable or cached. | Docker image pull may require network; no Grafana/model credentials. | Starts a temporary Collector container with loopback-published ports and stops it in test cleanup. | OTLP export to Collector debug pipeline for representative traces, metrics, and logs. |
+| `npm run test:integration:active-agent-lease` | No | Docker daemon available; pinned Collector Contrib and Prometheus images pullable or cached; producer and Prometheus clocks within 5 seconds. | Docker image pull may require network; no project, Grafana, or model credentials. | Creates labeled Collector/Prometheus containers and a dedicated network plus local child producers; bounded `finally` cleanup removes all resources. | Clean shutdown, `SIGTERM`, and cancellation-oriented `SIGKILL`; cached raw claim remains while lease-aware activity reaches zero without Collector restart. |
 | `npm run test:integration:grafana-stack` | No | Docker Compose available; `observability-stack/docker-compose.yml` and test overlay available; required Grafana/Tempo/Loki/Prometheus images pullable or cached. | Docker image pull may require network; no external Grafana credentials. Test-created local Grafana admin secret is synthetic. | Creates isolated Docker Compose project/networks/containers. May create `observability-stack/secrets/grafana_admin_password` when missing and removes the test-created file during cleanup; also creates temporary command-project files under the OS temp directory. If interrupted, run `docker compose` with the printed project name or remove stale `observme-grafana-it-*` resources. | Live local stack ingestion, datasource queries, `/obs` query commands, and dashboard provisioning imports. |
 | `npm run validate:grafana-obs` | No | Running Pi session with ObservMe loaded, reachable Collector/Grafana stack, and recent telemetry. See `docs/validation-flow.md`. | Requires Grafana read credentials from environment variables (`OBSERVME_GRAFANA_TOKEN` or username/password), datasource UIDs, and `OBSERVME_VALIDATION_SESSION_ID`; may access the configured Grafana URL. | Read-only against the configured stack; no file writes expected. Keep terminal output sanitized and omit secrets. | Operator-facing validation that Grafana has ObservMe data and representative `/obs` commands work against the same stack. |
 
@@ -76,6 +77,22 @@ This record was produced for the final-pass task "Run and record bounded live Pi
 
 Post-run cleanup check: `docker ps --filter name=observme-grafana-it --format '{{.Names}}'` and `docker network ls --filter name=observme-grafana-it --format '{{.Name}}'` returned no resources.
 
+## Active-agent lease release validation — 2026-07-14
+
+This sanitized record covers the production active-agent lease release candidate on local Node.js v26.0.0, npm 11.12.1, Docker 29.5.2, and Docker Compose 5.1.4. The checkout contains the lease implementation and documentation changes. No project, Grafana, model, or registry credentials were required or printed.
+
+| Command | Result | Evidence and notes |
+| --- | --- | --- |
+| `npm run validate` | Pass | Source/test typechecks, ESLint, formatting, script checks, 493 unit/contract tests, package-content checks, packaged-install smoke, handler smoke, Pi lifecycle smoke, and Pi runtime smoke passed. |
+| `npm run test:integration:collector` | Pass | The pinned Collector Contrib 0.104.0 debug pipeline received representative traces, metrics, and logs without default content capture. |
+| `npm run test:integration:active-agent-lease` | Pass | Clean shutdown, `SIGTERM`, and cancellation-oriented `SIGKILL` passed against Collector Contrib 0.104.0 and Prometheus 2.53.1. The raw claim remained cached while lease-aware activity reached zero with the same Collector instance and no restart. |
+| `npm run test:integration:grafana-stack` | Fail/unrelated pre-existing broad-stack blocker | The current run timed out when the Prometheus total-token query returned an empty vector. This token-ingestion/query path is unrelated to active-agent lease arithmetic, PromQL, or abrupt-termination convergence and does not invalidate the focused passing lease evidence. The earlier 2026-07-09 broad-stack run was already blocked later in its Tempo content assertion. |
+| `docker compose -f observability-stack/docker-compose.yml config --quiet` | Pass | Compose interpolation validated without rendering secret values. |
+| Pinned Collector Contrib 0.104.0 `validate --config=/etc/otel/otel-collector.yaml` | Pass | The shipped `observability-stack/config/otel/otel-collector.yaml` parsed successfully with the deployed distribution. |
+| `npm run pack:dry-run` | Pass | The 124-file package contained the lease source/metric convention, dashboards, alerts, examples, target user/operator/reference docs, and packaged `observme-docs` skill. |
+
+Post-run cleanup check found no containers or networks labeled for the active-agent lease integration and no `observme-grafana-it-*` containers or networks. Release remains blocked on the unrelated broad Grafana-stack validation unless the release owner accepts the explicitly allowed pre-existing blocker; all lease-specific release criteria passed.
+
 ## Review-closure evidence categories
 
 - **Read-only/check-only** — commands that inspect, type-check, lint, audit, run tests, dry-run packaging, or smoke local deterministic fixtures without publishing or writing tracked files.

package/examples/README.md

@@ -11,6 +11,7 @@ The npm package ships configuration examples and an extension-integration exampl
 - Local self-signed TLS and IPv4 preference are enabled for that profile.
 - Prompt, response, thinking, tool, Bash, and path capture remain disabled.
 - Redaction remains enabled and unsafe capture remains disabled.
+- Metrics export every 15 seconds and renew a 60-second active-agent lease; keep producer and Prometheus clocks synchronized within 5 seconds.
 
 To use it in a trusted project:
 
@@ -27,13 +28,17 @@ Read [`../docs/configuration.md`](../docs/configuration.md) for the quick guide
 
 - traces are exported to Tempo;
 - logs are exported to Loki;
-- metrics are remote-written to Mimir;
+- metrics are exposed on a Prometheus scrape endpoint;
+- the five-minute `metric_expiration` applies to all exported metric types only as stale-series/cardinality cleanup, not active-agent liveness;
+- lease-aware PromQL joins the positive lifecycle claim and lease on generated `observme_instance_id`; raw `observme_active_agents` sums are diagnostics, not production live totals;
 - high-cardinality workflow, agent, session, and spawn attributes are removed from metrics;
 - accidental content attributes are removed from logs as defense in depth.
 
 The example uses Collector components commonly provided by the Collector Contrib distribution. Verify that your deployed distribution includes every receiver, processor, and exporter, then replace backend endpoints and security settings for your environment.
 
-Read [`../docs/reference/05-otel-pipeline-and-collector.md`](../docs/reference/05-otel-pipeline-and-collector.md) for pipeline design and [`../docs/reference/11-deployment-runbooks.md`](../docs/reference/11-deployment-runbooks.md) for deployment and incident checks.
+Read [`../docs/reference/05-otel-pipeline-and-collector.md`](../docs/reference/05-otel-pipeline-and-collector.md) for pipeline design, [`../docs/reference/09-dashboards-alerts-slos.md`](../docs/reference/09-dashboards-alerts-slos.md#131-canonical-active-agent-promql) for canonical queries and raw-query migration, and [`../docs/reference/11-deployment-runbooks.md`](../docs/reference/11-deployment-runbooks.md) for deployment and incident checks.
+
+For GitHub Actions, prefer graceful Pi/sidecar shutdown and an `if: always()` cleanup step, but do not depend on either after force cancellation. Lease expiry provides correctness without a Collector restart. GitHub-hosted runner clocks meet the supported expectation; self-hosted runners must use reliable time synchronization.
 
 ## `integrations/subagent-runner.ts`