@senad-d/observme 0.1.3 → 0.1.4

package/.env.example

@@ -13,6 +13,10 @@ OBSERVME_TENANT=local-dev
 OBSERVME_OTLP_ENDPOINT=http://localhost:4318
 OBSERVME_OTLP_PROTOCOL=http/protobuf
 
+# Active-agent lease duration. Must be 10000-300000 ms and at least
+# twice the metric export interval plus 5000 ms of clock-skew margin.
+OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS=60000
+
 # Optional OTLP bearer token for remote/secured collectors. Leave blank for local dev.
 OBSERVME_OTLP_TOKEN=
 

package/CHANGELOG.md

@@ -4,6 +4,12 @@
 
 ### Added
 
+- Added a bounded GitHub-hosted Linux CI job for active-agent lease contracts and cancellation-oriented Docker integration coverage, with resource-labeled unconditional cleanup.
+- Added Docker-backed active-agent lease integration coverage for clean shutdown, `SIGTERM`, and `SIGKILL`, proving cached raw claims outlive lease-aware activity without a Collector restart.
+- Added a session-scoped active-agent lease controller that renews from the SDK metric collection cycle with an injectable wall clock and deterministic disposal.
+- Added the `observme_agent_lease_expires_unixtime_seconds` asynchronous gauge contract with unit metadata and deterministic observable callback test support.
+- Added bounded `metrics.activeAgentLeaseDurationMillis` configuration with layered `OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS` loading, fail-safe validation, and generated/example configuration coverage.
+- Added a production-readiness task plan for lease-based active-agent accounting that remains accurate after crashes, forced termination, and cancelled GitHub Actions jobs.
 - Added a packaged `observme-docs` Pi skill that routes natural-language ObservMe questions to focused user, operator, reference, example, and contributor documentation through Pi's normal skill discovery, without an ObservMe system-prompt hook.
 - Added a versioned `@senad-d/observme/integration` event-bus API and transport-agnostic child-runner example for parent-side spawn/wait/join telemetry and child process lineage propagation.
 - Added README tables cataloging available metrics, trace spans, and structured log events, including opt-in and reserved signals.
@@ -15,6 +21,11 @@
 
 ### Changed
 
+- Completed production active-agent lease documentation, raw-query migration guidance, GitHub Actions/self-hosted clock and cleanup runbooks, missing/expired-lease troubleshooting, Collector restart semantics, and sanitized release-validation evidence.
+- Reframed the Collector's five-minute Prometheus `metric_expiration` as exporter-wide stale-series/cardinality cleanup, longer than the default active-agent lease and independent of leased liveness.
+- Migrated active-agent dashboard totals, bounded breakdowns, aggregate topology inputs, and stuck-high alerts to leased activity, with raw/expired-claim diagnostics and a deployment-tunable stale-claim alert.
+- Defined and enforced canonical lease-aware active-agent PromQL for totals, bounded breakdowns, topology inputs, and alerts, including replica deduplication, future-lease rejection, and zero-safe idle states.
+- Documented the production active-agent lease, clock, convergence, instance-join, failure-mode, and backward-compatibility contract used by upcoming runtime and dashboard integration.
 - Moved the detailed technical reference into `docs/reference/` and updated package, documentation, examples, dashboards, tests, and skill routes to use the new location.
 - Made the packaged `observme-docs` skill resolve routed references from its installed package root instead of the caller's working directory or a repository checkout.
 - Reorganized documentation around `docs/README.md`, a categorized technical-reference index, and an example guide with explicit usage and safety notes.
@@ -27,6 +38,8 @@
 
 ### Fixed
 
+- Restored clean npm dependency resolution by pinning TypeScript to the latest release supported by `typescript-eslint`.
+- Integrated active-agent lease activation and deactivation with clean shutdown, duplicate/reload replacement, resume, and failed-start cleanup so final flushes cannot renew stale activity.
 - Fixed session, workflow, agent, turn, LLM, tool, Bash, histogram, active-span, failure/recovery, and session-count telemetry accuracy.
 - Fixed lifecycle serialization, duplicate session replacement, bounded shutdown/flush behavior, backfill cancellation, parallel tool correlation, and bounded agent-tree state cleanup.
 - Hardened configuration validation, project-root path confinement, endpoint security, custom redaction patterns, tenant-salted hashing, and environment propagation.

package/README.md

@@ -62,7 +62,7 @@ This checkout implements the ObservMe MVP scope described by the packaged [`docs
 
 - Extension load checks and `/obs health` backend checks.
 - Session, workflow, agent-run, turn, LLM, tool, bash, subagent-spawn, wait/join, compaction, branch, model-change, and thinking-level telemetry.
-- Multi-agent tree metrics for active agents, depth, fan-out, orphan agents, trace-context propagation failures, child failures, and workflow duration.
+- Lease-qualified multi-agent activity plus tree metrics for depth, fan-out, orphan agents, trace-context propagation failures, child failures, and workflow duration; abrupt exits converge without a Collector restart.
 - Session-scoped OTLP trace, metric, and log exporters with bounded queues, timeouts, and shutdown flushing.
 - Configurable capture controls, redaction, path scrubbing, hashing, truncation, and high-cardinality metric-label guards.
 - Grafana dashboard JSON, alert rules, SLO definitions, Collector examples, and compatibility matrix artifacts.
@@ -230,10 +230,17 @@ Metric labels are intentionally low-cardinality. Session, workflow, agent, trace
 | Histogram | Workflow and agent tree | `observme_workflow_duration_ms`<br>`observme_agent_run_duration_ms`<br>`observme_subagent_spawn_duration_ms`<br>`observme_agent_wait_duration_ms`<br>`observme_agent_join_duration_ms`<br>`observme_agent_tree_depth`<br>`observme_agent_tree_width`<br>`observme_agent_fanout_count` | Workflow/agent latency and multi-agent tree shape. |
 | Histogram | Operation latency | `observme_turn_duration_ms`<br>`observme_llm_request_duration_ms`<br>`observme_tool_duration_ms`<br>`observme_bash_duration_ms`<br>`observme_handler_duration_ms` | End-to-end operation and extension-handler duration in milliseconds. |
 | Histogram | Context and payload size | `observme_compaction_tokens_before`<br>`observme_prompt_size_chars`<br>`observme_response_size_chars`<br>`observme_tool_result_size_chars` | Context size before compaction and payload sizes in characters. Size metrics do not contain payload content. |
-| Up-down counter | Current activity | `observme_active_spans`<br>`observme_active_agents` | Active operations and agents for the current exported stream. |
+| Up-down counter | Current activity | `observme_active_spans`<br>`observme_active_agents` | Active operations and the compatibility lifecycle claim for agents in the current exported stream. |
+| Asynchronous gauge | Agent liveness lease | `observme_agent_lease_expires_unixtime_seconds` | Absolute Unix expiry renewed on each metric collection while the session is active; operational active-agent queries join it to the positive lifecycle claim. |
 | Registered, not yet recorded | Agent lifetime | `observme_agent_lifetime_duration_ms` | Reserved instrument; the current live handlers do not record measurements yet. |
 | Registered, not yet recorded | Official GenAI compatibility | `gen_ai.client.token.usage`<br>`gen_ai.client.operation.duration` | Reserved OpenTelemetry GenAI instruments; use the `observme_llm_*` metrics above for current data. |
 
+### Active-agent liveness
+
+`observme_active_agents` remains a clean-start/clean-shutdown lifecycle claim, but raw sums are not authoritative after a crash, `SIGKILL`, cancelled GitHub Actions job, or lost runner. The shipped dashboards and alerts count an instance only when that claim is positive and its `observme_agent_lease_expires_unixtime_seconds` value is current and within the supported future horizon. Clean shutdown reaches zero after export/scrape propagation; an ungraceful exit converges within the configured lease plus up to 5 seconds of supported clock skew and one Prometheus scrape/evaluation interval (80 seconds with the default 60-second lease and a 15-second scrape).
+
+Missing, expired, malformed, or pathologically future leases fail closed. GitHub-hosted runners satisfy the clock requirement; self-hosted runners must keep the producer and Prometheus clocks synchronized within 5 seconds. Collector restart is not required for convergence, even when its Prometheus exporter continues to expose a cached raw claim. Migrate custom panels, alerts, and recording rules from raw `sum(observme_active_agents)` queries to the [canonical lease-aware PromQL](docs/reference/09-dashboards-alerts-slos.md#131-canonical-active-agent-promql), and use raw/expired claims only for diagnostics.
+
 ### Traces
 
 | Span name | Created for | Notes |
@@ -383,7 +390,7 @@ These assets are included in the npm package:
 
 Open **ObservMe Overview** first for health/SLO chips, workload, cost, latency, agent-lineage status, and time-preserving links to focused dashboards. Use **Trace Journey** to follow a session/workflow/agent/run across Prometheus aggregates, Loki logs, and Tempo traces. Use **LLM Conversations** only for redacted opt-in content; raw prompt, response, command, path, and error-message values should never be placed in dashboard URLs or Prometheus labels. Empty failure tables normally mean no matching failures in the selected range, while optional content panels can be empty because capture is disabled by default.
 
-The full dashboard map, standard variables, drill-down workflow, threshold colors, and zero-state semantics are documented in `docs/reference/09-dashboards-alerts-slos.md`.
+The full dashboard map, canonical lease-aware active-agent queries, raw-query migration, standard variables, drill-down workflow, threshold colors, and zero-state semantics are documented in `docs/reference/09-dashboards-alerts-slos.md`. For an unexpected zero or stale raw claim, follow the active-lease troubleshooting flow in `docs/reference/11-deployment-runbooks.md` and check Export Health before concluding that the producer stopped.
 
 ## Documentation Set
 
@@ -415,8 +422,10 @@ npm run lint:fix    # ESLint auto-fix pass
 npm run format:check
 npm run test
 npm run test:integration:collector
+npm run test:integration:active-agent-lease
 npm run test:integration:grafana-stack
 npm run check:pack
+npm run pack:dry-run
 npm run smoke:pi-runtime
 npm run validate:grafana-obs
 pi --no-extensions -e .

package/dashboards/observme-agent-node-graphs.json

@@ -45,7 +45,7 @@
         "type": "prometheus",
         "uid": "prometheus"
       },
-      "description": "Aggregate Pi lineage topology built only from bounded Prometheus labels; it is not a single trace tree. Numeric main stats are current gauges or selected-range counts. Red nodes and edges identify spawn failures, orphan lineage, and trace-context propagation failures. During healthy idle ranges, only the Pi root may render because no aggregate activity exists; use Agents for ratios and Trace Journey for one execution.",
+      "description": "Aggregate fleet topology built only from bounded Prometheus labels; it is not per-workflow truth or a single trace tree. Role nodes combine canonical leased-current activity with selected-range run/spawn counts, so node and edge values must not be read as one execution. Clean shutdown removes leased activity after export/scrape propagation; after a crash, SIGKILL, cancelled job, or runner loss, leased activity converges to zero within the default 60-second lease plus one Prometheus scrape while raw claims may stay cached. Red nodes and edges identify spawn, lineage, or propagation failures. /obs agents is current-process child state; use Trace Journey for one execution.",
       "fieldConfig": {
         "defaults": {
           "mappings": [],
@@ -88,7 +88,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(vector(1), \"id\", \"pi-root\", \"__name__\", \".*\"), \"title\", \"Pi root\", \"id\", \".*\"), \"subTitle\", \"workflow entry\", \"id\", \".*\"), \"mainStat\", \"entry\", \"id\", \".*\"), \"secondaryStat\", \"aggregate\", \"id\", \".*\"), \"color\", \"blue\", \"id\", \".*\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(observme_active_agents) by (agent_role) or sum(increase(observme_agent_runs_total[$__range])) by (agent_role) or sum(increase(observme_subagents_spawned_total[$__range])) by (agent_role)), \"id\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"title\", \"Agent: $1\", \"agent_role\", \"(.+)\"), \"subTitle\", \"active or observed role\", \"agent_role\", \"(.+)\"), \"mainStat\", \"count\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"role\", \"agent_role\", \"(.+)\"), \"color\", \"green\", \"agent_role\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (subagent_depth), \"id\", \"subagents_depth_$1\", \"subagent_depth\", \"(.+)\"), \"title\", \"Subagents depth $1\", \"subagent_depth\", \"(.+)\"), \"subtitle\", \"spawned in range\", \"subagent_depth\", \"(.+)\"), \"mainStat\", \"spawns\", \"subagent_depth\", \"(.+)\"), \"secondaryStat\", \"depth\", \"subagent_depth\", \"(.+)\"), \"color\", \"orange\", \"subagent_depth\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) by (error_class) > 0), \"id\", \"spawn_failure_$1\", \"error_class\", \"(.+)\"), \"title\", \"Spawn failure: $1\", \"error_class\", \"(.+)\"), \"subtitle\", \"failed spawns\", \"error_class\", \"(.+)\"), \"mainStat\", \"failures\", \"error_class\", \"(.+)\"), \"secondaryStat\", \"error\", \"error_class\", \"(.+)\"), \"color\", \"red\", \"error_class\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_orphan_agents_total[$__range])) by (reason) > 0), \"id\", \"orphan_$1\", \"reason\", \"(.+)\"), \"title\", \"Orphan: $1\", \"reason\", \"(.+)\"), \"subtitle\", \"lineage missing\", \"reason\", \"(.+)\"), \"mainStat\", \"orphans\", \"reason\", \"(.+)\"), \"secondaryStat\", \"reason\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_trace_context_propagation_failures_total[$__range])) by (reason) > 0), \"id\", \"propagation_$1\", \"reason\", \"(.+)\"), \"title\", \"Propagation: $1\", \"reason\", \"(.+)\"), \"subtitle\", \"trace context failures\", \"reason\", \"(.+)\"), \"mainStat\", \"failures\", \"reason\", \"(.+)\"), \"secondaryStat\", \"reason\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\")",
+          "expr": "label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(vector(1), \"id\", \"pi-root\", \"__name__\", \".*\"), \"title\", \"Pi root\", \"id\", \".*\"), \"subTitle\", \"workflow entry\", \"id\", \".*\"), \"mainStat\", \"entry\", \"id\", \".*\"), \"secondaryStat\", \"aggregate\", \"id\", \".*\"), \"color\", \"blue\", \"id\", \".*\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum by (agent_role) (max by (observme_instance_id, agent_role) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or sum(increase(observme_agent_runs_total[$__range])) by (agent_role) or sum(increase(observme_subagents_spawned_total[$__range])) by (agent_role)), \"id\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"title\", \"Agent: $1\", \"agent_role\", \"(.+)\"), \"subTitle\", \"active or observed role\", \"agent_role\", \"(.+)\"), \"mainStat\", \"count\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"role\", \"agent_role\", \"(.+)\"), \"color\", \"green\", \"agent_role\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (subagent_depth), \"id\", \"subagents_depth_$1\", \"subagent_depth\", \"(.+)\"), \"title\", \"Subagents depth $1\", \"subagent_depth\", \"(.+)\"), \"subtitle\", \"spawned in range\", \"subagent_depth\", \"(.+)\"), \"mainStat\", \"spawns\", \"subagent_depth\", \"(.+)\"), \"secondaryStat\", \"depth\", \"subagent_depth\", \"(.+)\"), \"color\", \"orange\", \"subagent_depth\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) by (error_class) > 0), \"id\", \"spawn_failure_$1\", \"error_class\", \"(.+)\"), \"title\", \"Spawn failure: $1\", \"error_class\", \"(.+)\"), \"subtitle\", \"failed spawns\", \"error_class\", \"(.+)\"), \"mainStat\", \"failures\", \"error_class\", \"(.+)\"), \"secondaryStat\", \"error\", \"error_class\", \"(.+)\"), \"color\", \"red\", \"error_class\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_orphan_agents_total[$__range])) by (reason) > 0), \"id\", \"orphan_$1\", \"reason\", \"(.+)\"), \"title\", \"Orphan: $1\", \"reason\", \"(.+)\"), \"subtitle\", \"lineage missing\", \"reason\", \"(.+)\"), \"mainStat\", \"orphans\", \"reason\", \"(.+)\"), \"secondaryStat\", \"reason\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_trace_context_propagation_failures_total[$__range])) by (reason) > 0), \"id\", \"propagation_$1\", \"reason\", \"(.+)\"), \"title\", \"Propagation: $1\", \"reason\", \"(.+)\"), \"subtitle\", \"trace context failures\", \"reason\", \"(.+)\"), \"mainStat\", \"failures\", \"reason\", \"(.+)\"), \"secondaryStat\", \"reason\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\")",
           "format": "table",
           "instant": true,
           "legendFormat": "nodes",
@@ -101,7 +101,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(observme_active_agents) by (agent_role) or sum(increase(observme_agent_runs_total[$__range])) by (agent_role)), \"source\", \"pi-root\", \"agent_role\", \"(.+)\"), \"target\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"mainStat\", \"count\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"active/runs\", \"agent_role\", \"(.+)\"), \"color\", \"gray\", \"agent_role\", \"(.+)\"), \"id\", \":\", \"source\", \"target\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (agent_role, subagent_depth, spawn_type, spawn_reason), \"source\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"target\", \"subagents_depth_$1\", \"subagent_depth\", \"(.+)\"), \"mainStat\", \"spawns\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"spawn_type/reason\", \"agent_role\", \"(.+)\"), \"color\", \"gray\", \"agent_role\", \"(.+)\"), \"id\", \":\", \"source\", \"target\", \"spawn_type\", \"spawn_reason\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) by (error_class) > 0), \"source\", \"pi-root\", \"error_class\", \"(.+)\"), \"target\", \"spawn_failure_$1\", \"error_class\", \"(.+)\"), \"mainStat\", \"failures\", \"error_class\", \"(.+)\"), \"secondaryStat\", \"spawn\", \"error_class\", \"(.+)\"), \"color\", \"red\", \"error_class\", \"(.+)\"), \"id\", \":\", \"source\", \"target\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_orphan_agents_total[$__range])) by (reason) > 0), \"source\", \"pi-root\", \"reason\", \"(.+)\"), \"target\", \"orphan_$1\", \"reason\", \"(.+)\"), \"mainStat\", \"orphans\", \"reason\", \"(.+)\"), \"secondaryStat\", \"lineage\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\"), \"id\", \":\", \"source\", \"target\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_trace_context_propagation_failures_total[$__range])) by (reason) > 0), \"source\", \"pi-root\", \"reason\", \"(.+)\"), \"target\", \"propagation_$1\", \"reason\", \"(.+)\"), \"mainStat\", \"failures\", \"reason\", \"(.+)\"), \"secondaryStat\", \"trace context\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\"), \"id\", \":\", \"source\", \"target\")",
+          "expr": "label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum by (agent_role) (max by (observme_instance_id, agent_role) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or sum(increase(observme_agent_runs_total[$__range])) by (agent_role)), \"source\", \"pi-root\", \"agent_role\", \"(.+)\"), \"target\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"mainStat\", \"count\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"active/runs\", \"agent_role\", \"(.+)\"), \"color\", \"gray\", \"agent_role\", \"(.+)\"), \"id\", \":\", \"source\", \"target\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (agent_role, subagent_depth, spawn_type, spawn_reason), \"source\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"target\", \"subagents_depth_$1\", \"subagent_depth\", \"(.+)\"), \"mainStat\", \"spawns\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"spawn_type/reason\", \"agent_role\", \"(.+)\"), \"color\", \"gray\", \"agent_role\", \"(.+)\"), \"id\", \":\", \"source\", \"target\", \"spawn_type\", \"spawn_reason\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) by (error_class) > 0), \"source\", \"pi-root\", \"error_class\", \"(.+)\"), \"target\", \"spawn_failure_$1\", \"error_class\", \"(.+)\"), \"mainStat\", \"failures\", \"error_class\", \"(.+)\"), \"secondaryStat\", \"spawn\", \"error_class\", \"(.+)\"), \"color\", \"red\", \"error_class\", \"(.+)\"), \"id\", \":\", \"source\", \"target\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_orphan_agents_total[$__range])) by (reason) > 0), \"source\", \"pi-root\", \"reason\", \"(.+)\"), \"target\", \"orphan_$1\", \"reason\", \"(.+)\"), \"mainStat\", \"orphans\", \"reason\", \"(.+)\"), \"secondaryStat\", \"lineage\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\"), \"id\", \":\", \"source\", \"target\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_trace_context_propagation_failures_total[$__range])) by (reason) > 0), \"source\", \"pi-root\", \"reason\", \"(.+)\"), \"target\", \"propagation_$1\", \"reason\", \"(.+)\"), \"mainStat\", \"failures\", \"reason\", \"(.+)\"), \"secondaryStat\", \"trace context\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\"), \"id\", \":\", \"source\", \"target\")",
           "format": "table",
           "instant": true,
           "legendFormat": "edges",
@@ -137,7 +137,7 @@
         "type": "prometheus",
         "uid": "prometheus"
       },
-      "description": "Aggregate delegation topology by bounded agent role, spawn reason, and resulting subagent depth; it is not a single trace tree. Numeric main stats are selected-range counts. Red health nodes and edges summarize spawn failures by emitted error class because the failure counter does not carry spawn reason. An empty graph is a healthy idle state when no spawns or failures occurred; use Agents for aggregate analysis and Trace Journey for one execution.",
+      "description": "Aggregate fleet delegation topology by bounded role, spawn reason, and resulting depth; it is not per-workflow truth or a single trace tree. Role nodes may combine canonical leased-current activity with selected-range run/spawn counts, while other values are selected-range aggregates. Clean shutdown removes leased activity after export/scrape propagation; after an ungraceful exit, leased activity converges to zero within the default 60-second lease plus one Prometheus scrape while raw claims may stay cached. Red health nodes and edges summarize spawn failures. /obs agents is local current-process child state; use Trace Journey for one execution.",
       "fieldConfig": {
         "defaults": {
           "mappings": [],
@@ -180,7 +180,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(observme_active_agents) by (agent_role) or sum(increase(observme_agent_runs_total[$__range])) by (agent_role) or sum(increase(observme_subagents_spawned_total[$__range])) by (agent_role)), \"id\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"title\", \"Agent: $1\", \"agent_role\", \"(.+)\"), \"subTitle\", \"active or observed role\", \"agent_role\", \"(.+)\"), \"mainStat\", \"count\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"role\", \"agent_role\", \"(.+)\"), \"color\", \"green\", \"agent_role\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (spawn_reason), \"id\", \"spawn_reason_$1\", \"spawn_reason\", \"(.+)\"), \"title\", \"Spawn: $1\", \"spawn_reason\", \"(.+)\"), \"subTitle\", \"spawn reason\", \"spawn_reason\", \"(.+)\"), \"mainStat\", \"spawns\", \"spawn_reason\", \"(.+)\"), \"secondaryStat\", \"reason\", \"spawn_reason\", \"(.+)\"), \"color\", \"purple\", \"spawn_reason\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (subagent_depth), \"id\", \"subagents_depth_$1\", \"subagent_depth\", \"(.+)\"), \"title\", \"Subagents depth $1\", \"subagent_depth\", \"(.+)\"), \"subTitle\", \"spawned in range\", \"subagent_depth\", \"(.+)\"), \"mainStat\", \"spawns\", \"subagent_depth\", \"(.+)\"), \"secondaryStat\", \"depth\", \"subagent_depth\", \"(.+)\"), \"color\", \"orange\", \"subagent_depth\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) > 0), \"id\", \"spawn-health\", \"__name__\", \".*\"), \"title\", \"Spawn failure health\", \"id\", \".*\"), \"subTitle\", \"failure-only aggregate\", \"id\", \".*\"), \"mainStat\", \"failures\", \"id\", \".*\"), \"secondaryStat\", \"health\", \"id\", \".*\"), \"color\", \"red\", \"id\", \".*\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) by (error_class) > 0), \"id\", \"spawn_failure_$1\", \"error_class\", \"(.+)\"), \"title\", \"Spawn failure: $1\", \"error_class\", \"(.+)\"), \"subTitle\", \"failed spawns\", \"error_class\", \"(.+)\"), \"mainStat\", \"failures\", \"error_class\", \"(.+)\"), \"secondaryStat\", \"error class\", \"error_class\", \"(.+)\"), \"color\", \"red\", \"error_class\", \"(.+)\")",
+          "expr": "label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum by (agent_role) (max by (observme_instance_id, agent_role) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or sum(increase(observme_agent_runs_total[$__range])) by (agent_role) or sum(increase(observme_subagents_spawned_total[$__range])) by (agent_role)), \"id\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"title\", \"Agent: $1\", \"agent_role\", \"(.+)\"), \"subTitle\", \"active or observed role\", \"agent_role\", \"(.+)\"), \"mainStat\", \"count\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"role\", \"agent_role\", \"(.+)\"), \"color\", \"green\", \"agent_role\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (spawn_reason), \"id\", \"spawn_reason_$1\", \"spawn_reason\", \"(.+)\"), \"title\", \"Spawn: $1\", \"spawn_reason\", \"(.+)\"), \"subTitle\", \"spawn reason\", \"spawn_reason\", \"(.+)\"), \"mainStat\", \"spawns\", \"spawn_reason\", \"(.+)\"), \"secondaryStat\", \"reason\", \"spawn_reason\", \"(.+)\"), \"color\", \"purple\", \"spawn_reason\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (subagent_depth), \"id\", \"subagents_depth_$1\", \"subagent_depth\", \"(.+)\"), \"title\", \"Subagents depth $1\", \"subagent_depth\", \"(.+)\"), \"subTitle\", \"spawned in range\", \"subagent_depth\", \"(.+)\"), \"mainStat\", \"spawns\", \"subagent_depth\", \"(.+)\"), \"secondaryStat\", \"depth\", \"subagent_depth\", \"(.+)\"), \"color\", \"orange\", \"subagent_depth\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) > 0), \"id\", \"spawn-health\", \"__name__\", \".*\"), \"title\", \"Spawn failure health\", \"id\", \".*\"), \"subTitle\", \"failure-only aggregate\", \"id\", \".*\"), \"mainStat\", \"failures\", \"id\", \".*\"), \"secondaryStat\", \"health\", \"id\", \".*\"), \"color\", \"red\", \"id\", \".*\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) by (error_class) > 0), \"id\", \"spawn_failure_$1\", \"error_class\", \"(.+)\"), \"title\", \"Spawn failure: $1\", \"error_class\", \"(.+)\"), \"subTitle\", \"failed spawns\", \"error_class\", \"(.+)\"), \"mainStat\", \"failures\", \"error_class\", \"(.+)\"), \"secondaryStat\", \"error class\", \"error_class\", \"(.+)\"), \"color\", \"red\", \"error_class\", \"(.+)\")",
           "format": "table",
           "instant": true,
           "legendFormat": "nodes",

package/dashboards/observme-agents.json

@@ -1065,7 +1065,7 @@
         "type": "prometheus",
         "uid": "prometheus"
       },
-      "description": "Current active-agent gauge grouped by the emitted agent_role label, plus the global total. The tunable total=100 reference matches ObservMeActiveAgentsStuckHigh; a missing gauge is unknown telemetry, not healthy zero.",
+      "description": "Canonical leased-active agents grouped by emitted agent_role, plus the fleet total. Clean shutdown reaches zero after export/scrape propagation; after a crash, SIGKILL, cancelled job, or runner loss, the leased total converges to zero within the default 60-second lease plus one Prometheus scrape even when raw claims stay cached. Zero can also mean healthy idle or a failed-closed lease, so compare Export Health. /obs agents reports current-process child state, not this aggregate fleet view. The total=100 reference matches the deployment-tunable ObservMeActiveAgentsStuckHigh threshold.",
       "fieldConfig": {
         "defaults": {
           "color": {
@@ -1115,7 +1115,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "sum(observme_active_agents) by (agent_role)",
+          "expr": "sum by (agent_role) (max by (observme_instance_id, agent_role) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or on() vector(0)",
           "range": true,
           "refId": "A",
           "legendFormat": "{{agent_role}}"
@@ -1126,7 +1126,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "sum(observme_active_agents)",
+          "expr": "sum(max by (observme_instance_id) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or vector(0)",
           "range": true,
           "refId": "B",
           "legendFormat": "total active agents"
@@ -1697,6 +1697,171 @@
         "x": 0,
         "y": 57
       },
+      "id": 105,
+      "panels": [],
+      "title": "Lease health diagnostics",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "description": "Diagnostic positive lifecycle claims before lease qualification. These are not live-agent totals and may remain cached after a crash, SIGKILL, cancelled job, or runner loss; leased activity converges to zero within the default 60-second lease plus one Prometheus scrape. Clean shutdown exports zero immediately. Compare Expired active claims and Export Health. /obs agents reports only current-process child state, not fleet liveness.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 58
+      },
+      "id": 22,
+      "links": [
+        {
+          "targetBlank": false,
+          "title": "Open Export Health",
+          "url": "/d/observme-export-health/observme-export-health?${__url_time_range}"
+        }
+      ],
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showPercentChange": false,
+        "textMode": "auto",
+        "wideLayout": true
+      },
+      "pluginVersion": "11.1.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum(max by (observme_instance_id) (observme_active_agents > 0)) or vector(0)",
+          "instant": true,
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Raw active claims (diagnostic)",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "description": "Positive raw claims whose lease has expired. They are stale diagnostics and never contribute to leased active totals; clean shutdown normally avoids them, while ungraceful exits appear after the default 60-second lease plus one Prometheus scrape and may remain until Collector cleanup. Yellow at 1 and red at 5 are deployment-tunable. Compare Raw active claims and Export Health. /obs agents reports only current-process child state, not fleet liveness.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 58
+      },
+      "id": 23,
+      "links": [
+        {
+          "targetBlank": false,
+          "title": "Open Export Health",
+          "url": "/d/observme-export-health/observme-export-health?${__url_time_range}"
+        }
+      ],
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showPercentChange": false,
+        "textMode": "auto",
+        "wideLayout": true
+      },
+      "pluginVersion": "11.1.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum(max by (observme_instance_id) ((observme_active_agents > 0) and on (observme_instance_id) (observme_agent_lease_expires_unixtime_seconds <= time()))) or vector(0)",
+          "instant": true,
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Expired active claims (diagnostic)",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 66
+      },
       "id": 104,
       "panels": [],
       "title": "Lineage logs",
@@ -1768,7 +1933,7 @@
         "h": 10,
         "w": 24,
         "x": 0,
-        "y": 58
+        "y": 67
       },
       "id": 6,
       "options": {

package/dashboards/observme-alerts.yaml

@@ -103,11 +103,24 @@ groups:
           runbook: "Severity guidance from production docs: warning."
       - alert: ObservMeActiveAgentsStuckHigh
         expr: >-
-          sum(observme_active_agents) > 100
+          (sum(max by (observme_instance_id) ((observme_active_agents > 0) and on (observme_instance_id)
+          ((observme_agent_lease_expires_unixtime_seconds > time()) and
+          (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or vector(0)) > 100
         for: 30m
         labels:
           severity: deployment-dependent
         annotations:
           summary: Active agents stuck high
-          description: ObservMe active-agent gauge remains above 100.
-          runbook: "Severity guidance from production docs: depends on normal fleet size; tune per deployment."
+          description: ObservMe leased active-agent count remains above 100; expired, missing, and pathological leases fail closed.
+          runbook: "Severity guidance from production docs: depends on normal fleet size; tune per deployment. Compare leased activity with raw and expired claims plus Export Health."
+      - alert: ObservMeExpiredActiveAgentClaims
+        expr: >-
+          (sum(max by (observme_instance_id) ((observme_active_agents > 0) and on (observme_instance_id)
+          (observme_agent_lease_expires_unixtime_seconds <= time()))) or vector(0)) > 5
+        for: 15m
+        labels:
+          severity: deployment-dependent
+        annotations:
+          summary: Expired active-agent claims detected
+          description: More than five positive lifecycle claims have expired leases for 15 minutes; they are stale diagnostics and excluded from leased active totals.
+          runbook: "Deployment-tunable diagnostic: compare Raw active claims, Expired active claims, leased Active agents, and Export Health; investigate abrupt producer death, export gaps, and Collector cleanup. /obs agents is local child state, not fleet truth."

package/dashboards/observme-overview.json

@@ -1075,7 +1075,7 @@
         "type": "prometheus",
         "uid": "prometheus"
       },
-      "description": "Currently active ObservMe agent runtimes. Zero can mean healthy idle. Red at 100 references the tunable ObservMeActiveAgentsStuckHigh default; yellow at 50 is an early-warning assumption. Compare sustained values with fan-out and depth.",
+      "description": "Canonical leased-active ObservMe runtimes. Clean shutdown reaches zero after export/scrape propagation; after a crash, SIGKILL, cancelled job, or runner loss, the leased total converges to zero within the default 60-second lease plus one Prometheus scrape even if raw claims stay cached. Zero can also mean healthy idle or a failed-closed lease, so check Export Health. /obs agents reports only current-process child state, not fleet liveness. Red at 100 matches the deployment-tunable ObservMeActiveAgentsStuckHigh threshold; yellow at 50 is an early warning.",
       "fieldConfig": {
         "defaults": {
           "color": {
@@ -1141,7 +1141,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "sum(observme_active_agents) or vector(0)",
+          "expr": "sum(max by (observme_instance_id) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or vector(0)",
           "range": true,
           "refId": "A"
         }
@@ -1998,7 +1998,7 @@
         "type": "prometheus",
         "uid": "prometheus"
       },
-      "description": "Current active agents by bounded role and depth. No rows can mean healthy idle or an absent active-agent gauge; compare with the ungrouped Active agents chip and Export Health before treating it as healthy.",
+      "description": "Canonical leased-active agents by emitted bounded role and environment. Clean shutdown reaches zero after export/scrape propagation; after an ungraceful exit, leased totals converge to zero within the default 60-second lease plus one Prometheus scrape while raw claims may remain cached. Depth is omitted because active signals do not emit subagent_depth. An unlabeled zero means healthy idle or no qualifying lease when Prometheus is reachable; compare Export Health. /obs agents is current-process child state, not this aggregate fleet view.",
       "fieldConfig": {
         "defaults": {
           "color": {
@@ -2058,14 +2058,14 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "sum(observme_active_agents) by (agent_role, subagent_depth)",
+          "expr": "sum by (agent_role, environment) (max by (observme_instance_id, agent_role, environment) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or on() vector(0)",
           "instant": true,
-          "legendFormat": "{{agent_role}} depth={{subagent_depth}}",
+          "legendFormat": "{{agent_role}} / {{environment}}",
           "range": false,
           "refId": "A"
         }
       ],
-      "title": "Active agents by role/depth",
+      "title": "Active agents by role/environment",
       "type": "bargauge"
     },
     {

package/dashboards/observme-trace-journey.json

@@ -144,7 +144,7 @@
         "type": "prometheus",
         "uid": "prometheus"
       },
-      "description": "Current ObservMe agent runtimes from the active-agent gauge. Zero can mean healthy idle or an absent gauge; check Export Health before treating an unexpected zero as healthy.",
+      "description": "Canonical leased-active fleet total; Prometheus summary values are aggregate and are not filtered to the selected workflow. Clean shutdown reaches zero after export/scrape propagation; after a crash, SIGKILL, cancelled job, or runner loss, the leased total converges to zero within the default 60-second lease plus one Prometheus scrape even if raw claims stay cached. Zero can also mean healthy idle or a failed-closed lease, so check Export Health. /obs agents is current-process child state, not fleet liveness.",
       "fieldConfig": {
         "defaults": {
           "color": {
@@ -195,7 +195,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "(sum(observme_active_agents) or vector(0))",
+          "expr": "sum(max by (observme_instance_id) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or vector(0)",
           "instant": true,
           "range": false,
           "refId": "A"
@@ -1005,7 +1005,7 @@
         "type": "prometheus",
         "uid": "prometheus"
       },
-      "description": "Shape of the agent tree over time without using workflow or agent IDs as metric labels. Constant reference lines mirror the depth=5 and fan-out=20 alert thresholds.",
+      "description": "Aggregate fleet shape over time, not per-workflow truth: the leased-active role series is canonical current activity, while tree histograms summarize all matching telemetry without workflow or agent IDs as metric labels. Clean shutdown removes leased activity after export/scrape propagation; after an ungraceful exit, leased activity converges to zero within the default 60-second lease plus one Prometheus scrape while raw claims may remain cached. /obs agents is local current-process child state. Constant lines mirror the depth=5 and fan-out=20 alert thresholds.",
       "fieldConfig": {
         "defaults": {
           "color": {
@@ -1083,7 +1083,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "sum(observme_active_agents) by (agent_role)",
+          "expr": "sum by (agent_role) (max by (observme_instance_id, agent_role) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or on() vector(0)",
           "legendFormat": "active {{agent_role}}",
           "range": true,
           "refId": "A"

package/docs/agent-subagent-observability-requirements.md

@@ -647,27 +647,35 @@ histogram_quantile(0.95, sum(rate(observme_subagent_spawn_duration_ms_bucket[$__
 
 ```text
 observme_active_agents
+observme_agent_lease_expires_unixtime_seconds
 observme_agent_tree_depth
 observme_agent_tree_width
 observme_agent_fanout_count
 ```
 
-Expected labels:
-
-```text
-agent_role
-subagent_depth
-```
+The active claim and lease carry the bounded `agent_role` and `environment` dimensions plus generated resource-instance identity normalized by the Collector to `observme_instance_id`. They do not currently carry `subagent_depth`; never derive a replacement depth label from workflow, session, logical-agent, trace, span, or job-run IDs. The tree histograms retain their documented bounded tree dimensions.
 
 Dashboard examples:
 
 ```promql
-sum(observme_active_agents) by (agent_role, subagent_depth)
+sum by (agent_role) (
+  max by (observme_instance_id, agent_role) (
+    (observme_active_agents > 0)
+    and on (observme_instance_id)
+    (
+      (observme_agent_lease_expires_unixtime_seconds > time())
+      and
+      (observme_agent_lease_expires_unixtime_seconds <= time() + 305)
+    )
+  )
+) or on() vector(0)
 histogram_quantile(0.95, sum(rate(observme_agent_tree_depth_bucket[$__rate_interval])) by (subagent_depth, le))
 histogram_quantile(0.95, sum(rate(observme_agent_tree_width_bucket[$__rate_interval])) by (subagent_depth, le))
 histogram_quantile(0.95, sum(rate(observme_agent_fanout_count_bucket[$__rate_interval])) by (subagent_depth, le))
 ```
 
+Use the canonical total, role, environment, diagnostics, and migration guidance in [`reference/09-dashboards-alerts-slos.md` §1.3](reference/09-dashboards-alerts-slos.md#13-active-agent-leased-state-contract); raw active-claim sums are not current-liveness queries after ungraceful termination.
+
 ### Lineage health metrics
 
 ```text

package/docs/compatibility-matrix.md

@@ -1,8 +1,8 @@
 # ObservMe Compatibility Matrix
 
-Last updated: 2026-07-07
+Last updated: 2026-07-14
 
-This matrix records the ObservMe runtime and observability-stack versions that have been exercised or pinned for the current release line. A row marked **validated** has been covered by a named local or CI validation command. A row marked **pinned, pending integration** is present in the reference configuration but must be promoted to validated after the integration tests in tasks 48-50 exercise it.
+This matrix records the ObservMe runtime and observability-stack versions exercised or pinned for the current release line. A component is marked **validated** only when the named command actually exercised it; configured CI targets remain distinct from completed local or workflow evidence.
 
 ## Validation record
 
@@ -12,6 +12,14 @@ This matrix records the ObservMe runtime and observability-stack versions that h
 | 2026-07-07 | Local Docker 29.5.2 with Collector Contrib 0.104.0 | `npm run test:integration:collector` | Starts a local debug-exporter Collector container, exports representative ObservMe traces/metrics/logs, and asserts default-disabled content capture is absent. |
 | 2026-07-07 | Local Docker 29.5.2 with the pinned Grafana stack | `npm run test:integration:grafana-stack` | Starts `observability-stack/` services, exports representative ObservMe telemetry, queries Tempo by trace ID and lineage attributes, queries Loki by session ID, queries Prometheus token totals, and validates Grafana dashboard provisioning imports. |
 | Continuous integration target | GitHub Actions `ubuntu-latest`, Node.js 22.19.0 | `npm ci --ignore-scripts` then `npm run validate` | CI target from `.github/workflows/ci.yml`; record the workflow run URL here after each release validation. |
+| Lease cancellation CI profile (configured 2026-07-14) | GitHub-hosted Linux `ubuntu-latest` (exact `ImageOS`/`ImageVersion` recorded in the Actions runner log); Node.js 22.19.0; Collector Contrib 0.104.0; Prometheus 2.53.1; Grafana 11.1.0 | `npm run test:active-agent-lease:contracts` then `npm run test:integration:active-agent-lease` | The bounded `lease-cancellation` job validates clean shutdown, `SIGTERM`, and cancellation-oriented `SIGKILL` convergence against the pinned Collector and Prometheus without secrets or a Grafana dependency. Grafana 11.1.0 is the recorded dashboard compatibility baseline validated separately by `npm run test:integration:grafana-stack`. `if: always()` cleanup is best effort; lease expiry is the asserted correctness mechanism. |
+| 2026-07-14 | Local Node.js v26.0.0, npm 11.12.1 | `npm run validate` | Passed 493 unit/contract tests plus typecheck, ESLint, formatting, script, package-content, packaged-install, handler, Pi lifecycle, and Pi runtime checks for the lease release candidate. |
+| 2026-07-14 | Local Docker 29.5.2, Compose 5.1.4, Collector Contrib 0.104.0 | `npm run test:integration:collector`; Compose config; pinned Collector `validate` | Passed representative OTLP export, Compose interpolation, and the shipped Collector config validation. |
+| 2026-07-14 | Local Docker 29.5.2, Collector Contrib 0.104.0, Prometheus 2.53.1 | `npm run test:integration:active-agent-lease` | Passed clean shutdown, `SIGTERM`, and cancellation-oriented `SIGKILL`; the raw claim remained cached while leased activity converged without Collector restart. Labeled containers and the network were absent after cleanup. |
+| 2026-07-14 | Local Docker 29.5.2 with the pinned Grafana stack | `npm run test:integration:grafana-stack` | Unrelated pre-existing broad-stack blocker remains: this run timed out with an empty Prometheus total-token vector. That token path does not validate active-agent leases. Cleanup removed all integration containers and networks; focused lease evidence is unaffected. |
+| 2026-07-14 | npm package dry run for `@senad-d/observme` 0.1.3 | `npm run pack:dry-run` | Passed with 124 files; metric source, dashboards, alerts, examples, target docs, and `skills/observme-docs/SKILL.md` were present. |
+
+Active-agent lease interpretation additionally requires producer and Prometheus wall clocks to remain within 5 seconds. GitHub-hosted runners satisfy this expectation. Self-hosted runners are supported only with reliable NTP or equivalent synchronization; clock health must be monitored separately from the ObservMe version matrix.
 
 ## Runtime and library matrix
 

package/docs/configuration.md

@@ -24,6 +24,9 @@ Edit the project config file (`.pi/observme.yaml` in the standard distribution)
 - `privacy` — redaction, unsafe-capture acknowledgement, insecure transport, hash salt env var, and path handling. Keep `redactionEnabled: true`; set `allowUnsafeCapture: true` only when you intentionally accept unredacted sensitive-content export from this trusted project. Live telemetry and `/obs backfill` use the same policy: disabled capture omits content, enabled redaction redacts then truncates, redaction failures drop content, and `redactionEnabled: false` with `allowUnsafeCapture: true` exports raw truncated content.
 - `query.grafana` — Grafana URL, datasource UIDs, TLS, and IPv4 transport settings for `/obs` query commands.
 - `query.links.traceUrlTemplate` — the Grafana Explore trace-link template used by `/obs trace` and `/obs link`.
+- `metrics.activeAgentLeaseDurationMillis` — how long the last exported active-agent lease remains valid. The default is `60000` ms, the supported range is `10000`–`300000` ms, and the value must be at least `(2 * metrics.exportIntervalMillis) + 5000` ms. `OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS` overrides YAML through the normal precedence rules.
+
+The shipped dashboards and alerts combine a positive `observme_active_agents` lifecycle claim with a current lease; raw active-claim sums are diagnostic only after an ungraceful exit. Clean shutdown reaches zero after normal export/scrape propagation. Crash, `SIGKILL`, forced GitHub Actions cancellation, or runner loss converges within the lease plus up to 5 seconds of supported clock skew and one Prometheus scrape/evaluation interval, without restarting the Collector. Keep producer and Prometheus clocks synchronized within 5 seconds; GitHub-hosted runners meet this expectation, while self-hosted runners require reliable NTP or an equivalent time service.
 
 Keep credentials out of YAML. Reference environment variables such as `OBSERVME_OTLP_TOKEN`, `OBSERVME_GRAFANA_TOKEN`, `OBSERVME_GRAFANA_PASSWORD`, and `OBSERVME_HASH_SALT`, then set those values in the shell or a trusted project `.env` file. When any content capture flag is enabled with `privacy.redactionEnabled: true`, `OBSERVME_HASH_SALT` must be set before the event occurs; otherwise content capture fails closed and emits `redaction.failed` diagnostics instead of conversation rows.
 
@@ -45,3 +48,5 @@ Invalid or unsafe merged configuration falls back to safe defaults. `/obs status
 - [Example configurations](../examples/README.md)
 - [Security, privacy, and redaction](reference/06-security-privacy-redaction.md)
 - [Grafana and `/obs` validation flow](validation-flow.md)
+- [Canonical active-agent queries and raw-query migration](reference/09-dashboards-alerts-slos.md#131-canonical-active-agent-promql)
+- [Active-lease deployment and troubleshooting runbook](reference/11-deployment-runbooks.md)