@senad-d/observme 0.1.3 → 0.1.5

package/.env.example

@@ -13,6 +13,10 @@ OBSERVME_TENANT=local-dev
 OBSERVME_OTLP_ENDPOINT=http://localhost:4318
 OBSERVME_OTLP_PROTOCOL=http/protobuf
 
+# Active-agent lease duration. Must be 10000-300000 ms and at least
+# twice the metric export interval plus 5000 ms of clock-skew margin.
+OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS=60000
+
 # Optional OTLP bearer token for remote/secured collectors. Leave blank for local dev.
 OBSERVME_OTLP_TOKEN=
 

package/CHANGELOG.md

@@ -4,6 +4,12 @@
 
 ### Added
 
+- Added a bounded GitHub-hosted Linux CI job for active-agent lease contracts and cancellation-oriented Docker integration coverage, with resource-labeled unconditional cleanup.
+- Added Docker-backed active-agent lease integration coverage for clean shutdown, `SIGTERM`, and `SIGKILL`, proving cached raw claims outlive lease-aware activity without a Collector restart.
+- Added a session-scoped active-agent lease controller that renews from the SDK metric collection cycle with an injectable wall clock and deterministic disposal.
+- Added the `observme_agent_lease_expires_unixtime_seconds` asynchronous gauge contract with unit metadata and deterministic observable callback test support.
+- Added bounded `metrics.activeAgentLeaseDurationMillis` configuration with layered `OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS` loading, fail-safe validation, and generated/example configuration coverage.
+- Added a production-readiness task plan for lease-based active-agent accounting that remains accurate after crashes, forced termination, and cancelled GitHub Actions jobs.
 - Added a packaged `observme-docs` Pi skill that routes natural-language ObservMe questions to focused user, operator, reference, example, and contributor documentation through Pi's normal skill discovery, without an ObservMe system-prompt hook.
 - Added a versioned `@senad-d/observme/integration` event-bus API and transport-agnostic child-runner example for parent-side spawn/wait/join telemetry and child process lineage propagation.
 - Added README tables cataloging available metrics, trace spans, and structured log events, including opt-in and reserved signals.
@@ -15,6 +21,15 @@
 
 ### Changed
 
+- Made the 0.1.5 hardening patch self-contained with explicit review units, all imported production modules included, and unrelated specification history preserved.
+- Established a typed Pi event registration contract, removed non-event legacy registrations, pinned the release-resolved Pi API, and added minimum/release compatibility validation with pre-registration diagnostics.
+- Centralized embedded-credential Grafana URL diagnostics so configuration validation, readiness checks, URL construction, and transports share one safe failure class and operator guidance.
+- Documented credential-free Grafana base URLs and the fail-closed canonical project-file boundary across configuration, security, query, and troubleshooting guidance.
+- Completed production active-agent lease documentation, raw-query migration guidance, GitHub Actions/self-hosted clock and cleanup runbooks, missing/expired-lease troubleshooting, Collector restart semantics, and sanitized release-validation evidence.
+- Reframed the Collector's five-minute Prometheus `metric_expiration` as exporter-wide stale-series/cardinality cleanup, longer than the default active-agent lease and independent of leased liveness.
+- Migrated active-agent dashboard totals, bounded breakdowns, aggregate topology inputs, and stuck-high alerts to leased activity, with raw/expired-claim diagnostics and a deployment-tunable stale-claim alert.
+- Defined and enforced canonical lease-aware active-agent PromQL for totals, bounded breakdowns, topology inputs, and alerts, including replica deduplication, future-lease rejection, and zero-safe idle states.
+- Documented the production active-agent lease, clock, convergence, instance-join, failure-mode, and backward-compatibility contract used by upcoming runtime and dashboard integration.
 - Moved the detailed technical reference into `docs/reference/` and updated package, documentation, examples, dashboards, tests, and skill routes to use the new location.
 - Made the packaged `observme-docs` skill resolve routed references from its installed package root instead of the caller's working directory or a repository checkout.
 - Reorganized documentation around `docs/README.md`, a categorized technical-reference index, and an example guide with explicit usage and safety notes.
@@ -27,6 +42,39 @@
 
 ### Fixed
 
+- Simplified Pi SemVer validation into focused numeric-core and metadata checks, and made missing project-path component reversal explicit.
+- Accepted supported stable Pi versions with SemVer build metadata while preserving bounded diagnostics and prerelease rejection.
+- Rejected prerelease Pi builds excluded by the declared stable compatibility range before extension registration.
+- Applied one control-safe 64-row/8,192-character policy to every `/obs` notification and capped query result counts at configuration and runtime boundaries.
+- Coupled canonical project config, `.env`, and starter-config I/O to identity-verified file handles so concurrent symlink and ancestor swaps fail closed without exposing external paths.
+- Rejected credentials embedded in Grafana base URLs during config validation, query readiness, and transport preflight without exposing their values.
+- Rejected active and retained child-agent identifier collisions before creating integration spans, tree state, metrics, or propagation envelopes.
+- Retained ownership of timed-out OpenTelemetry shutdowns, observed late settlement safely, and deferred session replacement until prior exporter cleanup completes.
+- Replaced custom-redaction regex heuristics with bounded structural validation that rejects nested, ambiguous-alternative, and overlapping sequential repetition while preserving safe disjoint quantified alternatives.
+- Used concise word-character syntax for single-brace unresolved trace-link placeholders.
+- Used concise word-character syntax for dollar-brace unresolved trace-link placeholders.
+- Used concise word-character syntax for double-brace unresolved trace-link placeholders.
+- Split unresolved trace-link placeholder validation into simple per-syntax patterns while preserving rejection behavior.
+- Documented why best-effort Grafana stream cancellation failures are intentionally ignored.
+- Removed redundant explicit `undefined` support from the optional lifecycle model context property.
+- Simplified lifecycle recovery header selection by replacing a nested conditional expression with explicit branches.
+- Removed redundant awaiting of Pi's synchronous lifecycle status update.
+- Replaced deprecated proxy tracer construction with an isolated always-off provider for disabled and shutdown trace states.
+- Validated OTLP endpoints as secret-free absolute HTTP(S) URLs and constructed signal exporter paths with deterministic URL pathname semantics.
+- Implemented opt-in, versioned, active-branch correlation persistence with bounded validation and idempotent reload recovery, and removed unsupported automatic replay configuration and synthetic duplicate startup telemetry.
+- Bound live telemetry, query commands, and backfill correlation to Pi's typed session manager, preserved identity across reload, adopted replacement-session identity, and refreshed active metadata on session rename.
+- Bounded and sanitized backend-derived `/obs cost`, `/obs tools`, and `/obs agents` labels, rows, and notification output with visible Unicode-safe truncation.
+- Unified `/obs session`, `/obs trace`, and `/obs link` on one validated trace-link builder with canonical placeholders, structured Grafana fallback URLs, and bounded configuration diagnostics.
+- Made multi-signal OpenTelemetry startup transactional with bounded rollback, a terminal failed controller state, sanitized Pi diagnostics, and clean later-session recovery.
+- Preserved malformed environment and file configuration as bounded rejection diagnostics, with source-specific `/obs status` reporting and strict trusted `.env` parsing.
+- Enforced coherent terminal subagent transitions across the integration API, agent tree, spans, events, metrics, runtime state, waits, and joins.
+- Enforced production acknowledgement for Grafana and OTLP certificate-verification bypasses, wired retained OTLP TLS behavior into every exporter, and exposed effective transport security in `/obs status` and `/obs health`.
+- Kept live-session and backfill OpenTelemetry providers scoped to ObservMe instead of replacing process-global providers.
+- Redacted complete supported PEM private-key blocks, including malformed/truncated input, across live and backfilled content capture.
+- Bounded all Grafana health and datasource response bodies before JSON parsing across default, custom Node, and injected fetch transports.
+- Enforced top-level ObservMe disablement across lifecycle startup, runtime state, integration availability, and all OpenTelemetry signal factories.
+- Restored clean npm dependency resolution by pinning TypeScript to the latest release supported by `typescript-eslint`.
+- Integrated active-agent lease activation and deactivation with clean shutdown, duplicate/reload replacement, resume, and failed-start cleanup so final flushes cannot renew stale activity.
 - Fixed session, workflow, agent, turn, LLM, tool, Bash, histogram, active-span, failure/recovery, and session-count telemetry accuracy.
 - Fixed lifecycle serialization, duplicate session replacement, bounded shutdown/flush behavior, backfill cancellation, parallel tool correlation, and bounded agent-tree state cleanup.
 - Hardened configuration validation, project-root path confinement, endpoint security, custom redaction patterns, tenant-salted hashing, and environment propagation.

package/README.md

@@ -62,7 +62,7 @@ This checkout implements the ObservMe MVP scope described by the packaged [`docs
 
 - Extension load checks and `/obs health` backend checks.
 - Session, workflow, agent-run, turn, LLM, tool, bash, subagent-spawn, wait/join, compaction, branch, model-change, and thinking-level telemetry.
-- Multi-agent tree metrics for active agents, depth, fan-out, orphan agents, trace-context propagation failures, child failures, and workflow duration.
+- Lease-qualified multi-agent activity plus tree metrics for depth, fan-out, orphan agents, trace-context propagation failures, child failures, and workflow duration; abrupt exits converge without a Collector restart.
 - Session-scoped OTLP trace, metric, and log exporters with bounded queues, timeouts, and shutdown flushing.
 - Configurable capture controls, redaction, path scrubbing, hashing, truncation, and high-cardinality metric-label guards.
 - Grafana dashboard JSON, alert rules, SLO definitions, Collector examples, and compatibility matrix artifacts.
@@ -162,12 +162,28 @@ const started = observme?.startSubagent({
 });
 
 if (started?.ok) {
-  await launchChildPi({ env: started.env });
-  observme.completeSubagent(started.spawnId, { childStatus: "active" });
+  let child;
+  try {
+    child = await launchChildPi({ env: started.env });
+  } catch (error) {
+    observme.failSubagent(started.spawnId, {
+      childAgentId: started.childAgentId,
+      errorClass: error instanceof Error ? error.name : "launcher_error",
+    });
+  }
+
+  if (child) {
+    const result = await waitForChildPi(child);
+    observme.completeSubagent(started.spawnId, {
+      childAgentId: started.childAgentId,
+      childStatus: result.status,
+      outcome: result.status,
+    });
+  }
 }
 ```
 
-The discovery helper fails open when the event bus or a provider is malformed. API methods also reject unsafe/oversized requests and duplicate active lifecycle IDs without replacing existing telemetry state. Handle every discriminated failure result locally and keep orchestration functional.
+The discovery helper fails open when the event bus or a provider is malformed. API methods also reject unsafe/oversized requests, duplicate active lifecycle IDs, and active or retained child-placeholder collisions without replacing existing telemetry state. Handle every discriminated failure result locally and keep orchestration functional.
 
 Use [`docs/extension-integration.md`](docs/extension-integration.md) for the complete lifecycle, validation limits, and failure contract. The shipped [`examples/integrations/subagent-runner.ts`](examples/integrations/subagent-runner.ts) wraps a generic child transport, while [`docs/agent-subagent-observability-requirements.md`](docs/agent-subagent-observability-requirements.md) covers the larger orchestration design.
 
@@ -230,10 +246,17 @@ Metric labels are intentionally low-cardinality. Session, workflow, agent, trace
 | Histogram | Workflow and agent tree | `observme_workflow_duration_ms`<br>`observme_agent_run_duration_ms`<br>`observme_subagent_spawn_duration_ms`<br>`observme_agent_wait_duration_ms`<br>`observme_agent_join_duration_ms`<br>`observme_agent_tree_depth`<br>`observme_agent_tree_width`<br>`observme_agent_fanout_count` | Workflow/agent latency and multi-agent tree shape. |
 | Histogram | Operation latency | `observme_turn_duration_ms`<br>`observme_llm_request_duration_ms`<br>`observme_tool_duration_ms`<br>`observme_bash_duration_ms`<br>`observme_handler_duration_ms` | End-to-end operation and extension-handler duration in milliseconds. |
 | Histogram | Context and payload size | `observme_compaction_tokens_before`<br>`observme_prompt_size_chars`<br>`observme_response_size_chars`<br>`observme_tool_result_size_chars` | Context size before compaction and payload sizes in characters. Size metrics do not contain payload content. |
-| Up-down counter | Current activity | `observme_active_spans`<br>`observme_active_agents` | Active operations and agents for the current exported stream. |
+| Up-down counter | Current activity | `observme_active_spans`<br>`observme_active_agents` | Active operations and the compatibility lifecycle claim for agents in the current exported stream. |
+| Asynchronous gauge | Agent liveness lease | `observme_agent_lease_expires_unixtime_seconds` | Absolute Unix expiry renewed on each metric collection while the session is active; operational active-agent queries join it to the positive lifecycle claim. |
 | Registered, not yet recorded | Agent lifetime | `observme_agent_lifetime_duration_ms` | Reserved instrument; the current live handlers do not record measurements yet. |
 | Registered, not yet recorded | Official GenAI compatibility | `gen_ai.client.token.usage`<br>`gen_ai.client.operation.duration` | Reserved OpenTelemetry GenAI instruments; use the `observme_llm_*` metrics above for current data. |
 
+### Active-agent liveness
+
+`observme_active_agents` remains a clean-start/clean-shutdown lifecycle claim, but raw sums are not authoritative after a crash, `SIGKILL`, cancelled GitHub Actions job, or lost runner. The shipped dashboards and alerts count an instance only when that claim is positive and its `observme_agent_lease_expires_unixtime_seconds` value is current and within the supported future horizon. Clean shutdown reaches zero after export/scrape propagation; an ungraceful exit converges within the configured lease plus up to 5 seconds of supported clock skew and one Prometheus scrape/evaluation interval (80 seconds with the default 60-second lease and a 15-second scrape).
+
+Missing, expired, malformed, or pathologically future leases fail closed. GitHub-hosted runners satisfy the clock requirement; self-hosted runners must keep the producer and Prometheus clocks synchronized within 5 seconds. Collector restart is not required for convergence, even when its Prometheus exporter continues to expose a cached raw claim. Migrate custom panels, alerts, and recording rules from raw `sum(observme_active_agents)` queries to the [canonical lease-aware PromQL](docs/reference/09-dashboards-alerts-slos.md#131-canonical-active-agent-promql), and use raw/expired claims only for diagnostics.
+
 ### Traces
 
 | Span name | Created for | Notes |
@@ -261,7 +284,7 @@ All operational logs use short event bodies plus structured attributes. Correlat
 | Configuration, session, and workflow | `config.rejected`<br>`session.started`<br>`session.shutdown`<br>`session.duplicate_start`<br>`workflow.started`<br>`workflow.completed`<br>`workflow.failed` | Configuration diagnostics and top-level lifecycle/outcome events. |
 | Reserved session events | `session.named`<br>`session.error` | Public event names reserved for compatibility; current live handlers do not emit them. |
 | Agent run | `agent.run.started`<br>`agent.run.completed`<br>`agent.run.failed` | Agent-run lifecycle and outcome. |
-| Subagent lineage | `agent.spawn.started`<br>`agent.spawn.completed`<br>`agent.spawn.failed`<br>`agent.wait.started`<br>`agent.wait.completed`<br>`agent.join.started`<br>`agent.join.completed`<br>`agent.orphaned`<br>`trace_context.propagation_failed` | Parent/child lifecycle, lineage gaps, propagation failures, and join outcomes. |
+| Subagent lineage | `agent.spawn.started`<br>`agent.spawn.completed`<br>`agent.spawn.failed`<br>`agent.spawn.cancelled`<br>`agent.wait.started`<br>`agent.wait.completed`<br>`agent.join.started`<br>`agent.join.completed`<br>`agent.orphaned`<br>`trace_context.propagation_failed` | Parent/child lifecycle, lineage gaps, propagation failures, and join outcomes. |
 | Turn | `turn.started`<br>`turn.completed` | Turn lifecycle with run/turn correlation. |
 | LLM lifecycle | `llm.request.started`<br>`llm.request.completed`<br>`llm.request.failed` | Content-free provider request lifecycle, usage, cost, stop reason, and bounded errors. |
 | LLM content | `llm.prompt.captured`<br>`llm.response.captured`<br>`llm.thinking.captured` | Emitted only when the corresponding prompt, response, or thinking capture flag is enabled and redaction succeeds. |
@@ -280,7 +303,7 @@ ObservMe supports layered configuration with this precedence:
 defaults → global ~/.pi/agent/observme.yaml → trusted project config → trusted project .env → system environment variables → runtime options
 ```
 
-Factory-safe loading uses defaults/global/system-environment/runtime options only. Session-scoped loading can add trusted project config and a project-local `.env` when Pi marks the project trusted. `/obs status` reports the effective config source, whether project-local config was loaded, skipped because the project is untrusted, or missing, plus bounded rejection issue codes, the configured Grafana URL, and query-readiness status without rendering tokens or passwords. In untrusted projects, ObservMe does not read project-local config or `.env` files and uses safe defaults/global/system-environment layers instead. Invalid or unsafe configuration emits a bounded `config.rejected` diagnostic and falls back safely without exposing rejected values.
+Factory-safe loading uses defaults/global/system-environment/runtime options only. Session-scoped loading can add trusted project config and a project-local `.env` when Pi marks the project trusted. These project files must remain inside the stable canonical project root: in-root symlinks are supported, while out-of-root, dangling, replaced, or unverifiable paths fail closed through identity-verified file I/O. `/obs status` reports the effective config source, whether project-local config was loaded, skipped because the project is untrusted, missing, or rejected, plus bounded rejection issue codes, effective OTLP/Grafana transport security, the configured Grafana URL, and query-readiness status without rendering tokens, passwords, canonical targets, or external paths. In untrusted projects, ObservMe does not read project-local config or `.env` files and uses safe defaults/global/system-environment layers instead. Invalid or unsafe configuration emits a bounded `config.rejected` diagnostic and falls back safely without exposing rejected values.
 
 Cross-process agent lineage has a separate boundary: only the Pi process environment available to the shipped extension, or explicit runtime options for controlled embedders, is eligible for parent provenance. Project-local `.env` values configure ObservMe but cannot establish lineage. A child accepts only a complete validated workflow/parent/root/depth/spawn envelope and valid W3C context; malformed or stale envelopes fail open to a root/orphan fallback with sanitized telemetry.
 
@@ -310,7 +333,7 @@ To show failed-tool output such as GuardMe denial messages in the Tools dashboar
 
 Metadata such as token counts, duration, status, model/provider, tool name, and agent role/depth is captured by default. High-cardinality identifiers (session IDs, workflow IDs, agent IDs, trace/span IDs, entry IDs) are allowed on spans/logs for drill-down but are blocked from Prometheus metric labels.
 
-Grafana-backed query commands use the Grafana HTTP API, so browser login cookies are irrelevant. Configure either a Grafana service-account token (`OBSERVME_GRAFANA_TOKEN`) or local Basic auth (`OBSERVME_GRAFANA_USERNAME`/`OBSERVME_GRAFANA_PASSWORD`); token auth takes precedence and secrets are never rendered in command errors. You can supply these values as system environment variables before starting Pi, or copy the `.env.example` shipped with ObservMe to `.env` in a trusted project; system environment variables override `.env` values.
+Grafana-backed query commands use the Grafana HTTP API, so browser login cookies are irrelevant. Keep `query.grafana.url` credential-free; a base URL containing a username or password is rejected before network I/O with a bounded diagnostic that does not render the URL or credentials. Configure either a Grafana service-account token (`OBSERVME_GRAFANA_TOKEN`) or local Basic auth (`OBSERVME_GRAFANA_USERNAME`/`OBSERVME_GRAFANA_PASSWORD`); token auth takes precedence and secrets are never rendered in command errors. You can supply these values as system environment variables before starting Pi, or copy the `.env.example` shipped with ObservMe to `.env` in a trusted project; system environment variables override `.env` values.
 
 ### Supported local-stack query profile
 
@@ -336,7 +359,7 @@ query:
       preferIPv4: true
 ```
 
-Create a Grafana service-account token in Grafana (Administration → Users and access → Service accounts → Add service account/token; Viewer is enough for read-only datasource queries) and export it as `OBSERVME_GRAFANA_TOKEN`, or for local-only Basic auth read the generated admin password from the repository-only local stack's secrets directory. If you prefer a project-local env file, run `cp .env.example .env`, fill either `OBSERVME_GRAFANA_TOKEN` or `OBSERVME_GRAFANA_PASSWORD`, then restart Pi from that project. The local Collector and Loki profile uses `service.name=observme-pi-extension`; Loki queries use normalized labels such as `service_name`, `pi_session_id`, `pi_agent_id`, `pi_agent_run_id`, `event_name`, and `event_category`. If data is visible in Grafana but `/obs` commands fail, run `/obs health` and check extension env loading, Grafana auth, datasource UIDs, TLS, and DNS details.
+Create a Grafana service-account token in Grafana (Administration → Users and access → Service accounts → Add service account/token; Viewer is enough for read-only datasource queries) and export it as `OBSERVME_GRAFANA_TOKEN`, or for local-only Basic auth read the generated admin password from the repository-only local stack's secrets directory. Certificate verification stays enabled for HTTPS by default. Production rejects OTLP or Grafana certificate-verification bypass unless `privacy.allowInsecureTransport: true` explicitly acknowledges the risk; `otlp.tls.enabled` is not a supported setting because each endpoint URL scheme selects HTTP or HTTPS. If you prefer a project-local env file, run `cp .env.example .env`, fill either `OBSERVME_GRAFANA_TOKEN` or `OBSERVME_GRAFANA_PASSWORD`, then restart Pi from that project. The local Collector and Loki profile uses `service.name=observme-pi-extension`; Loki queries use normalized labels such as `service_name`, `pi_session_id`, `pi_agent_id`, `pi_agent_run_id`, `event_name`, and `event_category`. If data is visible in Grafana but `/obs` commands fail, run `/obs health` and check extension env loading, Grafana auth, datasource UIDs, TLS, and DNS details.
 
 ### Show LLM chat content in Grafana
 
@@ -361,7 +384,7 @@ Full configuration schema: `docs/reference/12-configuration-reference.md`. Full
 - ObservMe does not execute shell commands itself; it only observes tool/bash execution events emitted by Pi.
 - ObservMe never blocks Pi agent execution when the observability backend is degraded or unreachable.
 - ObservMe starts exporters only for a trusted session and shuts them down with bounded timeouts.
-- ObservMe does not continuously tail the full Pi session file; startup recovery reads only minimal session/correlation state, and historical replay requires an explicit `/obs backfill` command.
+- ObservMe does not continuously tail the full Pi session file. Optional correlation persistence uses one bounded `observme.correlation` custom entry on the active Pi branch, never LLM context; historical replay is never automatic and requires an explicit, confirmed `/obs backfill` command.
 - Optional content capture is disabled by default and must pass through the redaction pipeline before export when enabled.
 - `/obs` query commands are read-only and are not imported by telemetry-emission code paths.
 - Invalid or unsafe configuration falls back to safe defaults with bounded `config.rejected`, `/obs status`, and available Pi UI diagnostics; rejected values are never rendered. Intentionally unsafe capture emits a visible warning.
@@ -383,7 +406,7 @@ These assets are included in the npm package:
 
 Open **ObservMe Overview** first for health/SLO chips, workload, cost, latency, agent-lineage status, and time-preserving links to focused dashboards. Use **Trace Journey** to follow a session/workflow/agent/run across Prometheus aggregates, Loki logs, and Tempo traces. Use **LLM Conversations** only for redacted opt-in content; raw prompt, response, command, path, and error-message values should never be placed in dashboard URLs or Prometheus labels. Empty failure tables normally mean no matching failures in the selected range, while optional content panels can be empty because capture is disabled by default.
 
-The full dashboard map, standard variables, drill-down workflow, threshold colors, and zero-state semantics are documented in `docs/reference/09-dashboards-alerts-slos.md`.
+The full dashboard map, canonical lease-aware active-agent queries, raw-query migration, standard variables, drill-down workflow, threshold colors, and zero-state semantics are documented in `docs/reference/09-dashboards-alerts-slos.md`. For an unexpected zero or stale raw claim, follow the active-lease troubleshooting flow in `docs/reference/11-deployment-runbooks.md` and check Export Health before concluding that the producer stopped.
 
 ## Documentation Set
 
@@ -415,8 +438,10 @@ npm run lint:fix    # ESLint auto-fix pass
 npm run format:check
 npm run test
 npm run test:integration:collector
+npm run test:integration:active-agent-lease
 npm run test:integration:grafana-stack
 npm run check:pack
+npm run pack:dry-run
 npm run smoke:pi-runtime
 npm run validate:grafana-obs
 pi --no-extensions -e .

package/dashboards/observme-agent-node-graphs.json

@@ -45,7 +45,7 @@
         "type": "prometheus",
         "uid": "prometheus"
       },
-      "description": "Aggregate Pi lineage topology built only from bounded Prometheus labels; it is not a single trace tree. Numeric main stats are current gauges or selected-range counts. Red nodes and edges identify spawn failures, orphan lineage, and trace-context propagation failures. During healthy idle ranges, only the Pi root may render because no aggregate activity exists; use Agents for ratios and Trace Journey for one execution.",
+      "description": "Aggregate fleet topology built only from bounded Prometheus labels; it is not per-workflow truth or a single trace tree. Role nodes combine canonical leased-current activity with selected-range run/spawn counts, so node and edge values must not be read as one execution. Clean shutdown removes leased activity after export/scrape propagation; after a crash, SIGKILL, cancelled job, or runner loss, leased activity converges to zero within the default 60-second lease plus one Prometheus scrape while raw claims may stay cached. Red nodes and edges identify spawn, lineage, or propagation failures. /obs agents is current-process child state; use Trace Journey for one execution.",
       "fieldConfig": {
         "defaults": {
           "mappings": [],
@@ -88,7 +88,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(vector(1), \"id\", \"pi-root\", \"__name__\", \".*\"), \"title\", \"Pi root\", \"id\", \".*\"), \"subTitle\", \"workflow entry\", \"id\", \".*\"), \"mainStat\", \"entry\", \"id\", \".*\"), \"secondaryStat\", \"aggregate\", \"id\", \".*\"), \"color\", \"blue\", \"id\", \".*\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(observme_active_agents) by (agent_role) or sum(increase(observme_agent_runs_total[$__range])) by (agent_role) or sum(increase(observme_subagents_spawned_total[$__range])) by (agent_role)), \"id\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"title\", \"Agent: $1\", \"agent_role\", \"(.+)\"), \"subTitle\", \"active or observed role\", \"agent_role\", \"(.+)\"), \"mainStat\", \"count\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"role\", \"agent_role\", \"(.+)\"), \"color\", \"green\", \"agent_role\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (subagent_depth), \"id\", \"subagents_depth_$1\", \"subagent_depth\", \"(.+)\"), \"title\", \"Subagents depth $1\", \"subagent_depth\", \"(.+)\"), \"subtitle\", \"spawned in range\", \"subagent_depth\", \"(.+)\"), \"mainStat\", \"spawns\", \"subagent_depth\", \"(.+)\"), \"secondaryStat\", \"depth\", \"subagent_depth\", \"(.+)\"), \"color\", \"orange\", \"subagent_depth\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) by (error_class) > 0), \"id\", \"spawn_failure_$1\", \"error_class\", \"(.+)\"), \"title\", \"Spawn failure: $1\", \"error_class\", \"(.+)\"), \"subtitle\", \"failed spawns\", \"error_class\", \"(.+)\"), \"mainStat\", \"failures\", \"error_class\", \"(.+)\"), \"secondaryStat\", \"error\", \"error_class\", \"(.+)\"), \"color\", \"red\", \"error_class\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_orphan_agents_total[$__range])) by (reason) > 0), \"id\", \"orphan_$1\", \"reason\", \"(.+)\"), \"title\", \"Orphan: $1\", \"reason\", \"(.+)\"), \"subtitle\", \"lineage missing\", \"reason\", \"(.+)\"), \"mainStat\", \"orphans\", \"reason\", \"(.+)\"), \"secondaryStat\", \"reason\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_trace_context_propagation_failures_total[$__range])) by (reason) > 0), \"id\", \"propagation_$1\", \"reason\", \"(.+)\"), \"title\", \"Propagation: $1\", \"reason\", \"(.+)\"), \"subtitle\", \"trace context failures\", \"reason\", \"(.+)\"), \"mainStat\", \"failures\", \"reason\", \"(.+)\"), \"secondaryStat\", \"reason\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\")",
+          "expr": "label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(vector(1), \"id\", \"pi-root\", \"__name__\", \".*\"), \"title\", \"Pi root\", \"id\", \".*\"), \"subTitle\", \"workflow entry\", \"id\", \".*\"), \"mainStat\", \"entry\", \"id\", \".*\"), \"secondaryStat\", \"aggregate\", \"id\", \".*\"), \"color\", \"blue\", \"id\", \".*\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum by (agent_role) (max by (observme_instance_id, agent_role) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or sum(increase(observme_agent_runs_total[$__range])) by (agent_role) or sum(increase(observme_subagents_spawned_total[$__range])) by (agent_role)), \"id\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"title\", \"Agent: $1\", \"agent_role\", \"(.+)\"), \"subTitle\", \"active or observed role\", \"agent_role\", \"(.+)\"), \"mainStat\", \"count\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"role\", \"agent_role\", \"(.+)\"), \"color\", \"green\", \"agent_role\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (subagent_depth), \"id\", \"subagents_depth_$1\", \"subagent_depth\", \"(.+)\"), \"title\", \"Subagents depth $1\", \"subagent_depth\", \"(.+)\"), \"subtitle\", \"spawned in range\", \"subagent_depth\", \"(.+)\"), \"mainStat\", \"spawns\", \"subagent_depth\", \"(.+)\"), \"secondaryStat\", \"depth\", \"subagent_depth\", \"(.+)\"), \"color\", \"orange\", \"subagent_depth\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) by (error_class) > 0), \"id\", \"spawn_failure_$1\", \"error_class\", \"(.+)\"), \"title\", \"Spawn failure: $1\", \"error_class\", \"(.+)\"), \"subtitle\", \"failed spawns\", \"error_class\", \"(.+)\"), \"mainStat\", \"failures\", \"error_class\", \"(.+)\"), \"secondaryStat\", \"error\", \"error_class\", \"(.+)\"), \"color\", \"red\", \"error_class\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_orphan_agents_total[$__range])) by (reason) > 0), \"id\", \"orphan_$1\", \"reason\", \"(.+)\"), \"title\", \"Orphan: $1\", \"reason\", \"(.+)\"), \"subtitle\", \"lineage missing\", \"reason\", \"(.+)\"), \"mainStat\", \"orphans\", \"reason\", \"(.+)\"), \"secondaryStat\", \"reason\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_trace_context_propagation_failures_total[$__range])) by (reason) > 0), \"id\", \"propagation_$1\", \"reason\", \"(.+)\"), \"title\", \"Propagation: $1\", \"reason\", \"(.+)\"), \"subtitle\", \"trace context failures\", \"reason\", \"(.+)\"), \"mainStat\", \"failures\", \"reason\", \"(.+)\"), \"secondaryStat\", \"reason\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\")",
           "format": "table",
           "instant": true,
           "legendFormat": "nodes",
@@ -101,7 +101,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(observme_active_agents) by (agent_role) or sum(increase(observme_agent_runs_total[$__range])) by (agent_role)), \"source\", \"pi-root\", \"agent_role\", \"(.+)\"), \"target\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"mainStat\", \"count\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"active/runs\", \"agent_role\", \"(.+)\"), \"color\", \"gray\", \"agent_role\", \"(.+)\"), \"id\", \":\", \"source\", \"target\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (agent_role, subagent_depth, spawn_type, spawn_reason), \"source\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"target\", \"subagents_depth_$1\", \"subagent_depth\", \"(.+)\"), \"mainStat\", \"spawns\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"spawn_type/reason\", \"agent_role\", \"(.+)\"), \"color\", \"gray\", \"agent_role\", \"(.+)\"), \"id\", \":\", \"source\", \"target\", \"spawn_type\", \"spawn_reason\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) by (error_class) > 0), \"source\", \"pi-root\", \"error_class\", \"(.+)\"), \"target\", \"spawn_failure_$1\", \"error_class\", \"(.+)\"), \"mainStat\", \"failures\", \"error_class\", \"(.+)\"), \"secondaryStat\", \"spawn\", \"error_class\", \"(.+)\"), \"color\", \"red\", \"error_class\", \"(.+)\"), \"id\", \":\", \"source\", \"target\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_orphan_agents_total[$__range])) by (reason) > 0), \"source\", \"pi-root\", \"reason\", \"(.+)\"), \"target\", \"orphan_$1\", \"reason\", \"(.+)\"), \"mainStat\", \"orphans\", \"reason\", \"(.+)\"), \"secondaryStat\", \"lineage\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\"), \"id\", \":\", \"source\", \"target\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_trace_context_propagation_failures_total[$__range])) by (reason) > 0), \"source\", \"pi-root\", \"reason\", \"(.+)\"), \"target\", \"propagation_$1\", \"reason\", \"(.+)\"), \"mainStat\", \"failures\", \"reason\", \"(.+)\"), \"secondaryStat\", \"trace context\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\"), \"id\", \":\", \"source\", \"target\")",
+          "expr": "label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum by (agent_role) (max by (observme_instance_id, agent_role) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or sum(increase(observme_agent_runs_total[$__range])) by (agent_role)), \"source\", \"pi-root\", \"agent_role\", \"(.+)\"), \"target\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"mainStat\", \"count\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"active/runs\", \"agent_role\", \"(.+)\"), \"color\", \"gray\", \"agent_role\", \"(.+)\"), \"id\", \":\", \"source\", \"target\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (agent_role, subagent_depth, spawn_type, spawn_reason), \"source\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"target\", \"subagents_depth_$1\", \"subagent_depth\", \"(.+)\"), \"mainStat\", \"spawns\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"spawn_type/reason\", \"agent_role\", \"(.+)\"), \"color\", \"gray\", \"agent_role\", \"(.+)\"), \"id\", \":\", \"source\", \"target\", \"spawn_type\", \"spawn_reason\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) by (error_class) > 0), \"source\", \"pi-root\", \"error_class\", \"(.+)\"), \"target\", \"spawn_failure_$1\", \"error_class\", \"(.+)\"), \"mainStat\", \"failures\", \"error_class\", \"(.+)\"), \"secondaryStat\", \"spawn\", \"error_class\", \"(.+)\"), \"color\", \"red\", \"error_class\", \"(.+)\"), \"id\", \":\", \"source\", \"target\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_orphan_agents_total[$__range])) by (reason) > 0), \"source\", \"pi-root\", \"reason\", \"(.+)\"), \"target\", \"orphan_$1\", \"reason\", \"(.+)\"), \"mainStat\", \"orphans\", \"reason\", \"(.+)\"), \"secondaryStat\", \"lineage\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\"), \"id\", \":\", \"source\", \"target\") or label_join(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_trace_context_propagation_failures_total[$__range])) by (reason) > 0), \"source\", \"pi-root\", \"reason\", \"(.+)\"), \"target\", \"propagation_$1\", \"reason\", \"(.+)\"), \"mainStat\", \"failures\", \"reason\", \"(.+)\"), \"secondaryStat\", \"trace context\", \"reason\", \"(.+)\"), \"color\", \"red\", \"reason\", \"(.+)\"), \"id\", \":\", \"source\", \"target\")",
           "format": "table",
           "instant": true,
           "legendFormat": "edges",
@@ -137,7 +137,7 @@
         "type": "prometheus",
         "uid": "prometheus"
       },
-      "description": "Aggregate delegation topology by bounded agent role, spawn reason, and resulting subagent depth; it is not a single trace tree. Numeric main stats are selected-range counts. Red health nodes and edges summarize spawn failures by emitted error class because the failure counter does not carry spawn reason. An empty graph is a healthy idle state when no spawns or failures occurred; use Agents for aggregate analysis and Trace Journey for one execution.",
+      "description": "Aggregate fleet delegation topology by bounded role, spawn reason, and resulting depth; it is not per-workflow truth or a single trace tree. Role nodes may combine canonical leased-current activity with selected-range run/spawn counts, while other values are selected-range aggregates. Clean shutdown removes leased activity after export/scrape propagation; after an ungraceful exit, leased activity converges to zero within the default 60-second lease plus one Prometheus scrape while raw claims may stay cached. Red health nodes and edges summarize spawn failures. /obs agents is local current-process child state; use Trace Journey for one execution.",
       "fieldConfig": {
         "defaults": {
           "mappings": [],
@@ -180,7 +180,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(observme_active_agents) by (agent_role) or sum(increase(observme_agent_runs_total[$__range])) by (agent_role) or sum(increase(observme_subagents_spawned_total[$__range])) by (agent_role)), \"id\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"title\", \"Agent: $1\", \"agent_role\", \"(.+)\"), \"subTitle\", \"active or observed role\", \"agent_role\", \"(.+)\"), \"mainStat\", \"count\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"role\", \"agent_role\", \"(.+)\"), \"color\", \"green\", \"agent_role\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (spawn_reason), \"id\", \"spawn_reason_$1\", \"spawn_reason\", \"(.+)\"), \"title\", \"Spawn: $1\", \"spawn_reason\", \"(.+)\"), \"subTitle\", \"spawn reason\", \"spawn_reason\", \"(.+)\"), \"mainStat\", \"spawns\", \"spawn_reason\", \"(.+)\"), \"secondaryStat\", \"reason\", \"spawn_reason\", \"(.+)\"), \"color\", \"purple\", \"spawn_reason\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (subagent_depth), \"id\", \"subagents_depth_$1\", \"subagent_depth\", \"(.+)\"), \"title\", \"Subagents depth $1\", \"subagent_depth\", \"(.+)\"), \"subTitle\", \"spawned in range\", \"subagent_depth\", \"(.+)\"), \"mainStat\", \"spawns\", \"subagent_depth\", \"(.+)\"), \"secondaryStat\", \"depth\", \"subagent_depth\", \"(.+)\"), \"color\", \"orange\", \"subagent_depth\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) > 0), \"id\", \"spawn-health\", \"__name__\", \".*\"), \"title\", \"Spawn failure health\", \"id\", \".*\"), \"subTitle\", \"failure-only aggregate\", \"id\", \".*\"), \"mainStat\", \"failures\", \"id\", \".*\"), \"secondaryStat\", \"health\", \"id\", \".*\"), \"color\", \"red\", \"id\", \".*\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) by (error_class) > 0), \"id\", \"spawn_failure_$1\", \"error_class\", \"(.+)\"), \"title\", \"Spawn failure: $1\", \"error_class\", \"(.+)\"), \"subTitle\", \"failed spawns\", \"error_class\", \"(.+)\"), \"mainStat\", \"failures\", \"error_class\", \"(.+)\"), \"secondaryStat\", \"error class\", \"error_class\", \"(.+)\"), \"color\", \"red\", \"error_class\", \"(.+)\")",
+          "expr": "label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum by (agent_role) (max by (observme_instance_id, agent_role) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or sum(increase(observme_agent_runs_total[$__range])) by (agent_role) or sum(increase(observme_subagents_spawned_total[$__range])) by (agent_role)), \"id\", \"agent_$1\", \"agent_role\", \"(.+)\"), \"title\", \"Agent: $1\", \"agent_role\", \"(.+)\"), \"subTitle\", \"active or observed role\", \"agent_role\", \"(.+)\"), \"mainStat\", \"count\", \"agent_role\", \"(.+)\"), \"secondaryStat\", \"role\", \"agent_role\", \"(.+)\"), \"color\", \"green\", \"agent_role\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (spawn_reason), \"id\", \"spawn_reason_$1\", \"spawn_reason\", \"(.+)\"), \"title\", \"Spawn: $1\", \"spawn_reason\", \"(.+)\"), \"subTitle\", \"spawn reason\", \"spawn_reason\", \"(.+)\"), \"mainStat\", \"spawns\", \"spawn_reason\", \"(.+)\"), \"secondaryStat\", \"reason\", \"spawn_reason\", \"(.+)\"), \"color\", \"purple\", \"spawn_reason\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace(sum(increase(observme_subagents_spawned_total[$__range])) by (subagent_depth), \"id\", \"subagents_depth_$1\", \"subagent_depth\", \"(.+)\"), \"title\", \"Subagents depth $1\", \"subagent_depth\", \"(.+)\"), \"subTitle\", \"spawned in range\", \"subagent_depth\", \"(.+)\"), \"mainStat\", \"spawns\", \"subagent_depth\", \"(.+)\"), \"secondaryStat\", \"depth\", \"subagent_depth\", \"(.+)\"), \"color\", \"orange\", \"subagent_depth\", \"(.+)\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) > 0), \"id\", \"spawn-health\", \"__name__\", \".*\"), \"title\", \"Spawn failure health\", \"id\", \".*\"), \"subTitle\", \"failure-only aggregate\", \"id\", \".*\"), \"mainStat\", \"failures\", \"id\", \".*\"), \"secondaryStat\", \"health\", \"id\", \".*\"), \"color\", \"red\", \"id\", \".*\") or label_replace(label_replace(label_replace(label_replace(label_replace(label_replace((sum(increase(observme_subagent_spawn_failures_total[$__range])) by (error_class) > 0), \"id\", \"spawn_failure_$1\", \"error_class\", \"(.+)\"), \"title\", \"Spawn failure: $1\", \"error_class\", \"(.+)\"), \"subTitle\", \"failed spawns\", \"error_class\", \"(.+)\"), \"mainStat\", \"failures\", \"error_class\", \"(.+)\"), \"secondaryStat\", \"error class\", \"error_class\", \"(.+)\"), \"color\", \"red\", \"error_class\", \"(.+)\")",
           "format": "table",
           "instant": true,
           "legendFormat": "nodes",

package/dashboards/observme-agents.json

@@ -1065,7 +1065,7 @@
         "type": "prometheus",
         "uid": "prometheus"
       },
-      "description": "Current active-agent gauge grouped by the emitted agent_role label, plus the global total. The tunable total=100 reference matches ObservMeActiveAgentsStuckHigh; a missing gauge is unknown telemetry, not healthy zero.",
+      "description": "Canonical leased-active agents grouped by emitted agent_role, plus the fleet total. Clean shutdown reaches zero after export/scrape propagation; after a crash, SIGKILL, cancelled job, or runner loss, the leased total converges to zero within the default 60-second lease plus one Prometheus scrape even when raw claims stay cached. Zero can also mean healthy idle or a failed-closed lease, so compare Export Health. /obs agents reports current-process child state, not this aggregate fleet view. The total=100 reference matches the deployment-tunable ObservMeActiveAgentsStuckHigh threshold.",
       "fieldConfig": {
         "defaults": {
           "color": {
@@ -1115,7 +1115,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "sum(observme_active_agents) by (agent_role)",
+          "expr": "sum by (agent_role) (max by (observme_instance_id, agent_role) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or on() vector(0)",
           "range": true,
           "refId": "A",
           "legendFormat": "{{agent_role}}"
@@ -1126,7 +1126,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "sum(observme_active_agents)",
+          "expr": "sum(max by (observme_instance_id) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or vector(0)",
           "range": true,
           "refId": "B",
           "legendFormat": "total active agents"
@@ -1697,6 +1697,171 @@
         "x": 0,
         "y": 57
       },
+      "id": 105,
+      "panels": [],
+      "title": "Lease health diagnostics",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "description": "Diagnostic positive lifecycle claims before lease qualification. These are not live-agent totals and may remain cached after a crash, SIGKILL, cancelled job, or runner loss; leased activity converges to zero within the default 60-second lease plus one Prometheus scrape. Clean shutdown exports zero immediately. Compare Expired active claims and Export Health. /obs agents reports only current-process child state, not fleet liveness.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 58
+      },
+      "id": 22,
+      "links": [
+        {
+          "targetBlank": false,
+          "title": "Open Export Health",
+          "url": "/d/observme-export-health/observme-export-health?${__url_time_range}"
+        }
+      ],
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showPercentChange": false,
+        "textMode": "auto",
+        "wideLayout": true
+      },
+      "pluginVersion": "11.1.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum(max by (observme_instance_id) (observme_active_agents > 0)) or vector(0)",
+          "instant": true,
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Raw active claims (diagnostic)",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "prometheus"
+      },
+      "description": "Positive raw claims whose lease has expired. They are stale diagnostics and never contribute to leased active totals; clean shutdown normally avoids them, while ungraceful exits appear after the default 60-second lease plus one Prometheus scrape and may remain until Collector cleanup. Yellow at 1 and red at 5 are deployment-tunable. Compare Raw active claims and Export Health. /obs agents reports only current-process child state, not fleet liveness.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 58
+      },
+      "id": 23,
+      "links": [
+        {
+          "targetBlank": false,
+          "title": "Open Export Health",
+          "url": "/d/observme-export-health/observme-export-health?${__url_time_range}"
+        }
+      ],
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showPercentChange": false,
+        "textMode": "auto",
+        "wideLayout": true
+      },
+      "pluginVersion": "11.1.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "editorMode": "code",
+          "expr": "sum(max by (observme_instance_id) ((observme_active_agents > 0) and on (observme_instance_id) (observme_agent_lease_expires_unixtime_seconds <= time()))) or vector(0)",
+          "instant": true,
+          "range": false,
+          "refId": "A"
+        }
+      ],
+      "title": "Expired active claims (diagnostic)",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 66
+      },
       "id": 104,
       "panels": [],
       "title": "Lineage logs",
@@ -1768,7 +1933,7 @@
         "h": 10,
         "w": 24,
         "x": 0,
-        "y": 58
+        "y": 67
       },
       "id": 6,
       "options": {

package/dashboards/observme-alerts.yaml

@@ -103,11 +103,24 @@ groups:
           runbook: "Severity guidance from production docs: warning."
       - alert: ObservMeActiveAgentsStuckHigh
         expr: >-
-          sum(observme_active_agents) > 100
+          (sum(max by (observme_instance_id) ((observme_active_agents > 0) and on (observme_instance_id)
+          ((observme_agent_lease_expires_unixtime_seconds > time()) and
+          (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or vector(0)) > 100
         for: 30m
         labels:
           severity: deployment-dependent
         annotations:
           summary: Active agents stuck high
-          description: ObservMe active-agent gauge remains above 100.
-          runbook: "Severity guidance from production docs: depends on normal fleet size; tune per deployment."
+          description: ObservMe leased active-agent count remains above 100; expired, missing, and pathological leases fail closed.
+          runbook: "Severity guidance from production docs: depends on normal fleet size; tune per deployment. Compare leased activity with raw and expired claims plus Export Health."
+      - alert: ObservMeExpiredActiveAgentClaims
+        expr: >-
+          (sum(max by (observme_instance_id) ((observme_active_agents > 0) and on (observme_instance_id)
+          (observme_agent_lease_expires_unixtime_seconds <= time()))) or vector(0)) > 5
+        for: 15m
+        labels:
+          severity: deployment-dependent
+        annotations:
+          summary: Expired active-agent claims detected
+          description: More than five positive lifecycle claims have expired leases for 15 minutes; they are stale diagnostics and excluded from leased active totals.
+          runbook: "Deployment-tunable diagnostic: compare Raw active claims, Expired active claims, leased Active agents, and Export Health; investigate abrupt producer death, export gaps, and Collector cleanup. /obs agents is local child state, not fleet truth."

package/dashboards/observme-overview.json

@@ -1075,7 +1075,7 @@
         "type": "prometheus",
         "uid": "prometheus"
       },
-      "description": "Currently active ObservMe agent runtimes. Zero can mean healthy idle. Red at 100 references the tunable ObservMeActiveAgentsStuckHigh default; yellow at 50 is an early-warning assumption. Compare sustained values with fan-out and depth.",
+      "description": "Canonical leased-active ObservMe runtimes. Clean shutdown reaches zero after export/scrape propagation; after a crash, SIGKILL, cancelled job, or runner loss, the leased total converges to zero within the default 60-second lease plus one Prometheus scrape even if raw claims stay cached. Zero can also mean healthy idle or a failed-closed lease, so check Export Health. /obs agents reports only current-process child state, not fleet liveness. Red at 100 matches the deployment-tunable ObservMeActiveAgentsStuckHigh threshold; yellow at 50 is an early warning.",
       "fieldConfig": {
         "defaults": {
           "color": {
@@ -1141,7 +1141,7 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "sum(observme_active_agents) or vector(0)",
+          "expr": "sum(max by (observme_instance_id) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or vector(0)",
           "range": true,
           "refId": "A"
         }
@@ -1998,7 +1998,7 @@
         "type": "prometheus",
         "uid": "prometheus"
       },
-      "description": "Current active agents by bounded role and depth. No rows can mean healthy idle or an absent active-agent gauge; compare with the ungrouped Active agents chip and Export Health before treating it as healthy.",
+      "description": "Canonical leased-active agents by emitted bounded role and environment. Clean shutdown reaches zero after export/scrape propagation; after an ungraceful exit, leased totals converge to zero within the default 60-second lease plus one Prometheus scrape while raw claims may remain cached. Depth is omitted because active signals do not emit subagent_depth. An unlabeled zero means healthy idle or no qualifying lease when Prometheus is reachable; compare Export Health. /obs agents is current-process child state, not this aggregate fleet view.",
       "fieldConfig": {
         "defaults": {
           "color": {
@@ -2058,14 +2058,14 @@
             "uid": "prometheus"
           },
           "editorMode": "code",
-          "expr": "sum(observme_active_agents) by (agent_role, subagent_depth)",
+          "expr": "sum by (agent_role, environment) (max by (observme_instance_id, agent_role, environment) ((observme_active_agents > 0) and on (observme_instance_id) ((observme_agent_lease_expires_unixtime_seconds > time()) and (observme_agent_lease_expires_unixtime_seconds <= time() + 305)))) or on() vector(0)",
           "instant": true,
-          "legendFormat": "{{agent_role}} depth={{subagent_depth}}",
+          "legendFormat": "{{agent_role}} / {{environment}}",
           "range": false,
           "refId": "A"
         }
       ],
-      "title": "Active agents by role/depth",
+      "title": "Active agents by role/environment",
       "type": "bargauge"
     },
     {