@senad-d/observme 0.1.3 → 0.1.5

package/docs/reference/09-dashboards-alerts-slos.md

@@ -129,9 +129,126 @@ Zero-state interpretation checklist:
 - Empty failure-only Loki tables mean no matching failures in the selected range, not that the log pipeline is broken.
 - `or vector(0)` stats must say whether zero means healthy idle/no activity or a true zero failure ratio.
 - Optional redacted content panels can be empty when capture is disabled or when the Collector dropped content attributes by policy.
-- Active-session estimates are approximate across process restarts/counter resets; `observme_active_agents` is the authoritative active agent gauge.
+- Active-session estimates are approximate across process restarts and counter resets. Raw `observme_active_agents` values are lifecycle claims, not authoritative liveness after an ungraceful exit.
 - High-cardinality execution identifiers are allowed as Loki/Tempo filter variables only and must never be introduced into Prometheus queries or panel links from aggregate panels.
 
+### 1.3 Active-agent leased-state contract
+
+The authoritative metric, duration, export-interval relationship, clock-skew allowance, future-timestamp horizon, convergence target, and failure-mode behavior are defined once in [`04-telemetry-semantic-conventions.md` §12.4.1](04-telemetry-semantic-conventions.md#1241-active-agent-lease-and-clock-contract). Dashboards, alerts, examples, and tests must use that contract without introducing panel-local lease or skew windows.
+
+Operational active-agent views combine a positive `observme_active_agents` lifecycle claim with a currently valid `observme_agent_lease_expires_unixtime_seconds` value on the generated `observme_instance_id` resource label. The instance label is an internal join/deduplication key only: it must not be exposed as a dashboard variable, group, legend, URL value, or topology identity. Duplicate scrape/exporter representations of one instance count once.
+
+Interpret active-agent states as follows:
+
+- positive claim plus a valid current lease: active;
+- clean-shutdown claim of zero: inactive after normal export/scrape propagation, without waiting for lease expiry;
+- positive claim plus an expired lease: inactive stale claim, available only as a diagnostic;
+- missing, malformed, non-finite, or pathologically future lease: inactive (fail closed);
+- no matching series: zero-safe healthy idle/no activity when the telemetry backend itself is reachable.
+
+A crash, `SIGKILL`, cancelled job, or lost runner therefore converges to zero within the semantic-convention bound even if the Collector continues to expose the cached raw claim. A Collector/export outage can intentionally undercount after the last delivered lease expires; use Export Health and backend reachability to distinguish that state from process termination. Clean lifecycle handling and GitHub Actions `if: always()` cleanup remain useful latency optimizations, but neither is the correctness mechanism.
+
+### 1.3.1 Canonical active-agent PromQL
+
+Use the following total everywhere current live-agent state is required:
+
+```promql
+sum(
+  max by (observme_instance_id) (
+    (observme_active_agents > 0)
+    and on (observme_instance_id)
+    (
+      (observme_agent_lease_expires_unixtime_seconds > time())
+      and
+      (observme_agent_lease_expires_unixtime_seconds <= time() + 305)
+    )
+  )
+) or vector(0)
+```
+
+Both comparisons are required. The lower bound expires stale claims; the upper bound rejects leases beyond the frozen 305-second future horizon. `and on (observme_instance_id)` fails closed when the lease is absent or non-comparable, and `max by (observme_instance_id)` removes duplicate Collector/exporter/scrape replicas before the outer sum. The final `or vector(0)` makes no-series results render as zero; when Prometheus and Export Health are reachable, zero can mean healthy idle, clean shutdown, an expired claim, or a lease that failed closed.
+
+For a bounded emitted dimension, retain that dimension in both aggregations. Grouped queries use `or on() vector(0)` so the unlabeled zero appears only when no labeled breakdown exists. Canonical role and environment breakdowns are:
+
+```promql
+sum by (agent_role) (
+  max by (observme_instance_id, agent_role) (
+    (observme_active_agents > 0)
+    and on (observme_instance_id)
+    (
+      (observme_agent_lease_expires_unixtime_seconds > time())
+      and
+      (observme_agent_lease_expires_unixtime_seconds <= time() + 305)
+    )
+  )
+) or on() vector(0)
+```
+
+```promql
+sum by (environment) (
+  max by (observme_instance_id, environment) (
+    (observme_active_agents > 0)
+    and on (observme_instance_id)
+    (
+      (observme_agent_lease_expires_unixtime_seconds > time())
+      and
+      (observme_agent_lease_expires_unixtime_seconds <= time() + 305)
+    )
+  )
+) or on() vector(0)
+```
+
+The shipped active claim and lease currently emit `agent_role` and `environment`; they do not emit the metric label `subagent_depth`. Therefore current-active depth panels must not group by `subagent_depth`. For a compatible pipeline that actually emits the same bounded `subagent_depth` on the active claim, this is the supported depth shape:
+
+```promql
+sum by (subagent_depth) (
+  max by (observme_instance_id, subagent_depth) (
+    (observme_active_agents > 0)
+    and on (observme_instance_id)
+    (
+      (observme_agent_lease_expires_unixtime_seconds > time())
+      and
+      (observme_agent_lease_expires_unixtime_seconds <= time() + 305)
+    )
+  )
+) or on() vector(0)
+```
+
+Do not substitute resource-derived or high-cardinality execution labels for these dimensions. Use the bounded spawn/tree metrics for depth views in the shipped dashboards until `subagent_depth` is emitted on the active signals themselves.
+
+Raw positive lifecycle claims are diagnostic only:
+
+```promql
+sum(max by (observme_instance_id) (observme_active_agents > 0)) or vector(0)
+```
+
+Expired positive claims are diagnostic and never contribute to the live total:
+
+```promql
+sum(
+  max by (observme_instance_id) (
+    (observme_active_agents > 0)
+    and on (observme_instance_id)
+    (observme_agent_lease_expires_unixtime_seconds <= time())
+  )
+) or vector(0)
+```
+
+Compare raw claims, expired claims, the canonical live total, and Export Health when diagnosing a producer crash or export gap. A raw-minus-live difference can also include a missing, malformed, or pathologically future lease, so it must not be labeled strictly as “expired.” Never expose `observme_instance_id` as a variable or legend in these diagnostics.
+
+### 1.3.2 Migration from raw active-agent queries
+
+Use this rollout order for custom dashboards, recording rules, alerts, and API consumers:
+
+1. Deploy ObservMe producers that emit both `observme_active_agents` and `observme_agent_lease_expires_unixtime_seconds` with the same generated `observme_instance_id` resource label.
+2. Confirm the lease metric is present and clocks are synchronized before changing consumers. A missing lease fails closed, so a mixed-version producer without the new metric will not appear in the leased total.
+3. Replace raw `sum(observme_active_agents)`, raw role sums, and any equivalent positive-claim aggregation with the canonical total or bounded-dimension form above. Preserve the future-horizon check, instance join, deduplication, and zero-safe fallback together.
+4. Keep raw and expired-claim queries only in panels or runbooks explicitly labeled as diagnostics. Do not use them for paging, capacity decisions, or current topology.
+5. Keep Collector `metric_expiration` longer than the lease. Do not shorten expiration or schedule Collector restarts to correct liveness; an expired lease stops counting while the cached raw claim remains available for diagnosis.
+6. During rollback, old raw consumers continue to see the compatibility lifecycle claim, but they regain the known ghost-agent behavior after ungraceful exits. Treat rollback as a temporary loss of crash-safe liveness semantics.
+
+Validate the migrated consumer with live+valid, clean-shutdown, expired, missing, pathologically future, duplicate-replica, and no-series cases before production use.
+
 ## 2. Overview Dashboard
 
 Purpose: act as the operator landing page. Users should be able to answer whether ObservMe is healthy, workload is normal, agents/subagents are healthy, spend or latency is high, and which focused dashboard to open next without scanning every raw metric series.
@@ -584,10 +701,22 @@ Severity: warning.
 ### Active Agents Stuck High
 
 ```promql
-sum(observme_active_agents) > 100
+(
+  sum(
+    max by (observme_instance_id) (
+      (observme_active_agents > 0)
+      and on (observme_instance_id)
+      (
+        (observme_agent_lease_expires_unixtime_seconds > time())
+        and
+        (observme_agent_lease_expires_unixtime_seconds <= time() + 305)
+      )
+    )
+  ) or vector(0)
+) > 100
 ```
 
-Severity depends on normal fleet size; tune per deployment.
+Severity depends on normal fleet size; tune per deployment. This alert uses leased live agents, not cached raw claims.
 
 ## 12. SLOs
 

package/docs/reference/10-testing-release-operations.md

@@ -14,7 +14,9 @@ ObservMe must be tested at five levels:
 
 Required areas:
 
-- Config loading and precedence
+- Config loading, precedence, and active-agent lease/export-interval bounds
+- Active-agent lease renewal, deactivation, disposal, clock movement, and clean/abrupt lifecycle behavior
+- Canonical lease-aware PromQL, replica deduplication, and missing/expired/future lease cases
 - Redaction patterns
 - Hashing stability
 - Truncation behavior
@@ -123,6 +125,29 @@ Tests:
 - Prometheus metric query for token totals
 - Grafana dashboard import validates
 
+### 6.1 Active-agent lease integration
+
+Run the focused cached-exporter test separately from the broad Grafana stack:
+
+```bash
+npm run test:integration:active-agent-lease
+```
+
+The test starts pinned Collector and Prometheus containers, then runs real Node telemetry producers with a test-safe 2-second export interval, 10-second lease, and 1-second Prometheus scrape interval. It validates clean shutdown and `SIGTERM` first, then kills a producer with `SIGKILL` and requires all of the following:
+
+- the raw `observme_active_agents` claim remains positive across fresh Prometheus scrapes;
+- the canonical lease-aware count reaches zero without a shutdown event;
+- convergence stays within the documented 10-second lease, 5-second clock-skew allowance, and one 1-second scrape interval;
+- the Collector container identity and zero restart count remain unchanged.
+
+The test uses no credentials or content capture. Its parent process owns bounded waits and unconditional cleanup for child processes, containers, anonymous storage, and the dedicated Docker network after success, assertion failure, startup failure, or timeout. A one-minute test-only `metric_expiration` deliberately remains much longer than the lease so exporter caching reproduces the old raw-query failure; production cleanup guidance remains five minutes for the default one-minute lease.
+
+### 6.2 GitHub Actions cancellation validation
+
+The `lease-cancellation` job in `.github/workflows/ci.yml` runs lease unit/contract tests and the focused integration on `ubuntu-latest` with a bounded timeout. It simulates cancellation by sending `SIGKILL` to a producer so the workflow can assert convergence; cancelling the validation job itself could not produce evidence about its own result. An `if: always()` step removes labeled Docker resources when GitHub still schedules cleanup, but the test must reach zero through lease expiry before that step runs.
+
+Production workflows should likewise prefer graceful Pi and sidecar shutdown and use `if: always()` cleanup where available. Force cancellation, runner loss, or power loss can bypass both, so neither cleanup nor a shutdown callback may be the active-count correctness mechanism. GitHub-hosted runners meet the synchronized-clock expectation; self-hosted CI must keep the producer and Prometheus clocks within the supported 5-second skew.
+
 ## 7. Failure Tests
 
 ### Collector Down
@@ -228,15 +253,16 @@ Reject or sanitize dynamically generated labels.
 
 ## 10. Release Process
 
-1. Update version.
-2. Run unit tests.
-3. Run integration tests.
-4. Run redaction corpus tests.
-5. Run dashboard JSON validation.
-6. Run Collector config validation.
-7. Generate changelog.
-8. Publish package/artifact.
-9. Tag release.
+1. Update the version and `CHANGELOG.md`.
+2. Run `npm run validate` for source/test typechecks, ESLint, formatting, script checks, unit/contract/dashboard/alert tests, package-content checks, packaged-install smoke, handler smoke, Pi lifecycle smoke, and Pi runtime smoke.
+3. Run `npm run test:integration:collector` against the pinned Collector distribution.
+4. Run `npm run test:integration:active-agent-lease` and require clean shutdown, `SIGTERM`, and `SIGKILL` convergence without Collector restart while the raw claim remains cached.
+5. Run `npm run test:integration:grafana-stack` for live backends and dashboard provisioning. Record any unrelated pre-existing blocker precisely; it must not replace focused lease evidence.
+6. Validate Compose interpolation and the Collector config with the pinned distribution when Collector/example configuration changed.
+7. Run `npm run pack:dry-run` and confirm the lease metric source, dashboards, alerts, examples, user/operator/reference docs, and packaged `observme-docs` skill are present. Do not publish during this check.
+8. Record sanitized date, environment versions, pass/fail status, cleanup result, and any blocker in `docs/compatibility-matrix.md` and `docs/review-validation.md`; never record credentials, raw content, private paths, commands from interactive Pi, or full high-cardinality identifiers.
+9. Publish the package/artifact only after lease-related failures are resolved.
+10. Tag the release.
 
 ## 11. Compatibility Matrix
 
@@ -266,6 +292,7 @@ observme_export_errors_total
 observme_redaction_failures_total
 observme_active_spans
 observme_active_agents
+observme_agent_lease_expires_unixtime_seconds
 observme_workflows_started_total
 observme_workflows_completed_total
 observme_workflow_errors_total
@@ -313,7 +340,10 @@ When troubleshooting:
 3. Run `/obs agents` when troubleshooting workflow, parent/subagent lineage, fan-out, depth, or orphan agents.
 4. Check Collector health endpoint.
 5. Check Collector logs.
-6. Query `observme_export_errors_total`.
-7. Query Loki for `event_name="export.failed"` (or equivalent normalized structured metadata).
-8. Temporarily enable debug exporter in Collector.
-9. Never enable raw content capture in production without approval.
+6. For active-agent incidents, compare the canonical leased count, raw positive claims, expired claims, and direct presence of `observme_agent_lease_expires_unixtime_seconds`; then verify producer/Prometheus clock skew, `observme_instance_id` preservation, and the configured lease/export relationship.
+7. Query `observme_export_errors_total` and inspect Export Health before deciding that a missing or expired lease means the producer stopped.
+8. Query Loki for `event_name="export.failed"` (or equivalent normalized structured metadata).
+9. Temporarily enable the debug exporter in the Collector.
+10. Never enable raw content capture in production without approval.
+
+See [`11-deployment-runbooks.md` §5](11-deployment-runbooks.md#5-common-incidents) for expired-claim and missing-lease decision trees.

package/docs/reference/11-deployment-runbooks.md

@@ -88,6 +88,26 @@ Recommended ports:
 9090 prometheus
 ```
 
+### Prometheus exporter cleanup policy
+
+The bundled Collector exposes all metrics through one Prometheus exporter with `metric_expiration: 5m`. Treat that value only as stale-series/cardinality cleanup, not as active-agent liveness. It applies to every exported gauge, counter, and histogram. The leased active-agent queries determine liveness and converge after the 60-second default lease even while a raw positive claim remains cached.
+
+When changing `metrics.activeAgentLeaseDurationMillis`, `metrics.exportIntervalMillis`, scrape timing, or expected outage tolerance:
+
+1. Keep `metric_expiration` longer than the active-agent lease; five minutes is the recommendation for the default 60-second lease and 15-second export interval.
+2. Leave enough margin for ordinary export delays and scrape gaps. Shorter cleanup lowers Collector memory/cardinality but can remove every metric type during a temporary producer outage; longer cleanup retains abandoned series longer.
+3. Preserve generated `observme.instance.id`/`service.instance.id` resource identity for lease joins, while continuing to drop workflow, session, logical-agent, trace, span, spawn, and job-run identifiers from metric labels.
+4. Validate the configuration with the deployed Collector distribution, then use its supported reload mechanism or restart it. The bundled Compose stack requires a Collector restart.
+5. Expect a restart to clear the exporter's in-memory cache only. Prometheus history remains stored until its normal retention or deletion policy removes it. Traces and logs are unaffected.
+
+For the pinned local distribution, validate before restart:
+
+```bash
+docker compose -f observability-stack/docker-compose.yml config
+docker compose -f observability-stack/docker-compose.yml exec -T otel-collector \
+  /otelcol-contrib validate --config=/etc/otel/otel-collector.yaml
+```
+
 ## 3. CI Agent Runbook
 
 CI agents are short lived. Use one of:
@@ -108,13 +128,17 @@ Pi CI job -> sidecar collector -> central collector
 
 Best for high-volume jobs because Pi hands off telemetry locally.
 
-Shutdown steps:
+Preferred shutdown steps:
 
 1. Parent Pi passes trace context, `OBSERVME_WORKFLOW_ID`, and `OBSERVME_PARENT_AGENT_ID`/`OBSERVME_ROOT_AGENT_ID` to any child Pi processes it launches.
-2. Pi exits.
-3. ObservMe flushes with timeout.
-4. Sidecar collector drains with timeout.
-5. CI job ends.
+2. Stop Pi and child producers gracefully so ObservMe can deactivate the lease, export the zero lifecycle claim, and flush within its timeout.
+3. Drain the sidecar Collector with a timeout.
+4. Put best-effort process, container, and network cleanup in an `if: always()` GitHub Actions step.
+5. End the CI job.
+
+Graceful shutdown and unconditional cleanup reduce latency and resource leakage, but forced cancellation, `SIGKILL`, runner loss, or power loss can skip both. The active count still converges from the last lease; never make a post-job step, Pi shutdown callback, or Collector restart the correctness mechanism. Test cancellation by killing a child producer from a still-running validation job, because a cancelled workflow cannot assert its own convergence.
+
+GitHub-hosted runners satisfy the supported clock expectation. A self-hosted runner must keep its wall clock and the Prometheus clock synchronized within 5 seconds using reliable NTP or an equivalent managed time service. Monitor synchronization health and clock steps; a producer clock behind can fail closed early, while one ahead can delay expiry until the bounded future-horizon rule rejects it.
 
 ## 4. Production Configuration Checklist
 
@@ -133,6 +157,10 @@ Shutdown steps:
 - [ ] Collector batch processor enabled.
 - [ ] Loki structured metadata enabled when required by the deployed Loki version.
 - [ ] Prometheus OTLP receiver enabled with `--web.enable-otlp-receiver` if sending directly to Prometheus.
+- [ ] Prometheus scrape-exporter `metric_expiration` is longer than the active-agent lease and is treated only as exporter-wide cleanup.
+- [ ] Lease-aware queries, rather than raw active claims or Collector expiration, determine active-agent liveness.
+- [ ] Producer and Prometheus clocks are synchronized within 5 seconds; self-hosted runner time synchronization is monitored.
+- [ ] GitHub Actions uses graceful shutdown and `if: always()` cleanup where possible without depending on either for active-count correctness.
 - [ ] Backend retention configured.
 - [ ] Dashboards imported.
 - [ ] Alerts enabled.
@@ -141,6 +169,27 @@ Shutdown steps:
 
 ## 5. Common Incidents
 
+### Grafana-Backed Commands Reject the Base URL
+
+If `/obs status`, `/obs health`, or a query-backed command reports `embedded_grafana_url_credentials` or `embedded_credentials`:
+
+1. Configure `query.grafana.url` as an absolute HTTP(S) base URL without an embedded username or password component.
+2. Put bearer authentication in `query.grafana.token`, or configure both `query.grafana.username` and `query.grafana.password`. Prefer environment-variable references rather than secret values in YAML.
+3. Restart Pi, then rerun `/obs status` and `/obs health`.
+
+ObservMe rejects this URL class before default-fetch or custom-transport network I/O. Diagnostics intentionally identify only the safe failure class and setting names, never the rejected URL or credentials.
+
+### Project Config or `.env` Is Rejected
+
+Project-local configuration is loaded only for a trusted project and only while its lexical and canonical paths remain inside the stable canonical project root.
+
+1. Confirm Pi marks the project trusted.
+2. Use a normal in-root config directory and `.env` file, or symlinks whose existing targets also remain inside the same project root.
+3. Replace out-of-root, dangling, or changing symlinks and retry the session start. Do not bypass the containment check.
+4. Review `/obs status` for `config_source_rejected`; bootstrap failures appear as a bounded warning.
+
+ObservMe fails closed if it cannot verify the root, target, or opened-file identity. A rejected project config or `.env` is not read, starter creation never overwrites an existing target, and diagnostics omit canonical and external path details. Other accepted configuration layers continue to apply.
+
 ### No Traces in Tempo
 
 Check:
@@ -172,6 +221,34 @@ Check:
 4. Metric names visible
 5. Resource attributes not unexpectedly dropped
 
+### Active Count Is Correct but Raw Claims Remain
+
+This is expected after an ungraceful producer exit. Check:
+
+1. The dashboard uses the leased active-agent query, not `sum(observme_active_agents)`.
+2. The expired-claims diagnostic shows the abandoned stream.
+3. The configured lease has elapsed and producer/Prometheus clocks are synchronized within 5 seconds.
+4. `metric_expiration` remains longer than the lease. Do not shorten it to force correctness; the exporter will remove the raw gauge, counter, and histogram series together after the cleanup window.
+5. Collector memory/cardinality is within budget. If cleanup timing changes, validate and reload/restart the Collector; stored Prometheus history will remain.
+
+No Collector restart is required. Restarting clears the exporter's in-memory cache and can hide the diagnostic raw claim, but it does not improve lease convergence or delete Prometheus history.
+
+### Live Producer Is Missing from the Leased Count
+
+A missing, expired, malformed, or pathologically future lease fails closed. Diagnose in this order:
+
+1. Confirm Prometheus and the Export Health dashboard are reachable. If the backend is unavailable, zero is not a reliable health result.
+2. Compare the canonical leased total, raw positive claims, and expired claims. Query `observme_agent_lease_expires_unixtime_seconds` directly only for diagnosis; do not expose `observme_instance_id` in a public legend or variable.
+3. If the raw claim and lease are both missing, check `metrics.enabled`, `/obs status`, `/obs health`, OTLP reachability, `observme_export_errors_total`, Collector logs, and the metrics pipeline.
+4. If the raw claim exists but the lease is missing, confirm every expected producer runs a lease-capable ObservMe version and that the Collector preserves the same generated `observme_instance_id` label on both metrics. During a mixed-version rollout, missing leases intentionally do not count.
+5. If the lease is expired while the producer runs, verify metric collection/export is still occurring, then inspect Export Health for drops or exporter failures. Confirm `metrics.activeAgentLeaseDurationMillis >= (2 * metrics.exportIntervalMillis) + 5000`.
+6. If the lease is rejected as future, compare producer and Prometheus clocks. Restore reliable synchronization within 5 seconds; do not widen panel-local PromQL or remove the `time() + 305` safety bound.
+7. After export recovers, wait for the next metric collection and Prometheus scrape. The lease renews automatically while the session remains active; neither workload activity nor Collector restart is required.
+
+### Expired Claims Are Missing from Diagnostics
+
+The live total can still be correct. Check whether `metric_expiration` elapsed, the Collector restarted, or the deployment uses remote write rather than the scrape exporter. Any of those can remove the current raw series while historical Prometheus samples remain. Use backend history and Export Health for incident timing; do not infer that graceful shutdown occurred merely because the expired-claim panel is empty.
+
 ### Subagents Not Linked to Parent
 
 Check:

package/docs/reference/12-configuration-reference.md

@@ -15,8 +15,9 @@ observme:
     headers:
       Authorization: "Bearer ${OBSERVME_OTLP_TOKEN}"
     tls:
-      enabled: true
       insecureSkipVerify: false
+    # The endpoint scheme selects HTTP or HTTPS. HTTPS certificate verification
+    # stays enabled unless insecureSkipVerify is explicitly acknowledged.
     # Derived SDK URLs for OTLP/HTTP exporters:
     # traces:  https://otel-collector.example.com:4318/v1/traces
     # metrics: https://otel-collector.example.com:4318/v1/metrics
@@ -64,6 +65,7 @@ observme:
     enabled: true
     exportIntervalMillis: 15000
     exportTimeoutMillis: 3000
+    activeAgentLeaseDurationMillis: 60000
 
   logs:
     enabled: true
@@ -134,6 +136,16 @@ observme:
     flushTimeoutMs: 3000
 ```
 
+`otlp.endpoint` and every `otlp.signalEndpoints` value must be an absolute HTTP(S) URL without userinfo, unresolved placeholders, a query, or a fragment; credentials belong in `otlp.headers`. The base endpoint may contain an intentional path, to which ObservMe appends exactly one `/v1/{signal}` suffix using URL pathname semantics. Explicit signal endpoints must already end in their matching `/v1/traces`, `/v1/metrics`, or `/v1/logs` path.
+
+`query.grafana.url` must be an absolute HTTP(S) base URL without a username or password component. Authentication belongs only in `query.grafana.token` or the complete `query.grafana.username` and `query.grafana.password` pair. Credential-bearing base URLs fail configuration validation and query/transport preflight before network I/O. Diagnostics identify the bounded `embedded_credentials` failure class without rendering the rejected URL or either credential component.
+
+`otlp.tls.enabled` is intentionally unsupported and rejected as an unknown setting: the `http://` or `https://` endpoint scheme is the single source of truth for whether TLS is used. The retained `otlp.tls.insecureSkipVerify` option is passed to every OTLP HTTP exporter as its certificate-verification behavior.
+
+`query.links.traceUrlTemplate` is shared by `/obs session`, `/obs trace`, and `/obs link`. Absolute HTTP(S) templates support `{traceId}`, `{{traceId}}` (with optional inner whitespace), `${traceId}`, and `%TRACE_ID%`; optional Tempo UID placeholders use the matching `{tempoDatasourceUid}`, `{{tempoDatasourceUid}}`, `${tempoDatasourceUid}`, and `%TEMPO_DATASOURCE_UID%` forms. An empty template or one containing `...` selects the structured Grafana Explore fallback built from `query.grafana.url` and `query.grafana.datasourceUids.tempo`. Unsupported placeholders and invalid URLs fail with a bounded diagnostic.
+
+`query.maxLogs`, `query.maxTraces`, `query.maxMetricSeries`, and `query.maxAgents` accept integers from 1 through 100. Query clients enforce the same upper bound at runtime before issuing backend requests.
+
 ## 2. Environment Variables
 
 ```text
@@ -146,6 +158,7 @@ OBSERVME_OTLP_TRACES_ENDPOINT
 OBSERVME_OTLP_METRICS_ENDPOINT
 OBSERVME_OTLP_LOGS_ENDPOINT
 OBSERVME_OTLP_TOKEN
+OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS
 OBSERVME_WORKFLOW_ID
 OBSERVME_WORKFLOW_MAX_DEPTH_WARNING
 OBSERVME_WORKFLOW_MAX_FANOUT_WARNING
@@ -184,6 +197,17 @@ OBSERVME_ALLOW_UNSAFE_CAPTURE
 OBSERVME_ALLOW_INSECURE_TRANSPORT
 ```
 
+### 2.1 Active-agent lease configuration
+
+| Setting | Default | Supported values | Purpose |
+| --- | --- | --- | --- |
+| `metrics.activeAgentLeaseDurationMillis` | `60000` | Integer `10000`–`300000`, and at least `(2 * metrics.exportIntervalMillis) + 5000` | Validity window renewed on each metric collection while the session is active. |
+| `OBSERVME_ACTIVE_AGENT_LEASE_DURATION_MS` | unset | Same as the YAML field | Environment override through the normal trusted `.env` / system environment precedence. |
+
+The value controls failure-convergence latency, not a background timer. A clean shutdown deactivates the lease before final flush and reaches zero after normal export/scrape propagation. An ungraceful exit reaches zero within the configured lease plus up to 5 seconds of supported clock skew and one Prometheus scrape/evaluation interval. Missing or invalid leases fail closed. Keep producer and Prometheus clocks synchronized within 5 seconds; GitHub-hosted runners meet this requirement, while self-hosted runners need reliable time synchronization.
+
+Do not tune this field by shortening Collector `metric_expiration`. The recommended five-minute exporter cleanup remains longer than the default lease and applies to all gauges, counters, and histograms. Changing Collector configuration may require reload or restart, but restart is not required for active-agent correctness.
+
 ## 3. Workflow and Agent Lineage Configuration
 
 The `workflow:` and `agent:` config blocks control generated workflow identity, generated agent identity, and parent/child propagation. `pi.workflow.id`, `pi.agent.id`, `pi.agent.parent_id`, and related values are high-cardinality identifiers for traces/logs only; they must not be metric labels by default.
@@ -198,7 +222,8 @@ Rules:
 - Continue a valid W3C parent context explicitly on the child `pi.session` span. If trusted lineage has no usable continuation, start a new trace and add a validated parent span link when metadata exists, otherwise emit the documented propagation-failure log/counter fallback.
 - When `propagateTraceContext` is true, propagate W3C `traceparent`/`tracestate` to child processes launched by ObservMe-aware subagent wrappers.
 - When `workflow.enabled` is true, propagate `OBSERVME_WORKFLOW_ID` to child processes and report depth/fan-out/orphan metrics.
-- `writeCorrelationEntry` may append a minimal `custom` session entry for recovery, but it must remain disabled by default and must never use `custom_message`.
+- `writeCorrelationEntry` defaults to `false`. When enabled, successful startup appends at most one versioned `observme.correlation` `custom` entry to the active branch. Reload/resume/fork recovery scans only `ctx.sessionManager.getBranch()`, accepts the latest valid bounded lineage entry, ignores corrupt or abandoned-branch data, and does not duplicate an unchanged entry. ObservMe never uses `custom_message`, so this state cannot enter LLM context.
+- Historical telemetry is never replayed automatically. Use the explicit, confirmed `/obs backfill` command. The removed `replayOnStart` YAML field is rejected as unknown, and `OBSERVME_REPLAY_ON_START` is no longer recognized; neither can emit a synthetic duplicate startup record.
 
 ## 4. Config Precedence
 
@@ -211,6 +236,17 @@ Rules:
 
 Copy `.env.example` to `.env` for project-local extension variables, or export the same `OBSERVME_*` names in the shell before starting Pi. System environment variables override `.env` values, and `.env` must never be committed. If redacted content capture is enabled, set `OBSERVME_HASH_SALT` in the shell or trusted project `.env`; missing salts make capture fail closed.
 
+### 4.1 Project-Local Path and Symlink Policy
+
+Project config, project `.env`, and starter-config creation use the same fail-closed filesystem policy:
+
+- Pi must mark the project trusted before ObservMe reads or creates a project-local file.
+- The lexical path and resolved canonical target must both remain inside one stable canonical project root.
+- Existing file or directory symlinks are supported when their targets remain inside that root. Out-of-root, dangling, or unverifiable symlinks are rejected.
+- Reads and creates use opened file handles whose file identity, root identity, and canonical containment are verified before bytes are consumed or written. Concurrent replacement of a file, ancestor, or project root therefore rejects the operation instead of following the substituted target.
+- Normal missing in-root paths remain supported. Starter creation is exclusive and idempotent, never overwrites an existing file, and removes an unverified empty target if post-open validation fails.
+- Rejected project sources use the bounded `config_source_rejected` classification. Warnings and status diagnostics do not include canonical targets, external paths, file contents, or environment values; the rejected layer is skipped and other accepted layers continue to apply.
+
 Automatic project starter file:
 
 - On `session_start`, the extension creates `<cwd>/<CONFIG_DIR_NAME>/observme.yaml` (`<cwd>/.pi/observme.yaml` in the standard distribution) when the project is trusted and the file is missing.
@@ -243,6 +279,9 @@ agent:
   writeCorrelationEntry: false
 workflow:
   enabled: true
+metrics:
+  exportIntervalMillis: 15000
+  activeAgentLeaseDurationMillis: 60000
 ```
 
 ## 6. Development Defaults
@@ -291,7 +330,7 @@ Rules:
 - When the token is blank or an unresolved placeholder, a resolved `query.grafana.username` and `query.grafana.password` are sent as Basic auth for local development.
 - Query-backed commands and `/obs health` must fail fast before Grafana calls when Grafana auth is unresolved/missing/incomplete, `query.grafana.url` is invalid, or a required datasource UID is blank.
 - `/obs health` must report Grafana `401`/`403` responses as authentication failures and must not print token or password values.
-- `tls.insecureSkipVerify=true` is only for local self-signed certificates; production should trust the CA instead.
+- `query.grafana.tls.insecureSkipVerify=true` is only for local self-signed certificates; production requires `privacy.allowInsecureTransport=true` as an explicit acknowledgement and should trust the CA instead.
 - `transport.preferIPv4=true` uses Node's local HTTP(S) transport with IPv4 DNS lookup for Grafana calls.
 - Provisioned datasource UIDs are `tempo`, `loki`, and `prometheus`; Loki selectors use normalized labels such as `service_name`, `pi_session_id`, `event_name`, and `event_category` for ObservMe data.
 
@@ -319,10 +358,14 @@ ObservMe must display a warning when unsafe capture is active. Redacted capture
 Reject config when:
 
 - `allowUnsafeCapture=false` and `redactionEnabled=false` while any content capture is true.
-- Production endpoint uses `http://` and `allowInsecureTransport` is not true.
+- A production OTLP or Grafana endpoint uses `http://` and `privacy.allowInsecureTransport` is not true.
+- `query.grafana.url` contains an embedded username or password instead of using the dedicated token or username/password settings.
+- Production sets `otlp.tls.insecureSkipVerify=true` or `query.grafana.tls.insecureSkipVerify=true` and `privacy.allowInsecureTransport` is not true.
 - `otlp.protocol=http/protobuf` but a signal-specific exporter URL omits the required `/v1/traces`, `/v1/metrics`, or `/v1/logs` path.
-- Metric labels include high-cardinality fields such as workflow IDs, session IDs, agent IDs, parent/child agent IDs, trace IDs, span IDs, entry IDs, spawn IDs, or spawn tool-call IDs.
+- `metrics.activeAgentLeaseDurationMillis` is fractional, non-numeric, below `10000`, above `300000`, or less than `(2 * metrics.exportIntervalMillis) + 5000`.
+- Metric labels include high-cardinality fields such as workflow IDs, session IDs, agent IDs, parent/child agent IDs, trace IDs, span IDs, entry IDs, spawn IDs, or spawn tool-call IDs. The generated `observme.instance.id` / `service.instance.id` remains a resource identity used by the Collector for the `observme_instance_id` lease join; it must not be configured as an execution-derived label.
 - Project-local config is read while `ctx.isProjectTrusted()` is false.
+- A project config, project `.env`, or starter-config path escapes the canonical project root, changes identity during I/O, or cannot be verified safely.
 - Propagated workflow or agent lineage values are malformed, too long, or contain unsafe characters.
 - Queue sizes exceed configured memory guardrails.
 

package/docs/review-validation.md

@@ -55,6 +55,7 @@ Use this checklist when a review, release candidate, or local operator needs cov
 | `npm run smoke:pi-runtime` | Yes when the `pi` CLI is installed in the test image | A working local `pi` executable, dependencies installed, and permission to spawn a local child process. | No credentials. It deletes inherited `OBSERVME_*` values and uses a deterministic loopback backend. | Creates a temporary project and extension files under the OS temp directory, then removes them; local loopback server only. | Real Pi RPC extension load, `/obs` discovery/routing, reload/new-session replacement, and credential-free event-shape smoke coverage. |
 | `npm run smoke:packaged` | Usually release/local only | `npm` available and package dependencies resolvable for a temporary install. | May need npm registry/cache access to install package dependencies; no project credentials. | Runs `npm pack`, creates a temporary project under the OS temp directory, installs the tarball with `--ignore-scripts --no-audit --no-fund --package-lock=false`, and removes the temp directory. | Packaged install layout plus declared Pi extension and skill resources in the installed package. |
 | `npm run test:integration:collector` | No | Docker daemon available; Collector image `OBSERVME_COLLECTOR_IMAGE` or `otel/opentelemetry-collector-contrib:0.104.0` pullable or cached. | Docker image pull may require network; no Grafana/model credentials. | Starts a temporary Collector container with loopback-published ports and stops it in test cleanup. | OTLP export to Collector debug pipeline for representative traces, metrics, and logs. |
+| `npm run test:integration:active-agent-lease` | No | Docker daemon available; pinned Collector Contrib and Prometheus images pullable or cached; producer and Prometheus clocks within 5 seconds. | Docker image pull may require network; no project, Grafana, or model credentials. | Creates labeled Collector/Prometheus containers and a dedicated network plus local child producers; bounded `finally` cleanup removes all resources. | Clean shutdown, `SIGTERM`, and cancellation-oriented `SIGKILL`; cached raw claim remains while lease-aware activity reaches zero without Collector restart. |
 | `npm run test:integration:grafana-stack` | No | Docker Compose available; `observability-stack/docker-compose.yml` and test overlay available; required Grafana/Tempo/Loki/Prometheus images pullable or cached. | Docker image pull may require network; no external Grafana credentials. Test-created local Grafana admin secret is synthetic. | Creates isolated Docker Compose project/networks/containers. May create `observability-stack/secrets/grafana_admin_password` when missing and removes the test-created file during cleanup; also creates temporary command-project files under the OS temp directory. If interrupted, run `docker compose` with the printed project name or remove stale `observme-grafana-it-*` resources. | Live local stack ingestion, datasource queries, `/obs` query commands, and dashboard provisioning imports. |
 | `npm run validate:grafana-obs` | No | Running Pi session with ObservMe loaded, reachable Collector/Grafana stack, and recent telemetry. See `docs/validation-flow.md`. | Requires Grafana read credentials from environment variables (`OBSERVME_GRAFANA_TOKEN` or username/password), datasource UIDs, and `OBSERVME_VALIDATION_SESSION_ID`; may access the configured Grafana URL. | Read-only against the configured stack; no file writes expected. Keep terminal output sanitized and omit secrets. | Operator-facing validation that Grafana has ObservMe data and representative `/obs` commands work against the same stack. |
 
@@ -76,6 +77,22 @@ This record was produced for the final-pass task "Run and record bounded live Pi
 
 Post-run cleanup check: `docker ps --filter name=observme-grafana-it --format '{{.Names}}'` and `docker network ls --filter name=observme-grafana-it --format '{{.Name}}'` returned no resources.
 
+## Active-agent lease release validation — 2026-07-14
+
+This sanitized record covers the production active-agent lease release candidate on local Node.js v26.0.0, npm 11.12.1, Docker 29.5.2, and Docker Compose 5.1.4. The checkout contains the lease implementation and documentation changes. No project, Grafana, model, or registry credentials were required or printed.
+
+| Command | Result | Evidence and notes |
+| --- | --- | --- |
+| `npm run validate` | Pass | Source/test typechecks, ESLint, formatting, script checks, 493 unit/contract tests, package-content checks, packaged-install smoke, handler smoke, Pi lifecycle smoke, and Pi runtime smoke passed. |
+| `npm run test:integration:collector` | Pass | The pinned Collector Contrib 0.104.0 debug pipeline received representative traces, metrics, and logs without default content capture. |
+| `npm run test:integration:active-agent-lease` | Pass | Clean shutdown, `SIGTERM`, and cancellation-oriented `SIGKILL` passed against Collector Contrib 0.104.0 and Prometheus 2.53.1. The raw claim remained cached while lease-aware activity reached zero with the same Collector instance and no restart. |
+| `npm run test:integration:grafana-stack` | Fail/unrelated pre-existing broad-stack blocker | The current run timed out when the Prometheus total-token query returned an empty vector. This token-ingestion/query path is unrelated to active-agent lease arithmetic, PromQL, or abrupt-termination convergence and does not invalidate the focused passing lease evidence. The earlier 2026-07-09 broad-stack run was already blocked later in its Tempo content assertion. |
+| `docker compose -f observability-stack/docker-compose.yml config --quiet` | Pass | Compose interpolation validated without rendering secret values. |
+| Pinned Collector Contrib 0.104.0 `validate --config=/etc/otel/otel-collector.yaml` | Pass | The shipped `observability-stack/config/otel/otel-collector.yaml` parsed successfully with the deployed distribution. |
+| `npm run pack:dry-run` | Pass | The 124-file package contained the lease source/metric convention, dashboards, alerts, examples, target user/operator/reference docs, and packaged `observme-docs` skill. |
+
+Post-run cleanup check found no containers or networks labeled for the active-agent lease integration and no `observme-grafana-it-*` containers or networks. Release remains blocked on the unrelated broad Grafana-stack validation unless the release owner accepts the explicitly allowed pre-existing blocker; all lease-specific release criteria passed.
+
 ## Review-closure evidence categories
 
 - **Read-only/check-only** — commands that inspect, type-check, lint, audit, run tests, dry-run packaging, or smoke local deterministic fixtures without publishing or writing tracked files.

package/examples/README.md

@@ -11,6 +11,7 @@ The npm package ships configuration examples and an extension-integration exampl
 - Local self-signed TLS and IPv4 preference are enabled for that profile.
 - Prompt, response, thinking, tool, Bash, and path capture remain disabled.
 - Redaction remains enabled and unsafe capture remains disabled.
+- Metrics export every 15 seconds and renew a 60-second active-agent lease; keep producer and Prometheus clocks synchronized within 5 seconds.
 
 To use it in a trusted project:
 
@@ -27,13 +28,17 @@ Read [`../docs/configuration.md`](../docs/configuration.md) for the quick guide
 
 - traces are exported to Tempo;
 - logs are exported to Loki;
-- metrics are remote-written to Mimir;
+- metrics are exposed on a Prometheus scrape endpoint;
+- the five-minute `metric_expiration` applies to all exported metric types only as stale-series/cardinality cleanup, not active-agent liveness;
+- lease-aware PromQL joins the positive lifecycle claim and lease on generated `observme_instance_id`; raw `observme_active_agents` sums are diagnostics, not production live totals;
 - high-cardinality workflow, agent, session, and spawn attributes are removed from metrics;
 - accidental content attributes are removed from logs as defense in depth.
 
 The example uses Collector components commonly provided by the Collector Contrib distribution. Verify that your deployed distribution includes every receiver, processor, and exporter, then replace backend endpoints and security settings for your environment.
 
-Read [`../docs/reference/05-otel-pipeline-and-collector.md`](../docs/reference/05-otel-pipeline-and-collector.md) for pipeline design and [`../docs/reference/11-deployment-runbooks.md`](../docs/reference/11-deployment-runbooks.md) for deployment and incident checks.
+Read [`../docs/reference/05-otel-pipeline-and-collector.md`](../docs/reference/05-otel-pipeline-and-collector.md) for pipeline design, [`../docs/reference/09-dashboards-alerts-slos.md`](../docs/reference/09-dashboards-alerts-slos.md#131-canonical-active-agent-promql) for canonical queries and raw-query migration, and [`../docs/reference/11-deployment-runbooks.md`](../docs/reference/11-deployment-runbooks.md) for deployment and incident checks.
+
+For GitHub Actions, prefer graceful Pi/sidecar shutdown and an `if: always()` cleanup step, but do not depend on either after force cancellation. Lease expiry provides correctness without a Collector restart. GitHub-hosted runner clocks meet the supported expectation; self-hosted runners must use reliable time synchronization.
 
 ## `integrations/subagent-runner.ts`
 

package/examples/collector.yaml

@@ -1,4 +1,4 @@
-# Production-oriented Collector reference for Tempo, Loki, and Mimir.
+# Production-oriented Collector reference for Tempo, Loki, and Prometheus.
 # Verify that your Collector distribution contains every configured component,
 # replace backend endpoints/security for your deployment, and never add raw
 # content or high-cardinality execution identifiers to metric labels.
@@ -97,14 +97,14 @@ exporters:
     retry_on_failure:
       enabled: true
 
-  prometheusremotewrite/mimir:
-    endpoint: http://mimir:9009/api/v1/push
+  prometheus:
+    endpoint: 0.0.0.0:8889
+    # Exporter-wide stale-series/cardinality cleanup for gauges, counters, and
+    # histograms. Five minutes exceeds ObservMe's default one-minute active-agent
+    # lease; lease-aware PromQL, not expiration, determines liveness.
+    metric_expiration: 5m
     resource_to_telemetry_conversion:
       enabled: true
-    sending_queue:
-      enabled: true
-    retry_on_failure:
-      enabled: true
 
   debug:
     verbosity: basic
@@ -120,7 +120,7 @@ service:
     metrics:
       receivers: [otlp]
       processors: [memory_limiter, resource/observme, resource/drop_high_cardinality_metric_attrs, batch]
-      exporters: [prometheusremotewrite/mimir]
+      exporters: [prometheus]
 
     logs:
       receivers: [otlp]