better-dsh 0.2.4 → 0.2.5

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/developer-guide/gateway-internals.md

@@ -0,0 +1,297 @@
+---
+sidebar_position: 7
+title: "Gateway Internals"
+description: "How the messaging gateway boots, authorizes users, routes sessions, and delivers messages"
+---
+
+# Gateway Internals
+
+The messaging gateway is the long-running process that connects Hermes to 20+ external messaging platforms through a unified architecture.
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `gateway/run.py` | `GatewayRunner` facade — composes the `gateway/run_*.py` sibling mixins (startup, adapters, inbound, turn, busy, goals, notifications, shutdown, …) and `gateway/slash_commands_*.py` handlers |
+| `gateway/session.py` | `SessionStore` — conversation persistence and session key construction |
+| `gateway/delivery.py` | Outbound message delivery to target platforms/channels |
+| `gateway/pairing.py` | DM pairing flow for user authorization |
+| `gateway/channel_directory.py` | Maps chat IDs to human-readable names for cron delivery |
+| `gateway/hooks.py` | Hook discovery, loading, and lifecycle event dispatch |
+| `gateway/mirror.py` | Cross-session message mirroring for `send_message` |
+| `gateway/status.py` | Token lock management for profile-scoped gateway instances |
+| `gateway/builtin_hooks/` | Extension point for always-registered hooks (none shipped) |
+| `gateway/platform_registry.py` | Adapter registry, factories, and deferred (lazy) loaders for bundled platform plugins |
+| `plugins/platforms/<name>/` | Bundled messaging adapters (most platforms: `adapter.py` + `plugin.yaml`) |
+| `gateway/platforms/` | Shared `base.py` plus legacy/direct adapters (Signal, API server, webhooks, …) |
+
+## Architecture Overview
+
+```text
+┌─────────────────────────────────────────────────┐
+│                  GatewayRunner                  │
+│                                                 │
+│  ┌──────────┐  ┌──────────┐  ┌──────────┐       │
+│  │ Telegram │  │ Discord  │  │  Slack   │       │
+│  │ Adapter  │  │ Adapter  │  │ Adapter  │       │
+│  └────┬─────┘  └────┬─────┘  └────┬─────┘       │
+│       │             │             │             │
+│       └─────────────┼─────────────┘             │
+│                     ▼                           │
+│              _handle_message()                  │
+│                     │                           │
+│         ┌───────────┼───────────┐               │
+│         ▼           ▼           ▼               │
+│  Slash command   AIAgent    Queue/BG            │
+│    dispatch      creation   sessions            │
+│                     │                           │
+│                     ▼                           │
+│                 SessionStore                    │
+│              (SQLite persistence)               │
+└───────┴─────────────┴─────────────┴─────────────┘
+```
+
+## Message Flow
+
+When a message arrives from any platform:
+
+1. **Platform adapter** receives raw event, normalizes it into a `MessageEvent`
+2. **Base adapter** checks active session guard:
+   - If agent is running for this session → queue message, set interrupt event
+   - If `/approve`, `/deny`, `/stop` → bypass guard (dispatched inline)
+3. **GatewayRunner._handle_message()** receives the event:
+   - Resolve session key via `_session_key_for_source()` (format: `agent:{namespace}:{platform}:{chat_type}:{chat_id}`; the namespace is `main` for the default profile, `<profile>` under multiplexing — see [Multiplexed profiles](#multiplexed-profiles))
+   - Check authorization (see Authorization below)
+   - Check if it's a slash command → dispatch to command handler
+   - Check if agent is already running → intercept commands like `/stop`, `/status`
+   - Otherwise → create `AIAgent` instance and run conversation
+4. **Response** is sent back through the platform adapter
+
+### Session Key Format
+
+Session keys encode the full routing context:
+
+```
+agent:{namespace}:{platform}:{chat_type}:{chat_id}
+```
+
+For example: `agent:main:telegram:private:123456789` for the default profile, or
+`agent:work:telegram:private:123456789` when the multiplexer routes that chat to profile `work`
+(`gateway/session.py::_session_key_namespace`; a profile literally named `main` is marked `main~`).
+
+Thread-aware platforms (Telegram forum topics, Discord threads, Slack threads) may include thread IDs in the chat_id portion. **Never construct session keys manually** — always use `build_session_key()` from `gateway/session.py`.
+
+### Two-Level Message Guard
+
+When an agent is actively running, incoming messages pass through two sequential guards:
+
+1. **Level 1 — Base adapter** (`gateway/platforms/base.py`): Checks `_active_sessions`. If the session is active, queues the message in `_pending_messages` and sets an interrupt event. This catches messages *before* they reach the gateway runner.
+
+2. **Level 2 — Gateway runner** (`gateway/run_inbound.py`): Checks `_running_agents`. Intercepts specific commands (`/stop`, `/new`, `/queue`, `/status`, `/approve`, `/deny`) and routes them appropriately. Everything else triggers `running_agent.interrupt()`.
+
+Commands that must reach the runner while the agent is blocked (like `/approve`) are dispatched **inline** via `await self._message_handler(event)` — they bypass the background task system to avoid race conditions.
+
+## Authorization
+
+The gateway uses a multi-layer authorization check, evaluated in order:
+
+1. **Per-platform allow-all flag** (e.g., `TELEGRAM_ALLOW_ALL_USERS`) — if set, all users on that platform are authorized
+2. **Platform allowlist** (e.g., `TELEGRAM_ALLOWED_USERS`) — comma-separated user IDs
+3. **DM pairing** — authenticated users can pair new users via a pairing code
+4. **Global allow-all** (`GATEWAY_ALLOW_ALL_USERS`, or `gateway.allow_all_users` in `config.yaml`, bridged to the env var by `gateway/config_loader.py::bridge_core_env_settings`) — if set, all users across all platforms are authorized
+5. **Default: deny** — unauthorized users are rejected
+
+### DM Pairing Flow
+
+```text
+Admin: /pair
+Gateway: "Pairing code: ABC123. Share with the user."
+New user: ABC123
+Gateway: "Paired! You're now authorized."
+```
+
+Pairing state is persisted in `gateway/pairing.py` and survives restarts.
+
+## Slash Command Dispatch
+
+All slash commands in the gateway flow through the same resolution pipeline:
+
+1. `resolve_command()` from `hermes_cli/commands.py` maps input to canonical name (handles aliases, prefix matching)
+2. The canonical name is checked against `GATEWAY_KNOWN_COMMANDS`
+3. `_handle_message()` (`gateway/run_inbound.py`) looks the handler up by name — `_handle_<name>_command` on the `gateway/slash_commands_*.py` mixins — via `_command_handler_table` over `_IDLE_COMMANDS` / `_PLAIN_COMMANDS` in `gateway/run_busy.py`; there is no `if canonical == ...` chain
+4. Some commands are gated on config (`gateway_config_gate` on `CommandDef`)
+
+### Running-Agent Guard
+
+Commands that must NOT execute while the agent is processing are rejected early:
+
+While `_quick_key in self._running_agents`, `_dispatch_busy_slash_command()` in `gateway/run_busy.py` routes each recognized command by its `CommandDef.busy_policy` / `busy_handler`: a mid-run variant (`_busy_<key>_command`) if one exists, otherwise the normal handler when `busy_policy` allows it, otherwise a reject message ("⏳ Agent is running — `/model` can't run mid-turn…").
+
+Bypass commands (`/stop`, `/new`, `/approve`, `/deny`, `/queue`, `/status`) have mid-run handlers and are dispatched inline.
+
+## Config Sources
+
+The gateway reads configuration from multiple sources:
+
+| Source | What it provides |
+|--------|-----------------|
+| `~/.hermes/.env` | API keys, bot tokens, platform credentials |
+| `~/.hermes/config.yaml` | Model settings, tool configuration, display options |
+| Environment variables | Override any of the above |
+
+Unlike the CLI (which uses `load_cli_config()` with hardcoded defaults), the gateway reads `config.yaml` directly via YAML loader. This means config keys that exist in the CLI's defaults dict but not in the user's config file may behave differently between CLI and gateway.
+
+## Platform Adapters
+
+Most messaging platforms ship as plugin adapters under `plugins/platforms/<name>/adapter.py`; a few legacy adapters still live directly in `gateway/platforms/`. All extend `BasePlatformAdapter` from `gateway/platforms/base.py`:
+
+```text
+plugins/platforms/                  # plugin-packaged adapters (one dir each)
+├── telegram/adapter.py     # Telegram Bot API (long polling or webhook)
+├── discord/adapter.py      # Discord bot via discord.py
+├── slack/adapter.py        # Slack Socket Mode
+├── whatsapp/adapter.py     # WhatsApp Business Cloud API
+├── matrix/adapter.py       # Matrix via mautrix (optional E2EE)
+├── mattermost/adapter.py   # Mattermost WebSocket API
+├── email/adapter.py        # Email via IMAP/SMTP
+├── sms/adapter.py          # SMS via Twilio
+├── dingtalk/adapter.py     # DingTalk WebSocket
+├── feishu/adapter.py       # Feishu/Lark WebSocket or webhook
+├── wecom/adapter.py        # WeCom (WeChat Work) callback
+├── line/adapter.py         # LINE Messaging API
+├── teams/adapter.py        # Microsoft Teams
+├── irc/adapter.py          # IRC (canonical scoped-lock example)
+├── homeassistant/adapter.py # Home Assistant conversation integration
+└── …                       # google_chat, ntfy, photon, raft, simplex, …
+
+gateway/platforms/                  # core base + legacy direct adapters
+├── base.py              # BasePlatformAdapter — shared logic for all platforms
+├── signal.py            # Signal via signal-cli REST API
+├── weixin.py            # Weixin (personal WeChat) via iLink Bot API
+├── bluebubbles.py       # Apple iMessage via BlueBubbles macOS server
+├── qqbot/               # QQ Bot (Tencent QQ) via Official API v2 (sub-package)
+├── yuanbao.py           # Yuanbao (Tencent) DM/group adapter
+├── msgraph_webhook.py   # Microsoft Graph change-notification webhook (Teams, Outlook, etc.)
+├── webhook.py           # Inbound/outbound webhook adapter
+└── api_server.py        # REST API server adapter
+```
+
+**Deferred loading:** Bundled `kind: platform` plugins register cheap `register_deferred` loaders in `gateway/platform_registry.py` (via `hermes_cli/plugins.py`) so platform SDKs import only when the gateway starts, delivers, or runs setup/status — not on plain `hermes chat`. Resolution loads one adapter on lookup; full enumeration runs pending loaders only on paths that need every platform.
+
+Experimental connector-backed platforms use the generic relay adapter in `gateway/relay/` instead of a direct platform module. When `GATEWAY_RELAY_URL` or `gateway.relay_url` is configured, the gateway registers the `relay` platform, dials the connector over an outbound WebSocket, and receives `descriptor`, `inbound`, and `interrupt_inbound` frames on that same socket. The connector advertises a `CapabilityDescriptor`; Hermes can send normal outbound replies, token-less `follow_up` operations, and interrupt frames back through the relay. The source-grounded wire contract lives in [Relay ↔ Connector contract](relay-connector-contract.md).
+
+Adapters implement a common interface:
+- `connect()` / `disconnect()` — lifecycle management
+- `send()` — outbound message delivery
+- inbound events are normalized into a `MessageEvent` and forwarded via `handle_message()`
+
+Internal push wakes use `gateway.wake.admit_internal_event`: the public
+`handle_message()` still returns `None`, but the event's process-local
+`_gateway_accepted` receipt is set only after scheduling or queue insertion.
+A missing handler, mismatched explicit session key, or queue-cap drop is not
+acceptance. Custom adapters overriding ingress should delegate internal events to
+`BasePlatformAdapter.handle_message()` (or explicitly record actual admission),
+not equate a consumed/dropped callback with acceptance. This receipt is separate
+from heartbeat execution accounting and does not bypass authorization, emergency
+stop, or later turn-preparation gates.
+
+### Token Locks
+
+Adapters that connect with unique credentials call `acquire_scoped_lock()` in `connect()` and `release_scoped_lock()` in `disconnect()`. This prevents two profiles from using the same bot token simultaneously.
+
+A lock conflict is emitted as `{scope}_lock` with `retryable=True` so a **mid-run** reconnect can recover once the other holder exits. At **startup**, though, a live foreign holder is a configuration conflict: `gateway/restart.py::is_global_startup_conflict()` recognizes the `*_lock` / `lock_conflict` code families and the startup router parks the platform `fatal` instead of retry-queueing it. With nothing else connected the gateway exits `78` (`EX_CONFIG`, `gateway_state=startup_failed`) so the supervisor stops restarting it: systemd via `RestartPreventExitStatus=78`, s6 via finish→125, launchd via `KeepAlive.SuccessfulExit=false` after the stderr wrapper maps 78→0. Alongside a genuinely transient peer failure the gateway stays alive and only the peer retries.
+
+## Delivery Path
+
+Outgoing deliveries (`gateway/delivery.py`) handle:
+
+- **Direct reply** — send response back to the originating chat
+- **Home channel delivery** — route cron job outputs and background results to a configured home channel
+- **Explicit target delivery** — the send engine specifying `telegram:-1001234567890`, exposed via the [`hermes send` CLI](/guides/pipe-script-output) for shell scripts and via cron `deliver:` targets
+- **Cross-platform delivery** — deliver to a different platform than the originating message
+
+Cron job deliveries are NOT mirrored into gateway session history — they live in their own cron session only. This is a deliberate design choice to avoid message alternation violations.
+
+## Hooks
+
+Gateway hooks are Python modules that respond to lifecycle events:
+
+### Gateway Hook Events
+
+| Event | When fired |
+|-------|-----------|
+| `gateway:startup` | Gateway process starts |
+| `session:start` | New conversation session begins |
+| `session:end` | Session completes or times out |
+| `session:reset` | User resets session with `/new` |
+| `agent:start` | Agent begins processing a message |
+| `agent:step` | Agent completes one tool-calling iteration |
+| `agent:end` | Agent finishes and returns response |
+| `command:*` | Any slash command is executed |
+
+Hooks are discovered from `gateway/builtin_hooks/` (an extension point — currently empty in the shipped distribution; `_register_builtin_hooks()` is a no-op stub) and `<profile home>/hooks/` (user-installed; `~/.hermes/hooks/` for the default profile, one directory per served profile under multiplexing — paths resolve at call time, never at import). Each hook is a directory with a `HOOK.yaml` manifest and `handler.py`.
+
+## Memory Provider Integration
+
+When a memory provider plugin (e.g., Honcho) is enabled:
+
+1. Gateway creates an `AIAgent` per message with the session ID
+2. The `MemoryManager` initializes the provider with the session context
+3. Provider tools (e.g., `honcho_profile`, `viking_search`) are routed through:
+
+```text
+AIAgent._invoke_tool()
+  → self._memory_manager.handle_tool_call(name, args)
+    → provider.handle_tool_call(name, args)
+```
+
+4. On session end/reset, `on_session_end()` fires for cleanup and final data flush
+
+### Memory Flush Lifecycle
+
+Explicit conversation boundaries (such as `/new`, `/reset`, or `/resume`) flush and finalize the outgoing session. Idle time and daily boundaries never finalize it.
+
+Resource-only TTL, LRU, and memory-pressure eviction commits the cached transcript to configured memory providers before releasing the agent's clients. It does not close the durable conversation: the next turn reloads the same transcript and identity.
+
+## Background Maintenance
+
+The gateway runs periodic maintenance alongside message handling:
+
+- **Cron ticking** — checks job schedules and fires due jobs
+- **Session housekeeping** — reclaims cached resources without ending transcripts
+- **Memory flush** — commits memory before soft cache eviction
+- **Cache refresh** — refreshes model lists and provider status
+
+## Process Management
+
+The gateway runs as a long-lived process, managed via:
+
+- `hermes gateway start` / `hermes gateway stop` — manual control
+- `systemctl` (Linux) or `launchctl` (macOS) — service management
+- PID file at `~/.hermes/gateway.pid` — profile-scoped process tracking
+
+**Profile-scoped vs global**: `start_gateway()` uses profile-scoped PID files. Standalone (one gateway per profile), `hermes -p x gateway stop` stops only that profile's gateway. Under multiplexing there is ONE gateway process, owned by the default profile: `hermes gateway stop` on the default takes every served profile down, and `hermes -p x gateway stop` for a served secondary refuses with exit 78 (it has no gateway of its own). `hermes gateway stop --all` uses global `ps aux` scanning to kill all gateway processes (used during updates). Liveness is decided by `gateway.status.live_gateway_pid_for_home` (PID + start-time fingerprint), never bare PID existence.
+
+## Multiplexed profiles
+
+With `gateway.multiplex_profiles: true` one process serves the default profile plus every live directory under `profiles/` (`hermes_cli/profiles.py::profiles_to_serve(multiplex=True)`). `os.environ` and module globals hold the **launch** profile's values, so every activity for a secondary binds its scope explicitly — a profile is home + secret scope + terminal scope together:
+
+| Activity | Binding |
+|---|---|
+| Routed turn | `gateway/run.py::_profile_runtime_scope(home)` via `run_turn.py::_profile_scope_for_source` |
+| Agent release / eviction (TTL, LRU, memory pressure) | `gateway/run_agent_cache.py::_run_release_in_profile_scope` |
+| Shutdown | `gateway/run_shutdown.py::_finalize_session` |
+| Post-turn media delivery | `gateway/platforms/base.py::_media_delivery_scope` |
+| Cron tick | `cron/scheduler_provider.py::_profile_cron_scope(home)` (one ticker, profiles in sequence) |
+| Child processes (`hermes -p X` workers, relay turns, browser drivers) | `tools/environments/local.py::served_profile_child_env` |
+| Background threads | `agent/memory_provider.py::spawn_context_thread` |
+
+Secret reads fail closed (`agent.secret_scope.get_secret` raises `UnscopedSecretError`) only after `set_multiplex_active(True)`, which the gateway, cron, `gateway migrate` and the Desktop/dashboard `serve` backend set. Adapter YAML never reaches `os.environ` under multiplex: `gateway/platforms/_shared.py::apply_yaml_bridge` seeds `PlatformConfig.extra` and skips the environ write under a secondary's scope; gates read through `platform_gate_env`. Shared-ingress platforms (WhatsApp bridge, Relay) run on the default profile only; a secondary that enables one is logged once and stamped into runtime status (`run_adapters.py::_note_unserved_secondary_platform`). Per-profile isolation as the user sees it: [Multi-profile gateways § What is isolated per profile](/user-guide/multi-profile-gateways#what-is-isolated-per-profile).
+
+## Related Docs
+
+- [Session Storage](./session-storage.md)
+- [Cron Internals](./cron-internals.md)
+- [ACP Internals](./acp-internals.md)
+- [Agent Loop Internals](./agent-loop.md)
+- [Messaging Gateway (User Guide)](/user-guide/messaging)

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/developer-guide/gateway-monitoring.md

@@ -0,0 +1,309 @@
+---
+title: "Gateway Monitoring"
+description: "Health export, structured diagnostics, fleet queries and how to extend the monitoring plane"
+---
+
+# Gateway Monitoring
+
+Service health monitoring plus structured operational diagnostics for the
+Hermes gateway daemon, exported over OTLP/HTTP to an operator-configured
+endpoint (OpenTelemetry Collector, DataDog, or any OTLP receiver).
+
+This plane is content-free by construction. It exports gateway and cron
+lifecycle state, platform connector health, and content-free warning/error
+diagnostics. It never exports prompts, messages, tool arguments or results,
+job names, destinations, schedules, raw errors, session history, usage
+analytics, audit logs, or detailed execution traces. Run/model/tool trajectory
+capture is a separate plane served by Hermes's native NeMo Relay SDK
+integration and explicitly configured Relay subscribers or exporters.
+
+## What gets exported
+
+| Signal | OTLP route | Content |
+| --- | --- | --- |
+| Gateway gauges | `/v1/metrics` | `hermes.gateway.up/state/busy/drainable/active_agents/background_work/background_delegations/restart_requested`, `hermes.platform.up/degraded` with bounded `error_code` attributes |
+| Health/lifecycle events | `/v1/traces` | `gateway.lifecycle` state transitions (`starting -> running -> draining -> stopped`, `startup_failed`, exit), `gateway.health_snapshot`, platform state changes |
+| Diagnostics | `/v1/logs` | Warning/error gateway events with a constant body and bounded subsystem, severity, error class, and error code attributes; rendered log messages are never exported |
+| Cron scheduler gauges | `/v1/metrics` | Ticker heartbeat and last-success age (omitted when unavailable), a monotonic catch-up-occurrence count from the scheduler's stale-window branch, enabled/running job counts, and overdue count derived from persisted `next_run_at` plus the scheduler's existing grace rule |
+| Cron execution lifecycle | `/v1/traces` | Durable `claimed/running/completed/failed/unknown` states, bounded source and error class, opaque hashed job key, elapsed duration when timestamps exist, and delivery outcome when the scheduler knows it; terminal states make a fail-open flush attempt that can delay completion by up to one second |
+
+Signals carry `service.name`, version, supervision mode, and a stable one-way
+hash of the install id so an operator can distinguish instances without
+exporting account/profile identity or the raw install identifier.
+
+`hermes.gateway.active_agents`, `hermes.gateway.background_work`, and
+`hermes.gateway.background_delegations` are complementary. `active_agents`
+counts foreground message turns plus in-flight cron jobs plus API runs — the
+work the gateway drains on shutdown. `background_work` counts detached work that
+`active_agents` never includes: backgrounded `delegate_task` subagents,
+`terminal(background=true)` processes, and kanban workers; it is
+**task-granular** — a fan-out batch of N subagents counts as N — so it reflects
+real concurrent subagent load. `background_delegations` counts only async
+delegation **units** (each `delegate_task` dispatch is one, a fan-out batch is
+one), matching the async pool's capacity accounting; alert it against
+`delegation.max_concurrent_children` to see slot pressure. Sum `active_agents`
+and `background_work` for total live work per instance; use
+`background_delegations` for pool-saturation.
+
+## Enabling
+
+```yaml
+# config.yaml
+monitoring:
+  gateway_health_export:
+    enabled: true
+  export:
+    otlp:
+      enabled: true
+      endpoint: http://collector-host:4318/v1/traces   # metrics/logs derive
+      headers_env: {}   # header name -> ENV VAR NAME (values never stored)
+```
+
+Check the posture any time:
+
+```bash
+hermes monitoring status
+```
+
+The OpenTelemetry SDK is an optional extra (`pip install 'hermes-agent[otlp]'`),
+lazy-installed on first use. When the SDK is missing or the endpoint is down,
+the gateway runs unaffected: metric collection and ordinary event export stay
+off the hot path, while terminal cron events make one bounded fail-open flush
+attempt of up to one second so the final state is less likely to be lost.
+
+Works identically under systemd/launchd/s6 supervision, containers, tmux, or
+a plain `hermes gateway run`: the exporter lives in the gateway process, so
+no sidecar, agent, or collector is required on the host.
+
+## Collecting into DataDog
+
+Run a customer-owned OpenTelemetry Collector and forward:
+
+```yaml
+# otel-collector config
+receivers:
+  otlp:
+    protocols:
+      http:
+exporters:
+  datadog:
+    api:
+      key: ${env:DD_API_KEY}
+service:
+  pipelines:
+    metrics:   {receivers: [otlp], exporters: [datadog]}
+    traces:    {receivers: [otlp], exporters: [datadog]}
+    logs:      {receivers: [otlp], exporters: [datadog]}
+```
+
+Point `monitoring.export.otlp.endpoint` at the collector. Alerts belong on
+`hermes.gateway.up`, `hermes.platform.up`, and `hermes.platform.degraded`.
+
+## Generic fleet queries and alerts
+
+The exact syntax depends on the customer's observability backend. The examples
+below use PromQL-style expressions and intentionally avoid vendor-specific
+routing, destinations, or customer inventory.
+
+Group fleet views by the opaque `service.instance.id` resource attribute. A
+process that has died cannot emit its own zero, so every deployment needs both
+explicit-state and missing-series detection.
+
+```promql
+# Explicit gateway failure.
+hermes_gateway_up == 0
+
+# Box disappeared or stopped exporting. Choose a window longer than the
+# configured export interval and collector retry allowance.
+absent_over_time(hermes_gateway_up[5m])
+
+# Locally owned bridge is explicitly down.
+hermes_platform_up == 0
+
+# Scheduler thread is stale even though the gateway may still be alive.
+hermes_cron_scheduler_heartbeat_age_seconds > 180
+
+# Ticker loops but has not completed a successful tick recently.
+hermes_cron_scheduler_last_success_age_seconds > 300
+
+# One or more jobs are beyond their existing scheduler grace window.
+hermes_cron_jobs_overdue > 0
+
+# Catch-up counter increased, proving at least one stale occurrence was
+# collapsed and run once after a delay.
+increase(hermes_cron_scheduler_catch_up_occurrences[15m]) > 0
+```
+
+Cron execution lifecycle records arrive as `hermes.cron_execution` spans.
+Alert or derive events from bounded attributes such as:
+
+```text
+hermes.status = failed|unknown
+hermes.delivery_outcome = failed|not_configured
+hermes.error_class = auth_failed|rate_limited|timeout|network_error|
+                     dispatch_failed|interrupted|empty_response|
+                     invalid_config|unknown
+```
+
+Recommended operator views:
+
+1. one row per `service.instance.id` with gateway and configured local-platform
+   state;
+2. scheduler heartbeat, last-success age, running count, overdue count, and
+   catch-up increase;
+3. a cron lifecycle feed keyed only by opaque `hermes.job_key`;
+4. separate alerts for box absence, local bridge down, scheduler stale, cron
+   failed/unknown, delivery failure, and overdue/catch-up activity.
+
+Keep alert thresholds and routing in deployment-owned configuration. Do not add
+job names, prompts, outputs, schedules, destinations, raw errors, profile names,
+or account identity merely to make a dashboard easier to read.
+
+## Release-validation scenarios
+
+Before accepting a deployment, force and verify all five cases through the real
+collector and backend:
+
+1. **Cron success:** observe `claimed -> running -> completed`, duration, and a
+   truthful delivery outcome.
+2. **Cron failure:** observe `failed` plus a bounded error class, with no raw
+   exception or content in the decoded OTLP payload.
+3. **Cron interruption:** stop the owning gateway during execution, restart it,
+   and observe recovery to `unknown`.
+4. **Locally owned bridge outage:** break one native connector, observe its
+   bounded down/retrying/fatal state and recovery, and verify unaffected boxes
+   remain healthy.
+5. **Killed gateway:** terminate one canary, verify missing-series detection,
+   restart it, and confirm the same opaque instance identity returns.
+
+Hermes Agent-owned Relay transport health remains in scope. A separate gateway
+or connector service remains authoritative for any shared connected-platform
+state that it owns and should export that state through its own telemetry path.
+
+For every scenario, verify the signal and alert clear on recovery, other boxes
+remain unaffected, collector failure stays fail-open, and decoded metrics,
+spans, logs, and resource attributes remain content-free.
+
+## Local smoke test (no Docker)
+
+```bash
+# terminal 1: capture collector on :4318
+python scripts/observability/otel_capture_collector.py \
+  --host 127.0.0.1 --port 4318 --log /tmp/hermes_otel_capture.jsonl
+
+# terminal 2: drive the real exporter through lifecycle transitions,
+# a fatal platform, and a structured warning event, then flush
+python scripts/observability/gateway_health_export_probe.py \
+  --endpoint http://127.0.0.1:4318/v1/traces \
+  --log /tmp/hermes_otel_capture.jsonl --wait 8
+# exit 0 prints: {"requests": 6, "paths": ["/v1/logs", "/v1/metrics", "/v1/traces"]}
+```
+
+## Maintaining and extending this plane
+
+This plane is a **fixed, enumerated, content-free vocabulary** by design. Adding
+a signal is not just "emit a new metric" — every new name and attribute must be
+declared in each layer that enforces the bounded vocabulary, or it is silently
+dropped downstream. Follow the checklist for the change you are making. The
+golden rule: **a new signal that is emitted but not declared in every layer
+looks like a code bug but is a vocabulary-registration bug — nothing errors, the
+signal just never arrives.**
+
+### Content-free invariant (applies to every change)
+
+Before adding anything, confirm it cannot carry content. Numbers, booleans,
+ages, durations, monotonic counts, and one-way hashes are safe. **Never** add an
+attribute that can hold a job name, prompt, output, schedule, destination, raw
+exception text, file path, profile name, account id, or free-form string. When
+you must key a record to a job/entity, hash it (`sha256(...)[:24]`, see
+`_job_key` in `agent/monitoring/cron_health.py`) — never emit the raw id. All
+string attributes that could touch user input must pass through
+`redaction.redact_for_export` and be truncated (see `_span_attrs` in
+`agent/monitoring/otlp_exporter.py`).
+
+### Adding a new gauge/metric
+
+1. Emit it in the snapshot builder (`agent/monitoring/gateway_health.py`
+   `build_gateway_health_snapshot`, `cron_health.py` `build_cron_health_snapshot`,
+   or a sibling reader wired into `_read_runtime_snapshot` in
+   `gateway_health_export.py`). Best-effort: never let a reader raise into the
+   collection loop — wrap it and log a **content-free WARNING with the exception
+   TYPE name only** (the pattern the cron and background-work readers use), so a
+   future regression is visible instead of silently dropping the signal.
+2. Register the dotted metric name in the observable-gauge `metric_names` list in
+   `gateway_health_export.py::_start_metric_provider`. **A gauge that is emitted
+   in the snapshot but not registered here is never observed.**
+3. Add the export-table row and an alert example in this file.
+4. If the deployment fronts the exporter with an OpenTelemetry Collector that
+   uses a metric-name allowlist (a `filter/...` processor with `name != "..."`
+   guards), add the new name there too — otherwise the collector drops it before
+   the backend. This is not repo code, but it is the single most common reason a
+   correctly-emitted new metric never appears; call it out in the PR so the
+   deploying operator updates their collector config.
+
+### Adding a new subsystem (a new family of signals)
+
+Mirror the cron pattern (`cron_health.py` + its wiring): put the read/projection
+logic in its own module, expose one `build_<subsystem>_health_snapshot()` that
+returns bounded `GatewayMetric`s (and events if any), and extend it into
+`_read_runtime_snapshot` with the same best-effort try/except-WARNING guard.
+Then do the "adding a metric" checklist for each new name, and the "adding an
+attribute" checklist for each new event attribute. Add a release-validation
+scenario below for the subsystem's failure mode.
+
+### Extending the error-class / status / source / state vocabularies
+
+These are the closed enums that keep the plane bounded. Extend the SET, then the
+classifier, never one without the other:
+
+- **Cron** (`agent/monitoring/cron_health.py`): `_KNOWN_STATUSES`,
+  `_KNOWN_SOURCES`, `_KNOWN_DELIVERY_OUTCOMES`, and the `classify_cron_error`
+  keyword buckets. Anything not in the set is coerced to `unknown` on the way
+  out, so a new value that is not added to the set is invisible.
+- **Gateway/platform** (`agent/monitoring/gateway_health.py`):
+  `_KNOWN_GATEWAY_STATES`, `_KNOWN_PLATFORM_STATES`, and `classify_gateway_error`.
+
+Rules: keep the vocabulary SMALL and operationally meaningful (an error class
+should map to an operator action, not to an exception subclass); a new bucket
+must match on a stable keyword, not on message text that could vary; update the
+`hermes.error_class = ...` list in this file's alert section and the enum's unit
+test so the contract is asserted, not frozen as a count.
+
+### Adding a content-free attribute to an existing event/span
+
+Add the key to the emitter's per-kind `keep_by_kind` allowlist in
+`agent/monitoring/otlp_exporter.py::_span_attrs` (unlisted keys are dropped), run
+it through redaction if it is ever string-shaped, and — as with metrics — if the
+deployment's collector has a span-attribute `keep_keys(...)` allowlist, add the
+attribute there too or it is stripped in transit.
+
+### Verify the whole chain, not just emission
+
+Emitting is necessary but not sufficient. Confirm the signal survives all the
+way to the backend, because the enums, the `metric_names` registration, the
+emitter attribute allowlist, and any collector allowlist each drop unlisted
+values with no error:
+
+```bash
+hermes monitoring status                 # posture
+python scripts/observability/gateway_health_export_probe.py \
+  --endpoint http://127.0.0.1:4318/v1/traces \
+  --log /tmp/cap.jsonl --wait 8          # drive the real exporter
+```
+
+Decode the captured OTLP payload and assert the new name/attribute is present
+AND that no content leaked. When a real collector sits in front, add its
+allowlist entries and re-verify against the backend, not just the local capture.
+
+## Boundaries and roadmap
+
+The `hermes monitoring` CLI intentionally exposes `status` only. This first
+release covers only Hermes Agent-owned service-health and operational-diagnostic
+signals, including Hermes Agent-owned Relay transport health. Team Gateway's
+authoritative shared connector/platform state is explicitly out of scope, as
+are product analytics, audit/quality reporting, and detailed execution traces.
+Shared client usage metrics and enterprise trace telemetry are being designed on
+the NeMo Relay integration with their own consent, policy, and export
+boundaries; this monitoring plane stays narrow so an operator can enable it
+without touching any content-bearing signal. The telemetry surface may be
+reorganized as that lands.