better-dsh 0.2.5 → 0.3.0-c

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/developer-guide/gateway-internals.md

@@ -1,297 +0,0 @@
----
-sidebar_position: 7
-title: "Gateway Internals"
-description: "How the messaging gateway boots, authorizes users, routes sessions, and delivers messages"
----
-
-# Gateway Internals
-
-The messaging gateway is the long-running process that connects Hermes to 20+ external messaging platforms through a unified architecture.
-
-## Key Files
-
-| File | Purpose |
-|------|---------|
-| `gateway/run.py` | `GatewayRunner` facade — composes the `gateway/run_*.py` sibling mixins (startup, adapters, inbound, turn, busy, goals, notifications, shutdown, …) and `gateway/slash_commands_*.py` handlers |
-| `gateway/session.py` | `SessionStore` — conversation persistence and session key construction |
-| `gateway/delivery.py` | Outbound message delivery to target platforms/channels |
-| `gateway/pairing.py` | DM pairing flow for user authorization |
-| `gateway/channel_directory.py` | Maps chat IDs to human-readable names for cron delivery |
-| `gateway/hooks.py` | Hook discovery, loading, and lifecycle event dispatch |
-| `gateway/mirror.py` | Cross-session message mirroring for `send_message` |
-| `gateway/status.py` | Token lock management for profile-scoped gateway instances |
-| `gateway/builtin_hooks/` | Extension point for always-registered hooks (none shipped) |
-| `gateway/platform_registry.py` | Adapter registry, factories, and deferred (lazy) loaders for bundled platform plugins |
-| `plugins/platforms/<name>/` | Bundled messaging adapters (most platforms: `adapter.py` + `plugin.yaml`) |
-| `gateway/platforms/` | Shared `base.py` plus legacy/direct adapters (Signal, API server, webhooks, …) |
-
-## Architecture Overview
-
-```text
-┌─────────────────────────────────────────────────┐
-│                  GatewayRunner                  │
-│                                                 │
-│  ┌──────────┐  ┌──────────┐  ┌──────────┐       │
-│  │ Telegram │  │ Discord  │  │  Slack   │       │
-│  │ Adapter  │  │ Adapter  │  │ Adapter  │       │
-│  └────┬─────┘  └────┬─────┘  └────┬─────┘       │
-│       │             │             │             │
-│       └─────────────┼─────────────┘             │
-│                     ▼                           │
-│              _handle_message()                  │
-│                     │                           │
-│         ┌───────────┼───────────┐               │
-│         ▼           ▼           ▼               │
-│  Slash command   AIAgent    Queue/BG            │
-│    dispatch      creation   sessions            │
-│                     │                           │
-│                     ▼                           │
-│                 SessionStore                    │
-│              (SQLite persistence)               │
-└───────┴─────────────┴─────────────┴─────────────┘
-```
-
-## Message Flow
-
-When a message arrives from any platform:
-
-1. **Platform adapter** receives raw event, normalizes it into a `MessageEvent`
-2. **Base adapter** checks active session guard:
-   - If agent is running for this session → queue message, set interrupt event
-   - If `/approve`, `/deny`, `/stop` → bypass guard (dispatched inline)
-3. **GatewayRunner._handle_message()** receives the event:
-   - Resolve session key via `_session_key_for_source()` (format: `agent:{namespace}:{platform}:{chat_type}:{chat_id}`; the namespace is `main` for the default profile, `<profile>` under multiplexing — see [Multiplexed profiles](#multiplexed-profiles))
-   - Check authorization (see Authorization below)
-   - Check if it's a slash command → dispatch to command handler
-   - Check if agent is already running → intercept commands like `/stop`, `/status`
-   - Otherwise → create `AIAgent` instance and run conversation
-4. **Response** is sent back through the platform adapter
-
-### Session Key Format
-
-Session keys encode the full routing context:
-
-```
-agent:{namespace}:{platform}:{chat_type}:{chat_id}
-```
-
-For example: `agent:main:telegram:private:123456789` for the default profile, or
-`agent:work:telegram:private:123456789` when the multiplexer routes that chat to profile `work`
-(`gateway/session.py::_session_key_namespace`; a profile literally named `main` is marked `main~`).
-
-Thread-aware platforms (Telegram forum topics, Discord threads, Slack threads) may include thread IDs in the chat_id portion. **Never construct session keys manually** — always use `build_session_key()` from `gateway/session.py`.
-
-### Two-Level Message Guard
-
-When an agent is actively running, incoming messages pass through two sequential guards:
-
-1. **Level 1 — Base adapter** (`gateway/platforms/base.py`): Checks `_active_sessions`. If the session is active, queues the message in `_pending_messages` and sets an interrupt event. This catches messages *before* they reach the gateway runner.
-
-2. **Level 2 — Gateway runner** (`gateway/run_inbound.py`): Checks `_running_agents`. Intercepts specific commands (`/stop`, `/new`, `/queue`, `/status`, `/approve`, `/deny`) and routes them appropriately. Everything else triggers `running_agent.interrupt()`.
-
-Commands that must reach the runner while the agent is blocked (like `/approve`) are dispatched **inline** via `await self._message_handler(event)` — they bypass the background task system to avoid race conditions.
-
-## Authorization
-
-The gateway uses a multi-layer authorization check, evaluated in order:
-
-1. **Per-platform allow-all flag** (e.g., `TELEGRAM_ALLOW_ALL_USERS`) — if set, all users on that platform are authorized
-2. **Platform allowlist** (e.g., `TELEGRAM_ALLOWED_USERS`) — comma-separated user IDs
-3. **DM pairing** — authenticated users can pair new users via a pairing code
-4. **Global allow-all** (`GATEWAY_ALLOW_ALL_USERS`, or `gateway.allow_all_users` in `config.yaml`, bridged to the env var by `gateway/config_loader.py::bridge_core_env_settings`) — if set, all users across all platforms are authorized
-5. **Default: deny** — unauthorized users are rejected
-
-### DM Pairing Flow
-
-```text
-Admin: /pair
-Gateway: "Pairing code: ABC123. Share with the user."
-New user: ABC123
-Gateway: "Paired! You're now authorized."
-```
-
-Pairing state is persisted in `gateway/pairing.py` and survives restarts.
-
-## Slash Command Dispatch
-
-All slash commands in the gateway flow through the same resolution pipeline:
-
-1. `resolve_command()` from `hermes_cli/commands.py` maps input to canonical name (handles aliases, prefix matching)
-2. The canonical name is checked against `GATEWAY_KNOWN_COMMANDS`
-3. `_handle_message()` (`gateway/run_inbound.py`) looks the handler up by name — `_handle_<name>_command` on the `gateway/slash_commands_*.py` mixins — via `_command_handler_table` over `_IDLE_COMMANDS` / `_PLAIN_COMMANDS` in `gateway/run_busy.py`; there is no `if canonical == ...` chain
-4. Some commands are gated on config (`gateway_config_gate` on `CommandDef`)
-
-### Running-Agent Guard
-
-Commands that must NOT execute while the agent is processing are rejected early:
-
-While `_quick_key in self._running_agents`, `_dispatch_busy_slash_command()` in `gateway/run_busy.py` routes each recognized command by its `CommandDef.busy_policy` / `busy_handler`: a mid-run variant (`_busy_<key>_command`) if one exists, otherwise the normal handler when `busy_policy` allows it, otherwise a reject message ("⏳ Agent is running — `/model` can't run mid-turn…").
-
-Bypass commands (`/stop`, `/new`, `/approve`, `/deny`, `/queue`, `/status`) have mid-run handlers and are dispatched inline.
-
-## Config Sources
-
-The gateway reads configuration from multiple sources:
-
-| Source | What it provides |
-|--------|-----------------|
-| `~/.hermes/.env` | API keys, bot tokens, platform credentials |
-| `~/.hermes/config.yaml` | Model settings, tool configuration, display options |
-| Environment variables | Override any of the above |
-
-Unlike the CLI (which uses `load_cli_config()` with hardcoded defaults), the gateway reads `config.yaml` directly via YAML loader. This means config keys that exist in the CLI's defaults dict but not in the user's config file may behave differently between CLI and gateway.
-
-## Platform Adapters
-
-Most messaging platforms ship as plugin adapters under `plugins/platforms/<name>/adapter.py`; a few legacy adapters still live directly in `gateway/platforms/`. All extend `BasePlatformAdapter` from `gateway/platforms/base.py`:
-
-```text
-plugins/platforms/                  # plugin-packaged adapters (one dir each)
-├── telegram/adapter.py     # Telegram Bot API (long polling or webhook)
-├── discord/adapter.py      # Discord bot via discord.py
-├── slack/adapter.py        # Slack Socket Mode
-├── whatsapp/adapter.py     # WhatsApp Business Cloud API
-├── matrix/adapter.py       # Matrix via mautrix (optional E2EE)
-├── mattermost/adapter.py   # Mattermost WebSocket API
-├── email/adapter.py        # Email via IMAP/SMTP
-├── sms/adapter.py          # SMS via Twilio
-├── dingtalk/adapter.py     # DingTalk WebSocket
-├── feishu/adapter.py       # Feishu/Lark WebSocket or webhook
-├── wecom/adapter.py        # WeCom (WeChat Work) callback
-├── line/adapter.py         # LINE Messaging API
-├── teams/adapter.py        # Microsoft Teams
-├── irc/adapter.py          # IRC (canonical scoped-lock example)
-├── homeassistant/adapter.py # Home Assistant conversation integration
-└── …                       # google_chat, ntfy, photon, raft, simplex, …
-
-gateway/platforms/                  # core base + legacy direct adapters
-├── base.py              # BasePlatformAdapter — shared logic for all platforms
-├── signal.py            # Signal via signal-cli REST API
-├── weixin.py            # Weixin (personal WeChat) via iLink Bot API
-├── bluebubbles.py       # Apple iMessage via BlueBubbles macOS server
-├── qqbot/               # QQ Bot (Tencent QQ) via Official API v2 (sub-package)
-├── yuanbao.py           # Yuanbao (Tencent) DM/group adapter
-├── msgraph_webhook.py   # Microsoft Graph change-notification webhook (Teams, Outlook, etc.)
-├── webhook.py           # Inbound/outbound webhook adapter
-└── api_server.py        # REST API server adapter
-```
-
-**Deferred loading:** Bundled `kind: platform` plugins register cheap `register_deferred` loaders in `gateway/platform_registry.py` (via `hermes_cli/plugins.py`) so platform SDKs import only when the gateway starts, delivers, or runs setup/status — not on plain `hermes chat`. Resolution loads one adapter on lookup; full enumeration runs pending loaders only on paths that need every platform.
-
-Experimental connector-backed platforms use the generic relay adapter in `gateway/relay/` instead of a direct platform module. When `GATEWAY_RELAY_URL` or `gateway.relay_url` is configured, the gateway registers the `relay` platform, dials the connector over an outbound WebSocket, and receives `descriptor`, `inbound`, and `interrupt_inbound` frames on that same socket. The connector advertises a `CapabilityDescriptor`; Hermes can send normal outbound replies, token-less `follow_up` operations, and interrupt frames back through the relay. The source-grounded wire contract lives in [Relay ↔ Connector contract](relay-connector-contract.md).
-
-Adapters implement a common interface:
-- `connect()` / `disconnect()` — lifecycle management
-- `send()` — outbound message delivery
-- inbound events are normalized into a `MessageEvent` and forwarded via `handle_message()`
-
-Internal push wakes use `gateway.wake.admit_internal_event`: the public
-`handle_message()` still returns `None`, but the event's process-local
-`_gateway_accepted` receipt is set only after scheduling or queue insertion.
-A missing handler, mismatched explicit session key, or queue-cap drop is not
-acceptance. Custom adapters overriding ingress should delegate internal events to
-`BasePlatformAdapter.handle_message()` (or explicitly record actual admission),
-not equate a consumed/dropped callback with acceptance. This receipt is separate
-from heartbeat execution accounting and does not bypass authorization, emergency
-stop, or later turn-preparation gates.
-
-### Token Locks
-
-Adapters that connect with unique credentials call `acquire_scoped_lock()` in `connect()` and `release_scoped_lock()` in `disconnect()`. This prevents two profiles from using the same bot token simultaneously.
-
-A lock conflict is emitted as `{scope}_lock` with `retryable=True` so a **mid-run** reconnect can recover once the other holder exits. At **startup**, though, a live foreign holder is a configuration conflict: `gateway/restart.py::is_global_startup_conflict()` recognizes the `*_lock` / `lock_conflict` code families and the startup router parks the platform `fatal` instead of retry-queueing it. With nothing else connected the gateway exits `78` (`EX_CONFIG`, `gateway_state=startup_failed`) so the supervisor stops restarting it: systemd via `RestartPreventExitStatus=78`, s6 via finish→125, launchd via `KeepAlive.SuccessfulExit=false` after the stderr wrapper maps 78→0. Alongside a genuinely transient peer failure the gateway stays alive and only the peer retries.
-
-## Delivery Path
-
-Outgoing deliveries (`gateway/delivery.py`) handle:
-
-- **Direct reply** — send response back to the originating chat
-- **Home channel delivery** — route cron job outputs and background results to a configured home channel
-- **Explicit target delivery** — the send engine specifying `telegram:-1001234567890`, exposed via the [`hermes send` CLI](/guides/pipe-script-output) for shell scripts and via cron `deliver:` targets
-- **Cross-platform delivery** — deliver to a different platform than the originating message
-
-Cron job deliveries are NOT mirrored into gateway session history — they live in their own cron session only. This is a deliberate design choice to avoid message alternation violations.
-
-## Hooks
-
-Gateway hooks are Python modules that respond to lifecycle events:
-
-### Gateway Hook Events
-
-| Event | When fired |
-|-------|-----------|
-| `gateway:startup` | Gateway process starts |
-| `session:start` | New conversation session begins |
-| `session:end` | Session completes or times out |
-| `session:reset` | User resets session with `/new` |
-| `agent:start` | Agent begins processing a message |
-| `agent:step` | Agent completes one tool-calling iteration |
-| `agent:end` | Agent finishes and returns response |
-| `command:*` | Any slash command is executed |
-
-Hooks are discovered from `gateway/builtin_hooks/` (an extension point — currently empty in the shipped distribution; `_register_builtin_hooks()` is a no-op stub) and `<profile home>/hooks/` (user-installed; `~/.hermes/hooks/` for the default profile, one directory per served profile under multiplexing — paths resolve at call time, never at import). Each hook is a directory with a `HOOK.yaml` manifest and `handler.py`.
-
-## Memory Provider Integration
-
-When a memory provider plugin (e.g., Honcho) is enabled:
-
-1. Gateway creates an `AIAgent` per message with the session ID
-2. The `MemoryManager` initializes the provider with the session context
-3. Provider tools (e.g., `honcho_profile`, `viking_search`) are routed through:
-
-```text
-AIAgent._invoke_tool()
-  → self._memory_manager.handle_tool_call(name, args)
-    → provider.handle_tool_call(name, args)
-```
-
-4. On session end/reset, `on_session_end()` fires for cleanup and final data flush
-
-### Memory Flush Lifecycle
-
-Explicit conversation boundaries (such as `/new`, `/reset`, or `/resume`) flush and finalize the outgoing session. Idle time and daily boundaries never finalize it.
-
-Resource-only TTL, LRU, and memory-pressure eviction commits the cached transcript to configured memory providers before releasing the agent's clients. It does not close the durable conversation: the next turn reloads the same transcript and identity.
-
-## Background Maintenance
-
-The gateway runs periodic maintenance alongside message handling:
-
-- **Cron ticking** — checks job schedules and fires due jobs
-- **Session housekeeping** — reclaims cached resources without ending transcripts
-- **Memory flush** — commits memory before soft cache eviction
-- **Cache refresh** — refreshes model lists and provider status
-
-## Process Management
-
-The gateway runs as a long-lived process, managed via:
-
-- `hermes gateway start` / `hermes gateway stop` — manual control
-- `systemctl` (Linux) or `launchctl` (macOS) — service management
-- PID file at `~/.hermes/gateway.pid` — profile-scoped process tracking
-
-**Profile-scoped vs global**: `start_gateway()` uses profile-scoped PID files. Standalone (one gateway per profile), `hermes -p x gateway stop` stops only that profile's gateway. Under multiplexing there is ONE gateway process, owned by the default profile: `hermes gateway stop` on the default takes every served profile down, and `hermes -p x gateway stop` for a served secondary refuses with exit 78 (it has no gateway of its own). `hermes gateway stop --all` uses global `ps aux` scanning to kill all gateway processes (used during updates). Liveness is decided by `gateway.status.live_gateway_pid_for_home` (PID + start-time fingerprint), never bare PID existence.
-
-## Multiplexed profiles
-
-With `gateway.multiplex_profiles: true` one process serves the default profile plus every live directory under `profiles/` (`hermes_cli/profiles.py::profiles_to_serve(multiplex=True)`). `os.environ` and module globals hold the **launch** profile's values, so every activity for a secondary binds its scope explicitly — a profile is home + secret scope + terminal scope together:
-
-| Activity | Binding |
-|---|---|
-| Routed turn | `gateway/run.py::_profile_runtime_scope(home)` via `run_turn.py::_profile_scope_for_source` |
-| Agent release / eviction (TTL, LRU, memory pressure) | `gateway/run_agent_cache.py::_run_release_in_profile_scope` |
-| Shutdown | `gateway/run_shutdown.py::_finalize_session` |
-| Post-turn media delivery | `gateway/platforms/base.py::_media_delivery_scope` |
-| Cron tick | `cron/scheduler_provider.py::_profile_cron_scope(home)` (one ticker, profiles in sequence) |
-| Child processes (`hermes -p X` workers, relay turns, browser drivers) | `tools/environments/local.py::served_profile_child_env` |
-| Background threads | `agent/memory_provider.py::spawn_context_thread` |
-
-Secret reads fail closed (`agent.secret_scope.get_secret` raises `UnscopedSecretError`) only after `set_multiplex_active(True)`, which the gateway, cron, `gateway migrate` and the Desktop/dashboard `serve` backend set. Adapter YAML never reaches `os.environ` under multiplex: `gateway/platforms/_shared.py::apply_yaml_bridge` seeds `PlatformConfig.extra` and skips the environ write under a secondary's scope; gates read through `platform_gate_env`. Shared-ingress platforms (WhatsApp bridge, Relay) run on the default profile only; a secondary that enables one is logged once and stamped into runtime status (`run_adapters.py::_note_unserved_secondary_platform`). Per-profile isolation as the user sees it: [Multi-profile gateways § What is isolated per profile](/user-guide/multi-profile-gateways#what-is-isolated-per-profile).
-
-## Related Docs
-
-- [Session Storage](./session-storage.md)
-- [Cron Internals](./cron-internals.md)
-- [ACP Internals](./acp-internals.md)
-- [Agent Loop Internals](./agent-loop.md)
-- [Messaging Gateway (User Guide)](/user-guide/messaging)

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/developer-guide/gateway-monitoring.md

@@ -1,309 +0,0 @@
----
-title: "Gateway Monitoring"
-description: "Health export, structured diagnostics, fleet queries and how to extend the monitoring plane"
----
-
-# Gateway Monitoring
-
-Service health monitoring plus structured operational diagnostics for the
-Hermes gateway daemon, exported over OTLP/HTTP to an operator-configured
-endpoint (OpenTelemetry Collector, DataDog, or any OTLP receiver).
-
-This plane is content-free by construction. It exports gateway and cron
-lifecycle state, platform connector health, and content-free warning/error
-diagnostics. It never exports prompts, messages, tool arguments or results,
-job names, destinations, schedules, raw errors, session history, usage
-analytics, audit logs, or detailed execution traces. Run/model/tool trajectory
-capture is a separate plane served by Hermes's native NeMo Relay SDK
-integration and explicitly configured Relay subscribers or exporters.
-
-## What gets exported
-
-| Signal | OTLP route | Content |
-| --- | --- | --- |
-| Gateway gauges | `/v1/metrics` | `hermes.gateway.up/state/busy/drainable/active_agents/background_work/background_delegations/restart_requested`, `hermes.platform.up/degraded` with bounded `error_code` attributes |
-| Health/lifecycle events | `/v1/traces` | `gateway.lifecycle` state transitions (`starting -> running -> draining -> stopped`, `startup_failed`, exit), `gateway.health_snapshot`, platform state changes |
-| Diagnostics | `/v1/logs` | Warning/error gateway events with a constant body and bounded subsystem, severity, error class, and error code attributes; rendered log messages are never exported |
-| Cron scheduler gauges | `/v1/metrics` | Ticker heartbeat and last-success age (omitted when unavailable), a monotonic catch-up-occurrence count from the scheduler's stale-window branch, enabled/running job counts, and overdue count derived from persisted `next_run_at` plus the scheduler's existing grace rule |
-| Cron execution lifecycle | `/v1/traces` | Durable `claimed/running/completed/failed/unknown` states, bounded source and error class, opaque hashed job key, elapsed duration when timestamps exist, and delivery outcome when the scheduler knows it; terminal states make a fail-open flush attempt that can delay completion by up to one second |
-
-Signals carry `service.name`, version, supervision mode, and a stable one-way
-hash of the install id so an operator can distinguish instances without
-exporting account/profile identity or the raw install identifier.
-
-`hermes.gateway.active_agents`, `hermes.gateway.background_work`, and
-`hermes.gateway.background_delegations` are complementary. `active_agents`
-counts foreground message turns plus in-flight cron jobs plus API runs — the
-work the gateway drains on shutdown. `background_work` counts detached work that
-`active_agents` never includes: backgrounded `delegate_task` subagents,
-`terminal(background=true)` processes, and kanban workers; it is
-**task-granular** — a fan-out batch of N subagents counts as N — so it reflects
-real concurrent subagent load. `background_delegations` counts only async
-delegation **units** (each `delegate_task` dispatch is one, a fan-out batch is
-one), matching the async pool's capacity accounting; alert it against
-`delegation.max_concurrent_children` to see slot pressure. Sum `active_agents`
-and `background_work` for total live work per instance; use
-`background_delegations` for pool-saturation.
-
-## Enabling
-
-```yaml
-# config.yaml
-monitoring:
-  gateway_health_export:
-    enabled: true
-  export:
-    otlp:
-      enabled: true
-      endpoint: http://collector-host:4318/v1/traces   # metrics/logs derive
-      headers_env: {}   # header name -> ENV VAR NAME (values never stored)
-```
-
-Check the posture any time:
-
-```bash
-hermes monitoring status
-```
-
-The OpenTelemetry SDK is an optional extra (`pip install 'hermes-agent[otlp]'`),
-lazy-installed on first use. When the SDK is missing or the endpoint is down,
-the gateway runs unaffected: metric collection and ordinary event export stay
-off the hot path, while terminal cron events make one bounded fail-open flush
-attempt of up to one second so the final state is less likely to be lost.
-
-Works identically under systemd/launchd/s6 supervision, containers, tmux, or
-a plain `hermes gateway run`: the exporter lives in the gateway process, so
-no sidecar, agent, or collector is required on the host.
-
-## Collecting into DataDog
-
-Run a customer-owned OpenTelemetry Collector and forward:
-
-```yaml
-# otel-collector config
-receivers:
-  otlp:
-    protocols:
-      http:
-exporters:
-  datadog:
-    api:
-      key: ${env:DD_API_KEY}
-service:
-  pipelines:
-    metrics:   {receivers: [otlp], exporters: [datadog]}
-    traces:    {receivers: [otlp], exporters: [datadog]}
-    logs:      {receivers: [otlp], exporters: [datadog]}
-```
-
-Point `monitoring.export.otlp.endpoint` at the collector. Alerts belong on
-`hermes.gateway.up`, `hermes.platform.up`, and `hermes.platform.degraded`.
-
-## Generic fleet queries and alerts
-
-The exact syntax depends on the customer's observability backend. The examples
-below use PromQL-style expressions and intentionally avoid vendor-specific
-routing, destinations, or customer inventory.
-
-Group fleet views by the opaque `service.instance.id` resource attribute. A
-process that has died cannot emit its own zero, so every deployment needs both
-explicit-state and missing-series detection.
-
-```promql
-# Explicit gateway failure.
-hermes_gateway_up == 0
-
-# Box disappeared or stopped exporting. Choose a window longer than the
-# configured export interval and collector retry allowance.
-absent_over_time(hermes_gateway_up[5m])
-
-# Locally owned bridge is explicitly down.
-hermes_platform_up == 0
-
-# Scheduler thread is stale even though the gateway may still be alive.
-hermes_cron_scheduler_heartbeat_age_seconds > 180
-
-# Ticker loops but has not completed a successful tick recently.
-hermes_cron_scheduler_last_success_age_seconds > 300
-
-# One or more jobs are beyond their existing scheduler grace window.
-hermes_cron_jobs_overdue > 0
-
-# Catch-up counter increased, proving at least one stale occurrence was
-# collapsed and run once after a delay.
-increase(hermes_cron_scheduler_catch_up_occurrences[15m]) > 0
-```
-
-Cron execution lifecycle records arrive as `hermes.cron_execution` spans.
-Alert or derive events from bounded attributes such as:
-
-```text
-hermes.status = failed|unknown
-hermes.delivery_outcome = failed|not_configured
-hermes.error_class = auth_failed|rate_limited|timeout|network_error|
-                     dispatch_failed|interrupted|empty_response|
-                     invalid_config|unknown
-```
-
-Recommended operator views:
-
-1. one row per `service.instance.id` with gateway and configured local-platform
-   state;
-2. scheduler heartbeat, last-success age, running count, overdue count, and
-   catch-up increase;
-3. a cron lifecycle feed keyed only by opaque `hermes.job_key`;
-4. separate alerts for box absence, local bridge down, scheduler stale, cron
-   failed/unknown, delivery failure, and overdue/catch-up activity.
-
-Keep alert thresholds and routing in deployment-owned configuration. Do not add
-job names, prompts, outputs, schedules, destinations, raw errors, profile names,
-or account identity merely to make a dashboard easier to read.
-
-## Release-validation scenarios
-
-Before accepting a deployment, force and verify all five cases through the real
-collector and backend:
-
-1. **Cron success:** observe `claimed -> running -> completed`, duration, and a
-   truthful delivery outcome.
-2. **Cron failure:** observe `failed` plus a bounded error class, with no raw
-   exception or content in the decoded OTLP payload.
-3. **Cron interruption:** stop the owning gateway during execution, restart it,
-   and observe recovery to `unknown`.
-4. **Locally owned bridge outage:** break one native connector, observe its
-   bounded down/retrying/fatal state and recovery, and verify unaffected boxes
-   remain healthy.
-5. **Killed gateway:** terminate one canary, verify missing-series detection,
-   restart it, and confirm the same opaque instance identity returns.
-
-Hermes Agent-owned Relay transport health remains in scope. A separate gateway
-or connector service remains authoritative for any shared connected-platform
-state that it owns and should export that state through its own telemetry path.
-
-For every scenario, verify the signal and alert clear on recovery, other boxes
-remain unaffected, collector failure stays fail-open, and decoded metrics,
-spans, logs, and resource attributes remain content-free.
-
-## Local smoke test (no Docker)
-
-```bash
-# terminal 1: capture collector on :4318
-python scripts/observability/otel_capture_collector.py \
-  --host 127.0.0.1 --port 4318 --log /tmp/hermes_otel_capture.jsonl
-
-# terminal 2: drive the real exporter through lifecycle transitions,
-# a fatal platform, and a structured warning event, then flush
-python scripts/observability/gateway_health_export_probe.py \
-  --endpoint http://127.0.0.1:4318/v1/traces \
-  --log /tmp/hermes_otel_capture.jsonl --wait 8
-# exit 0 prints: {"requests": 6, "paths": ["/v1/logs", "/v1/metrics", "/v1/traces"]}
-```
-
-## Maintaining and extending this plane
-
-This plane is a **fixed, enumerated, content-free vocabulary** by design. Adding
-a signal is not just "emit a new metric" — every new name and attribute must be
-declared in each layer that enforces the bounded vocabulary, or it is silently
-dropped downstream. Follow the checklist for the change you are making. The
-golden rule: **a new signal that is emitted but not declared in every layer
-looks like a code bug but is a vocabulary-registration bug — nothing errors, the
-signal just never arrives.**
-
-### Content-free invariant (applies to every change)
-
-Before adding anything, confirm it cannot carry content. Numbers, booleans,
-ages, durations, monotonic counts, and one-way hashes are safe. **Never** add an
-attribute that can hold a job name, prompt, output, schedule, destination, raw
-exception text, file path, profile name, account id, or free-form string. When
-you must key a record to a job/entity, hash it (`sha256(...)[:24]`, see
-`_job_key` in `agent/monitoring/cron_health.py`) — never emit the raw id. All
-string attributes that could touch user input must pass through
-`redaction.redact_for_export` and be truncated (see `_span_attrs` in
-`agent/monitoring/otlp_exporter.py`).
-
-### Adding a new gauge/metric
-
-1. Emit it in the snapshot builder (`agent/monitoring/gateway_health.py`
-   `build_gateway_health_snapshot`, `cron_health.py` `build_cron_health_snapshot`,
-   or a sibling reader wired into `_read_runtime_snapshot` in
-   `gateway_health_export.py`). Best-effort: never let a reader raise into the
-   collection loop — wrap it and log a **content-free WARNING with the exception
-   TYPE name only** (the pattern the cron and background-work readers use), so a
-   future regression is visible instead of silently dropping the signal.
-2. Register the dotted metric name in the observable-gauge `metric_names` list in
-   `gateway_health_export.py::_start_metric_provider`. **A gauge that is emitted
-   in the snapshot but not registered here is never observed.**
-3. Add the export-table row and an alert example in this file.
-4. If the deployment fronts the exporter with an OpenTelemetry Collector that
-   uses a metric-name allowlist (a `filter/...` processor with `name != "..."`
-   guards), add the new name there too — otherwise the collector drops it before
-   the backend. This is not repo code, but it is the single most common reason a
-   correctly-emitted new metric never appears; call it out in the PR so the
-   deploying operator updates their collector config.
-
-### Adding a new subsystem (a new family of signals)
-
-Mirror the cron pattern (`cron_health.py` + its wiring): put the read/projection
-logic in its own module, expose one `build_<subsystem>_health_snapshot()` that
-returns bounded `GatewayMetric`s (and events if any), and extend it into
-`_read_runtime_snapshot` with the same best-effort try/except-WARNING guard.
-Then do the "adding a metric" checklist for each new name, and the "adding an
-attribute" checklist for each new event attribute. Add a release-validation
-scenario below for the subsystem's failure mode.
-
-### Extending the error-class / status / source / state vocabularies
-
-These are the closed enums that keep the plane bounded. Extend the SET, then the
-classifier, never one without the other:
-
-- **Cron** (`agent/monitoring/cron_health.py`): `_KNOWN_STATUSES`,
-  `_KNOWN_SOURCES`, `_KNOWN_DELIVERY_OUTCOMES`, and the `classify_cron_error`
-  keyword buckets. Anything not in the set is coerced to `unknown` on the way
-  out, so a new value that is not added to the set is invisible.
-- **Gateway/platform** (`agent/monitoring/gateway_health.py`):
-  `_KNOWN_GATEWAY_STATES`, `_KNOWN_PLATFORM_STATES`, and `classify_gateway_error`.
-
-Rules: keep the vocabulary SMALL and operationally meaningful (an error class
-should map to an operator action, not to an exception subclass); a new bucket
-must match on a stable keyword, not on message text that could vary; update the
-`hermes.error_class = ...` list in this file's alert section and the enum's unit
-test so the contract is asserted, not frozen as a count.
-
-### Adding a content-free attribute to an existing event/span
-
-Add the key to the emitter's per-kind `keep_by_kind` allowlist in
-`agent/monitoring/otlp_exporter.py::_span_attrs` (unlisted keys are dropped), run
-it through redaction if it is ever string-shaped, and — as with metrics — if the
-deployment's collector has a span-attribute `keep_keys(...)` allowlist, add the
-attribute there too or it is stripped in transit.
-
-### Verify the whole chain, not just emission
-
-Emitting is necessary but not sufficient. Confirm the signal survives all the
-way to the backend, because the enums, the `metric_names` registration, the
-emitter attribute allowlist, and any collector allowlist each drop unlisted
-values with no error:
-
-```bash
-hermes monitoring status                 # posture
-python scripts/observability/gateway_health_export_probe.py \
-  --endpoint http://127.0.0.1:4318/v1/traces \
-  --log /tmp/cap.jsonl --wait 8          # drive the real exporter
-```
-
-Decode the captured OTLP payload and assert the new name/attribute is present
-AND that no content leaked. When a real collector sits in front, add its
-allowlist entries and re-verify against the backend, not just the local capture.
-
-## Boundaries and roadmap
-
-The `hermes monitoring` CLI intentionally exposes `status` only. This first
-release covers only Hermes Agent-owned service-health and operational-diagnostic
-signals, including Hermes Agent-owned Relay transport health. Team Gateway's
-authoritative shared connector/platform state is explicitly out of scope, as
-are product analytics, audit/quality reporting, and detailed execution traces.
-Shared client usage metrics and enterprise trace telemetry are being designed on
-the NeMo Relay integration with their own consent, policy, and export
-boundaries; this monitoring plane stays narrow so an operator can enable it
-without touching any content-bearing signal. The telemetry surface may be
-reorganized as that lands.