nodmix 2026.5.25

package/docs/gateway/pairing.md

@@ -0,0 +1,207 @@
+---
+summary: "Gateway-owned node pairing (Option B) for iOS and other remote nodes"
+read_when:
+  - Implementing node pairing approvals without macOS UI
+  - Adding CLI flows for approving remote nodes
+  - Extending gateway protocol with node management
+title: "Gateway-owned pairing"
+---
+
+In Gateway-owned pairing, the **Gateway** is the source of truth for which nodes
+are allowed to join. UIs (macOS app, future clients) are just frontends that
+approve or reject pending requests.
+
+**Important:** WS nodes use **device pairing** (role `node`) during `connect`.
+`node.pair.*` is a separate pairing store and does **not** gate the WS handshake.
+Only clients that explicitly call `node.pair.*` use this flow.
+
+## Concepts
+
+- **Pending request**: a node asked to join; requires approval.
+- **Paired node**: approved node with an issued auth token.
+- **Transport**: the Gateway WS endpoint forwards requests but does not decide
+  membership. (Legacy TCP bridge support has been removed.)
+
+## How pairing works
+
+1. A node connects to the Gateway WS and requests pairing.
+2. The Gateway stores a **pending request** and emits `node.pair.requested`.
+3. You approve or reject the request (CLI or UI).
+4. On approval, the Gateway issues a **new token** (tokens are rotated on re-pair).
+5. The node reconnects using the token and is now "paired".
+
+Pending requests expire automatically after **5 minutes**.
+
+## CLI workflow (headless friendly)
+
+```bash
+nodmix nodes pending
+nodmix nodes approve <requestId>
+nodmix nodes reject <requestId>
+nodmix nodes status
+nodmix nodes remove --node <id|name|ip>
+nodmix nodes rename --node <id|name|ip> --name "Living Room iPad"
+```
+
+`nodes status` shows paired/connected nodes and their capabilities.
+
+## API surface (gateway protocol)
+
+Events:
+
+- `node.pair.requested` - emitted when a new pending request is created.
+- `node.pair.resolved` - emitted when a request is approved/rejected/expired.
+
+Methods:
+
+- `node.pair.request` - create or reuse a pending request.
+- `node.pair.list` - list pending + paired nodes (`operator.pairing`).
+- `node.pair.approve` - approve a pending request (issues token).
+- `node.pair.reject` - reject a pending request.
+- `node.pair.remove` - remove a stale paired node entry.
+- `node.pair.verify` - verify `{ nodeId, token }`.
+
+Notes:
+
+- `node.pair.request` is idempotent per node: repeated calls return the same
+  pending request.
+- Repeated requests for the same pending node also refresh the stored node
+  metadata and the latest allowlisted declared command snapshot for operator visibility.
+- Approval **always** generates a fresh token; no token is ever returned from
+  `node.pair.request`.
+- Operator scope levels and approval-time checks are summarized in
+  [Operator scopes](/gateway/operator-scopes).
+- Requests may include `silent: true` as a hint for auto-approval flows.
+- `node.pair.approve` uses the pending request's declared commands to enforce
+  extra approval scopes:
+  - commandless request: `operator.pairing`
+  - non-exec command request: `operator.pairing` + `operator.write`
+  - `system.run` / `system.run.prepare` / `system.which` request:
+    `operator.pairing` + `operator.admin`
+
+<Warning>
+Node pairing is a trust and identity flow plus token issuance. It does **not** pin the live node command surface per node.
+
+- Live node commands come from what the node declares on connect after the gateway's global node command policy (`gateway.nodes.allowCommands` and `denyCommands`) is applied.
+- Per-node `system.run` allow and ask policy lives on the node in `exec.approvals.node.*`, not in the pairing record.
+
+</Warning>
+
+## Node command gating (2026.3.31+)
+
+<Warning>
+**Breaking change:** Starting with `2026.3.31`, node commands are disabled until node pairing is approved. Device pairing alone is no longer enough to expose declared node commands.
+</Warning>
+
+When a node connects for the first time, pairing is requested automatically. Until the pairing request is approved, all pending node commands from that node are filtered and will not execute. Once trust is established through pairing approval, the node's declared commands become available subject to the normal command policy.
+
+This means:
+
+- Nodes that were previously relying on device pairing alone to expose commands must now complete node pairing.
+- Commands queued before pairing approval are dropped, not deferred.
+
+## Node event trust boundaries (2026.3.31+)
+
+<Warning>
+**Breaking change:** Node-originated runs now stay on a reduced trusted surface.
+</Warning>
+
+Node-originated summaries and related session events are restricted to the intended trusted surface. Notification-driven or node-triggered flows that previously relied on broader host or session tool access may need adjustment. This hardening ensures that node events cannot escalate into host-level tool access beyond what the node's trust boundary permits.
+
+Durable node presence updates follow the same identity boundary. The `node.presence.alive` event is
+accepted only from authenticated node device sessions and updates pairing metadata only when the
+device/node identity is already paired. Self-declared `client.id` values are not enough to write
+last-seen state.
+
+## Auto-approval (macOS app)
+
+The macOS app can optionally attempt a **silent approval** when:
+
+- the request is marked `silent`, and
+- the app can verify an SSH connection to the gateway host using the same user.
+
+If silent approval fails, it falls back to the normal "Approve/Reject" prompt.
+
+## Trusted-CIDR device auto-approval
+
+WS device pairing for `role: node` remains manual by default. For private
+node networks where the Gateway already trusts the network path, operators can
+opt in with explicit CIDRs or exact IPs:
+
+```json5
+{
+  gateway: {
+    nodes: {
+      pairing: {
+        autoApproveCidrs: ["192.168.1.0/24"],
+      },
+    },
+  },
+}
+```
+
+Security boundary:
+
+- Disabled when `gateway.nodes.pairing.autoApproveCidrs` is unset.
+- No blanket LAN or private-network auto-approve mode exists.
+- Only fresh `role: node` device pairing with no requested scopes is eligible.
+- Operator, browser, Control UI, and WebChat clients stay manual.
+- Role, scope, metadata, and public-key upgrades stay manual.
+- Same-host loopback trusted-proxy header paths are not eligible because that
+  path can be spoofed by local callers.
+
+## Metadata-upgrade auto-approval
+
+When an already paired device reconnects with only non-sensitive metadata
+changes (for example, display name or client platform hints), Nodmix treats
+that as a `metadata-upgrade`. Silent auto-approval is narrow: it applies only
+to trusted non-browser local reconnects that already proved possession of local
+or shared credentials, including same-host native app reconnects after OS
+version metadata changes. Browser/Control UI clients and remote clients still
+use the explicit re-approval flow. Scope upgrades (read to write/admin) and
+public key changes are **not** eligible for metadata-upgrade auto-approval -
+they stay as explicit re-approval requests.
+
+## QR pairing helpers
+
+`/pair qr` renders the pairing payload as structured media so mobile and
+browser clients can scan it directly.
+
+Deleting a device also sweeps any stale pending pairing requests for that
+device id, so `nodes pending` does not show orphaned rows after a revoke.
+
+## Locality and forwarded headers
+
+Gateway pairing treats a connection as loopback only when both the raw socket
+and any upstream proxy evidence agree. If a request arrives on loopback but
+carries `Forwarded`, any `X-Forwarded-*`, or `X-Real-IP` header evidence, that
+forwarded-header evidence disqualifies the loopback locality claim. The pairing
+path then requires explicit approval instead of silently treating the request as
+a same-host connect. See [Trusted Proxy Auth](/gateway/trusted-proxy-auth) for
+the equivalent rule on operator auth.
+
+## Storage (local, private)
+
+Pairing state is stored under the Gateway state directory (default `~/.nodmix`):
+
+- `~/.nodmix/nodes/paired.json`
+- `~/.nodmix/nodes/pending.json`
+
+If you override `NODMIX_STATE_DIR`, the `nodes/` folder moves with it.
+
+Security notes:
+
+- Tokens are secrets; treat `paired.json` as sensitive.
+- Rotating a token requires re-approval (or deleting the node entry).
+
+## Transport behavior
+
+- The transport is **stateless**; it does not store membership.
+- If the Gateway is offline or pairing is disabled, nodes cannot pair.
+- If the Gateway is in remote mode, pairing still happens against the remote Gateway's store.
+
+## Related
+
+- [Channel pairing](/channels/pairing)
+- [Nodes](/nodes)
+- [Devices CLI](/cli/devices)

package/docs/gateway/prometheus.md

@@ -0,0 +1,230 @@
+---
+summary: "Expose Nodmix diagnostics as Prometheus text metrics through the diagnostics-prometheus plugin"
+title: "Prometheus metrics"
+sidebarTitle: "Prometheus"
+read_when:
+  - You want Prometheus, Grafana, VictoriaMetrics, or another scraper to collect Nodmix Gateway metrics
+  - You need the Prometheus metric names and label policy for dashboards or alerts
+  - You want metrics without running an OpenTelemetry collector
+---
+
+Nodmix can expose diagnostics metrics through the official `diagnostics-prometheus` plugin. It listens to trusted internal diagnostics and renders a Prometheus text endpoint at:
+
+```text
+GET /api/diagnostics/prometheus
+```
+
+Content type is `text/plain; version=0.0.4; charset=utf-8`, the standard Prometheus exposition format.
+
+<Warning>
+The route uses Gateway authentication (operator scope). Do not expose it as a public unauthenticated `/metrics` endpoint. Scrape it through the same auth path you use for other operator APIs.
+</Warning>
+
+For traces, logs, OTLP push, and OpenTelemetry GenAI semantic attributes, see [OpenTelemetry export](/gateway/opentelemetry).
+
+## Quick start
+
+<Steps>
+  <Step title="Install the plugin">
+    ```bash
+    nodmix plugins install clawhub:@nodmix/diagnostics-prometheus
+    ```
+  </Step>
+  <Step title="Enable the plugin">
+    <Tabs>
+      <Tab title="Config">
+        ```json5
+        {
+          plugins: {
+            allow: ["diagnostics-prometheus"],
+            entries: {
+              "diagnostics-prometheus": { enabled: true },
+            },
+          },
+          diagnostics: {
+            enabled: true,
+          },
+        }
+        ```
+      </Tab>
+      <Tab title="CLI">
+        ```bash
+        nodmix plugins enable diagnostics-prometheus
+        ```
+      </Tab>
+    </Tabs>
+  </Step>
+  <Step title="Restart the Gateway">
+    The HTTP route is registered at plugin startup, so reload after enabling.
+  </Step>
+  <Step title="Scrape the protected route">
+    Send the same gateway auth your operator clients use:
+
+    ```bash
+    curl -H "Authorization: Bearer $NODMIX_GATEWAY_TOKEN" \
+      http://127.0.0.1:18789/api/diagnostics/prometheus
+    ```
+
+  </Step>
+  <Step title="Wire Prometheus">
+    ```yaml
+    # prometheus.yml
+    scrape_configs:
+      - job_name: nodmix
+        scrape_interval: 30s
+        metrics_path: /api/diagnostics/prometheus
+        authorization:
+          credentials_file: /etc/prometheus/nodmix-gateway-token
+        static_configs:
+          - targets: ["nodmix-gateway:18789"]
+    ```
+  </Step>
+</Steps>
+
+<Note>
+`diagnostics.enabled: true` is required. Without it, the plugin still registers the HTTP route but no diagnostic events flow into the exporter, so the response is empty.
+</Note>
+
+## Metrics exported
+
+| Metric                                        | Type      | Labels                                                                                    |
+| --------------------------------------------- | --------- | ----------------------------------------------------------------------------------------- |
+| `nodmix_run_completed_total`                | counter   | `channel`, `model`, `outcome`, `provider`, `trigger`                                      |
+| `nodmix_run_duration_seconds`               | histogram | `channel`, `model`, `outcome`, `provider`, `trigger`                                      |
+| `nodmix_model_call_total`                   | counter   | `api`, `error_category`, `model`, `outcome`, `provider`, `transport`                      |
+| `nodmix_model_call_duration_seconds`        | histogram | `api`, `error_category`, `model`, `outcome`, `provider`, `transport`                      |
+| `nodmix_model_tokens_total`                 | counter   | `agent`, `channel`, `model`, `provider`, `token_type`                                     |
+| `nodmix_gen_ai_client_token_usage`          | histogram | `model`, `provider`, `token_type`                                                         |
+| `nodmix_model_cost_usd_total`               | counter   | `agent`, `channel`, `model`, `provider`                                                   |
+| `nodmix_tool_execution_total`               | counter   | `error_category`, `outcome`, `params_kind`, `tool`                                        |
+| `nodmix_tool_execution_duration_seconds`    | histogram | `error_category`, `outcome`, `params_kind`, `tool`                                        |
+| `nodmix_harness_run_total`                  | counter   | `channel`, `error_category`, `harness`, `model`, `outcome`, `phase`, `plugin`, `provider` |
+| `nodmix_harness_run_duration_seconds`       | histogram | `channel`, `error_category`, `harness`, `model`, `outcome`, `phase`, `plugin`, `provider` |
+| `nodmix_message_received_total`             | counter   | `channel`, `source`                                                                       |
+| `nodmix_message_dispatch_started_total`     | counter   | `channel`, `source`                                                                       |
+| `nodmix_message_dispatch_completed_total`   | counter   | `channel`, `outcome`, `reason`, `source`                                                  |
+| `nodmix_message_dispatch_duration_seconds`  | histogram | `channel`, `outcome`, `reason`, `source`                                                  |
+| `nodmix_message_processed_total`            | counter   | `channel`, `outcome`, `reason`                                                            |
+| `nodmix_message_processed_duration_seconds` | histogram | `channel`, `outcome`, `reason`                                                            |
+| `nodmix_message_delivery_started_total`     | counter   | `channel`, `delivery_kind`                                                                |
+| `nodmix_message_delivery_total`             | counter   | `channel`, `delivery_kind`, `error_category`, `outcome`                                   |
+| `nodmix_message_delivery_duration_seconds`  | histogram | `channel`, `delivery_kind`, `error_category`, `outcome`                                   |
+| `nodmix_talk_event_total`                   | counter   | `brain`, `event_type`, `mode`, `provider`, `transport`                                    |
+| `nodmix_talk_event_duration_seconds`        | histogram | `brain`, `event_type`, `mode`, `provider`, `transport`                                    |
+| `nodmix_talk_audio_bytes`                   | histogram | `brain`, `event_type`, `mode`, `provider`, `transport`                                    |
+| `nodmix_queue_lane_size`                    | gauge     | `lane`                                                                                    |
+| `nodmix_queue_lane_wait_seconds`            | histogram | `lane`                                                                                    |
+| `nodmix_session_state_total`                | counter   | `reason`, `state`                                                                         |
+| `nodmix_session_queue_depth`                | gauge     | `state`                                                                                   |
+| `nodmix_session_turn_created_total`         | counter   | `agent`, `channel`, `trigger`                                                             |
+| `nodmix_session_recovery_total`             | counter   | `action`, `active_work_kind`, `state`, `status`                                           |
+| `nodmix_session_recovery_age_seconds`       | histogram | `action`, `active_work_kind`, `state`, `status`                                           |
+| `nodmix_memory_bytes`                       | gauge     | `kind`                                                                                    |
+| `nodmix_memory_rss_bytes`                   | histogram | none                                                                                      |
+| `nodmix_memory_pressure_total`              | counter   | `level`, `reason`                                                                         |
+| `nodmix_telemetry_exporter_total`           | counter   | `exporter`, `reason`, `signal`, `status`                                                  |
+| `nodmix_prometheus_series_dropped_total`    | counter   | none                                                                                      |
+
+## Label policy
+
+<AccordionGroup>
+  <Accordion title="Bounded, low-cardinality labels">
+    Prometheus labels stay bounded and low-cardinality. The exporter does not emit raw diagnostic identifiers such as `runId`, `sessionKey`, `sessionId`, `callId`, `toolCallId`, message IDs, chat IDs, or provider request IDs.
+
+    Label values are redacted and must match Nodmix's low-cardinality character policy. Values that fail the policy are replaced with `unknown`, `other`, or `none`, depending on the metric.
+
+  </Accordion>
+  <Accordion title="Series cap and overflow accounting">
+    The exporter caps retained time series in memory at **2048** series across counters, gauges, and histograms combined. New series beyond that cap are dropped, and `nodmix_prometheus_series_dropped_total` increments by one each time.
+
+    Watch this counter as a hard signal that an attribute upstream is leaking high-cardinality values. The exporter never lifts the cap automatically; if it climbs, fix the source rather than disabling the cap.
+
+  </Accordion>
+  <Accordion title="What never appears in Prometheus output">
+    - prompt text, response text, tool inputs, tool outputs, system prompts
+    - Talk transcripts, audio payloads, call ids, room ids, handoff tokens, turn ids, and raw session ids
+    - raw provider request IDs (only bounded hashes, where applicable, on spans — never on metrics)
+    - session keys and session IDs
+    - hostnames, file paths, secret values
+
+  </Accordion>
+</AccordionGroup>
+
+## PromQL recipes
+
+```promql
+# Tokens per minute, split by provider
+sum by (provider) (rate(nodmix_model_tokens_total[1m]))
+
+# Spend (USD) over the last hour, by model
+sum by (model) (increase(nodmix_model_cost_usd_total[1h]))
+
+# 95th percentile model run duration
+histogram_quantile(
+  0.95,
+  sum by (le, provider, model)
+    (rate(nodmix_run_duration_seconds_bucket[5m]))
+)
+
+# Queue wait time SLO (95p under 2s)
+histogram_quantile(
+  0.95,
+  sum by (le, lane) (rate(nodmix_queue_lane_wait_seconds_bucket[5m]))
+) < 2
+
+# Dropped Prometheus series (cardinality alarm)
+increase(nodmix_prometheus_series_dropped_total[15m]) > 0
+```
+
+<Tip>
+Prefer `gen_ai_client_token_usage` for cross-provider dashboards: it follows the OpenTelemetry GenAI semantic conventions and is consistent with metrics from non-Nodmix GenAI services.
+</Tip>
+
+## Choosing between Prometheus and OpenTelemetry export
+
+Nodmix supports both surfaces independently. You can run either, both, or neither.
+
+<Tabs>
+  <Tab title="diagnostics-prometheus">
+    - **Pull** model: Prometheus scrapes `/api/diagnostics/prometheus`.
+    - No external collector required.
+    - Authenticated through normal Gateway auth.
+    - Surface is metrics only (no traces or logs).
+    - Best for stacks already standardized on Prometheus + Grafana.
+
+  </Tab>
+  <Tab title="diagnostics-otel">
+    - **Push** model: Nodmix sends OTLP/HTTP to a collector or OTLP-compatible backend.
+    - Surface includes metrics, traces, and logs.
+    - Bridges to Prometheus through an OpenTelemetry Collector (`prometheus` or `prometheusremotewrite` exporter) when you need both.
+    - See [OpenTelemetry export](/gateway/opentelemetry) for the full catalog.
+
+  </Tab>
+</Tabs>
+
+## Troubleshooting
+
+<AccordionGroup>
+  <Accordion title="Empty response body">
+    - Check `diagnostics.enabled: true` in config.
+    - Confirm the plugin is enabled and loaded with `nodmix plugins list --enabled`.
+    - Generate some traffic; counters and histograms only emit lines after at least one event.
+
+  </Accordion>
+  <Accordion title="401 / unauthorized">
+    The endpoint requires the Gateway operator scope (`auth: "gateway"` with `gatewayRuntimeScopeSurface: "trusted-operator"`). Use the same token or password Prometheus uses for any other Gateway operator route. There is no public unauthenticated mode.
+  </Accordion>
+  <Accordion title="`nodmix_prometheus_series_dropped_total` is climbing">
+    A new attribute is exceeding the **2048**-series cap. Inspect recent metrics for an unexpectedly high-cardinality label and fix it at the source. The exporter intentionally drops new series instead of silently rewriting labels.
+  </Accordion>
+  <Accordion title="Prometheus shows stale series after a restart">
+    The plugin keeps state in memory only. After a Gateway restart, counters reset to zero and gauges restart at their next reported value. Use PromQL `rate()` and `increase()` to handle resets cleanly.
+  </Accordion>
+</AccordionGroup>
+
+## Related
+
+- [Diagnostics export](/gateway/diagnostics) — local diagnostics zip for support bundles
+- [Health and readiness](/gateway/health) — `/healthz` and `/readyz` probes
+- [Logging](/logging) — file-based logging
+- [OpenTelemetry export](/gateway/opentelemetry) — OTLP push for traces, metrics, and logs