nodmix 2026.5.25

package/docs/gateway/diagnostics.md

@@ -0,0 +1,213 @@
+---
+summary: "Create shareable Gateway diagnostics bundles for bug reports"
+title: "Diagnostics export"
+read_when:
+  - Preparing a bug report or support request
+  - Debugging Gateway crashes, restarts, memory pressure, or oversized payloads
+  - Reviewing what diagnostics data is recorded or redacted
+---
+
+Nodmix can create a local diagnostics zip for bug reports. It combines
+sanitized Gateway status, health, logs, config shape, and recent payload-free
+stability events.
+
+Treat diagnostics bundles like secrets until you have reviewed them. They are
+designed to omit or redact payloads and credentials, but they still summarize
+local Gateway logs and host-level runtime state.
+
+## Quick start
+
+```bash
+nodmix gateway diagnostics export
+```
+
+The command prints the written zip path. To choose a path:
+
+```bash
+nodmix gateway diagnostics export --output nodmix-diagnostics.zip
+```
+
+For automation:
+
+```bash
+nodmix gateway diagnostics export --json
+```
+
+## Chat command
+
+Owners can use `/diagnostics [note]` in chat to request a local Gateway export.
+Use this when the bug happened in a real conversation and you want one
+copy-pasteable report for support:
+
+1. Send `/diagnostics` in the conversation where you noticed the problem. Add a
+   short note if it helps, for example `/diagnostics bad tool choice`.
+2. Nodmix sends the diagnostics preamble and asks for one explicit exec
+   approval. The approval runs `nodmix gateway diagnostics export --json`.
+   Do not approve diagnostics through an allow-all rule.
+3. After approval, Nodmix replies with a pasteable report containing the local
+   bundle path, manifest summary, privacy notes, and relevant session ids.
+
+In group chats, an owner can still run `/diagnostics`, but Nodmix does not
+post the diagnostic details back into the shared chat. It sends the preamble,
+approval prompts, Gateway export result, and Codex session/thread breakdown to
+the owner through the private approval route. The group only gets a short notice
+that the diagnostics flow was sent privately. If Nodmix cannot find a private
+owner route, the command fails closed and asks the owner to run it from a DM.
+
+When the active Nodmix session is using the native OpenAI Codex harness,
+the same exec approval also covers an OpenAI feedback upload for the Codex
+runtime threads Nodmix knows about. That upload is separate from the local
+Gateway zip and appears only for Codex harness sessions. Before approval, the
+prompt explains that approving diagnostics will also send Codex feedback, but it
+does not list Codex session or thread ids. After approval, the chat reply lists
+the channels, Nodmix session ids, Codex thread ids, and local resume commands
+for the threads that were sent to OpenAI servers. If you deny or ignore the
+approval, Nodmix does not run the export, does not send Codex feedback, and
+does not print the Codex ids.
+
+That makes the common Codex debugging loop short: notice the bad behavior in
+Telegram, Discord, or another channel, run `/diagnostics`, approve once, share
+the report with support, then run the printed `codex resume <thread-id>` command
+locally if you want to inspect the native Codex thread yourself. See
+[Codex harness](/plugins/codex-harness#inspect-codex-threads-locally) for
+that inspection workflow.
+
+## What the export contains
+
+The zip includes:
+
+- `summary.md`: human-readable overview for support.
+- `diagnostics.json`: machine-readable summary of config, logs, status, health,
+  and stability data.
+- `manifest.json`: export metadata and file list.
+- Sanitized config shape and non-secret config details.
+- Sanitized log summaries and recent redacted log lines.
+- Best-effort Gateway status and health snapshots.
+- `stability/latest.json`: newest persisted stability bundle, when available.
+
+The export is useful even when the Gateway is unhealthy. If the Gateway cannot
+answer status or health requests, the local logs, config shape, and latest
+stability bundle are still collected when available.
+
+## Privacy model
+
+Diagnostics are designed to be shareable. The export keeps operational data
+that helps debugging, such as:
+
+- subsystem names, plugin ids, provider ids, channel ids, and configured modes
+- status codes, durations, byte counts, queue state, and memory readings
+- sanitized log metadata and redacted operational messages
+- config shape and non-secret feature settings
+
+The export omits or redacts:
+
+- chat text, prompts, instructions, webhook bodies, and tool outputs
+- credentials, API keys, tokens, cookies, and secret values
+- raw request or response bodies
+- account ids, message ids, raw session ids, hostnames, and local usernames
+
+When a log message looks like user, chat, prompt, or tool payload text, the
+export keeps only that a message was omitted and the byte count.
+
+## Stability recorder
+
+The Gateway records a bounded, payload-free stability stream by default when
+diagnostics are enabled. It is for operational facts, not content.
+
+The same diagnostic heartbeat records liveness samples when the Gateway keeps
+running but the Node.js event loop or CPU looks saturated. These
+`diagnostic.liveness.warning` events include event-loop delay, event-loop
+utilization, CPU-core ratio, active/waiting/queued session counts, the current
+startup/runtime phase when known, recent phase spans, and bounded active/queued
+work labels. Idle samples stay in telemetry at `info` level. Liveness samples
+become Gateway warnings only when work is waiting or queued, or when active work
+overlaps with sustained event-loop delay. Transient max-delay spikes during
+otherwise healthy background work stay in debug logs. They do not restart the
+Gateway by themselves.
+
+Startup phases also emit `diagnostic.phase.completed` events with wall-clock and
+CPU timing. Stalled embedded-run diagnostics mark `terminalProgressStale=true`
+when the last bridge progress looked terminal, such as a raw response item or
+response completion event, but the Gateway still considers the embedded run
+active.
+
+Inspect the live recorder:
+
+```bash
+nodmix gateway stability
+nodmix gateway stability --type payload.large
+nodmix gateway stability --json
+```
+
+Inspect the newest persisted stability bundle after a fatal exit, shutdown
+timeout, or restart startup failure:
+
+```bash
+nodmix gateway stability --bundle latest
+```
+
+Create a diagnostics zip from the newest persisted bundle:
+
+```bash
+nodmix gateway stability --bundle latest --export
+```
+
+Persisted bundles live under `~/.nodmix/logs/stability/` when events exist.
+
+## Useful options
+
+```bash
+nodmix gateway diagnostics export \
+  --output nodmix-diagnostics.zip \
+  --log-lines 5000 \
+  --log-bytes 1000000
+```
+
+- `--output <path>`: write to a specific zip path.
+- `--log-lines <count>`: maximum sanitized log lines to include.
+- `--log-bytes <bytes>`: maximum log bytes to inspect.
+- `--url <url>`: Gateway WebSocket URL for status and health snapshots.
+- `--token <token>`: Gateway token for status and health snapshots.
+- `--password <password>`: Gateway password for status and health snapshots.
+- `--timeout <ms>`: status and health snapshot timeout.
+- `--no-stability-bundle`: skip persisted stability bundle lookup.
+- `--json`: print machine-readable export metadata.
+
+## Disable diagnostics
+
+Diagnostics are enabled by default. To disable the stability recorder and
+diagnostic event collection:
+
+```json5
+{
+  diagnostics: {
+    enabled: false,
+  },
+}
+```
+
+Disabling diagnostics reduces bug-report detail. It does not affect normal
+Gateway logging.
+
+Critical memory pressure snapshots are off by default. To keep diagnostics
+events and also capture the pre-OOM stability snapshot:
+
+```json5
+{
+  diagnostics: {
+    memoryPressureSnapshot: true,
+  },
+}
+```
+
+Use this only on hosts that can tolerate the extra file-system scan and snapshot
+write during critical memory pressure. Normal memory pressure events still
+record RSS, heap, threshold, and growth facts when the snapshot is off.
+
+## Related
+
+- [Health checks](/gateway/health)
+- [Gateway CLI](/cli/gateway#gateway-diagnostics-export)
+- [Gateway protocol](/gateway/protocol#system-and-identity)
+- [Logging](/logging)
+- [OpenTelemetry export](/gateway/opentelemetry) — separate flow for streaming diagnostics to a collector

package/docs/gateway/discovery.md

@@ -0,0 +1,154 @@
+---
+summary: "Node discovery and transports (Bonjour, Tailscale, SSH) for finding the gateway"
+read_when:
+  - Implementing or changing Bonjour discovery/advertising
+  - Adjusting remote connection modes (direct vs SSH)
+  - Designing node discovery + pairing for remote nodes
+title: "Discovery and transports"
+---
+
+Nodmix has two distinct problems that look similar on the surface:
+
+1. **Operator remote control**: the macOS menu bar app controlling a gateway running elsewhere.
+2. **Node pairing**: iOS/Android (and future nodes) finding a gateway and pairing securely.
+
+The design goal is to keep all network discovery/advertising in the **Node Gateway** (`nodmix gateway`) and keep clients (mac app, iOS) as consumers.
+
+## Terms
+
+- **Gateway**: a single long-running gateway process that owns state (sessions, pairing, node registry) and runs channels. Most setups use one per host; isolated multi-gateway setups are possible.
+- **Gateway WS (control plane)**: the WebSocket endpoint on `127.0.0.1:18789` by default; can be bound to LAN/tailnet via `gateway.bind`.
+- **Direct WS transport**: a LAN/tailnet-facing Gateway WS endpoint (no SSH).
+- **SSH transport (fallback)**: remote control by forwarding `127.0.0.1:18789` over SSH.
+- **Legacy TCP bridge (removed)**: older node transport (see
+  [Bridge protocol](/gateway/bridge-protocol)); no longer advertised for
+  discovery and no longer part of current builds.
+
+Protocol details:
+
+- [Gateway protocol](/gateway/protocol)
+- [Bridge protocol (legacy)](/gateway/bridge-protocol)
+
+## Why we keep both direct and SSH
+
+- **Direct WS** is the best UX on the same network and within a tailnet:
+  - auto-discovery on LAN via Bonjour
+  - pairing tokens + ACLs owned by the gateway
+  - no shell access required; protocol surface can stay tight and auditable
+- **SSH** remains the universal fallback:
+  - works anywhere you have SSH access (even across unrelated networks)
+  - survives multicast/mDNS issues
+  - requires no new inbound ports besides SSH
+
+## Discovery inputs (how clients learn where the gateway is)
+
+### 1) Bonjour / DNS-SD discovery
+
+Multicast Bonjour is best-effort and does not cross networks. Nodmix can also browse the
+same gateway beacon via a configured wide-area DNS-SD domain, so discovery can cover:
+
+- `local.` on the same LAN
+- a configured unicast DNS-SD domain for cross-network discovery
+
+Target direction:
+
+- The **gateway** advertises its WS endpoint via Bonjour when the bundled
+  `bonjour` plugin is enabled. The plugin auto-starts on macOS hosts and is
+  opt-in elsewhere.
+- Clients browse and show a "pick a gateway" list, then store the chosen endpoint.
+
+Troubleshooting and beacon details: [Bonjour](/gateway/bonjour).
+
+#### Service beacon details
+
+- Service types:
+  - `_nodmix-gw._tcp` (gateway transport beacon)
+- TXT keys (non-secret):
+  - `role=gateway`
+  - `transport=gateway`
+  - `displayName=<friendly name>` (operator-configured display name)
+  - `lanHost=<hostname>.local`
+  - `gatewayPort=18789` (Gateway WS + HTTP)
+  - `gatewayTls=1` (only when TLS is enabled)
+  - `gatewayTlsSha256=<sha256>` (only when TLS is enabled and fingerprint is available)
+  - `canvasPort=<port>` (canvas host port; currently the same as `gatewayPort` when the canvas host is enabled)
+  - `tailnetDns=<magicdns>` (optional hint; auto-detected when Tailscale is available)
+  - `sshPort=<port>` (mDNS full mode only; wide-area DNS-SD may omit it, in which case SSH defaults stay at `22`)
+  - `cliPath=<path>` (mDNS full mode only; wide-area DNS-SD still writes it as a remote-install hint)
+
+Security notes:
+
+- Bonjour/mDNS TXT records are **unauthenticated**. Clients must treat TXT values as UX hints only.
+- Routing (host/port) should prefer the **resolved service endpoint** (SRV + A/AAAA) over TXT-provided `lanHost`, `tailnetDns`, or `gatewayPort`.
+- TLS pinning must never allow an advertised `gatewayTlsSha256` to override a previously stored pin.
+- iOS/Android nodes should require an explicit "trust this fingerprint" confirmation before storing a first-time pin (out-of-band verification) whenever the chosen route is secure/TLS-based.
+
+Enable/disable/override:
+
+- `nodmix plugins enable bonjour` enables LAN multicast advertising.
+- `NODMIX_DISABLE_BONJOUR=1` disables advertising.
+- When the Bonjour plugin is enabled and `NODMIX_DISABLE_BONJOUR` is unset,
+  Bonjour advertises on normal hosts and auto-disables inside detected containers.
+  Empty-config macOS Gateway startup enables the plugin automatically; Linux,
+  Windows, and containerized deployments need explicit enablement.
+  Use `0` only on host, macvlan, or another mDNS-capable network; use `1` to
+  force-disable.
+- `gateway.bind` in `~/.nodmix/nodmix.json` controls the Gateway bind mode.
+- `NODMIX_SSH_PORT` overrides the SSH port advertised when `sshPort` is emitted.
+- `NODMIX_TAILNET_DNS` publishes a `tailnetDns` hint (MagicDNS).
+- `NODMIX_CLI_PATH` overrides the advertised CLI path.
+
+### 2) Tailnet (cross-network)
+
+For London/Vienna style setups, Bonjour won't help. The recommended "direct" target is:
+
+- Tailscale MagicDNS name (preferred) or a stable tailnet IP.
+
+If the gateway can detect it is running under Tailscale, it publishes `tailnetDns` as an optional hint for clients (including wide-area beacons).
+
+The macOS app now prefers MagicDNS names over raw Tailscale IPs for gateway discovery. This improves reliability when tailnet IPs change (for example after node restarts or CGNAT reassignment), because MagicDNS names resolve to the current IP automatically.
+
+For mobile node pairing, discovery hints do not relax transport security on tailnet/public routes:
+
+- iOS/Android still require a secure first-time tailnet/public connect path (`wss://` or Tailscale Serve/Funnel).
+- A discovered raw tailnet IP is a routing hint, not permission to use plaintext remote `ws://`.
+- Private LAN direct-connect `ws://` remains supported.
+- If you want the simplest Tailscale path for mobile nodes, use Tailscale Serve so discovery and the setup code both resolve to the same secure MagicDNS endpoint.
+
+### 3) Manual / SSH target
+
+When there is no direct route (or direct is disabled), clients can always connect via SSH by forwarding the loopback gateway port.
+
+See [Remote access](/gateway/remote).
+
+## Transport selection (client policy)
+
+Recommended client behavior:
+
+1. If a paired direct endpoint is configured and reachable, use it.
+2. Else, if discovery finds a gateway on `local.` or the configured wide-area domain, offer a one-tap "Use this gateway" choice and save it as the direct endpoint.
+3. Else, if a tailnet DNS/IP is configured, try direct.
+   For mobile nodes on tailnet/public routes, direct means a secure endpoint, not plaintext remote `ws://`.
+4. Else, fall back to SSH.
+
+## Pairing + auth (direct transport)
+
+The gateway is the source of truth for node/client admission.
+
+- Pairing requests are created/approved/rejected in the gateway (see [Gateway pairing](/gateway/pairing)).
+- The gateway enforces:
+  - auth (token / keypair)
+  - scopes/ACLs (the gateway is not a raw proxy to every method)
+  - rate limits
+
+## Responsibilities by component
+
+- **Gateway**: advertises discovery beacons, owns pairing decisions, and hosts the WS endpoint.
+- **macOS app**: helps you pick a gateway, shows pairing prompts, and uses SSH only as a fallback.
+- **iOS/Android nodes**: browse Bonjour as a convenience and connect to the paired Gateway WS.
+
+## Related
+
+- [Remote access](/gateway/remote)
+- [Tailscale](/gateway/tailscale)
+- [Bonjour discovery](/gateway/bonjour)