nodmix 2026.5.25

package/docs/help/debugging.md

@@ -0,0 +1,344 @@
+---
+summary: "Debugging tools: watch mode, raw model streams, and tracing reasoning leakage"
+read_when:
+  - You need to inspect raw model output for reasoning leakage
+  - You want to run the Gateway in watch mode while iterating
+  - You need a repeatable debugging workflow
+title: "Debugging"
+---
+
+Debugging helpers for streaming output, especially when a provider mixes reasoning into normal text.
+
+## Runtime debug overrides
+
+Use `/debug` in chat to set **runtime-only** config overrides (memory, not disk).
+`/debug` is disabled by default; enable with `commands.debug: true`.
+This is handy when you need to toggle obscure settings without editing `nodmix.json`.
+
+Examples:
+
+```
+/debug show
+/debug set messages.responsePrefix="[nodmix]"
+/debug unset messages.responsePrefix
+/debug reset
+```
+
+`/debug reset` clears all overrides and returns to the on-disk config.
+
+## Session trace output
+
+Use `/trace` when you want to see plugin-owned trace/debug lines in one session
+without turning on full verbose mode.
+
+Examples:
+
+```text
+/trace
+/trace on
+/trace off
+```
+
+Use `/trace` for plugin diagnostics such as Active Memory debug summaries.
+Keep using `/verbose` for normal verbose status/tool output, and keep using
+`/debug` for runtime-only config overrides.
+
+## Plugin lifecycle trace
+
+Use `NODMIX_PLUGIN_LIFECYCLE_TRACE=1` when plugin lifecycle commands feel slow
+and you need a built-in phase breakdown for plugin metadata, discovery, registry,
+runtime mirror, config mutation, and refresh work. The trace is opt-in and writes
+to stderr, so JSON command output remains parseable.
+
+Example:
+
+```bash
+NODMIX_PLUGIN_LIFECYCLE_TRACE=1 nodmix plugins install tokenjuice --force
+```
+
+Example output:
+
+```text
+[plugins:lifecycle] phase="config read" ms=6.83 status=ok command="install"
+[plugins:lifecycle] phase="slot selection" ms=94.31 status=ok command="install" pluginId="tokenjuice"
+[plugins:lifecycle] phase="registry refresh" ms=51.56 status=ok command="install" reason="source-changed"
+```
+
+Use this for plugin lifecycle investigation before reaching for a CPU profiler.
+If the command is running from a source checkout, prefer measuring the built
+runtime with `node dist/entry.js ...` after `pnpm build`; `pnpm nodmix ...`
+also measures source-runner overhead.
+
+## CLI startup and command profiling
+
+Use the checked-in startup benchmark when a command feels slow:
+
+```bash
+pnpm test:startup:bench:smoke
+pnpm tsx scripts/bench-cli-startup.ts --preset real --case status --runs 3
+pnpm tsx scripts/bench-cli-startup.ts --preset real --cpu-prof-dir .artifacts/cli-cpu
+```
+
+For one-off profiling through the normal source runner, set
+`NODMIX_RUN_NODE_CPU_PROF_DIR`:
+
+```bash
+NODMIX_RUN_NODE_CPU_PROF_DIR=.artifacts/cli-cpu pnpm nodmix status
+```
+
+The source runner adds Node CPU profile flags and writes a `.cpuprofile` for the
+command. Use this before adding temporary instrumentation to command code.
+
+For startup stalls that look like synchronous filesystem or module-loader work,
+add Node's sync I/O trace flag through the source runner:
+
+```bash
+NODMIX_TRACE_SYNC_IO=1 pnpm nodmix gateway --force
+```
+
+`pnpm gateway:watch` leaves this flag disabled by default for the watched
+Gateway child. Set `NODMIX_TRACE_SYNC_IO=1` when you explicitly want Node
+sync I/O trace output in watch mode.
+
+## Gateway watch mode
+
+For fast iteration, run the gateway under the file watcher:
+
+```bash
+pnpm gateway:watch
+```
+
+By default, this starts or restarts a tmux session named
+`nodmix-gateway-watch-main` (or a profile/port-specific variant such as
+`nodmix-gateway-watch-dev-19001`) and auto-attaches from interactive terminals.
+Non-interactive shells, CI, and agent exec calls stay detached and print attach
+instructions instead. Attach manually when needed:
+
+```bash
+tmux attach -t nodmix-gateway-watch-main
+```
+
+The tmux pane runs the raw watcher:
+
+```bash
+node scripts/watch-node.mjs gateway --force
+```
+
+Use foreground mode when tmux is not wanted:
+
+```bash
+pnpm gateway:watch:raw
+# or
+NODMIX_GATEWAY_WATCH_TMUX=0 pnpm gateway:watch
+```
+
+Disable auto-attach while keeping tmux management:
+
+```bash
+NODMIX_GATEWAY_WATCH_ATTACH=0 pnpm gateway:watch
+```
+
+Profile watched Gateway CPU time when debugging startup/runtime hotspots:
+
+```bash
+pnpm gateway:watch --benchmark
+```
+
+The watch wrapper consumes `--benchmark` before invoking the Gateway and writes
+one V8 `.cpuprofile` per Gateway child exit under
+`.artifacts/gateway-watch-profiles/`. Stop or restart the watched gateway to
+flush the current profile, then open it with Chrome DevTools or Speedscope:
+
+```bash
+npx speedscope .artifacts/gateway-watch-profiles/*.cpuprofile
+```
+
+Use `--benchmark-dir <path>` when you want profiles somewhere else.
+Use `--benchmark-no-force` when you want the benchmarked child to skip the
+default `--force` port cleanup and fail fast if the Gateway port is already in
+use.
+Benchmark mode suppresses sync-I/O trace spam by default. Set
+`NODMIX_TRACE_SYNC_IO=1` with `--benchmark` when you explicitly want both CPU
+profiles and Node sync-I/O stack traces. In benchmark mode those trace blocks
+are written to `gateway-watch-output.log` under the benchmark directory and
+filtered from the terminal pane; normal Gateway logs remain visible.
+
+The tmux wrapper carries common non-secret runtime selectors such as
+`NODMIX_PROFILE`, `NODMIX_CONFIG_PATH`, `NODMIX_STATE_DIR`,
+`NODMIX_GATEWAY_PORT`, and `NODMIX_SKIP_CHANNELS` into the pane. Put
+provider credentials in your normal profile/config, or use raw foreground mode
+for one-off ephemeral secrets.
+If the watched Gateway exits during startup, the watcher runs
+`nodmix doctor --fix --non-interactive` once and restarts the Gateway child.
+Use `NODMIX_GATEWAY_WATCH_AUTO_DOCTOR=0` when you want the original startup
+failure without the dev-only repair pass.
+The managed tmux pane also defaults to colored Gateway logs for readability;
+set `FORCE_COLOR=0` when starting `pnpm gateway:watch` to disable ANSI output.
+
+The watcher restarts on build-relevant files under `src/`, extension source files,
+extension `package.json` and `nodmix.plugin.json` metadata, `tsconfig.json`,
+`package.json`, and `tsdown.config.ts`. Extension metadata changes restart the
+gateway without forcing a `tsdown` rebuild; source and config changes still
+rebuild `dist` first.
+
+Add any gateway CLI flags after `gateway:watch` and they will be passed through on
+each restart. Re-running the same watch command respawns the named tmux pane, and
+the raw watcher still keeps its single-watcher lock so duplicate watcher parents
+are replaced instead of piling up.
+
+## Dev profile + dev gateway (--dev)
+
+Use the dev profile to isolate state and spin up a safe, disposable setup for
+debugging. There are **two** `--dev` flags:
+
+- **Global `--dev` (profile):** isolates state under `~/.nodmix-dev` and
+  defaults the gateway port to `19001` (derived ports shift with it).
+- **`gateway --dev`: tells the Gateway to auto-create a default config +
+  workspace** when missing (and skip BOOTSTRAP.md).
+
+Recommended flow (dev profile + dev bootstrap):
+
+```bash
+pnpm gateway:dev
+NODMIX_PROFILE=dev nodmix tui
+```
+
+If you don't have a global install yet, run the CLI via `pnpm nodmix ...`.
+
+What this does:
+
+1. **Profile isolation** (global `--dev`)
+   - `NODMIX_PROFILE=dev`
+   - `NODMIX_STATE_DIR=~/.nodmix-dev`
+   - `NODMIX_CONFIG_PATH=~/.nodmix-dev/nodmix.json`
+   - `NODMIX_GATEWAY_PORT=19001` (browser/canvas shift accordingly)
+
+2. **Dev bootstrap** (`gateway --dev`)
+   - Writes a minimal config if missing (`gateway.mode=local`, bind loopback).
+   - Sets `agent.workspace` to the dev workspace.
+   - Sets `agent.skipBootstrap=true` (no BOOTSTRAP.md).
+   - Seeds the workspace files if missing:
+     `AGENTS.md`, `SOUL.md`, `TOOLS.md`, `IDENTITY.md`, `USER.md`, `HEARTBEAT.md`.
+   - Default identity: **C3-PO** (protocol droid).
+   - Skips channel providers in dev mode (`NODMIX_SKIP_CHANNELS=1`).
+
+Reset flow (fresh start):
+
+```bash
+pnpm gateway:dev:reset
+```
+
+<Note>
+`--dev` is a **global** profile flag and gets eaten by some runners. If you need to spell it out, use the env var form:
+
+```bash
+NODMIX_PROFILE=dev nodmix gateway --dev --reset
+```
+
+</Note>
+
+`--reset` wipes config, credentials, sessions, and the dev workspace (using
+`trash`, not `rm`), then recreates the default dev setup.
+
+<Tip>
+If a non-dev gateway is already running (launchd or systemd), stop it first:
+
+```bash
+nodmix gateway stop
+```
+
+</Tip>
+
+## Raw stream logging (Nodmix)
+
+Nodmix can log the **raw assistant stream** before any filtering/formatting.
+This is the best way to see whether reasoning is arriving as plain text deltas
+(or as separate thinking blocks).
+
+Enable it via CLI:
+
+```bash
+pnpm gateway:watch --raw-stream
+```
+
+Optional path override:
+
+```bash
+pnpm gateway:watch --raw-stream --raw-stream-path ~/.nodmix/logs/raw-stream.jsonl
+```
+
+Equivalent env vars:
+
+```bash
+NODMIX_RAW_STREAM=1
+NODMIX_RAW_STREAM_PATH=~/.nodmix/logs/raw-stream.jsonl
+```
+
+Default file:
+
+`~/.nodmix/logs/raw-stream.jsonl`
+
+## Raw chunk logging (pi-mono)
+
+To capture **raw OpenAI-compat chunks** before they are parsed into blocks,
+pi-mono exposes a separate logger:
+
+```bash
+PI_RAW_STREAM=1
+```
+
+Optional path:
+
+```bash
+PI_RAW_STREAM_PATH=~/.pi-mono/logs/raw-openai-completions.jsonl
+```
+
+Default file:
+
+`~/.pi-mono/logs/raw-openai-completions.jsonl`
+
+> Note: this is only emitted by processes using pi-mono's
+> `openai-completions` provider.
+
+## Safety notes
+
+- Raw stream logs can include full prompts, tool output, and user data.
+- Keep logs local and delete them after debugging.
+- If you share logs, scrub secrets and PII first.
+
+## Debugging in VSCode
+
+Source maps are required to enable debugging in VSCode-based IDEs because many of the generated files end up with hashed names as part of the build process. The included `launch.json` configurations target the Gateway service, but can be adapted quickly for other purposes:
+
+1. **Rebuild and Debug Gateway** - Debugs the Gateway service after creating a new build
+2. **Debug Gateway** - Debugs the Gateway service of a pre-existing build
+
+### Setup
+
+The default **Rebuild and Debug Gateway** configuration is batteries-included, it will automatically delete the `/dist` folder and rebuild the project with debugging enabled:
+
+1. Open the **Run and Debug** panel from the Activity Bar or press `Ctrl`+`Shift`+`D`
+2. In the IDE, ensure **Rebuild and Debug Gateway** is selected in the configuration dropdown and then press the **Start Debugging** button
+
+Alternatively - if you prefer to manage the build and debug processes manually:
+
+1. Open a terminal and enable source maps:
+   - **Linux/macOS**: `export OUTPUT_SOURCE_MAPS=1`
+   - **Windows (PowerShell)**: `$env:OUTPUT_SOURCE_MAPS="1"`
+   - **Windows (CMD)**: `set OUTPUT_SOURCE_MAPS=1`
+2. In the same terminal, rebuild the project: `pnpm clean:dist && pnpm build`
+3. In the IDE, select the **Debug Gateway** option in the **Run and Debug** configuration dropdown and then press the **Start Debugging** button
+
+You can now set breakpoints in your TypeScript source files (`src/` directory) and the debugger will correctly map breakpoints to the compiled JavaScript via source maps. You'll be able to inspect variables, step through code, and examine call stacks as expected.
+
+### Notes
+
+- If using the **"Rebuild and Debug Gateway"** option - each time the debugger is launched it will completely delete the `/dist` folder and run a full `pnpm build` with source maps enabled before starting the Gateway
+- If using the **"Debug Gateway"** option - debug sessions can be started and stopped at any time without affecting the `/dist` folder, but you must use a separate terminal process to both enable debugging and manage the build cycle
+- Modify the `launch.json` settings for `args` to debug other sections of the project
+- If you need to use the built Nodmix CLI for other tasks (i.e. `dashboard --no-open` if your debug session spawns a new auth token), you can execute it in another terminal as `node ./nodmix.mjs` or create a shell alias like `alias nodmix-build="node $(pwd)/nodmix.mjs"`
+
+## Related
+
+- [Troubleshooting](/help/troubleshooting)
+- [FAQ](/help/faq)

package/docs/help/environment.md

@@ -0,0 +1,214 @@
+---
+summary: "Where Nodmix loads environment variables and the precedence order"
+read_when:
+  - You need to know which env vars are loaded, and in what order
+  - You are debugging missing API keys in the Gateway
+  - You are documenting provider auth or deployment environments
+title: "Environment variables"
+---
+
+Nodmix pulls environment variables from multiple sources. The rule is **never override existing values**.
+
+## Precedence (highest → lowest)
+
+1. **Process environment** (what the Gateway process already has from the parent shell/daemon).
+2. **`.env` in the current working directory** (dotenv default; does not override).
+3. **Global `.env`** at `~/.nodmix/.env` (aka `$NODMIX_STATE_DIR/.env`; does not override).
+4. **Config `env` block** in `~/.nodmix/nodmix.json` (applied only if missing).
+5. **Optional login-shell import** (`env.shellEnv.enabled` or `NODMIX_LOAD_SHELL_ENV=1`), applied only for missing expected keys.
+
+On Ubuntu fresh installs that use the default state dir, Nodmix also treats `~/.config/nodmix/gateway.env` as a compatibility fallback after the global `.env`. If both files exist and disagree, Nodmix keeps `~/.nodmix/.env` and prints a warning.
+
+If the config file is missing entirely, step 4 is skipped; shell import still runs if enabled.
+
+## Config `env` block
+
+Two equivalent ways to set inline env vars (both are non-overriding):
+
+```json5
+{
+  env: {
+    OPENROUTER_API_KEY: "sk-or-...",
+    vars: {
+      GROQ_API_KEY: "gsk-...",
+    },
+  },
+}
+```
+
+The config `env` block accepts literal string values only. It does not expand
+`file:...` values; for example, `XAI_API_KEY: "file:secrets/xai-api-key.txt"`
+is passed to providers as that exact string.
+
+For file-backed provider keys, use a SecretRef on the credential field that
+supports it:
+
+```json5
+{
+  secrets: {
+    providers: {
+      xai_key_file: {
+        source: "file",
+        path: "~/.nodmix/secrets/xai-api-key.txt",
+        mode: "singleValue",
+      },
+    },
+  },
+  models: {
+    providers: {
+      xai: {
+        apiKey: { source: "file", provider: "xai_key_file", id: "value" },
+      },
+    },
+  },
+}
+```
+
+See [Secrets Management](/gateway/secrets) and the
+[SecretRef credential surface](/reference/secretref-credential-surface) for
+supported fields.
+
+## Shell env import
+
+`env.shellEnv` runs your login shell and imports only **missing** expected keys:
+
+```json5
+{
+  env: {
+    shellEnv: {
+      enabled: true,
+      timeoutMs: 15000,
+    },
+  },
+}
+```
+
+Env var equivalents:
+
+- `NODMIX_LOAD_SHELL_ENV=1`
+- `NODMIX_SHELL_ENV_TIMEOUT_MS=15000`
+
+## Runtime-injected env vars
+
+Nodmix also injects context markers into spawned child processes:
+
+- `NODMIX_SHELL=exec`: set for commands run through the `exec` tool.
+- `NODMIX_SHELL=acp`: set for ACP runtime backend process spawns (for example `acpx`).
+- `NODMIX_SHELL=acp-client`: set for `nodmix acp client` when it spawns the ACP bridge process.
+- `NODMIX_SHELL=tui-local`: set for local TUI `!` shell commands.
+
+These are runtime markers (not required user config). They can be used in shell/profile logic
+to apply context-specific rules.
+
+## UI env vars
+
+- `NODMIX_THEME=light`: force the light TUI palette when your terminal has a light background.
+- `NODMIX_THEME=dark`: force the dark TUI palette.
+- `COLORFGBG`: if your terminal exports it, Nodmix uses the background color hint to auto-pick the TUI palette.
+
+## Env var substitution in config
+
+You can reference env vars directly in config string values using `${VAR_NAME}` syntax:
+
+```json5
+{
+  models: {
+    providers: {
+      "vercel-gateway": {
+        apiKey: "${VERCEL_GATEWAY_API_KEY}",
+      },
+    },
+  },
+}
+```
+
+See [Configuration: Env var substitution](/gateway/configuration-reference#env-var-substitution) for full details.
+
+## Secret refs vs `${ENV}` strings
+
+Nodmix supports two env-driven patterns:
+
+- `${VAR}` string substitution in config values.
+- SecretRef objects (`{ source: "env", provider: "default", id: "VAR" }`) for fields that support secrets references.
+
+Both resolve from process env at activation time. SecretRef details are documented in [Secrets Management](/gateway/secrets).
+The config `env` block itself does not resolve SecretRefs or `file:...`
+shorthand values.
+
+## Path-related env vars
+
+| Variable                 | Purpose                                                                                                                                                                          |
+| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `NODMIX_HOME`          | Override the home directory used for all internal path resolution (`~/.nodmix/`, agent dirs, sessions, credentials). Useful when running Nodmix as a dedicated service user. |
+| `NODMIX_STATE_DIR`     | Override the state directory (default `~/.nodmix`).                                                                                                                            |
+| `NODMIX_CONFIG_PATH`   | Override the config file path (default `~/.nodmix/nodmix.json`).                                                                                                             |
+| `NODMIX_INCLUDE_ROOTS` | Path-list of directories where `$include` directives may resolve files outside the config directory (default: none — `$include` is confined to the config dir). Tilde-expanded.  |
+
+## Logging
+
+| Variable                         | Purpose                                                                                                                                                                                      |
+| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `NODMIX_LOG_LEVEL`             | Override log level for both file and console (e.g. `debug`, `trace`). Takes precedence over `logging.level` and `logging.consoleLevel` in config. Invalid values are ignored with a warning. |
+| `NODMIX_DEBUG_MODEL_TRANSPORT` | Emit targeted model request/response timing diagnostics at `info` level without enabling global debug logs.                                                                                  |
+| `NODMIX_DEBUG_MODEL_PAYLOAD`   | Model payload diagnostics: `summary`, `tools`, or `full-redacted`. `full-redacted` is capped and redacted but may include prompt/message text.                                               |
+| `NODMIX_DEBUG_SSE`             | Streaming diagnostics: `events` for first/done timing, `peek` to include the first five redacted SSE events.                                                                                 |
+| `NODMIX_DEBUG_CODE_MODE`       | Code-mode model-surface diagnostics, including provider-tool hiding and exec/wait-only enforcement.                                                                                          |
+
+### `NODMIX_HOME`
+
+When set, `NODMIX_HOME` replaces the system home directory (`$HOME` / `os.homedir()`) for all internal path resolution. This enables full filesystem isolation for headless service accounts.
+
+**Precedence:** `NODMIX_HOME` > `$HOME` > `USERPROFILE` > `os.homedir()`
+
+**Example** (macOS LaunchDaemon):
+
+```xml
+<key>EnvironmentVariables</key>
+<dict>
+  <key>NODMIX_HOME</key>
+  <string>/Users/user</string>
+</dict>
+```
+
+`NODMIX_HOME` can also be set to a tilde path (e.g. `~/svc`), which gets expanded using `$HOME` before use.
+
+## nvm users: web_fetch TLS failures
+
+If Node.js was installed via **nvm** (not the system package manager), the built-in `fetch()` uses
+nvm's bundled CA store, which may be missing modern root CAs (ISRG Root X1/X2 for Let's Encrypt,
+DigiCert Global Root G2, etc.). This causes `web_fetch` to fail with `"fetch failed"` on most HTTPS sites.
+
+On Linux, Nodmix automatically detects nvm and applies the fix in the actual startup environment:
+
+- `nodmix gateway install` writes `NODE_EXTRA_CA_CERTS` into the systemd service environment
+- the `nodmix` CLI entrypoint re-execs itself with `NODE_EXTRA_CA_CERTS` set before Node startup
+
+**Manual fix (for older versions or direct `node ...` launches):**
+
+Export the variable before starting Nodmix:
+
+```bash
+export NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt
+nodmix gateway run
+```
+
+Do not rely on writing only to `~/.nodmix/.env` for this variable; Node reads
+`NODE_EXTRA_CA_CERTS` at process startup.
+
+## Legacy environment variables
+
+Nodmix only reads `NODMIX_*` environment variables. The legacy
+`CLAWDBOT_*` and `MOLTBOT_*` prefixes from earlier releases are silently
+ignored.
+
+If any are still set on the Gateway process at startup, Nodmix emits a
+single Node deprecation warning (`NODMIX_LEGACY_ENV_VARS`) listing the
+detected prefixes and the total count. Rename each value by replacing the
+legacy prefix with `NODMIX_` (for example `CLAWDBOT_GATEWAY_TOKEN` →
+`NODMIX_GATEWAY_TOKEN`); the old names take no effect.
+
+## Related
+
+- [Gateway configuration](/gateway/configuration)
+- [FAQ: env vars and .env loading](/help/faq#env-vars-and-env-loading)
+- [Models overview](/concepts/models)