fengming 0.3.9 → 0.3.11

package/docs/help/debugging.md

@@ -1,341 +0,0 @@
----
-summary: "Debugging tools: watch mode, raw model streams, and tracing reasoning leakage"
-read_when:
-  - You need to inspect raw model output for reasoning leakage
-  - You want to run the Gateway in watch mode while iterating
-  - You need a repeatable debugging workflow
-title: "Debugging"
----
-
-Debugging helpers for streaming output, especially when a provider mixes reasoning into normal text.
-
-## Runtime debug overrides
-
-Use `/debug` in chat to set **runtime-only** config overrides (memory, not disk).
-`/debug` is disabled by default; enable with `commands.debug: true`.
-This is handy when you need to toggle obscure settings without editing `fengming.json`.
-
-Examples:
-
-```
-/debug show
-/debug set messages.responsePrefix="[fengming]"
-/debug unset messages.responsePrefix
-/debug reset
-```
-
-`/debug reset` clears all overrides and returns to the on-disk config.
-
-## Session trace output
-
-Use `/trace` when you want to see plugin-owned trace/debug lines in one session
-without turning on full verbose mode.
-
-Examples:
-
-```text
-/trace
-/trace on
-/trace off
-```
-
-Use `/trace` for plugin diagnostics such as Active Memory debug summaries.
-Keep using `/verbose` for normal verbose status/tool output, and keep using
-`/debug` for runtime-only config overrides.
-
-## Plugin lifecycle trace
-
-Use `FENGMING_PLUGIN_LIFECYCLE_TRACE=1` when plugin lifecycle commands feel slow
-and you need a built-in phase breakdown for plugin metadata, discovery, registry,
-runtime mirror, config mutation, and refresh work. The trace is opt-in and writes
-to stderr, so JSON command output remains parseable.
-
-Example:
-
-```bash
-FENGMING_PLUGIN_LIFECYCLE_TRACE=1 fengming plugins install tokenjuice --force
-```
-
-Example output:
-
-```text
-[plugins:lifecycle] phase="config read" ms=6.83 status=ok command="install"
-[plugins:lifecycle] phase="slot selection" ms=94.31 status=ok command="install" pluginId="tokenjuice"
-[plugins:lifecycle] phase="registry refresh" ms=51.56 status=ok command="install" reason="source-changed"
-```
-
-Use this for plugin lifecycle investigation before reaching for a CPU profiler.
-If the command is running from a source checkout, prefer measuring the built
-runtime with `node dist/entry.js ...` after `pnpm build`; `pnpm fengming ...`
-also measures source-runner overhead.
-
-## CLI startup and command profiling
-
-Use the checked-in startup benchmark when a command feels slow:
-
-```bash
-pnpm test:startup:bench:smoke
-pnpm tsx scripts/bench-cli-startup.ts --preset real --case status --runs 3
-pnpm tsx scripts/bench-cli-startup.ts --preset real --cpu-prof-dir .artifacts/cli-cpu
-```
-
-For one-off profiling through the normal source runner, set
-`FENGMING_RUN_NODE_CPU_PROF_DIR`:
-
-```bash
-FENGMING_RUN_NODE_CPU_PROF_DIR=.artifacts/cli-cpu pnpm fengming status
-```
-
-The source runner adds Node CPU profile flags and writes a `.cpuprofile` for the
-command. Use this before adding temporary instrumentation to command code.
-
-For startup stalls that look like synchronous filesystem or module-loader work,
-add Node's sync I/O trace flag through the source runner:
-
-```bash
-FENGMING_TRACE_SYNC_IO=1 pnpm fengming gateway --force
-```
-
-`pnpm gateway:watch` leaves this flag disabled by default for the watched
-Gateway child. Set `FENGMING_TRACE_SYNC_IO=1` when you explicitly want Node
-sync I/O trace output in watch mode.
-
-## Gateway watch mode
-
-For fast iteration, run the gateway under the file watcher:
-
-```bash
-pnpm gateway:watch
-```
-
-By default, this starts or restarts a tmux session named
-`fengming-gateway-watch-main` (or a profile/port-specific variant such as
-`fengming-gateway-watch-dev-19001`) and auto-attaches from interactive terminals.
-Non-interactive shells, CI, and agent exec calls stay detached and print attach
-instructions instead. Attach manually when needed:
-
-```bash
-tmux attach -t fengming-gateway-watch-main
-```
-
-The tmux pane runs the raw watcher:
-
-```bash
-node scripts/watch-node.mjs gateway --force
-```
-
-Use foreground mode when tmux is not wanted:
-
-```bash
-pnpm gateway:watch:raw
-# or
-FENGMING_GATEWAY_WATCH_TMUX=0 pnpm gateway:watch
-```
-
-Disable auto-attach while keeping tmux management:
-
-```bash
-FENGMING_GATEWAY_WATCH_ATTACH=0 pnpm gateway:watch
-```
-
-Profile watched Gateway CPU time when debugging startup/runtime hotspots:
-
-```bash
-pnpm gateway:watch --benchmark
-```
-
-The watch wrapper consumes `--benchmark` before invoking the Gateway and writes
-one V8 `.cpuprofile` per Gateway child exit under
-`.artifacts/gateway-watch-profiles/`. Stop or restart the watched gateway to
-flush the current profile, then open it with Chrome DevTools or Speedscope:
-
-```bash
-npx speedscope .artifacts/gateway-watch-profiles/*.cpuprofile
-```
-
-Use `--benchmark-dir <path>` when you want profiles somewhere else.
-Use `--benchmark-no-force` when you want the benchmarked child to skip the
-default `--force` port cleanup and fail fast if the Gateway port is already in
-use.
-Benchmark mode suppresses sync-I/O trace spam by default. Set
-`FENGMING_TRACE_SYNC_IO=1` with `--benchmark` when you explicitly want both CPU
-profiles and Node sync-I/O stack traces. In benchmark mode those trace blocks
-are written to `gateway-watch-output.log` under the benchmark directory and
-filtered from the terminal pane; normal Gateway logs remain visible.
-
-The tmux wrapper carries common non-secret runtime selectors such as
-`FENGMING_PROFILE`, `FENGMING_CONFIG_PATH`, `FENGMING_STATE_DIR`,
-`FENGMING_GATEWAY_PORT`, and `FENGMING_SKIP_CHANNELS` into the pane. Put
-provider credentials in your normal profile/config, or use raw foreground mode
-for one-off ephemeral secrets.
-If the watched Gateway exits during startup, the watcher runs
-`fengming doctor --fix --non-interactive` once and restarts the Gateway child.
-Use `FENGMING_GATEWAY_WATCH_AUTO_DOCTOR=0` when you want the original startup
-failure without the dev-only repair pass.
-The managed tmux pane also defaults to colored Gateway logs for readability;
-set `FORCE_COLOR=0` when starting `pnpm gateway:watch` to disable ANSI output.
-
-The watcher restarts on build-relevant files under `src/`, extension source files,
-extension `package.json` and `fengming.plugin.json` metadata, `tsconfig.json`,
-`package.json`, and `tsdown.config.ts`. Extension metadata changes restart the
-gateway without forcing a `tsdown` rebuild; source and config changes still
-rebuild `dist` first.
-
-Add any gateway CLI flags after `gateway:watch` and they will be passed through on
-each restart. Re-running the same watch command respawns the named tmux pane, and
-the raw watcher still keeps its single-watcher lock so duplicate watcher parents
-are replaced instead of piling up.
-
-## Dev profile + dev gateway (--dev)
-
-Use the dev profile to isolate state and spin up a safe, disposable setup for
-debugging. There are **two** `--dev` flags:
-
-- **Global `--dev` (profile):** isolates state under `~/.fengming-dev` and
-  defaults the gateway port to `19001` (derived ports shift with it).
-- **`gateway --dev`: tells the Gateway to auto-create a default config +
-  workspace** when missing (and skip BOOTSTRAP.md).
-
-Recommended flow (dev profile + dev bootstrap):
-
-```bash
-pnpm gateway:dev
-FENGMING_PROFILE=dev fengming tui
-```
-
-If you don't have a global install yet, run the CLI via `pnpm fengming ...`.
-
-What this does:
-
-1. **Profile isolation** (global `--dev`)
-   - `FENGMING_PROFILE=dev`
-   - `FENGMING_STATE_DIR=~/.fengming-dev`
-   - `FENGMING_CONFIG_PATH=~/.fengming-dev/fengming.json`
-   - `FENGMING_GATEWAY_PORT=19001` (browser/canvas shift accordingly)
-
-2. **Dev bootstrap** (`gateway --dev`)
-   - Writes a minimal config if missing (`gateway.mode=local`, bind loopback).
-   - Sets `agent.workspace` to the dev workspace.
-   - Sets `agent.skipBootstrap=true` (no BOOTSTRAP.md).
-   - Seeds the workspace files if missing:
-     `AGENTS.md`, `SOUL.md`, `TOOLS.md`, `IDENTITY.md`, `USER.md`, `HEARTBEAT.md`.
-   - Default identity: **C3-PO** (protocol droid).
-   - Skips channel providers in dev mode (`FENGMING_SKIP_CHANNELS=1`).
-
-Reset flow (fresh start):
-
-```bash
-pnpm gateway:dev:reset
-```
-
-<Note>
-`--dev` is a **global** profile flag and gets eaten by some runners. If you need to spell it out, use the env var form:
-
-```bash
-FENGMING_PROFILE=dev fengming gateway --dev --reset
-```
-
-</Note>
-
-`--reset` wipes config, credentials, sessions, and the dev workspace (using
-`trash`, not `rm`), then recreates the default dev setup.
-
-<Tip>
-If a non-dev gateway is already running (launchd or systemd), stop it first:
-
-```bash
-fengming gateway stop
-```
-
-</Tip>
-
-## Raw stream logging (FengMing)
-
-FengMing can log the **raw assistant stream** before any filtering/formatting.
-This is the best way to see whether reasoning is arriving as plain text deltas
-(or as separate thinking blocks).
-
-Enable it via CLI:
-
-```bash
-pnpm gateway:watch --raw-stream
-```
-
-Optional path override:
-
-```bash
-pnpm gateway:watch --raw-stream --raw-stream-path ~/.fengming/logs/raw-stream.jsonl
-```
-
-Equivalent env vars:
-
-```bash
-FENGMING_RAW_STREAM=1
-FENGMING_RAW_STREAM_PATH=~/.fengming/logs/raw-stream.jsonl
-```
-
-Default file:
-
-`~/.fengming/logs/raw-stream.jsonl`
-
-## Raw OpenAI-compatible chunk logging
-
-To capture **raw OpenAI-compat chunks** before they are parsed into blocks,
-enable the transport logger:
-
-```bash
-FENGMING_RAW_STREAM=1
-```
-
-Optional path:
-
-```bash
-FENGMING_RAW_STREAM_PATH=~/.fengming/logs/raw-openai-completions.jsonl
-```
-
-Default file:
-
-`~/.fengming/logs/raw-openai-completions.jsonl`
-
-## Safety notes
-
-- Raw stream logs can include full prompts, tool output, and user data.
-- Keep logs local and delete them after debugging.
-- If you share logs, scrub secrets and PII first.
-
-## Debugging in VSCode
-
-Source maps are required to enable debugging in VSCode-based IDEs because many of the generated files end up with hashed names as part of the build process. The included `launch.json` configurations target the Gateway service, but can be adapted quickly for other purposes:
-
-1. **Rebuild and Debug Gateway** - Debugs the Gateway service after creating a new build
-2. **Debug Gateway** - Debugs the Gateway service of a pre-existing build
-
-### Setup
-
-The default **Rebuild and Debug Gateway** configuration is batteries-included, it will automatically delete the `/dist` folder and rebuild the project with debugging enabled:
-
-1. Open the **Run and Debug** panel from the Activity Bar or press `Ctrl`+`Shift`+`D`
-2. In the IDE, ensure **Rebuild and Debug Gateway** is selected in the configuration dropdown and then press the **Start Debugging** button
-
-Alternatively - if you prefer to manage the build and debug processes manually:
-
-1. Open a terminal and enable source maps:
-   - **Linux/macOS**: `export OUTPUT_SOURCE_MAPS=1`
-   - **Windows (PowerShell)**: `$env:OUTPUT_SOURCE_MAPS="1"`
-   - **Windows (CMD)**: `set OUTPUT_SOURCE_MAPS=1`
-2. In the same terminal, rebuild the project: `pnpm clean:dist && pnpm build`
-3. In the IDE, select the **Debug Gateway** option in the **Run and Debug** configuration dropdown and then press the **Start Debugging** button
-
-You can now set breakpoints in your TypeScript source files (`src/` directory) and the debugger will correctly map breakpoints to the compiled JavaScript via source maps. You'll be able to inspect variables, step through code, and examine call stacks as expected.
-
-### Notes
-
-- If using the **"Rebuild and Debug Gateway"** option - each time the debugger is launched it will completely delete the `/dist` folder and run a full `pnpm build` with source maps enabled before starting the Gateway
-- If using the **"Debug Gateway"** option - debug sessions can be started and stopped at any time without affecting the `/dist` folder, but you must use a separate terminal process to both enable debugging and manage the build cycle
-- Modify the `launch.json` settings for `args` to debug other sections of the project
-- If you need to use the built FengMing CLI for other tasks (i.e. `dashboard --no-open` if your debug session spawns a new auth token), you can execute it in another terminal as `node ./fengming.mjs` or create a shell alias like `alias fengming-build="node $(pwd)/fengming.mjs"`
-
-## Related
-
-- [Troubleshooting](/help/troubleshooting)
-- [FAQ](/help/faq)

package/docs/help/environment.md

@@ -1,233 +0,0 @@
----
-summary: "Where FengMing loads environment variables and the precedence order"
-read_when:
-  - You need to know which env vars are loaded, and in what order
-  - You are debugging missing API keys in the Gateway
-  - You are documenting provider auth or deployment environments
-title: "Environment variables"
----
-
-FengMing pulls environment variables from multiple sources. The rule is **never override existing values**.
-Workspace `.env` files are a lower-trust source: FengMing ignores provider credentials and protected runtime controls from workspace `.env` before applying precedence.
-
-## Precedence (highest → lowest)
-
-1. **Process environment** (what the Gateway process already has from the parent shell/daemon).
-2. **`.env` in the current working directory** (dotenv default; does not override; provider credentials and protected runtime controls are ignored).
-3. **Global `.env`** at `~/.fengming/.env` (aka `$FENGMING_STATE_DIR/.env`; recommended for provider API keys; does not override).
-4. **Config `env` block** in `~/.fengming/fengming.json` (applied only if missing).
-5. **Optional login-shell import** (`env.shellEnv.enabled` or `FENGMING_LOAD_SHELL_ENV=1`), applied only for missing expected keys.
-
-On Ubuntu fresh installs that use the default state dir, FengMing also treats `~/.config/fengming/gateway.env` as a compatibility fallback after the global `.env`. If both files exist and disagree, FengMing keeps `~/.fengming/.env` and prints a warning.
-
-If the config file is missing entirely, step 4 is skipped; shell import still runs if enabled.
-
-## Provider credentials and workspace `.env`
-
-Do not keep provider API keys only in a workspace `.env`. FengMing ignores provider credential environment variables from workspace `.env` files, including common keys such as `GEMINI_API_KEY`, `GOOGLE_API_KEY`, `XAI_API_KEY`, `MISTRAL_API_KEY`, `GROQ_API_KEY`, `DEEPSEEK_API_KEY`, `PERPLEXITY_API_KEY`, `BRAVE_API_KEY`, `TAVILY_API_KEY`, `EXA_API_KEY`, and `FIRECRAWL_API_KEY`.
-
-Use one of these trusted sources for provider credentials:
-
-- The Gateway process environment, such as a shell, launchd/systemd unit, container secret, or CI secret.
-- The global runtime dotenv file at `~/.fengming/.env` or `$FENGMING_STATE_DIR/.env`.
-- The config `env` block in `~/.fengming/fengming.json`.
-- Optional login-shell import when `env.shellEnv.enabled` or `FENGMING_LOAD_SHELL_ENV=1` is enabled.
-
-If you previously stored provider keys only in a workspace `.env`, move them to one of the trusted sources above. Workspace `.env` can still provide ordinary project variables that are not credentials, endpoint redirects, host overrides, or `FENGMING_*` runtime controls.
-
-See [Workspace `.env` files](/gateway/security#workspace-env-files) for the security rationale.
-
-## Config `env` block
-
-Two equivalent ways to set inline env vars (both are non-overriding):
-
-```json5
-{
-  env: {
-    OPENROUTER_API_KEY: "sk-or-...",
-    vars: {
-      GROQ_API_KEY: "gsk-...",
-    },
-  },
-}
-```
-
-The config `env` block accepts literal string values only. It does not expand
-`file:...` values; for example, `XAI_API_KEY: "file:secrets/xai-api-key.txt"`
-is passed to providers as that exact string.
-
-For file-backed provider keys, use a SecretRef on the credential field that
-supports it:
-
-```json5
-{
-  secrets: {
-    providers: {
-      xai_key_file: {
-        source: "file",
-        path: "~/.fengming/secrets/xai-api-key.txt",
-        mode: "singleValue",
-      },
-    },
-  },
-  models: {
-    providers: {
-      xai: {
-        apiKey: { source: "file", provider: "xai_key_file", id: "value" },
-      },
-    },
-  },
-}
-```
-
-See [Secrets Management](/gateway/secrets) and the
-[SecretRef credential surface](/reference/secretref-credential-surface) for
-supported fields.
-
-## Shell env import
-
-`env.shellEnv` runs your login shell and imports only **missing** expected keys:
-
-```json5
-{
-  env: {
-    shellEnv: {
-      enabled: true,
-      timeoutMs: 15000,
-    },
-  },
-}
-```
-
-Env var equivalents:
-
-- `FENGMING_LOAD_SHELL_ENV=1`
-- `FENGMING_SHELL_ENV_TIMEOUT_MS=15000`
-
-## Runtime-injected env vars
-
-FengMing also injects context markers into spawned child processes:
-
-- `FENGMING_SHELL=exec`: set for commands run through the `exec` tool.
-- `FENGMING_SHELL=acp`: set for ACP runtime backend process spawns (for example `acpx`).
-- `FENGMING_SHELL=acp-client`: set for `fengming acp client` when it spawns the ACP bridge process.
-- `FENGMING_SHELL=tui-local`: set for local TUI `!` shell commands.
-- `FENGMING_CLI=1`: set for child processes spawned by the CLI entry point.
-
-These are runtime markers (not required user config). They can be used in shell/profile logic
-to apply context-specific rules.
-
-## UI env vars
-
-- `FENGMING_THEME=light`: force the light TUI palette when your terminal has a light background.
-- `FENGMING_THEME=dark`: force the dark TUI palette.
-- `COLORFGBG`: if your terminal exports it, FengMing uses the background color hint to auto-pick the TUI palette.
-
-## Env var substitution in config
-
-You can reference env vars directly in config string values using `${VAR_NAME}` syntax:
-
-```json5
-{
-  models: {
-    providers: {
-      "vercel-gateway": {
-        apiKey: "${VERCEL_GATEWAY_API_KEY}",
-      },
-    },
-  },
-}
-```
-
-See [Configuration: Env var substitution](/gateway/configuration-reference#env-var-substitution) for full details.
-
-## Secret refs vs `${ENV}` strings
-
-FengMing supports two env-driven patterns:
-
-- `${VAR}` string substitution in config values.
-- SecretRef objects (`{ source: "env", provider: "default", id: "VAR" }`) for fields that support secrets references.
-
-Both resolve from process env at activation time. SecretRef details are documented in [Secrets Management](/gateway/secrets).
-The config `env` block itself does not resolve SecretRefs or `file:...`
-shorthand values.
-
-## Path-related env vars
-
-| Variable                 | Purpose                                                                                                                                                                                                                                 |
-| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `FENGMING_HOME`          | Override the home directory used for internal FengMing path defaults (`~/.fengming/`, agent dirs, sessions, credentials, installer onboarding, and the default dev checkout). Useful when running FengMing as a dedicated service user. |
-| `FENGMING_STATE_DIR`     | Override the state directory (default `~/.fengming`).                                                                                                                                                                                   |
-| `FENGMING_CONFIG_PATH`   | Override the config file path (default `~/.fengming/fengming.json`).                                                                                                                                                                    |
-| `FENGMING_INCLUDE_ROOTS` | Path-list of directories where `$include` directives may resolve files outside the config directory (default: none — `$include` is confined to the config dir). Tilde-expanded.                                                         |
-
-## Logging
-
-| Variable                         | Purpose                                                                                                                                                                                      |
-| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `FENGMING_LOG_LEVEL`             | Override log level for both file and console (e.g. `debug`, `trace`). Takes precedence over `logging.level` and `logging.consoleLevel` in config. Invalid values are ignored with a warning. |
-| `FENGMING_DEBUG_MODEL_TRANSPORT` | Emit targeted model request/response timing diagnostics at `info` level without enabling global debug logs.                                                                                  |
-| `FENGMING_DEBUG_MODEL_PAYLOAD`   | Model payload diagnostics: `summary`, `tools`, or `full-redacted`. `full-redacted` is capped and redacted but may include prompt/message text.                                               |
-| `FENGMING_DEBUG_SSE`             | Streaming diagnostics: `events` for first/done timing, `peek` to include the first five redacted SSE events.                                                                                 |
-| `FENGMING_DEBUG_CODE_MODE`       | Code-mode model-surface diagnostics, including provider-tool hiding and exec/wait-only enforcement.                                                                                          |
-
-### `FENGMING_HOME`
-
-When set, `FENGMING_HOME` replaces the system home directory (`$HOME` / `os.homedir()`) for internal FengMing path defaults. This includes the default state directory, config path, agent directories, credentials, installer onboarding workspace, and the default dev checkout used by `fengming update --channel dev`.
-
-**Precedence:** `FENGMING_HOME` > `$HOME` > `USERPROFILE` > Termux `PREFIX` home fallback on Android > `os.homedir()`
-
-**Example** (macOS LaunchDaemon):
-
-```xml
-<key>EnvironmentVariables</key>
-<dict>
-  <key>FENGMING_HOME</key>
-  <string>/Users/user</string>
-</dict>
-```
-
-`FENGMING_HOME` can also be set to a tilde path (e.g. `~/svc`), which gets expanded using the same OS home fallback chain before use.
-
-Explicit path variables such as `FENGMING_STATE_DIR`, `FENGMING_CONFIG_PATH`, and `FENGMING_GIT_DIR` still take precedence. OS-account tasks such as shell startup file detection, package-manager setup, and host `~` expansion may still use the real system home.
-
-## nvm users: web_fetch TLS failures
-
-If Node.js was installed via **nvm** (not the system package manager), the built-in `fetch()` uses
-nvm's bundled CA store, which may be missing modern root CAs (ISRG Root X1/X2 for Let's Encrypt,
-DigiCert Global Root G2, etc.). This causes `web_fetch` to fail with `"fetch failed"` on most HTTPS sites.
-
-On Linux, FengMing automatically detects nvm and applies the fix in the actual startup environment:
-
-- `fengming gateway install` writes `NODE_EXTRA_CA_CERTS` into the systemd service environment
-- the `fengming` CLI entrypoint re-execs itself with `NODE_EXTRA_CA_CERTS` set before Node startup
-
-**Manual fix (for older versions or direct `node ...` launches):**
-
-Export the variable before starting FengMing:
-
-```bash
-export NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt
-fengming gateway run
-```
-
-Do not rely on writing only to `~/.fengming/.env` for this variable; Node reads
-`NODE_EXTRA_CA_CERTS` at process startup.
-
-## Legacy environment variables
-
-FengMing only reads `FENGMING_*` environment variables. The legacy
-`CLAWDBOT_*` and `MOLTBOT_*` prefixes from earlier releases are silently
-ignored.
-
-If any are still set on the Gateway process at startup, FengMing emits a
-single Node deprecation warning (`FENGMING_LEGACY_ENV_VARS`) listing the
-detected prefixes and the total count. Rename each value by replacing the
-legacy prefix with `FENGMING_` (for example `CLAWDBOT_GATEWAY_TOKEN` →
-`FENGMING_GATEWAY_TOKEN`); the old names take no effect.
-
-## Related
-
-- [Gateway configuration](/gateway/configuration)
-- [FAQ: env vars and .env loading](/help/faq#env-vars-and-env-loading)
-- [Models overview](/concepts/models)