fengming 0.3.10 → 0.3.12

package/docs/gateway/troubleshooting.md

@@ -1,877 +0,0 @@
----
-summary: "Deep troubleshooting runbook for gateway, channels, automation, nodes, and browser"
-read_when:
-  - The troubleshooting hub pointed you here for deeper diagnosis
-  - You need stable symptom based runbook sections with exact commands
-title: "Troubleshooting"
-sidebarTitle: "Troubleshooting"
----
-
-This page is the deep runbook. Start at [/help/troubleshooting](/help/troubleshooting) if you want the fast triage flow first.
-
-## Command ladder
-
-Run these first, in this order:
-
-```bash
-fengming status
-fengming gateway status
-fengming logs --follow
-fengming doctor
-fengming channels status --probe
-```
-
-Expected healthy signals:
-
-- `fengming gateway status` shows `Runtime: running`, `Connectivity probe: ok`, and a `Capability: ...` line.
-- `fengming doctor` reports no blocking config/service issues.
-- `fengming channels status --probe` shows live per-account transport status and, where supported, probe/audit results such as `works` or `audit ok`.
-
-## After an update
-
-Use this when an update finishes but the Gateway is down, channels are empty, or
-model calls start failing with 401s.
-
-```bash
-fengming status --all
-fengming update status --json
-fengming gateway status --deep
-fengming doctor --fix
-fengming gateway restart
-```
-
-Look for:
-
-- `Update restart` in `fengming status` / `fengming status --all`. Pending or
-  failed handoffs include the next command to run.
-- `plugin load failed: dependency tree corrupted; run fengming doctor --fix`
-  under Channels. That means the channel config still exists, but plugin
-  registration failed before the channel could load.
-- provider 401s after re-auth. `fengming doctor --fix` checks for stale
-  per-agent OAuth auth shadows and removes the old copies so all agents resolve
-  the current shared profile.
-
-## Split brain installs and newer config guard
-
-Use this when a gateway service unexpectedly stops after an update, or logs show that one `fengming` binary is older than the version that last wrote `fengming.json`.
-
-FengMing stamps config writes with `meta.lastTouchedVersion`. Read-only commands can still inspect a config written by a newer FengMing, but process and service mutations refuse to continue from an older binary. Blocked actions include gateway service start, stop, restart, uninstall, forced service reinstall, service-mode gateway startup, and `gateway --force` port cleanup.
-
-```bash
-which fengming
-fengming --version
-fengming gateway status --deep
-fengming config get meta.lastTouchedVersion
-```
-
-<Steps>
-  <Step title="Fix PATH">
-    Fix `PATH` so `fengming` resolves to the newer install, then rerun the action.
-  </Step>
-  <Step title="Reinstall the gateway service">
-    Reinstall the intended gateway service from the newer install:
-
-    ```bash
-    fengming gateway install --force
-    fengming gateway restart
-    ```
-
-  </Step>
-  <Step title="Remove stale wrappers">
-    Remove stale system package or old wrapper entries that still point at an old `fengming` binary.
-  </Step>
-</Steps>
-
-<Warning>
-For intentional downgrade or emergency recovery only, set `FENGMING_ALLOW_OLDER_BINARY_DESTRUCTIVE_ACTIONS=1` for the single command. Leave it unset for normal operation.
-</Warning>
-
-## Protocol mismatch after rollback
-
-Use this when logs keep printing `protocol mismatch` after you downgrade or roll back FengMing. This means an older Gateway is running, but a newer local client process is still trying to reconnect with a protocol range that the older Gateway cannot speak.
-
-```bash
-fengming --version
-which -a fengming
-fengming gateway status --deep
-fengming doctor --deep
-fengming logs --follow
-```
-
-Look for:
-
-- `protocol mismatch ... client=... v<version> min=<n> max=<n> expected=<n>` in Gateway logs.
-- `Established clients:` in `fengming gateway status --deep` or `Gateway clients` in `fengming doctor --deep`. This lists active TCP clients connected to the Gateway port, including PIDs and command lines when the OS allows it.
-- A client process whose command line points at the newer FengMing install or wrapper you rolled back from.
-
-Fix:
-
-1. Stop or restart the stale FengMing client process shown by `gateway status --deep`.
-2. Restart apps or wrappers that embed FengMing, such as local dashboards, editors, app-server helpers, or long-running `fengming logs --follow` shells.
-3. Re-run `fengming gateway status --deep` or `fengming doctor --deep` and confirm the stale client PID is gone.
-
-Do not make an older Gateway accept a newer incompatible protocol. Protocol bumps protect the wire contract; rollback recovery is a process/version cleanup problem.
-
-## Skill symlink skipped as path escape
-
-Use this when logs include:
-
-```text
-Skipping escaped skill path outside its configured root: ... reason=symlink-escape
-```
-
-FengMing treats every skill root as a containment boundary. A symlink under
-`~/.agents/skills`, `<workspace>/.agents/skills`, `<workspace>/skills`, or
-`~/.fengming/skills` is skipped when its real target resolves outside that root
-unless the target is explicitly trusted.
-
-Inspect the link:
-
-```bash
-ls -l ~/.agents/skills/<name>
-realpath ~/.agents/skills/<name>
-fengming config get skills.load
-```
-
-If the target is intentional, configure both the direct skill root and the
-allowed symlink target:
-
-```json5
-{
-  skills: {
-    load: {
-      extraDirs: ["~/Projects/manager/skills"],
-      allowSymlinkTargets: ["~/Projects/manager/skills"],
-    },
-  },
-}
-```
-
-Then start a new session or wait for the skills watcher to refresh. Restart the
-gateway if the running process predates the config change.
-
-Do not use broad targets such as `~`, `/`, or a whole synced project folder.
-Keep `allowSymlinkTargets` scoped to the real skill root that contains trusted
-`SKILL.md` directories.
-
-Related:
-
-- [Skills config](/tools/skills-config#symlinked-sibling-repos)
-- [Configuration examples](/gateway/configuration-examples#symlinked-sibling-skill-repo)
-
-## Anthropic 429 extra usage required for long context
-
-Use this when logs/errors include: `HTTP 429: rate_limit_error: Extra usage is required for long context requests`.
-
-```bash
-fengming logs --follow
-fengming models status
-fengming config get agents.defaults.models
-```
-
-Look for:
-
-- Selected Anthropic model is a GA-capable 1M Claude 4.x model, or the model has legacy `params.context1m: true`.
-- Current Anthropic credential is not eligible for long-context usage.
-- Requests fail only on long sessions/model runs that need the 1M context path.
-
-Fix options:
-
-<Steps>
-  <Step title="Use a standard context window">
-    Switch to a standard-window model, or remove legacy `context1m` from older
-    model config that is not GA-capable for 1M context.
-  </Step>
-  <Step title="Use an eligible credential">
-    Use an Anthropic credential that is eligible for long-context requests, or switch to an Anthropic API key.
-  </Step>
-  <Step title="Configure fallback models">
-    Configure fallback models so runs continue when Anthropic long-context requests are rejected.
-  </Step>
-</Steps>
-
-Related:
-
-- [Anthropic](/providers/anthropic)
-- [Token use and costs](/reference/token-use)
-- [Why am I seeing HTTP 429 from Anthropic?](/help/faq-first-run#why-am-i-seeing-http-429-ratelimiterror-from-anthropic)
-
-## Upstream 403 blocked responses
-
-Use this when an upstream LLM provider returns a generic `403` such as
-`Your request was blocked`.
-
-Do not assume this is always an FengMing configuration issue. The response can
-come from an upstream security layer such as a CDN, WAF, bot-management rule, or
-reverse proxy in front of an OpenAI-compatible endpoint.
-
-```bash
-fengming status
-fengming gateway status
-fengming logs --follow
-```
-
-Look for:
-
-- multiple models under the same provider failing in the same way
-- HTML or generic security text instead of a normal provider API error
-- provider-side security events for the same request time
-- a tiny direct `curl` probe succeeding while normal SDK-shaped requests fail
-
-Fix the provider-side filtering first when the evidence points to a WAF/CDN
-block. Prefer a narrowly scoped allow or skip rule for the API path FengMing
-uses, and avoid disabling protection for the whole site.
-
-<Warning>
-A successful minimal `curl` does not guarantee that real SDK-style requests will
-pass through the same upstream security layer.
-</Warning>
-
-Related:
-
-- [OpenAI-compatible endpoints](/gateway/configuration-reference#openai-compatible-endpoints)
-- [Provider configuration](/providers)
-- [Logs](/logging)
-
-## Local OpenAI-compatible backend passes direct probes but agent runs fail
-
-Use this when:
-
-- `curl ... /v1/models` works
-- tiny direct `/v1/chat/completions` calls work
-- FengMing model runs fail only on normal agent turns
-
-```bash
-curl http://127.0.0.1:1234/v1/models
-curl http://127.0.0.1:1234/v1/chat/completions \
-  -H 'content-type: application/json' \
-  -d '{"model":"<id>","messages":[{"role":"user","content":"hi"}],"stream":false}'
-fengming infer model run --model <provider/model> --prompt "hi" --json
-fengming logs --follow
-```
-
-Look for:
-
-- direct tiny calls succeed, but FengMing runs fail only on larger prompts
-- `model_not_found` or 404 errors even though direct `/v1/chat/completions`
-  works with the same bare model id
-- backend errors about `messages[].content` expecting a string
-- intermittent `incomplete turn detected ... stopReason=stop payloads=0` warnings with an OpenAI-compatible local backend
-- backend crashes that appear only with larger prompt-token counts or full agent runtime prompts
-
-<AccordionGroup>
-  <Accordion title="Common signatures">
-    - `model_not_found` with a local MLX/vLLM-style server → verify `baseUrl` includes `/v1`, `api` is `"openai-completions"` for `/v1/chat/completions` backends, and `models.providers.<provider>.models[].id` is the bare provider-local id. Select it with the provider prefix once, for example `mlx/mlx-community/Qwen3-30B-A3B-6bit`; keep the catalog entry as `mlx-community/Qwen3-30B-A3B-6bit`.
-    - `messages[...].content: invalid type: sequence, expected a string` → backend rejects structured Chat Completions content parts. Fix: set `models.providers.<provider>.models[].compat.requiresStringContent: true`.
-    - `validation.keys` or allowed message keys like `["role","content"]` → backend rejects OpenAI-style replay metadata on Chat Completions messages. Fix: set `models.providers.<provider>.models[].compat.strictMessageKeys: true`.
-    - `incomplete turn detected ... stopReason=stop payloads=0` → the backend completed the Chat Completions request but returned no user-visible assistant text for that turn. FengMing retries replay-safe empty OpenAI-compatible turns once; persistent failures usually mean the backend is emitting empty/non-text content or suppressing final-answer text.
-    - direct tiny requests succeed, but FengMing agent runs fail with backend/model crashes (for example Gemma on some `inferrs` builds) → FengMing transport is likely already correct; the backend is failing on the larger agent-runtime prompt shape.
-    - failures shrink after disabling tools but do not disappear → tool schemas were part of the pressure, but the remaining issue is still upstream model/server capacity or a backend bug.
-
-  </Accordion>
-  <Accordion title="Fix options">
-    1. Set `compat.requiresStringContent: true` for string-only Chat Completions backends.
-    2. Set `compat.strictMessageKeys: true` for strict Chat Completions backends that only accept `role` and `content` on each message.
-    3. Set `compat.supportsTools: false` for models/backends that cannot handle FengMing's tool schema surface reliably.
-    4. Lower prompt pressure where possible: smaller workspace bootstrap, shorter session history, lighter local model, or a backend with stronger long-context support.
-    5. If tiny direct requests keep passing while FengMing agent turns still crash inside the backend, treat it as an upstream server/model limitation and file a repro there with the accepted payload shape.
-  </Accordion>
-</AccordionGroup>
-
-Related:
-
-- [Configuration](/gateway/configuration)
-- [Local models](/gateway/local-models)
-- [OpenAI-compatible endpoints](/gateway/configuration-reference#openai-compatible-endpoints)
-
-## No replies
-
-If channels are up but nothing answers, check routing and policy before reconnecting anything.
-
-```bash
-fengming status
-fengming channels status --probe
-fengming pairing list --channel <channel> [--account <id>]
-fengming config get channels
-fengming logs --follow
-```
-
-Look for:
-
-- Pairing pending for DM senders.
-- Group mention gating (`requireMention`, `mentionPatterns`).
-- Channel/group allowlist mismatches.
-
-Common signatures:
-
-- `drop guild message (mention required` → group message ignored until mention.
-- `pairing request` → sender needs approval.
-- `blocked` / `allowlist` → sender/channel was filtered by policy.
-
-Related:
-
-- [Channel troubleshooting](/channels/troubleshooting)
-- [Groups](/channels/groups)
-- [Pairing](/channels/pairing)
-
-## Dashboard control UI connectivity
-
-When dashboard/control UI will not connect, validate URL, auth mode, and secure context assumptions.
-
-```bash
-fengming gateway status
-fengming status
-fengming logs --follow
-fengming doctor
-fengming gateway status --json
-```
-
-Look for:
-
-- Correct probe URL and dashboard URL.
-- Auth mode/token mismatch between client and gateway.
-- HTTP usage where device identity is required.
-
-If a local browser cannot connect to `127.0.0.1:18789` after an update, first
-recover the local Gateway service and confirm it is serving the dashboard:
-
-```bash
-fengming gateway restart
-lsof -i :18789
-curl http://127.0.0.1:18789
-```
-
-If `curl` returns FengMing HTML, the Gateway is working and the remaining issue
-is likely browser cache, an old deep link, or stale tab state. Open
-`http://127.0.0.1:18789` directly and navigate from the dashboard. If restart
-does not leave the service running, run `fengming gateway start` and recheck
-`fengming gateway status`.
-
-<AccordionGroup>
-  <Accordion title="Connect / auth signatures">
-    - `device identity required` → non-secure context or missing device auth.
-    - `origin not allowed` → browser `Origin` is not in `gateway.controlUi.allowedOrigins` (or you are connecting from a non-loopback browser origin without an explicit allowlist).
-    - `device nonce required` / `device nonce mismatch` → client is not completing the challenge-based device auth flow (`connect.challenge` + `device.nonce`).
-    - `device signature invalid` / `device signature expired` → client signed the wrong payload (or stale timestamp) for the current handshake.
-    - `AUTH_TOKEN_MISMATCH` with `canRetryWithDeviceToken=true` → client can do one trusted retry with cached device token.
-    - That cached-token retry reuses the cached scope set stored with the paired device token. Explicit `deviceToken` / explicit `scopes` callers keep their requested scope set instead.
-    - `AUTH_SCOPE_MISMATCH` → the device token was recognized, but its approved scopes do not cover this connect request; re-pair or approve the requested scope contract instead of rotating a shared gateway token.
-    - Outside that retry path, connect auth precedence is explicit shared token/password first, then explicit `deviceToken`, then stored device token, then bootstrap token.
-    - On the async Tailscale Serve Control UI path, failed attempts for the same `{scope, ip}` are serialized before the limiter records the failure. Two bad concurrent retries from the same client can therefore surface `retry later` on the second attempt instead of two plain mismatches.
-    - `too many failed authentication attempts (retry later)` from a browser-origin loopback client → repeated failures from that same normalized `Origin` are locked out temporarily; another localhost origin uses a separate bucket.
-    - repeated `unauthorized` after that retry → shared token/device token drift; refresh token config and re-approve/rotate device token if needed.
-    - `gateway connect failed:` → wrong host/port/url target.
-
-  </Accordion>
-</AccordionGroup>
-
-### Auth detail codes quick map
-
-Use `error.details.code` from the failed `connect` response to pick the next action:
-
-| Detail code                  | Meaning                                                                                                                                                                                      | Recommended action                                                                                                                                                                                                                                                                       |
-| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `AUTH_TOKEN_MISSING`         | Client did not send a required shared token.                                                                                                                                                 | Paste/set token in the client and retry. For dashboard paths: `fengming config get gateway.auth.token` then paste into Control UI settings.                                                                                                                                              |
-| `AUTH_TOKEN_MISMATCH`        | Shared token did not match gateway auth token.                                                                                                                                               | If `canRetryWithDeviceToken=true`, allow one trusted retry. Cached-token retries reuse stored approved scopes; explicit `deviceToken` / `scopes` callers keep requested scopes. If still failing, run the [token drift recovery checklist](/cli/devices#token-drift-recovery-checklist). |
-| `AUTH_DEVICE_TOKEN_MISMATCH` | Cached per-device token is stale or revoked.                                                                                                                                                 | Rotate/re-approve device token using [devices CLI](/cli/devices), then reconnect.                                                                                                                                                                                                        |
-| `AUTH_SCOPE_MISMATCH`        | Device token is valid, but its approved role/scopes do not cover this connect request.                                                                                                       | Re-pair the device or approve the requested scope contract; do not treat this as shared-token drift.                                                                                                                                                                                     |
-| `PAIRING_REQUIRED`           | Device identity needs approval. Check `error.details.reason` for `not-paired`, `scope-upgrade`, `role-upgrade`, or `metadata-upgrade`, and use `requestId` / `remediationHint` when present. | Approve pending request: `fengming devices list` then `fengming devices approve <requestId>`. Scope/role upgrades use the same flow after you review the requested access.                                                                                                               |
-
-<Note>
-Direct loopback backend RPCs authenticated with the shared gateway token/password should not depend on the CLI's paired-device scope baseline. If subagents or other internal calls still fail with `scope-upgrade`, verify the caller is using `client.id: "gateway-client"` and `client.mode: "backend"` and is not forcing an explicit `deviceIdentity` or device token.
-</Note>
-
-Device auth v2 migration check:
-
-```bash
-fengming --version
-fengming doctor
-fengming gateway status
-```
-
-If logs show nonce/signature errors, update the connecting client and verify it:
-
-<Steps>
-  <Step title="Wait for connect.challenge">
-    Client waits for the gateway-issued `connect.challenge`.
-  </Step>
-  <Step title="Sign the payload">
-    Client signs the challenge-bound payload.
-  </Step>
-  <Step title="Send the device nonce">
-    Client sends `connect.params.device.nonce` with the same challenge nonce.
-  </Step>
-</Steps>
-
-If `fengming devices rotate` / `revoke` / `remove` is denied unexpectedly:
-
-- paired-device token sessions can manage only **their own** device unless the caller also has `operator.admin`
-- `fengming devices rotate --scope ...` can only request operator scopes that the caller session already holds
-
-Related:
-
-- [Configuration](/gateway/configuration) (gateway auth modes)
-- [Control UI](/web/control-ui)
-- [Devices](/cli/devices)
-- [Remote access](/gateway/remote)
-- [Trusted proxy auth](/gateway/trusted-proxy-auth)
-
-## Gateway service not running
-
-Use this when service is installed but process does not stay up.
-
-```bash
-fengming gateway status
-fengming status
-fengming logs --follow
-fengming doctor
-fengming gateway status --deep   # also scan system-level services
-```
-
-Look for:
-
-- `Runtime: stopped` with exit hints.
-- Service config mismatch (`Config (cli)` vs `Config (service)`).
-- Port/listener conflicts.
-- Extra launchd/systemd/schtasks installs when `--deep` is used.
-- `Other gateway-like services detected (best effort)` cleanup hints.
-
-<AccordionGroup>
-  <Accordion title="Common signatures">
-    - `Gateway start blocked: set gateway.mode=local` or `existing config is missing gateway.mode` → local gateway mode is not enabled, or the config file was clobbered and lost `gateway.mode`. Fix: set `gateway.mode="local"` in your config, or re-run `fengming onboard --mode local` / `fengming setup` to restamp the expected local-mode config. If you are running FengMing via Podman, the default config path is `~/.fengming/fengming.json`.
-    - `refusing to bind gateway ... without auth` → non-loopback bind without a valid gateway auth path (token/password, or trusted-proxy where configured).
-    - `another gateway instance is already listening` / `EADDRINUSE` → port conflict.
-    - `Other gateway-like services detected (best effort)` → stale or parallel launchd/systemd/schtasks units exist. Most setups should keep one gateway per machine; if you do need more than one, isolate ports + config/state/workspace. See [/gateway#multiple-gateways-same-host](/gateway#multiple-gateways-same-host).
-    - `System-level FengMing gateway service detected` from doctor → a systemd system unit exists while the user-level service is missing. Remove or disable the duplicate before allowing doctor to install a user service, or set `FENGMING_SERVICE_REPAIR_POLICY=external` if the system unit is the intended supervisor.
-    - `Gateway service port does not match current gateway config` → the installed supervisor still pins the old `--port`. Run `fengming doctor --fix` or `fengming gateway install --force`, then restart the gateway service.
-
-  </Accordion>
-</AccordionGroup>
-
-Related:
-
-- [Background exec and process tool](/gateway/background-process)
-- [Configuration](/gateway/configuration)
-- [Doctor](/gateway/doctor)
-
-## macOS gateway silently stops responding, then resumes when you touch the dashboard
-
-Use this when channels (Telegram, WhatsApp, etc.) on a macOS host go quiet for minutes to hours at a time, and the gateway appears to come back the moment you open the Control UI, SSH in, or otherwise interact with the host. There is usually no obvious symptom in `fengming status` because by the time you look the gateway is alive again.
-
-```bash
-ls ~/.fengming/logs/stability/ | tail -5
-fengming gateway stability --bundle latest
-pmset -g log | grep -iE "sleep|wake|maintenance" | tail -50
-launchctl print gui/$UID/ai.fengming.gateway | grep -E "state|last exit|runs"
-```
-
-Look for:
-
-- One or more `*-uncaught_exception.json` bundles in `~/.fengming/logs/stability/` with `error.code` set to a transient network code such as `ENETDOWN`, `ENETUNREACH`, `EHOSTUNREACH`, or `ECONNREFUSED`.
-- `pmset -g log` lines like `Entering Sleep state due to 'Maintenance Sleep'` or `en0 driver is slow (msg: WillChangeState to 0)` aligned with the crash timestamps. Power Nap / Maintenance Sleep briefly puts the Wi-Fi driver into state 0; any outbound `connect()` that lands in that window can fail with `ENETDOWN` even on a host that otherwise has full network connectivity.
-- `launchctl print` output showing `state = not running` with multiple recent `runs` and an exit code, especially when the gap between crash and the next launch is on the order of an hour rather than seconds. macOS launchd applies an undocumented respawn-protection gate after a crash burst that can stop honoring `KeepAlive=true` until an external trigger such as interactive login, dashboard connection, or `launchctl kickstart` re-arms it.
-
-Common signatures:
-
-- A stability bundle whose `error.code` is `ENETDOWN` or a sibling code, with the call stack pointing into Node `net` `lookupAndConnect` / `Socket.connect`. FengMing `2026.5.26` and newer classify these as benign transient network errors so they no longer propagate to the top-level uncaught handler; if you are on an older release, upgrade first.
-- Long quiet periods that end the instant you connect to the Control UI or SSH into the host: the user-visible activity is what re-arms launchd's respawn gate, not anything the dashboard does to the gateway.
-- `runs` count incrementing across the day with no corresponding `received SIG*; shutting down` line in `~/Library/Logs/fengming/gateway.log`: clean shutdowns log a signal; transient crashes do not.
-
-What to do:
-
-1. **Upgrade the gateway** if you are running a release before `2026.5.26`. After upgrading, future `ENETDOWN` errors are logged as warnings instead of terminating the process.
-2. **Reduce maintenance sleep activity** on Mac mini / desktop hosts that are meant to run as always-on servers:
-
-   ```bash
-   sudo pmset -a sleep 0 disksleep 0 standby 0 powernap 0
-   ```
-
-   This significantly reduces, but does not entirely eliminate, the underlying driver flap. The system can still perform some maintenance sleeps for TCP keepalive and mDNS upkeep regardless of these flags.
-
-3. **Add a liveness watchdog** so a future crash burst that gets parked by launchd is caught quickly:
-
-   ```bash
-   # Example launchd-aware liveness check, suitable for a 5-minute cron or LaunchAgent
-   state=$(launchctl print gui/$UID/ai.fengming.gateway 2>/dev/null | awk -F'= ' '/state =/ {print $2; exit}')
-   if [ "$state" != "running" ]; then
-     launchctl kickstart -k gui/$UID/ai.fengming.gateway
-   fi
-   ```
-
-   The point is to externally re-arm the respawn gate; `KeepAlive=true` alone is not sufficient on macOS after a crash burst.
-
-Related:
-
-- [macOS platform notes](/platforms/macos)
-- [Logging](/logging)
-- [Doctor](/gateway/doctor)
-
-## Gateway exits during high memory use
-
-Use this when the Gateway disappears under load, the supervisor reports an OOM-style restart, or logs mention `critical memory pressure bundle written`.
-
-```bash
-fengming gateway status --deep
-fengming logs --follow
-fengming gateway stability --bundle latest
-fengming gateway diagnostics export
-```
-
-Look for:
-
-- `Reason: diagnostic.memory.pressure.critical` in the latest stability bundle.
-- `Memory pressure:` with `critical/rss_threshold`, `critical/heap_threshold`, or `critical/rss_growth`.
-- `V8 heap:` values near the heap limit.
-- `Largest session files:` entries such as `agents/<agent>/sessions/<session>.jsonl` or `sessions/<session>.jsonl`.
-- Linux cgroup memory counters when the gateway runs inside a container or memory-limited service.
-
-Common signatures:
-
-- `critical memory pressure bundle written` appears shortly before restart → FengMing captured a pre-OOM stability bundle. Inspect it with `fengming gateway stability --bundle latest`.
-- `memory pressure: level=critical ... memoryPressureSnapshot=disabled` appears in gateway logs → FengMing detected critical memory pressure, but the pre-OOM stability snapshot is off.
-- `Largest session files:` points at a very large redacted transcript path → reduce retained session history, inspect session growth, or move old transcripts out of the active store before restarting.
-- `V8 heap:` used bytes are close to the heap limit → lower prompt/session pressure, reduce concurrent work, or raise the Node heap limit only after confirming the workload is expected.
-- `Memory pressure: critical/rss_growth` → memory grew quickly inside one sampling window. Check the latest logs for a large import, runaway tool output, repeated retries, or a batch of queued agent work.
-- Critical memory pressure appears in logs but no bundle exists → this is the default. Set `diagnostics.memoryPressureSnapshot: true` to capture the pre-OOM stability bundle on future critical memory pressure events.
-
-The stability bundle is payload-free. It includes operational memory evidence and redacted relative file paths, not message text, webhook bodies, credentials, tokens, cookies, or raw session ids. Attach the diagnostics export to bug reports instead of copying raw logs.
-
-Related:
-
-- [Gateway health](/gateway/health)
-- [Diagnostics export](/gateway/diagnostics)
-- [Sessions](/cli/sessions)
-
-## Gateway rejected invalid config
-
-Use this when Gateway startup fails with `Invalid config` or hot reload logs say
-it skipped an invalid edit.
-
-```bash
-fengming logs --follow
-fengming config file
-fengming config validate
-fengming doctor
-```
-
-Look for:
-
-- `Invalid config at ...`
-- `config reload skipped (invalid config): ...`
-- `Config write rejected: ...`
-- A timestamped `fengming.json.rejected.*` file beside the active config
-- A timestamped `fengming.json.clobbered.*` file if `doctor --fix` repaired a broken direct edit
-- FengMing keeps the latest 32 `.clobbered.*` files for each config path and rotates older ones
-
-<AccordionGroup>
-  <Accordion title="What happened">
-    - The config did not validate during startup, hot reload, or an FengMing-owned write.
-    - Gateway startup fails closed instead of rewriting `fengming.json`.
-    - Hot reload skips invalid external edits and keeps the current runtime config active.
-    - FengMing-owned writes reject invalid/destructive payloads before commit and save `.rejected.*`.
-    - `fengming doctor --fix` owns repair. It can remove non-JSON prefixes or restore the last-known-good copy while preserving the rejected payload as `.clobbered.*`.
-    - When many repairs happen for one config path, FengMing rotates older `.clobbered.*` files so the newest repaired payload is still available.
-
-  </Accordion>
-  <Accordion title="Inspect and repair">
-    ```bash
-    CONFIG="$(fengming config file)"
-    ls -lt "$CONFIG".clobbered.* "$CONFIG".rejected.* 2>/dev/null | head
-    diff -u "$CONFIG" "$(ls -t "$CONFIG".clobbered.* 2>/dev/null | head -n 1)"
-    fengming config validate
-    fengming doctor
-    ```
-  </Accordion>
-  <Accordion title="Common signatures">
-    - `.clobbered.*` exists → doctor preserved a broken external edit while repairing the active config.
-    - `.rejected.*` exists → an FengMing-owned config write failed schema or clobber checks before commit.
-    - `Config write rejected:` → the write tried to drop required shape, shrink the file sharply, or persist invalid config.
-    - `config reload skipped (invalid config):` → a direct edit failed validation and was ignored by the running Gateway.
-    - `Invalid config at ...` → startup failed before Gateway services booted.
-    - `missing-meta-vs-last-good`, `gateway-mode-missing-vs-last-good`, or `size-drop-vs-last-good:*` → an FengMing-owned write was rejected because it lost fields or size compared with the last-known-good backup.
-    - `Config last-known-good promotion skipped` → the candidate contained redacted secret placeholders such as `***`.
-
-  </Accordion>
-  <Accordion title="Fix options">
-    1. Run `fengming doctor --fix` to let doctor repair prefixed/clobbered config or restore last-known-good.
-    2. Copy only the intended keys from `.clobbered.*` or `.rejected.*`, then apply them with `fengming config set` or `config.patch`.
-    3. Run `fengming config validate` before restarting.
-    4. If you edit by hand, keep the full JSON5 config, not just the partial object you wanted to change.
-  </Accordion>
-</AccordionGroup>
-
-Related:
-
-- [Config](/cli/config)
-- [Configuration: hot reload](/gateway/configuration#config-hot-reload)
-- [Configuration: strict validation](/gateway/configuration#strict-validation)
-- [Doctor](/gateway/doctor)
-
-## Gateway probe warnings
-
-Use this when `fengming gateway probe` reaches something, but still prints a warning block.
-
-```bash
-fengming gateway probe
-fengming gateway probe --json
-fengming gateway probe --ssh user@gateway-host
-```
-
-Look for:
-
-- `warnings[].code` and `primaryTargetId` in JSON output.
-- Whether the warning is about SSH fallback, multiple gateways, missing scopes, or unresolved auth refs.
-
-Common signatures:
-
-- `SSH tunnel failed to start; falling back to direct probes.` → SSH setup failed, but the command still tried direct configured/loopback targets.
-- `multiple reachable gateways detected` → more than one target answered. Usually this means an intentional multi-gateway setup or stale/duplicate listeners.
-- `Read-probe diagnostics are limited by gateway scopes (missing operator.read)` → connect worked, but detail RPC is scope-limited; pair device identity or use credentials with `operator.read`.
-- `Gateway accepted the WebSocket connection, but follow-up read diagnostics failed` → connect worked, but the full diagnostic RPC set timed out or failed. Treat this as a reachable Gateway with degraded diagnostics; compare `connect.ok` and `connect.rpcOk` in `--json` output.
-- `Capability: pairing-pending` or `gateway closed (1008): pairing required` → the gateway answered, but this client still needs pairing/approval before normal operator access.
-- unresolved `gateway.auth.*` / `gateway.remote.*` SecretRef warning text → auth material was unavailable in this command path for the failed target.
-
-Related:
-
-- [Gateway](/cli/gateway)
-- [Multiple gateways on the same host](/gateway#multiple-gateways-same-host)
-- [Remote access](/gateway/remote)
-
-## Channel connected, messages not flowing
-
-If channel state is connected but message flow is dead, focus on policy, permissions, and channel specific delivery rules.
-
-```bash
-fengming channels status --probe
-fengming pairing list --channel <channel> [--account <id>]
-fengming status --deep
-fengming logs --follow
-fengming config get channels
-```
-
-Look for:
-
-- DM policy (`pairing`, `allowlist`, `open`, `disabled`).
-- Group allowlist and mention requirements.
-- Missing channel API permissions/scopes.
-
-Common signatures:
-
-- `mention required` → message ignored by group mention policy.
-- `pairing` / pending approval traces → sender is not approved.
-- `missing_scope`, `not_in_channel`, `Forbidden`, `401/403` → channel auth/permissions issue.
-
-Related:
-
-- [Channel troubleshooting](/channels/troubleshooting)
-- [Discord](/channels/discord)
-- [Telegram](/channels/telegram)
-- [WhatsApp](/channels/whatsapp)
-
-## Cron and heartbeat delivery
-
-If cron or heartbeat did not run or did not deliver, verify scheduler state first, then delivery target.
-
-```bash
-fengming cron status
-fengming cron list
-fengming cron runs --id <jobId> --limit 20
-fengming system heartbeat last
-fengming logs --follow
-```
-
-Look for:
-
-- Cron enabled and next wake present.
-- Job run history status (`ok`, `skipped`, `error`).
-- Heartbeat skip reasons (`quiet-hours`, `requests-in-flight`, `cron-in-progress`, `lanes-busy`, `alerts-disabled`, `empty-heartbeat-file`, `no-tasks-due`).
-
-<AccordionGroup>
-  <Accordion title="Common signatures">
-    - `cron: scheduler disabled; jobs will not run automatically` → cron disabled.
-    - `cron: timer tick failed` → scheduler tick failed; check file/log/runtime errors.
-    - `heartbeat skipped` with `reason=quiet-hours` → outside active hours window.
-    - `heartbeat skipped` with `reason=empty-heartbeat-file` → `HEARTBEAT.md` exists but only contains blank lines / markdown headers, so FengMing skips the model call.
-    - `heartbeat skipped` with `reason=no-tasks-due` → `HEARTBEAT.md` contains a `tasks:` block, but none of the tasks are due on this tick.
-    - `heartbeat: unknown accountId` → invalid account id for heartbeat delivery target.
-    - `heartbeat skipped` with `reason=dm-blocked` → heartbeat target resolved to a DM-style destination while `agents.defaults.heartbeat.directPolicy` (or per-agent override) is set to `block`.
-
-  </Accordion>
-</AccordionGroup>
-
-Related:
-
-- [Heartbeat](/gateway/heartbeat)
-- [Scheduled tasks](/automation/cron-jobs)
-- [Scheduled tasks: troubleshooting](/automation/cron-jobs#troubleshooting)
-
-## Node paired, tool fails
-
-If a node is paired but tools fail, isolate foreground, permission, and approval state.
-
-```bash
-fengming nodes status
-fengming nodes describe --node <idOrNameOrIp>
-fengming approvals get --node <idOrNameOrIp>
-fengming logs --follow
-fengming status
-```
-
-Look for:
-
-- Node online with expected capabilities.
-- OS permission grants for camera/mic/location/screen.
-- Exec approvals and allowlist state.
-
-Common signatures:
-
-- `NODE_BACKGROUND_UNAVAILABLE` → node app must be in foreground.
-- `*_PERMISSION_REQUIRED` / `LOCATION_PERMISSION_REQUIRED` → missing OS permission.
-- `SYSTEM_RUN_DENIED: approval required` → exec approval pending.
-- `SYSTEM_RUN_DENIED: allowlist miss` → command blocked by allowlist.
-
-Related:
-
-- [Exec approvals](/tools/exec-approvals)
-- [Node troubleshooting](/nodes/troubleshooting)
-- [Nodes](/nodes/index)
-
-## Browser tool fails
-
-Use this when browser tool actions fail even though the gateway itself is healthy.
-
-```bash
-fengming browser status
-fengming browser start --browser-profile fengming
-fengming browser profiles
-fengming logs --follow
-fengming doctor
-```
-
-Look for:
-
-- Whether `plugins.allow` is set and includes `browser`.
-- Valid browser executable path.
-- CDP profile reachability.
-- Local Chrome availability for `existing-session` / `user` profiles.
-
-<AccordionGroup>
-  <Accordion title="Plugin / executable signatures">
-    - `unknown command "browser"` or `unknown command 'browser'` → the bundled browser plugin is excluded by `plugins.allow`.
-    - browser tool missing / unavailable while `browser.enabled=true` → `plugins.allow` excludes `browser`, so the plugin never loaded.
-    - `Failed to start Chrome CDP on port` → browser process failed to launch.
-    - `browser.executablePath not found` → configured path is invalid.
-    - `browser.cdpUrl must be http(s) or ws(s)` → the configured CDP URL uses an unsupported scheme such as `file:` or `ftp:`.
-    - `browser.cdpUrl has invalid port` → the configured CDP URL has a bad or out-of-range port.
-    - `Playwright is not available in this gateway build; '<feature>' is unsupported.` → the current gateway install lacks the core browser runtime dependency; reinstall or update FengMing, then restart the gateway. ARIA snapshots and basic page screenshots can still work, but navigation, AI snapshots, CSS-selector element screenshots, and PDF export stay unavailable.
-
-  </Accordion>
-  <Accordion title="Chrome MCP / existing-session signatures">
-    - `Could not find DevToolsActivePort for chrome` → Chrome MCP existing-session could not attach to the selected browser data dir yet. Open the browser inspect page, enable remote debugging, keep the browser open, approve the first attach prompt, then retry. If signed-in state is not required, prefer the managed `fengming` profile.
-    - `No Chrome tabs found for profile="user"` → the Chrome MCP attach profile has no open local Chrome tabs.
-    - `Remote CDP for profile "<name>" is not reachable` → the configured remote CDP endpoint is not reachable from the gateway host.
-    - `Browser attachOnly is enabled ... not reachable` or `Browser attachOnly is enabled and CDP websocket ... is not reachable` → attach-only profile has no reachable target, or the HTTP endpoint answered but the CDP WebSocket still could not be opened.
-
-  </Accordion>
-  <Accordion title="Element / screenshot / upload signatures">
-    - `fullPage is not supported for element screenshots` → screenshot request mixed `--full-page` with `--ref` or `--element`.
-    - `element screenshots are not supported for existing-session profiles; use ref from snapshot.` → Chrome MCP / `existing-session` screenshot calls must use page capture or a snapshot `--ref`, not CSS `--element`.
-    - `existing-session file uploads do not support element selectors; use ref/inputRef.` → Chrome MCP upload hooks need snapshot refs, not CSS selectors.
-    - `existing-session file uploads currently support one file at a time.` → send one upload per call on Chrome MCP profiles.
-    - `existing-session dialog handling does not support timeoutMs.` → dialog hooks on Chrome MCP profiles do not support timeout overrides.
-    - `existing-session type does not support timeoutMs overrides.` → omit `timeoutMs` for `act:type` on `profile="user"` / Chrome MCP existing-session profiles, or use a managed/CDP browser profile when a custom timeout is required.
-    - `existing-session evaluate does not support timeoutMs overrides.` → omit `timeoutMs` for `act:evaluate` on `profile="user"` / Chrome MCP existing-session profiles, or use a managed/CDP browser profile when a custom timeout is required.
-    - `response body is not supported for existing-session profiles yet.` → `responsebody` still requires a managed browser or raw CDP profile.
-    - stale viewport / dark-mode / locale / offline overrides on attach-only or remote CDP profiles → run `fengming browser stop --browser-profile <name>` to close the active control session and release Playwright/CDP emulation state without restarting the whole gateway.
-
-  </Accordion>
-</AccordionGroup>
-
-Related:
-
-- [Browser (FengMing-managed)](/tools/browser)
-- [Browser troubleshooting](/tools/browser-linux-troubleshooting)
-
-## If you upgraded and something suddenly broke
-
-Most post-upgrade breakage is config drift or stricter defaults now being enforced.
-
-<AccordionGroup>
-  <Accordion title="1. Auth and URL override behavior changed">
-    ```bash
-    fengming gateway status
-    fengming config get gateway.mode
-    fengming config get gateway.remote.url
-    fengming config get gateway.auth.mode
-    ```
-
-    What to check:
-
-    - If `gateway.mode=remote`, CLI calls may be targeting remote while your local service is fine.
-    - Explicit `--url` calls do not fall back to stored credentials.
-
-    Common signatures:
-
-    - `gateway connect failed:` → wrong URL target.
-    - `unauthorized` → endpoint reachable but wrong auth.
-
-  </Accordion>
-  <Accordion title="2. Bind and auth guardrails are stricter">
-    ```bash
-    fengming config get gateway.bind
-    fengming config get gateway.auth.mode
-    fengming config get gateway.auth.token
-    fengming gateway status
-    fengming logs --follow
-    ```
-
-    What to check:
-
-    - Non-loopback binds (`lan`, `tailnet`, `custom`) need a valid gateway auth path: shared token/password auth, or a correctly configured non-loopback `trusted-proxy` deployment.
-    - Old keys like `gateway.token` do not replace `gateway.auth.token`.
-
-    Common signatures:
-
-    - `refusing to bind gateway ... without auth` → non-loopback bind without a valid gateway auth path.
-    - `Connectivity probe: failed` while runtime is running → gateway alive but inaccessible with current auth/url.
-
-  </Accordion>
-  <Accordion title="3. Pairing and device identity state changed">
-    ```bash
-    fengming devices list
-    fengming pairing list --channel <channel> [--account <id>]
-    fengming logs --follow
-    fengming doctor
-    ```
-
-    What to check:
-
-    - Pending device approvals for dashboard/nodes.
-    - Pending DM pairing approvals after policy or identity changes.
-
-    Common signatures:
-
-    - `device identity required` → device auth not satisfied.
-    - `pairing required` → sender/device must be approved.
-
-  </Accordion>
-</AccordionGroup>
-
-If the service config and runtime still disagree after checks, reinstall service metadata from the same profile/state directory:
-
-```bash
-fengming gateway install --force
-fengming gateway restart
-```
-
-Related:
-
-- [Authentication](/gateway/authentication)
-- [Background exec and process tool](/gateway/background-process)
-- [Gateway-owned pairing](/gateway/pairing)
-
-## Related
-
-- [Doctor](/gateway/doctor)
-- [FAQ](/help/faq)
-- [Gateway runbook](/gateway)