sensorium-mcp 3.0.80 → 3.0.84

package/templates/sensorium-repair.default.md

@@ -27,7 +27,8 @@ You are repairing **Sensorium itself** (this MCP server + its .NET supervisor)
 - **Root cause over symptoms.** A restart "fixes" nothing if the bug recurs. Find the exact file:line and the commit that introduced it.
 - **Verify file contents via terminal, never trust cached reads.** The `read_file` tool can serve a STALE copy (it served an old `package.json` version during a real incident). For anything you will edit or reason about critically, confirm with `git show HEAD:<path>` or `Get-Content <path>`.
 - **Never trust a subagent's described diff.** Subagents have hallucinated multi-hundred-line diffs for files that were clean. Always confirm with `git diff` / `git status` in a terminal yourself.
-- **Watch for concurrent autonomous agents.** A Sensorium worker may be committing to this repo *while you work*. Re-check `git HEAD`, branch, and `git log` before and after edits; a fix you were about to write may already be committed.
+- **Phantom working-tree changes = stale VS Code editor buffers.** Files silently reverting to OLD versions on disk (a committed fix undone, `package.json` version rolled back) is caused by VS Code `files.autoSave` + `files.hotExit` writing a stale editor buffer to disk — NOT an agent, NOT you editing manually. It's the same mechanism that "clobbers" AI-edited files mid-session. Confirm: `git status`/`git diff` shows a stale revert; check `%APPDATA%\Code\Backups\**` for hot-exit backups whose mtime is *after* your last save (⇒ an unreloaded window is holding a stale buffer). Fix permanently: set `"files.autoSave": "off"` + `"files.hotExit": "off"` in User settings, then `Revert File` on the stale tabs and reload the window. Until the window reloads, the risk persists for it.
+- **Watch for concurrent autonomous agents.** A Sensorium worker may be committing to this repo *while you work*. Re-check `git HEAD`, branch, and `git log` before and after edits; a fix you were about to write may already be committed. (But a spontaneous working-tree revert is usually the VS-Code-buffer issue above, not an agent — verify file mtimes: an agent write shows a recent mtime, a buffer flush may not.)
 - **Git safety:** stage explicit paths only. NEVER `git add -A/./-u`, `git commit -a`. NEVER commit `tmp_*`, dumps, screenshots, or binaries.
 - **Deploy ≠ push.** Pushing to git does not update the running system (see "Ship & deploy").
 
@@ -37,8 +38,8 @@ You are repairing **Sensorium itself** (this MCP server + its .NET supervisor)
 - **Logs** under `~/.remote-copilot-mcp/logs/`:
   - `mcp/server.log` — MCP app log (rotated ~5 MB).
   - `mcp/mcp-stderr-YYYYMMDD.log` — captured MCP stderr (fatal errors, startup banners, `[wait]`/`[telegram]` activity).
-  - `supervisor/supervisor-YYYYMMDD.log` — supervisor restart/health decisions (very noisy with Telegram `getUpdates`; filter it out).
-- **Process model:** the .NET supervisor process `sensorium-supervisor` supervises a Node MCP server (**HTTP port 3847**) + watcher (**3848**). MCP start command is either `node dist/index.js` (local build) or `npx -y sensorium-mcp@latest` (published). Workers are `claude`/`codex`/`copilot` processes whose command line references `.remote-copilot-mcp`.
+  - `supervisor/supervisor-YYYYMMDD.log` — supervisor restart/health decisions (very noisy with Telegram `getUpdates`; filter it out). **Timezone caveat:** MCP logs (`server.log`, `mcp-stderr`) are **UTC** (`...Z`); the supervisor log was historically **LOCAL**, but a `UtcTimestampFormatter` change can switch it to UTC — always cross-reference one shared event (e.g. "MCP server is ready" vs the node startup banner) to confirm which, before correlating timestamps.
+- **Process model:** the .NET supervisor process `sensorium-supervisor` supervises a Node MCP server (**HTTP port 3847**) + watcher (**3848**). MCP start command is either `node dist/index.js` (local build) or `npx -y sensorium-mcp@latest` (published). Workers are `claude`/`codex`/`copilot` processes whose command line references `.remote-copilot-mcp` — **a process check that queries only `node`/`sensorium-supervisor` will MISS every worker and falsely report "no threads running." Always include `claude,codex,copilot`.**
 - **`~/.remote-copilot-mcp/server.pid`** holds `{"pid":N}` — should match the port-3847 owner. A mismatch is the signature of a false-death loop.
 - **Windows: `git` is often NOT on PATH.** Locate it (`%LOCALAPPDATA%\Programs\Git\cmd\git.exe` or `C:\Program Files\Git\cmd\git.exe`) and call it by full path.
 - **Prior root causes are logged in memory:** read `/memories/repo/critical-fixes-log.md`, `/memories/repo/keeper-stuck-detection-bug.md`, `/memories/repo/sensorium-architecture.md` before theorizing — the bug may be known.
@@ -63,7 +64,8 @@ You are repairing **Sensorium itself** (this MCP server + its .NET supervisor)
 - **Keeper "stuck" kills keepAlive threads.** `wait_for_instructions` long-poll didn't refresh the per-thread heartbeat → keeper declared stuck → killed. Fix: write `writeThreadHeartbeat()` in the poll loop.
 - **Duplicate thread spawns.** Keeper checks the root threadId but `start_thread` creates a worker with a different id → always "not running" → re-spawns. Fix: track the worker id from the `start_thread` response.
 - **Copilot process dies / no memory.** `copilot.bat` (VS Code wrapper) spawned detached dies silently — prefer `copilot.exe`; and a missing `thread_id` column crashed bootstrap. See critical-fixes-log.
-- **Self-update port-handoff race.** New node self-registers its pid asynchronously, so `server.pid` transiently names a dead pid while HTTP is healthy — the trigger for the false-death loop above.
+- **Reads but no reply — Claude worker answers into invisible stdout.** 👀 is set on message *consume*, not on reply (so 👀 ≠ answered). With Claude tool search active, reply tools (`report_progress`/`send_voice`/`send_message_to_thread`) are deferred, so a conversational agent emits answers as plain `text` → its stdout transcript, never delivered. Signature: `server.log` `Read N messages … Processing…` with no following delivery `tool_use`. First rule out routing (operator's topic must equal the thread's `thread_registry.telegram_topic_id`; inbound `resolveThreadForTopic`/outbound `resolveTelegramTopicId` are symmetric). Fix: `ENABLE_TOOL_SEARCH=false` so tools load upfront, then **respawn the worker** (see Deploy).
+- **Self-update port-handoff race (real but often MIS-attributed — do not over-chase).** In theory a new node self-registers its pid asynchronously, so `server.pid` can transiently name a dead pid while HTTP is healthy. BUT the self-update poller is gated behind a ~10-min min-uptime **AND** `remoteVersion !== currentVersion`, so at the latest published version it is a silent no-op. Before blaming self-update, confirm it is ACTUALLY firing: look for `[self-update] New version detected` / `Closing HTTP server` and a non-empty `update-spawn.log`; a `[self-update] Deferring — uptime …s < min 600s` line proves it is NOT updating. In practice the recent restart loops were **event-loop freeze** (heavy startup work) and **network outages to Telegram**, not self-update.
 
 ## Emergency stop & restart
 
@@ -91,9 +93,19 @@ Then `Stop-Process -Id <pid> -Force` in the order above. Verify ports 3847/3848
 - **Ship:** `npm version patch --no-git-tag-version` → commit explicit paths (`fix(...)` then `chore(release): x.y.z`) → `git push origin main`.
 - **CI:** confirm green — `gh run watch <run-id> --repo <owner>/<repo> --exit-status`. Fix and re-push until green.
 - **Deploy at runtime (push alone does NOT deploy):**
-  - **MCP server** running via `npx sensorium-mcp@latest` picks up a new version only after **`npm publish`** (the watcher auto-updates). A local `node dist/index.js` needs `npm run build`.
+  - **MCP server** running via `npx sensorium-mcp@latest` picks up a new version only after **`npm publish`** (the watcher auto-updates). A local `node dist/index.js` needs `npm run build`. **Already-running workers keep their old spawn behavior even after the server updates — respawn them to apply spawn-side fixes.** `ENABLE_TOOL_SEARCH=false` is the right knob (Claude ≥2.1.7), but an `env` block in `~/.claude/settings.json` overrides the spawn env, and the spawn copies that file wholesale into the per-thread `CLAUDE_CONFIG_DIR` (dragging in the operator's own MCP servers).
   - **Supervisor** (.NET) needs a **rebuild + reinstall via `Install-Sensorium.ps1`** to apply supervisor code changes — a running supervisor keeps its old binary in memory.
 
+## Field notes (hard-won — check these before deep code work)
+
+- **"No threads connected" / a thread won't respond is usually NOT a dead thread.** First confirm liveness, don't assume: heartbeats in `~/.remote-copilot-mcp/heartbeats/{threadId}` fresh (age < ~5s) + a live `claude` worker per `pids/{threadId}.pid` = the thread IS alive. If threads are alive but the operator can't reach them, the break is the **Telegram delivery path** or the agent replying into stdout (see failure modes), not the thread.
+- **The Claude worker's stream-json transcript is ground truth for agent behavior** — `~/.remote-copilot-mcp/logs/threads/<Name>_<threadId>_<YYYY-MM-DD>.json` (NDJSON; file-fd stdout+stderr, survives server restarts). Per `type:"assistant"` block: `text` = invisible to operator, `tool_use` of a delivery tool = actually sent. `type:"rate_limit_event"` = Claude 5-hour cap (`resetsAt` = epoch secs); `subtype:"api_retry"` = transient API errors. Fresh MCP heartbeat + **stale transcript mtime** = worker idle in the long-poll (normal), not frozen.
+- **`[dispatcher] Poll error: fetch failed` (repeating every ~6s) + `.NET` supervisor `SocketException 10054 "connection forcibly closed by remote host"` = a NETWORK outage to `api.telegram.org`, not a code bug.** On laptops this correlates with Wi-Fi drops / Modern Standby / power-source flapping (see Windows Event Log `Kernel-Power` 105/506/507). Test reachability directly: `Resolve-DnsName api.telegram.org`, `Test-NetConnection api.telegram.org -Port 443`, `Invoke-WebRequest https://api.telegram.org -Method Head`. If those pass but the MCP still shows no successful polls, the dispatcher may not have recovered from the socket reset → **restart the MCP** to re-establish polling.
+- **Responsive vs frozen vs exited — three different states, check explicitly.** `Invoke-WebRequest -Method Options http://127.0.0.1:3847/{health,ready,mcp}` returning **204** + `Get-Process -Id <pid>` `Responding=True` = event loop ALIVE (so any "silence" is a stuck sub-loop like the Telegram poller, not a freeze). No HTTP answer but port still LISTENing = frozen. Port CLOSED (`connection refused`) = process EXITED.
+- **Use the lifecycle instrumentation (shipped 3.0.79+).** On a death, `server.log` gets `[lifecycle] process exit code=N uptime=Ns`; a real OS signal logs `[lifecycle] received signal ...`; a native fatal error writes `report*.json` in `logs/mcp/`; an internal self-kill attempt logs `[process] killProcessTree pid=… caller=…` or `[process] REFUSED kill … self-protection`. **`exit code=0 uptime=0-2s` markers are BENIGN** — short-lived launcher/duplicate processes that found the port taken and bailed cleanly, not the real server dying.
+- **Windows Event Log is decisive for native crash vs external kill.** No `Application Error` (Event 1000) for `node.exe` at the death time ⇒ **not** a native crash (e.g. not `better_sqlite3`). No `[lifecycle]`/`[shutdown]`/`[fatal]` line either ⇒ the process was hard-killed externally (`taskkill /F` / `TerminateProcess`, which bypass signal handlers and WER) — audit the internal kill paths (`killProcessTree caller=…`) and any second supervisor doing `KillByPort`.
+- **Event-loop freeze death-spiral.** Heavy synchronous startup work (better-sqlite3 consolidation, narrative backfill, per-thread `start_session` bootstrap embeddings across many reconnecting threads) blocks the loop past the supervisor's ~3s HTTP readiness probe → supervisor kills+respawns → the fresh process re-runs the same work → re-freeze. Mitigations: persist a per-day marker so consolidation/rotation run **once per day** across restarts (`src/data/daily-marker.ts`); defer/chunk heavy startup work off the readiness window.
+
 ## After an incident
 
 Record the root cause, the fix, the commit hash, and the version in `/memories/repo/critical-fixes-log.md` (append, don't rewrite) so the next repair starts from knowledge, not zero.