kenaz 0.0.0-stage → 0.3.2

package/core/dev/SYNAPSE_WORKER_PER_PROJECT.md

@@ -0,0 +1,881 @@
+# Synapse Worker Per Project — Design + PoC (TASK_1984)
+
+Status: **ON by default (TASK_1999, 2026-09-22).** Feature flag
+`KENAZ_SYNAPSE_WORKERS` defaults ON everywhere, including installed builds —
+`KENAZ_SYNAPSE_WORKERS=0` opts out. See §10 for the decision, the R1 status
+aggregation and R5 auto-regulation that were the prerequisites, and the
+rollback lever.
+
+Base: main @ `c6ad8ec83` (includes TASK_1983 `active_project_paths()`,
+TASK_1958/1960/1980).
+
+DDx this card resolves: `rune_d7a57528a7a94868b1b2a9e16931aa2b` (2026-09-21,
+user + Ansuz). Crash-floor evidence: `rune_2c28a731b02c4d2594a6478719f8adbb`
+(3+ live `kuzu::Database` instances in one process → ACCESS_VIOLATION,
+`CRASH_INVESTIGATION_2026-04-18.md`), `rune_e3ddf1b76b8f45bf90a11a84d2674515`
+(cross-project concurrency fails from ~6 concurrent calls; **sequential**
+cross-project access is cheap, 85-112ms/call — concurrency is what breaks,
+not project count).
+
+---
+
+## 0. Problem restated
+
+Users run several projects at once (several Ansuz sessions, one per
+project). Today the Dashboard backend is **one process** with
+`MAX_DB_POOL_SIZE=2` KuzuDB instances, one of which is permanently pinned to
+the cortex cross-project graph (`meta_graph` is excluded from eviction
+candidates). That leaves **one** project slot. Switching the Dashboard's
+current project evicts whatever was resident (drain ≤30s + CHECKPOINT), so a
+second live session's Synapse tools go cold or block.
+
+The cap cannot be raised: 3+ live `kuzu::Database` instances in **one
+process** crash on Windows regardless of buffer-pool size (confirmed against
+the kuzu 0.11.3 fork). The cap is a **per-process** floor. The only way to
+keep N projects resident and reachable at once is **N processes**.
+
+---
+
+## 1. R1 — Measurements
+
+All measured 2026-09-21 on the live dev backend (`kenaz-dashboard.exe`,
+pid 32900, port 18080), one project (KenazAI) already resident, via
+read-only `Get-Process`/`tasklist` and one MCP call. No live session was
+disrupted (Synapse tool calls only; no rebuild, no restart).
+
+| Measurement | Value | Method |
+|---|---|---|
+| Backend RSS, 1 project resident, idle | **834,124 KB** (~815 MB) | `tasklist /FI "PID eq 32900"` |
+| Backend RSS immediately after a `synapse_status` call for a **second, non-current** project (esp32) | **923,212 KB** (~902 MB) | same, after `POST /api/mcp` `synapse_status` with `projectPath=.../esp32` |
+| **Delta for opening a second project's KuzuDB pool slot** | **+89,088 KB (~87 MB)** | difference of the above |
+| Call wall time for that cross-project call | 211 ms | `time curl` |
+| ONNX embedding model on disk (arctic-embed-xs, the variant actually deployed — SFR-400M/bge-m3 are unused fallbacks, confirmed no files present for either) | **87 MB** (`model.onnx`) + tokenizer.json | `du -h` on `%LOCALAPPDATA%\Kenaz Dashboard\resources\onnx\arctic-embed-xs\model.onnx` |
+| ONNX runtime DLL | 12 MB (`onnxruntime.dll`) | same dir |
+| Is the embedding model loaded per-process or per-project? | **Per-process.** `INDEX_ENGINE` and `QUERY_ENGINE` (`features/synapse/embedding.rs:103-104`) are `static OnceLock<Result<EmbeddingEngine, String>>` — process-global singletons, keyed by nothing project-specific. One process pays the ONNX load cost exactly once no matter how many projects it touches (today, or in a pool large enough not to evict). | Source read: `resolve_model_dir()` / `create_engine()` take no project argument; the OnceLocks are declared at module scope. |
+
+**RSS delta interpretation.** +87 MB for a second project's KuzuDB slot
+almost exactly matches the on-disk ONNX model size — that is very likely
+coincidence, not causation: the embedding engines are process-global
+OnceLocks that were already initialized before this call (this backend had
+already served searches for the resident project), so this call could not
+have triggered a *second* embedding-engine load. The ~87 MB is far more
+likely KuzuDB opening a second buffer pool + catalog + mmap'd pages for the
+esp32 graph database. Both numbers happening to be ~87 MB is the kind of
+coincidence that deserves a second, cleaner measurement before being relied
+on (see "Honest limits" below) — but it does not change the qualitative
+conclusion.
+
+**The N-worker RAM math this implies (structural, not measured per-worker —
+no worker existed to measure until this PoC):**
+
+- Per-worker fixed cost ≈ ONNX model (~87 MB on disk, ~100-180 MB resident
+  once ORT allocates its session arena — two engines, INDEX + QUERY, each
+  `create_engine()`'d independently, so treat 180-250 MB as the realistic
+  per-worker floor for the embedding half alone) **+** one KuzuDB instance's
+  buffer pool for that project (~87 MB, this run's measured delta) **+**
+  axum/tokio/reqwest runtime overhead (tens of MB, no WebView — a worker has
+  no Tauri window, so it should sit well below the 815 MB measured for the
+  full GUI process).
+- **Estimated floor per worker: ~250-400 MB.** This is a structural estimate
+  from measured components, not a measured worker RSS (R4 must replace it
+  with a real number — the runbook below asks for exactly that).
+- 16 GB machine, ~8 GB realistically available to background work after OS +
+  browser + IDE: **~20-30 concurrent workers** before RAM pressure, i.e. more
+  than any single user plausibly has active projects — but each worker
+  **duplicates the ONNX cost that a single process pays once.** N=5 active
+  projects costs ~5× the embedding RAM that one resident process pays today.
+- 32 GB machine: proportionally higher, same duplication tax.
+- **This duplication is the central trade-off for R5**, not raw worker
+  count: worker-per-project trades a hard crash floor (unfixable in-process)
+  for a RAM tax that scales with concurrently-active project count, paid
+  once per worker regardless of how many tools that worker serves.
+
+**Honest limits of R1.** This is one measurement on one machine with one
+warm backend; it is a lower bound / plausibility check, not a
+production capacity model. The 87 MB/87 MB coincidence above should be
+treated with suspicion until R4 measures an actual standalone worker's RSS
+after `/health` goes ready (with and without a prior search, to separate
+ORT init cost from KuzuDB open cost). No native-exception or crash was
+observed during this measurement.
+
+---
+
+## 2. R2 — Design
+
+### 2.1 Process model
+
+| Runs in **main backend** (Tauri process, port 18080) | Runs in **a worker** (one per active project, `--synapse-worker`) |
+|---|---|
+| cortex (cross-project `meta_graph`, `global_memory`) | that project's KuzuDB graph + FTS5 + embedding index |
+| Alaya (built on cortex) | that project's heat/pulse/co-change caches (already per-project files under `<project>/.synapse/`) |
+| hippocampus for the **current** project only (see 2.6 — no per-connection session identity exists yet, so "current" is still the Dashboard's global selection for tools that omit `projectPath`) | that project's inscribe spool (`memory/inscribe.rs`, already per-project dir — TASK_1959) |
+| the forwarder (routing table + on-demand spawn) | that project's `IS_INDEXING` / `index_jobs` / `CONNECTION_GATE` state (all become per-**worker**-process automatically — no code change needed, they are already process statics; a worker just doesn't share them with the main process or other workers) |
+| the axum router's non-synapse routes (agents, scheduler, tasks, git status, dispatch-log, ...) — unaffected, always local | its own embedding OnceLocks (ONNX loaded once per worker — the R1 duplication cost) |
+| the Job Object that owns every worker's lifecycle | nothing cross-project; a worker never talks to another worker or to cortex |
+
+The worker binary is **the same `kenaz-dashboard.exe`**, launched with
+`--synapse-worker --project <path> --port <n>` (see 2.7 — deliberately not
+the existing `bin/synapse-headless.rs`, which is a dev-only cargo bin target
+not present in an installed bundle). A worker starts no Tauri `Builder`, no
+WebView, no plugin manager, no scheduler — just the existing axum
+`HttpServer` bound to `127.0.0.1:<port>` with a fresh `HttpAppState`, i.e.
+exactly the subset of routes `routes::build_api_router` already builds
+(`/api/mcp`, `/api/synapse/*`, `/api/synapse/temporal/*`, `/health`), unused
+routes present but inert (no scheduler ever runs since nothing calls it).
+
+### 2.2 Forwarding layer
+
+Hook point: `mcp.rs::execute_tool_inner`, immediately before the tool-name
+`match`. Every MCP `tools/call` already carries (or, per TASK_1983, is
+expected to carry) an explicit `projectPath` for project-scoped tools —
+`is_project_scoped_tool()` already defines that prefix set
+(`synapse_*` / `hippocampus_*` / `cortex_*`).
+
+Rule: **forward iff** (a) `KENAZ_SYNAPSE_WORKERS=1`, (b) the tool is in the
+forwardable set, (c) `projectPath` is present and non-empty, and (d) the
+resolved project is **not** the Dashboard's current project. The current
+project always stays local — no loopback hop for the common case, and it is
+also the one case the main process is guaranteed to already have a live
+pool slot for.
+
+**Resolution failure: serve locally, never forward (TASK_1986 R2).** The
+decision above is factored into a pure `forward_decision(flag_on,
+tool_forwardable, current, requested) -> ServeLocally(reason) | Forward` in
+`synapse_worker.rs`. When the Dashboard's current project cannot be resolved
+(`current: None` — e.g. `projects.json` missing/unparseable), the decision
+is always `ServeLocally("current_unresolved")`, never `Forward` — a forward
+target cannot be validated as "not the current project" if there is no
+resolved current project to compare against, so the fail-safe direction is
+local execution, not a blind hop to whatever worker the request asked for.
+Logged once per process at `warn` level under the `synapse_worker` tracing
+target (not per call — a broken `projects.json` must not spam the log).
+
+At backend startup (before the reaper is armed), the resolved current
+project is logged once under the `synapse_worker` tracing target:
+`tracing::info!(target: "synapse_worker", workers_enabled, current_project =
+%path_or_"<unresolved>", "synapse worker forwarder configuration")` — or,
+when `KENAZ_SYNAPSE_WORKERS` is not `1`, a single line noting the forwarder
+is disabled (no project resolution attempted in that case).
+
+The REST surface (`/api/synapse/*`, used directly by the frontend, not
+through MCP) needs the same treatment at its own resolution point — out of
+scope for this PoC (R3 only forwards MCP `tools/call`), called out
+explicitly as a gap below (2.8).
+
+**Error semantics:**
+
+- Worker not yet running → spawned on demand; the calling request blocks on
+  the HTTP readiness probe (bounded — see 2.3) rather than failing fast.
+  First-touch latency = spawn-to-ready time (R4 measures this).
+- Worker spawn fails (can't find exe, can't write log dir, OS refuses
+  `Command::spawn`) → tool call returns `McpToolResult::error(...)`
+  immediately, with the underlying OS error in the message. No silent
+  fallback to local execution — a silent fallback would service the request
+  against the WRONG process's — actually the current process has no data for
+  a project it never opened, so this would just surface as a different
+  error downstream; failing at the forward boundary keeps the error close to
+  its cause.
+- Worker crashes mid-flight (native exception, panic) → the forwarder's
+  `reqwest` call returns a connection error, surfaced as
+  `McpToolResult::error`. The PoC does **not** retry or respawn
+  automatically (R5 gap — production needs the crash-restart-with-backoff
+  from the requirements, not implemented here).
+- Timeout: the PoC forward call uses a 120s client timeout (headroom for a
+  first-touch spawn + a slow `synapse_index full`); the readiness probe
+  itself uses a ~15s budget (50 × 300ms) that R4 should replace with a
+  measured number rather than this guess.
+
+### 2.3 Worker lifecycle
+
+- **Spawn trigger (PoC):** first forwarded call for a project with no
+  tracked worker. **Production** should instead spawn from
+  `active_project_paths()` (TASK_1983 — the live-session-lease set) on a
+  timer/watcher, so a worker is warm *before* the first tool call needs it,
+  not on the request's own critical path. The PoC's on-demand spawn was
+  chosen because it needs no background task and is enough to prove R4's
+  numbers; the production version trades a warm-start guarantee for a
+  standing idle-RAM cost per active session even if that session's Synapse
+  tools are never called — a real trade-off, not a free upgrade.
+- **Readiness:** HTTP probe against the worker's own `/health` route
+  (TASK_1961 lesson — never a bare TCP-connect: the listener binds long
+  before axum, embedding init, or the first KuzuDB open are actually able to
+  answer, so a TCP-only probe would report "ready" while the first real
+  request still queues behind slow init).
+- **Idle reaper:** not implemented in the PoC (`WORKERS` map only grows).
+  Production needs one — same shape as the pool's own missing "nothing
+  tracks last_used, no idle release" gap (rune on hippocampus concurrency,
+  2026-09-04) that TASK_1925 addressed for in-process pooling; a
+  worker-level reaper is the direct analogue, keyed off last-forwarded-call
+  time instead of `Arc::strong_count`.
+- **Crash restart with backoff:** not implemented in the PoC. Production:
+  detect via failed forward + a stale/dead PID in the `WORKERS` map, restart
+  with exponential backoff, cap retries, surface the give-up state through
+  `synapse_status` per-project (2.5).
+- **Job Object cleanup:** implemented — the PoC calls the *existing* global
+  `job_object::assign_child_pid(pid)` (already initialized at Tauri setup
+  for the pty-server/nextjs-server sidecars — `lib.rs:1770`), so every
+  worker dies with the Dashboard even on a hard crash
+  (`JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE`), same precedent as those two
+  sidecars. A **scoped** Job Object per worker (`job_object::ScopedJobObject`,
+  already exists for Agent Hub launches) would let production kill one
+  wedged worker without taking down the others — worth adopting over the
+  PoC's use of the single global job if per-worker force-kill becomes a
+  requirement.
+
+### 2.4 Port allocation and auth
+
+- **PoC:** a process-global incrementing counter starting at 18100.
+  Acceptable for one Dashboard instance in a dev session; **not** acceptable
+  for production — no collision check against a port a user's other
+  service might already hold, no persistence across restarts (a worker
+  registry rebuilt from nothing on every Dashboard boot, needing every
+  worker respawned).
+- **Production recommendation:** bind with port `0` (OS-assigned ephemeral)
+  and have the worker report its actual bound port back to the parent
+  (stdout line, matching the `SERVER_READY port=N` pattern
+  `bin/synapse-headless.rs` already uses) rather than dictating a port
+  up front — removes the collision class entirely.
+- **Auth:** workers bind `127.0.0.1` only (the PoC explicitly overrides the
+  shared `HttpServerConfig` default of `0.0.0.0`, which exists for LAN
+  mobile access to the *main* backend — TASK_088 — and is wrong for a
+  worker). The existing `require_auth_with_state` middleware already exempts
+  loopback, non-tunnel requests (`auth.rs:241`, `is_loopback`), so the
+  forwarder's plain loopback POST needs no token. This reuses the main
+  backend's existing trust model as-is: **any local process can reach a
+  worker's port**, same trust boundary the main backend already has. That is
+  an explicit, inherited trust decision, not a new hole — flagging it here
+  so it is reviewed as a decision rather than discovered later. If it needs
+  tightening, the tunnel-bypass-closing mechanism already in `auth.rs`
+  (`x-kenaz-source: tunnel` requires a real token even from loopback) is the
+  template to extend to worker-to-worker traffic if that ever exists (it
+  does not in this design — only main→worker, never worker→worker).
+
+### 2.5 Status aggregation
+
+- **PoC:** none. `synapse_status` called with a non-current `projectPath`
+  is transparently forwarded and returns that worker's own
+  process-local status — indistinguishable from a local call except for
+  added latency, which is the intended PoC behavior (correctness first).
+- **Production:** the Dashboard's project switcher needs worker state
+  (running / starting / dead / never-spawned) surfaced *without* a live
+  tool call — i.e. a lightweight main-process registry endpoint
+  (`GET /api/synapse/workers` — new, not in this PoC) reporting
+  `{project_path, port, pid, state, last_forward_at}` per tracked worker,
+  read from the same `WORKERS` map this PoC introduces. `synapse_status` for
+  the *current* project should also learn to report "N other projects have
+  live workers" so a user can see the whole resident set, not just the one
+  they're looking at.
+
+### 2.6 What stays process-global vs goes per-worker
+
+| Global today | After this design |
+|---|---|
+| `CONNECTION_GATE` / `MAX_TOTAL_CONNECTIONS=6` (`connection/mod.rs:99,265`) | Per-**worker** automatically (each worker is its own process with its own statics) — main backend's gate now only bounds *its own* current-project + cortex traffic, not the whole fleet. |
+| `IS_INDEXING` / `is_indexing_project()` (`connection/mod.rs:241,432`) | Per-worker automatically. A worker indexing its one project cannot block another project's connections the way today's single shared flag can when it inadvertently means "the whole backend is busy." |
+| `index_jobs` registry (`mcp_tools/synapse/index_jobs.rs`) | Per-worker automatically — a job ID is only meaningful within the process that issued it; the forwarder must route a `synapse_index job_id=...` status poll to the *same* worker that started the job (routing by `projectPath`, not by job ID, already achieves this as long as the caller keeps passing `projectPath`). |
+| Inscribe spool (`memory/inscribe.rs`, TASK_1959) | Already per-project directory on disk — no change; a worker restarting for the same project still finds and drains its own spool. |
+| Heat / pulse / co-change caches | Already per-project files under `<project>/.synapse/` — no change. |
+| Cortex `meta_graph` / `global_memory` | **Stays in the main process.** This is the one thing that is inherently cross-project by design (it exists specifically to correlate across projects) — moving it to a worker would defeat its purpose. It keeps its pinned pool slot in the main process exactly as today. |
+| `~/.kenaz/projects.json` registry, `CURRENT_PROJECT_OVERRIDE` | Stays global/shared, read by both the main process and (indirectly, via explicit `projectPath` args) workers — no worker ever needs to know the *current* project, only its own. |
+
+### 2.7 Why the same binary, not `bin/synapse-headless.rs`
+
+`bin/synapse-headless.rs` (TASK_1589) already does ~90% of what a worker
+needs — standalone axum server, ONNX preload, no Tauri — and was the first
+thing this design checked for reuse. It is a **separate Cargo `[[bin]]`
+target**, built only when something explicitly asks for it; it is not part
+of the Tauri bundle an end user installs (NSIS ships `kenaz-dashboard.exe`
+plus its sidecars, not a second full Rust binary). A shipped Dashboard must
+be able to spawn its own workers from the one executable it *is*, so the
+worker mode lives as a CLI-flag branch in `main.rs`/`infrastructure` instead.
+`synapse-headless` remains valuable as a dev/CI tool for indexing a repo
+standalone (its own use case, unrelated to this design) and was not
+modified.
+
+### 2.8 Known gaps / explicitly out of scope for this PoC
+
+- REST `/api/synapse/*` calls (used directly by the frontend) are not
+  forwarded — only MCP `tools/call`. A production rollout needs the same
+  routing rule applied at that boundary (likely by extracting the
+  `maybe_forward`-equivalent check into a small tower middleware shared by
+  both entry points, rather than duplicating it).
+- No per-connection MCP session identity exists (confirmed unavailable by
+  TASK_1983's investigation — `get_session_id()` is one process-lifetime
+  UUID, not per-connection). A tool call that omits `projectPath` still
+  falls back to the Dashboard's single global "current project," exactly as
+  today — this design does not change that resolution gap, it only adds
+  routing for calls that *do* pass `projectPath` explicitly.
+- Idle reaper, crash-restart-with-backoff, and worker-state status
+  aggregation (2.3, 2.5) are designed but not implemented — R3 scope was
+  "enough for R4," not production-complete.
+
+### 2.9 Rollback story
+
+- `KENAZ_SYNAPSE_WORKERS` unset (default): `maybe_forward` returns `None`
+  unconditionally on its first line — zero code path change, byte-identical
+  behavior to pre-TASK_1984 main.
+- `--synapse-worker` is a new, additive CLI branch checked before any
+  existing startup code runs; omitting the flag (every existing launch path)
+  leaves `main()` unchanged from the very next line onward.
+- No schema, config file, or on-disk format changed. Turning the flag back
+  off after a rollout requires no cleanup beyond killing any live worker
+  processes (the Job Object handles that automatically on the next Dashboard
+  restart even if a user forgets).
+
+---
+
+## 3. R3 — PoC (what actually ships in this branch)
+
+New file `dashboard/src-tauri/src/infrastructure/synapse_worker.rs`:
+
+- `try_run_worker_mode()` — parses `--synapse-worker --project <path> --port
+  <n>` from argv; if present, does the same ONNX preload/eager-init sequence
+  as `bin/synapse-headless.rs`, forces `KENAZ_HTTP_HOST=127.0.0.1` +
+  `KENAZ_HTTP_PORT=<n>`, starts `HttpServer::from_env()` with a fresh
+  `HttpAppState::new()`, prints a `READY` line, waits for Ctrl-C, exits.
+  Called from `main.rs` as the very first line of `fn main()`, before the
+  release-mode stderr redirect and before `kenaz_dashboard_lib::run()`
+  (Tauri) — a worker never touches the GUI stack.
+- `maybe_forward(name, args)` — the forwarder described in 2.2-2.4:
+  allowlist check, current-project short-circuit, on-demand spawn via
+  `ensure_worker()` (Job-Object-assigned, logs to
+  `<project>/.synapse/worker_<port>.log`), HTTP readiness probe
+  (`wait_for_worker_ready`), then a plain JSON-RPC `tools/call` proxy over
+  `reqwest` to the worker's `/api/mcp`.
+- Forwardable tool set for this PoC: `synapse_status`, `synapse_recall`,
+  `synapse_search`, `synapse_inscribe`, `synapse_index`.
+
+Touched (minimal, both additive):
+
+- `dashboard/src-tauri/src/main.rs` — one early-return guard calling
+  `try_run_worker_mode()`.
+- `dashboard/src-tauri/src/infrastructure/http_server/routes/mcp.rs` — one
+  `maybe_forward` call inserted immediately before the tool-name `match` in
+  `execute_tool_inner`.
+- `dashboard/src-tauri/src/infrastructure/mod.rs` — `pub mod synapse_worker;`.
+
+Not touched: `connection/`, `lib.rs` (beyond nothing — no lib.rs change was
+needed once the flag lived in `main.rs`), any production default, any
+existing test.
+
+---
+
+## 4. R4 — Experiment runbook (Ansuz runs this after merge)
+
+Goal: with the flag OFF vs ON, run two projects concurrently and compare
+failure/eviction counts, per-project latency, worker RSS, spawn-to-ready
+time, and backend.log for native exceptions.
+
+An existing probe already does almost exactly this shape —
+`dev/scripts/probe/pool-multi-project.js` (built for TASK_1925) round-robins
+concurrent calls across N project paths against `/api/mcp` and classifies
+failures by message substring, reporting p50/p95/max latency. Reuse it
+rather than writing a new one.
+
+### 4.1 Baseline (flag OFF — today's behavior, for comparison)
+
+```bash
+# Ensure KENAZ_SYNAPSE_WORKERS is NOT set in the backend's environment.
+node dev/scripts/probe/pool-multi-project.js \
+  --base-url http://127.0.0.1:18080 \
+  --concurrency 12 --waves 3 \
+  --projects "C:/Users/user/Documents/GitHub/KenazAI,C:/Users/user/Documents/GitHub/esp32"
+```
+
+Expected observables (from prior measurement, rune on 2026-07-25 pool
+probe): eviction/"connection limit exceeded"/"read pool exhausted" errors
+appearing above ~6 concurrent, `/api/rust/health` still answering
+throughout (backend does not crash — the caps exist precisely to prevent
+that), no native exceptions in backend.log.
+
+### 4.2 Flag ON — two workers concurrent
+
+```bash
+# Set once, before the Dashboard process starts (env var read once, not hot-reloadable):
+set KENAZ_SYNAPSE_WORKERS=1
+# (start/restart the Dashboard backend)
+
+node dev/scripts/probe/pool-multi-project.js \
+  --base-url http://127.0.0.1:18080 \
+  --concurrency 12 --waves 3 \
+  --projects "C:/Users/user/Documents/GitHub/KenazAI,C:/Users/user/Documents/GitHub/esp32"
+```
+
+`KenazAI` is (in this environment) the Dashboard's current project, so its
+calls stay local; `esp32` calls are forwarded and spawn a worker on first
+touch.
+
+Expected observables, and how to get each number:
+
+- **Spawn + forward evidence**: `grep -c "\[synapse-worker\]" backend.log`
+  should show a `starting pid=...` / `READY pid=...` pair for the esp32
+  worker, and `maybe_forward` doesn't log by itself in the PoC — add a
+  temporary `tracing::info!` there if the runbook needs an explicit forward
+  marker beyond inferring it from the worker's own request log (out of
+  scope to add permanently, but cheap for one verification run).
+- **Worker RSS**: `tasklist /FI "IMAGENAME eq kenaz-dashboard.exe"` will now
+  list 2+ processes — the main backend and each worker — by PID; the
+  worker's own log line prints its `pid=` for correlation. Compare against
+  R1's structural estimate (~250-400 MB floor).
+- **Spawn-to-ready time**: timestamp the first esp32-scoped probe request
+  vs. the worker's `READY` log line (or instrument `ensure_worker` with an
+  `Instant::now()` around `wait_for_worker_ready` for this run only).
+- **Failures/evictions**: compare `pool-multi-project.js`'s failure
+  classification counts against the 4.1 baseline — under the flag,
+  cross-project failures should trend toward zero (each project now has its
+  own pool, not a shared 2-slot one), at the cost of first-touch spawn
+  latency the baseline never pays.
+- **Per-project latency**: the probe's own p50/p95/max, split by which
+  project each call targeted (its `--projects` round-robin already tags
+  results per project internally — check the script's report format).
+- **Native exceptions**: `grep -i "0xc0000005\|ACCESS_VIOLATION\|0xe06d7363"
+  backend.log` across both runs — should be absent in both; a hit under the
+  flag would mean the worker boundary did not actually eliminate the
+  crash-floor risk (it should, since it eliminates concurrent
+  `kuzu::Database` instances in one process by construction, but this is
+  exactly the kind of claim that needs a negative-result check, not just an
+  assumption).
+
+### 4.3 Scale check (optional, if time allows)
+
+Repeat 4.2 with a third and fourth project added to `--projects` to see
+where the "practical concurrent-worker ceiling" from R1's estimate actually
+falls on this machine — `tasklist` RSS sum vs. available RAM is the direct
+measurement R1 could not take without workers existing yet.
+
+---
+
+## 5. R5 — Decision record
+
+**Go/no-go criteria (numbers, checked against R4's actual results once
+run):**
+
+- GO if: cross-project failure/eviction rate under the flag drops
+  materially versus the 4.1 baseline (target: nonzero improvement is not
+  enough — it should approach zero for the failure classes that are purely
+  about the shared 2-slot pool, since worker-per-project removes that
+  sharing by construction) AND no native exception appears in either arm
+  AND measured per-worker RSS stays within the R1 structural estimate
+  (~250-400 MB) — a number far outside that range would mean the design's
+  RAM math is wrong and needs redoing before any rollout sizing advice is
+  trustworthy AND spawn-to-ready time is bounded well enough that a user
+  opening a second project does not perceive multi-second hangs on first
+  Synapse tool use (no fixed threshold yet — R4 measures the actual number
+  first).
+- NO-GO if: R4 finds worker RSS far above the structural estimate (e.g. if
+  ONNX sessions turn out to share less than assumed, or KuzuDB's buffer pool
+  is larger under real project sizes than esp32's), or spawn latency makes
+  first-touch unacceptable, or a native exception appears under the flag
+  (would mean some cross-process assumption in this design is wrong, not
+  just a tuning problem).
+- **Bounded alternative if NO-GO**: a worker **pool of size K** (not
+  unbounded N) with LRU-by-lease-liveness eviction — same worker-process
+  mechanism, but capped at K concurrent workers (K chosen from R4's
+  measured per-worker RSS against a target machine's available RAM), evicting
+  the worker whose `active_project_paths()` lease is oldest/dead first. This
+  keeps the crash-floor fix (still N *separate* processes, so the
+  3-Database-instances-in-one-process crash is still structurally
+  impossible) while bounding total RAM cost, at the price of the same
+  "still only K projects fully resident" limitation this design set out to
+  remove — just with K much larger than today's *effective* 1.
+
+This PoC did not reach a final go/no-go — that requires R4's actual
+numbers, which this runbook hands to Ansuz to produce. The design above
+should not be treated as validated until those numbers come back within the
+bounds this section sets.
+
+## 6. Post-merge smoke (Ansuz, 2026-09-21, verify-worktree debug build 8f3296bd5)
+
+Standalone worker mode works without the forwarder: `kenaz-dashboard.exe --synapse-worker --project <throwaway git repo> --port 18090` printed `[synapse-worker] READY` and answered `POST /api/mcp synapse_status` within **2 s** of spawn. Measured idle footprint of that worker (empty project, no embedding computed yet): **RSS 40 MB, private 10 MB** (PowerShell Get-Process). That is far below the structural 250-400 MB floor estimated in §1 because the KuzuDB buffer pool and the ONNX engine are both loaded lazily on first use; the estimate applies to a worker that has actually indexed and embedded. Consequence for the capacity model: resident-but-idle workers are cheap, so the ceiling is set by concurrently *active* workers, not by how many projects have a live lease. The R4 experiment (flag ON, two projects concurrent through the forwarder) still requires restarting the Dashboard backend with `KENAZ_SYNAPSE_WORKERS=1`; it was not run in this session because the live dev backend belongs to the user.
+
+## 7. R4 results (Ansuz, 2026-09-21, live dev backend restarted with `KENAZ_SYNAPSE_WORKERS=1`)
+
+Setup: `npm run tauri:dev` started with the flag in its environment (debug
+build), Dashboard current project = KenazAI, second project = esp32 (real
+indexed `.synapse`). Probe: `dev/scripts/probe/pool-multi-project.js
+--concurrency 12 --waves 3 --projects KenazAI,esp32`.
+
+### 7.1 The runbook's probe never reached the forwarder
+
+§4 told the runner to reuse the TASK_1925 probe as-is. Its two tool shapes
+(`hippocampus_episode_recall`, `synapse_cypher`) are not in
+`FORWARDABLE_TOOLS`, so the first flag-ON run served all 36 calls locally
+and produced zero worker evidence while reporting 0 failures — a probe that
+cannot fail is not a probe. The script now has `--tool-set worker`
+(`synapse_status` + `synapse_recall`, both forwardable) and a per-project
+split in its JSON summary. Every number below is from that tool set.
+
+### 7.2 Three defects found on the first real forwarded burst
+
+| # | Defect | Evidence | Fix |
+|---|--------|----------|-----|
+| 1 | **Spawn race**: `ensure_worker` checked the registry, spawned, then inserted — no lock across the spawn. | 12 concurrent esp32 calls → the main process spawned **6** workers (ports 18100–18105). | Registry now holds one `tokio::sync::OnceCell` per project; concurrent first-touch callers share it, `get_or_try_init` runs the spawn once, a failed spawn leaves the cell empty for retry. Test: 12 tasks → 1 spawn, all see the same port. |
+| 2 | **Recursive workers**: a worker inherits `KENAZ_SYNAPSE_WORKERS=1`, and the Dashboard's *current* project is not its project, so every forwarded call re-entered `maybe_forward` inside the worker and spawned a grandchild worker. | Process tree: each of the 6 workers had 1–3 children, **14** `--synapse-worker` processes total; grandchildren logged `assign_child_pid failed: Job Object not initialized` → orphans that survive Dashboard exit. | `try_run_worker_mode` sets a process-wide `IS_WORKER` flag (checked before the env var) and removes the env var. Test: flag set + env set → `maybe_forward` returns `None`. |
+| 3 | **Shared port**: `server.rs` binds with `SO_REUSEADDR`; on Windows that lets a second process bind + listen on an already-listening port. | `netstat`: **7** listeners on 127.0.0.1:18100; every esp32 call failed with `forward to port 18100 failed: error sending request` because the connection landed on a recursing worker. | Worker sets `KENAZ_HTTP_EXCLUSIVE_BIND=1`; `start_with_state` then uses `SO_EXCLUSIVEADDRUSE` (raw `setsockopt`, the SDK defines it as a macro so windows-sys has no constant) instead of `SO_REUSEADDR`, so a clash fails at bind time. Main backend keeps `SO_REUSEADDR` (restart-after-TIME_WAIT). |
+
+Per-project result of that first burst: KenazAI (local) 18/18 OK, esp32
+(forwarded) **0/18** OK, wave-1 esp32 mean 5.1 s (ready-wait budget burned
+on a recursing child).
+
+### 7.3 Numbers after the fixes (same probe, backend rebuilt by the dev watcher)
+
+| Wave | Burst wall | KenazAI (local) mean / max | esp32 (forwarded) mean / max | Failures |
+|------|-----------:|---------------------------:|-----------------------------:|---------:|
+| 1 (first touch, spawn included) | 896 ms | 320 / 440 ms | **823 / 894 ms** | 0 / 12 |
+| 2 | 437 ms | 305 / 437 ms | 106 / 132 ms | 0 / 12 |
+| 3 | 416 ms | 299 / 415 ms | 127 / 187 ms | 0 / 12 |
+
+- **Workers spawned**: exactly 1 (pid 24236, port 18100, child of the
+  backend, inside its Job Object). `[synapse-worker] starting` / `READY`
+  pair in `esp32/.synapse/worker_18100.log`.
+- **Spawn-to-ready**: < 0.9 s wall including the first forwarded
+  `synapse_recall` (wave-1 esp32 max), debug build. The §4 ready budget of
+  15 s is generous by >15×; leave it (a cold release-build machine under
+  antivirus scan is the real worst case).
+- **Worker RSS after serving 9 recalls + 9 status calls**: **130 MB**
+  (Get-CimInstance WorkingSetSize) — below the §1 structural estimate of
+  250–400 MB because esp32 is small; the estimate stays the planning
+  number for a KenazAI-sized project.
+- **Native exceptions** (`0xc0000005|ACCESS_VIOLATION|0xe06d7363`) in
+  backend.log across both runs: **0**. `/api/rust/health` p95 ≤ 12 ms
+  throughout.
+- **Forward hop cost**: esp32 steady-state mean ~110–130 ms vs local
+  KenazAI ~300 ms — not comparable (different projects, different graph
+  sizes); the hop itself is loopback + JSON re-encode, well under the
+  variance between projects.
+
+### 7.4 What R4 did NOT establish
+
+- **No flag-OFF baseline with the `worker` tool set.** The same burst
+  with the flag off would route esp32 through the shared 2-slot pool. It
+  needs the backend restarted without the flag; not done this round. The
+  first flag-ON run with the `pool` tool set (everything local) showed
+  0/36 failures too, so at concurrency 12 × 2 projects the single-process
+  pool does **not** currently fail — TASK_1960/1980 already cut pool demand.
+  The failure-rate criterion in §5 is therefore not the discriminating
+  number; the crash floor (3 `kuzu::Database` in one process, TASK_1925)
+  is structural and does not show up in a probe that only opens two.
+- **Release build not measured** (RSS, spawn latency, `CREATE_NO_WINDOW`).
+- **No worker lifecycle**: nothing reaps a worker when its project's lease
+  dies; the worker survives until the Dashboard exits (Job Object).
+
+### 7.5 Interim R5 reading
+
+Within the bounds §5 set: no native exception under the flag, per-worker
+RSS below the structural estimate, first-touch latency sub-second. The
+three defects were all in the PoC's process plumbing, not in the design's
+premise, and each now has a regression test on the exact path that failed.
+**Not yet GO**: the flag stays off by default until (a) a flag-OFF control
+with the same tool set is recorded, (b) a release-build run, and (c) a
+worker reaper tied to `active_projects` leases exists (else every project a
+user ever touched in a session keeps a process). Those are TASK_1985-scale
+follow-ups, not blockers for merging the fixes.
+
+### 7.6 The two missing arms (Ansuz, 2026-09-21, later the same session)
+
+Same probe (`--tool-set worker --concurrency 12 --waves 3`, KenazAI + esp32).
+
+**Release build, flag ON** (`kenaz-dashboard.exe` 0.2.234 started standalone with `KENAZ_SYNAPSE_WORKERS=1`, dev stack stopped):
+
+| Wave | Burst wall | KenazAI mean / max | esp32 mean / max | Failures |
+|------|-----------:|-------------------:|-----------------:|---------:|
+| 1 (first touch) | 1010 ms | 994 / 1009 ms | 852 / 896 ms | 0 / 12 |
+| 2 | 477 ms | 345 / 477 ms | 136 / 192 ms | 0 / 12 |
+| 3 | 400 ms | 294 / 400 ms | 115 / 161 ms | 0 / 12 |
+
+- Worker RSS after 3 waves: esp32 **114 MB**, KenazAI **156 MB**; main process 221 MB. Spawn-to-ready ≈ 0.9–1.0 s (wave-1 max). No `conhost` child under either worker — `CREATE_NO_WINDOW` holds in the `windows_subsystem="windows"` binary.
+- **Observation, not a defect**: the release process spawned a worker for KenazAI too (port 18101). Standalone, its `get_current_project()` did not resolve to KenazAI (the release app's own settings, not the dev tree's), so *both* projects counted as non-current and were forwarded. Under the flag, "current project stays local" depends on that resolution being right at first touch; a production rollout should log the resolved current project once at startup so this is visible.
+
+**Dev build, flag OFF** (control — the shared 2-slot pool serves both projects in one process):
+
+| Wave | Burst wall | KenazAI mean / max | esp32 mean / max | Failures |
+|------|-----------:|-------------------:|-----------------:|---------:|
+| 1 | 517 ms | 327 / 462 ms | 327 / 516 ms | 0 / 12 |
+| 2 | 423 ms | 312 / 423 ms | 144 / 262 ms | 0 / 12 |
+| 3 | 446 ms | 313 / 432 ms | 318 / 446 ms | 0 / 12 |
+
+- 0 failures, 0 native exceptions, no eviction/limit errors. At 12 concurrent × 2 projects the single-process pool no longer fails — the 2026-07-25 expectation in §4.1 predates TASK_1960/1980's demand cut.
+
+**Reading.** The failure-rate criterion of §5 cannot discriminate the arms at this load: both are 36/36. What the flag buys is (a) the structural crash floor — a third concurrently-open project cannot put three `kuzu::Database` instances in one process — and (b) per-project isolation of latency: esp32 steady-state is ~115–136 ms via its worker vs 144–327 ms when sharing the pool with KenazAI's heavier calls. What it costs is ~110–160 MB RSS per resident worker plus ~1 s on first touch. Remaining gate before default-ON: worker lifecycle (reap on lease death / idle), see §8.
+
+## 8. Worker lifecycle — reaper (Ansuz, 2026-09-21, follow-up (c) of §7.5)
+
+`synapse_worker.rs` §3. Armed only in the main backend when
+`KENAZ_SYNAPSE_WORKERS=1` (workers never reap; flag off ⇒ no task at all).
+Every `KENAZ_SYNAPSE_WORKER_REAP_TICK_SECS` (default 60) it reads
+`active_projects::active_project_paths()` (the TASK_1983 lease set) and
+kills — `TerminateProcess` on Windows, `SIGTERM` on Unix — every registered
+worker that satisfies all three of:
+
+1. no live lease names its project (same `normalize_key` as the registry);
+2. idle for at least `KENAZ_SYNAPSE_WORKER_IDLE_SECS` (default 300) since
+   its last forwarded call completed;
+3. zero forwarded calls in flight (an RAII guard in `maybe_forward` holds
+   the count across the whole HTTP exchange, so a worker mid-answer is
+   never killed).
+
+The registry entry is removed under the lock *before* the kill, so a call
+arriving during the kill re-spawns cleanly through the §7.2 once-cell path.
+Killing only an idle worker leaves its KuzuDB WAL quiescent — the same
+state a Dashboard exit puts it in via KILL_ON_JOB_CLOSE.
+
+Tests (`t1984_dedupe_tests`, 5/5): selection picks only the
+idle-leaseless-quiescent worker out of {leased-idle, leaseless-idle,
+leaseless-fresh, leaseless-busy, still-spawning}; nothing is selected one
+millisecond before the TTL and the same worker is selected one millisecond
+after.
+
+**Not yet exercised live.** The selection is unit-tested and the kill path
+is the same `OpenProcess`/`TerminateProcess` pair `job_object.rs` already
+uses, but no run has yet watched a real worker disappear on lease death:
+that needs the backend restarted with the flag plus a short TTL
+(`KENAZ_SYNAPSE_WORKER_IDLE_SECS=15 KENAZ_SYNAPSE_WORKER_REAP_TICK_SECS=5`),
+and the single-instance plugin prevents a second debug instance beside the
+running dev stack. First thing to do on the next flag-ON restart; until then
+the flag stays off by default.
+
+## 9. R1 reaper evidence + R3 decision (Ansuz, 2026-09-22, TASK_1986)
+
+### 9.1 Reaper exercised live (R1 / A1)
+
+Dev backend restarted with `KENAZ_SYNAPSE_WORKERS=1 KENAZ_SYNAPSE_WORKER_IDLE_SECS=15
+KENAZ_SYNAPSE_WORKER_REAP_TICK_SECS=5` (18:20:23Z). Probe: `pool-multi-project.js --tool-set worker
+--projects KenazAI,esp32 --concurrency 6 --waves 2` (12 + 12 calls, 0 failures, burst 763 / 340 ms).
+esp32 has no live session lease, so its worker is leaseless by construction; KenazAI is the current
+project and is served locally.
+
+| t | Event (tauri dev stderr, `[synapse-worker]`) | Evidence |
+|---|---|---|
+| +0 | `reaper armed: idle_ttl=15s tick=5s` | flag read, reaper task alive |
+| +~40 s | worker pid **17592** (port 18100, project esp32) spawned by wave 1 | `Get-CimInstance` listed it with `--synapse-worker --project` |
+| +~85 s | `reaped idle worker pid=17592 port=18100 project=c:/users/user/documents/github/esp32` | process list empty |
+| re-probe | wave of 6 calls, 0 failures, burst 814 ms (first-touch respawn cost) | new worker pid **34156** (port 18101) |
+| later | `reaped idle worker pid=34156 port=18101 project=...esp32` | cycle repeats; no orphan |
+
+A1 met: one worker reaped after lease-less idle TTL, re-spawned on the next call through the once-cell
+path, reaped again. No failures on either side of the kill. The reaper line is `eprintln!` only: it is
+visible in the dev stderr log but **not in backend.log** (GUI builds) — R2's tracing line covers the
+startup config; the reaped event itself should also go through `tracing` before any default-ON
+rollout (small follow-up, not done here).
+
+### 9.2 R2
+
+Landed separately (startup line `synapse worker forwarder configuration` with `workers_enabled` and
+`current_project`, and `forward_decision` with the `current_unresolved -> serve locally` fail-safe).
+
+### 9.3 R3 decision record — recommendation: **dev-only ON, production stays opt-in**
+
+Numbers it rests on: §7.6 release-build flag-ON 36/36 vs dev flag-OFF 36/36 (the failure criterion
+cannot discriminate at 12x2); worker RSS 114-156 MB per project on release; spawn-to-ready ~1 s;
+§9.1 reaper cycle verified; the structural gain is the crash floor (a third open project cannot take the
+main process down) plus latency isolation.
+
+- **Dev machines (multi-project, long sessions): ON.** This is where 3+ projects are open at once and
+  where the KuzuDB crash floor has actually bitten (TASK_1925 history). Mechanism: the `tauri:dev`
+  launcher sets `KENAZ_SYNAPSE_WORKERS=1` (or `.env.local`), not a code default — rollback is unsetting
+  it.
+- **Production installs: stay opt-in** until (a) `synapse_status` aggregates across workers (R4),
+  (b) the release standalone `get_current_project()` resolution observed in §7.6 is fixed so the
+  current project is never forwarded to a worker of itself, (c) the reaped event is in backend.log,
+  and (d) a two-week dev soak shows no orphaned worker/port leak. A typical end-user install has one
+  project open; the extra 115-155 MB process buys them nothing yet.
+- **Not "everywhere"**: without (a)-(c) an installed user would see status numbers that silently
+  exclude worker-served projects (Name-Preserving Narrowing, sigil_12331ef3).
+
+The default flip — even dev-only — changes shared developer infrastructure and is the user's call;
+this section records the recommendation and its evidence, it does not apply it.
+
+## 10. Default ON everywhere (Ansuz, 2026-09-22, TASK_1999)
+
+### 10.1 Decision
+
+**User decision, 2026-09-22: default ON everywhere, including installed builds** — wider than §9.3's
+"dev-only" recommendation. §9.3's three blocking prerequisites for production are landed by this card:
+
+- (a) `synapse_status` aggregates across workers — R1, below.
+- (b) release standalone `get_current_project()` resolution — the R2 fail-safe already landed in
+  TASK_1986 (`forward_decision`'s `current_unresolved -> ServeLocally`, 7850d67c8): an unresolved
+  current project now always serves locally, never forwards to (or spawns) a worker for itself. This
+  card's R2 is the **live verification** of that fail-safe on a release standalone exe — procedure in
+  §10.4; evidence recorded there once the orchestrator's release-build smoke runs (this card is
+  Rust-only and does not itself build/run cargo — see the dispatch note at the top of this section's
+  commit).
+- (c) reaped/spawned events already go through `tracing` (target `synapse_worker`) — landed in
+  TASK_1986 (§8/§9.1), confirmed still in place: `reap_once` and `spawn_reaper_if_enabled` both log via
+  `tracing::info!` in addition to `eprintln!`.
+
+§9.3's fourth item — "a two-week dev soak shows no orphaned worker/port leak" — is superseded by R5
+below: auto-regulation (leased-idle TTL, a hard concurrent-worker cap, an RSS watchdog) is landed as a
+*precondition* of the flip, not a follow-up, so an orphan/leak class of problem has a structural ceiling
+(≤`KENAZ_SYNAPSE_WORKER_MAX` processes, each capped at `KENAZ_SYNAPSE_WORKER_MAX_RSS_MB`) rather than
+depending on empirical soak time to rule out unbounded growth.
+
+### 10.2 R1 — synapse_status worker aggregation
+
+`infrastructure::synapse_worker::status_block()` is the one source of truth both surfaces call:
+
+- MCP `synapse_status` (`graph.rs::tool_synapse_status`): adds top-level `workers` (array) and
+  `worker_totals` (object), additive only — no existing key's meaning changes.
+- REST `GET /api/synapse/status` (`routes/synapse.rs::get_synapse_status`): `SynapseStatusResponse`
+  gains the same two fields (camelCase on the wire: `workers`, `workerTotals`), populated in all three
+  response branches (project-specific, `resolve_core_path` error, global registry) so no branch can
+  silently narrow the numbers.
+
+Each `workers[]` row: `project, port, pid, spawned_at, last_used, idle_secs, leased, in_flight, rss_mb,
+status`. `status` is fetched over loopback from the worker's OWN `synapse_status` for its project (2s
+bound per worker), then trimmed to a compact subset (`compact_worker_status`, TASK_1999 follow-up) —
+`phase, indexing_in_progress, symbol_entries, rune_count, spool_depth, last_indexed` — since embedding
+the full ~5KB status payload per row made the aggregate balloon with every additional worker; a caller
+who wants the rest calls `synapse_status(projectPath=<that project>)` directly, which forwards to the
+same worker. A hung/dead worker degrades `status` to the string `"unreachable"` rather than blocking (or
+failing) the aggregated call for every other worker. `worker_totals`: `worker_count, total_rss_mb,
+total_in_flight, unreachable_count`.
+
+Degrade path (A1): kill a worker's process out from under the registry between reaper ticks — the next
+`synapse_status` poll's forwarded call to that port fails (connection refused), so that row's `status`
+reads `"unreachable"` and `unreachable_count` increments; `rss_mb`/`idle_secs`/`leased` still report
+last-known registry state until the next reaper tick removes the entry entirely.
+
+### 10.3 R5 — auto-regulation (precondition of default ON, not a follow-up)
+
+Default ON means several projects can each hold open a worker indefinitely without this. Extends the
+existing reaper (`spawn_reaper_if_enabled` / `reap_once`) rather than adding a second loop:
+
+| Knob | Default | What it controls |
+|---|---|---|
+| `KENAZ_SYNAPSE_WORKERS` | unset = **ON** | `0` disables the whole forwarder/worker mechanism (§10.5 rollback). Any other value (unset, `1`, garbage) is ON. |
+| `KENAZ_SYNAPSE_WORKER_IDLE_SECS` | 300 | Leaseless-idle TTL (unchanged from TASK_1986) — a worker whose project has no live session lease. |
+| `KENAZ_SYNAPSE_WORKER_LEASED_IDLE_SECS` | 900 | **New (R5a).** Idle TTL for a worker whose project DOES have a live lease. Longer than the leaseless TTL — reaping an actively-worked project is a bigger interruption, but respawn costs only ~1s (§7.6/§9.1), so an idle-but-leased worker is not held forever either. |
+| `KENAZ_SYNAPSE_WORKER_REAP_TICK_SECS` | 60 | Reaper poll interval (unchanged). |
+| `KENAZ_SYNAPSE_WORKER_MAX` | 3 | **New (R5b).** Hard cap on concurrently resident workers. Spawning past the cap evicts the LRU idle worker first (reason `cap`); if every tracked worker is busy, the call is served locally instead of spawning a 4th process (logged, not silently degraded). |
+| `KENAZ_SYNAPSE_WORKER_MAX_RSS_MB` | 400 | **New (R5c).** Per-worker RSS cap, checked on the reaper tick. |
+| `KENAZ_SYNAPSE_WORKERS_TOTAL_RSS_MB` | 1200 | **New (R5c).** Fleet-wide RSS cap, checked on the reaper tick — over cap reaps the idlest worker(s) first. |
+
+A busy worker (`in_flight != 0`) is never evicted by any of these — cap and RSS checks skip it the same
+way idle-TTL selection always has (a worker mid-answer is never killed). Every eviction — idle, cap, or
+RSS — logs via `tracing::info!(target: "synapse_worker", ...)` with an explicit `reason` field
+(`idle_leaseless | idle_leased | cap | rss`), landing in `backend.log` on installed builds, not just dev
+stderr.
+
+Unit tests (`synapse_worker.rs`, no process spawning): `select_reapable` (leaseless vs. leased TTL
+selection), `decide_capacity` (room available / evict LRU idle / serve-locally-when-all-busy),
+`select_rss_reap_victims` (per-worker cap, fleet-total cap reaping idlest-first, busy workers spared in
+both), `build_worker_rows` (leased/idle/rss reporting from a fake registry), plus the flag-parsing and
+`is_forwardable_tool` suites for R3 below.
+
+**A4 (live, not yet run by this card — Rust-only dispatch, no cargo build/run):** open 4 projects via
+`dev/scripts/probe/pool-multi-project.js` → expect only 3 resident workers (4th evicts the LRU idle one,
+`reason=cap` in backend.log); leave one leased project idle past 900s → its worker is reaped
+(`reason=idle_leased`) and respawns on the next call; a synthetic RSS breach
+(`KENAZ_SYNAPSE_WORKER_MAX_RSS_MB` set artificially low) reaps the idlest worker (`reason=rss`);
+`synapse_status.workers[]` shows `rss_mb`/`idle_secs` throughout. Left for the orchestrator's live/
+release smoke pass.
+
+### 10.4 R2 — release standalone smoke procedure (for the orchestrator to run + record)
+
+1. Stop any running `tauri dev` backend by PID (never `taskkill` by image name — see repo-wide lesson).
+2. `cargo build --release` (in the fixed verify worktree — kuzu re-cmakes per worktree PATH, ~20 min the
+   first time per worktree, per `reference_worktree_rust_verify_recipe.md`).
+3. Launch the release exe standalone (not through the installer) with `KENAZ_SYNAPSE_WORKERS` unset
+   (default ON) and at least two projects known to `projects.json`.
+4. Call `synapse_status` (MCP or REST) for the **current** project — expect `workers[]` to show **no**
+   entry for it (R2 fail-safe: the current project is always served locally) and the
+   `log_forwarder_config` startup line (`backend.log`, target `synapse_worker`) to show
+   `current_project` resolved to that project's path, not `<unresolved>`.
+5. Call `synapse_status` for **one other** known project — expect exactly one worker entry to appear for
+   it, with `status` reporting real (not `"unreachable"`) counts.
+6. Record: both pids, `rss_mb` for the spawned worker, and the exact `forwarder configuration` /
+   `forward_decision` log lines from `backend.log`.
+7. Evidence goes in this section (§10.4) once run — not yet run as of this card (Rust-only dispatch;
+   the orchestrator runs it in the fixed verify worktree per the dispatch contract at the top of this
+   task).
+
+### 10.5 R3 — the flip + rollback
+
+`KENAZ_SYNAPSE_WORKERS` semantics inverted: **on unless set to exactly `0`** (`workers_enabled()` in
+`synapse_worker.rs`). Unset, `1`, or any other value all mean ON — the PoC's `== Some("1")` opt-in gate
+is retired. `FORWARDABLE_TOOLS`'s five-tool PoC allowlist is replaced by `is_forwardable_tool`, matching
+`mcp.rs::is_project_scoped_tool`'s full `synapse_*` / `hippocampus_*` / `cortex_*` prefix set (duplicated
+in `synapse_worker.rs` rather than calling that private function — `mcp.rs` is outside this card's scope
+paths; the two prefix sets must be kept in sync by hand until they share one home). Reaper defaults
+(`KENAZ_SYNAPSE_WORKER_IDLE_SECS=300`, `KENAZ_SYNAPSE_WORKER_REAP_TICK_SECS=60`) are unchanged.
+
+**Rollback:** set `KENAZ_SYNAPSE_WORKERS=0` and restart the backend. Every forwarded call serves locally
+again (`workers_enabled()` returns false), the reaper never arms, and `log_forwarder_config` logs the
+disabled state at startup. No worker processes survive a restart with the flag off (Job Object /
+`KILL_ON_JOB_CLOSE` owns their lifecycle either way).
+
+**Dev launcher:** verified already clean — `dashboard/package.json`'s `tauri:dev` script does not set
+`KENAZ_SYNAPSE_WORKERS` (grepped; only `dev/scripts/probe/pool-multi-project.js`, this doc, and the two
+Rust source files reference the var), so there is no launcher env to remove — the default-ON flag
+semantics apply to `tauri:dev` automatically, same as every other entry point.
+
+### 10.6 What this card did NOT touch
+
+- `mcp.rs::execute_tool_inner`'s comment above the `maybe_forward` call still says "behind env
+  KENAZ_SYNAPSE_WORKERS=1" — stale after the flip, but `mcp.rs` is outside this card's scope paths.
+  Small follow-up for whoever next edits that file.
+- The R2 `forward_decision` fail-safe itself (`current_unresolved -> ServeLocally`) is untouched, per
+  the dispatch instruction — §10.1(b) only points at it, does not modify it.
+
+### 10.7 Two defects found on the release standalone smoke (Ansuz, 2026-09-22, TASK_1999 R6)
+
+The §10.4 smoke procedure surfaced two defects, both fixed in `synapse_worker.rs`:
+
+**Defect 1 — `synapse_status` forwarded to a worker.** With the §10.5 widened forwardable prefix set,
+`synapse_status` for a foreign project was itself a forwardable tool — it got proxied to that project's
+worker, so the caller got the WORKER's view (`workers: [] / worker_count 0`, since a worker never spawns
+children) instead of the main process's aggregation across every resident worker. Fix: `is_forwardable_tool`
+now checks an explicit `MAIN_ONLY_TOOLS` deny-list first — `synapse_status`, `synapse_health`,
+`synapse_maintain`, `synapse_index`, `synapse_release_project`, and any `synapse_evolution_*` tool — all of
+which either answer a question about the process/pool/worker-set itself or trigger a long-running job, and
+so are always served locally regardless of which project they're asked about. `synapse_status` served
+locally on a foreign project is unaffected by this — it already gets that project's per-project numbers
+from `status_block()`'s own direct (non-forwarding) queries to each tracked worker's `/api/mcp` (§10.2);
+only the "hop the whole call over to a worker and return ITS answer" path is blocked.
+
+**Defect 2 — current-project identity split by spelling/bundling.** On the release exe started standalone
+from `target/release`, `get_current_project()` resolved to
+`<repo>/dashboard/src-tauri/target/release/kenaz-core` — the bundled Synapse binary's own exe path — so a
+call for the source repo (`C:/Users/user/Documents/GitHub/KenazAI`) compared as a FOREIGN project and
+spawned a 160MB self-worker. Separately, an esp32 worker path was observed logged with mixed slash
+spellings (`C:\...` vs `C:/...`), which a plain string-equality key would also treat as two projects. Fix,
+both in `normalize_key`/`forward_decision`:
+
+- `normalize_key` now resolves via `std::fs::canonicalize` when the path exists on disk (falling back to
+  the raw path when it doesn't — a worker's `--project` arg for a not-yet-touched path, or a key built from
+  a test fixture), strips Windows' verbatim (`\\?\`) prefix after the backslash→forward-slash pass, trims a
+  trailing slash, and lowercases only on Windows (`#[cfg(windows)]`) — so spelling alone never creates a
+  second identity for the same on-disk project.
+- A new `same_project_or_bundle(current, requested) -> bool` rule, consulted by `forward_decision` right
+  after the existing `current == requested` check: if `current` is a bundled `kenaz-core` exe path
+  (`is_bundled_kenaz_core` — normalized path contains `/target/` and ends with `/kenaz-core`, `.exe` suffix
+  stripped first), its real project identity is the source repo that built it — the ancestor 4 directory
+  levels above the exe's parent (`<repo>/dashboard/src-tauri/target/release/kenaz-core` -> `<repo>`, walking
+  up through `release`, `target`, `src-tauri`, `dashboard`, computed via 5 `Path::parent()` calls from the
+  full exe path and re-`normalize_key`'d). A `requested` path equal to that computed root is
+  `ServeLocally("current_bundle_alias")` — same project, two spellings, not forwarded and no self-worker
+  spawned. A genuinely different `requested` path (not the bundle's own source root) still `Forward`s
+  normally.
+
+Unit tests (`synapse_worker.rs`): `is_forwardable_tool_denies_main_only_tools` /
+`is_forwardable_tool_still_allows_recall` (Defect 1); `normalize_key_collapses_slash_and_case_spelling` +
+`same_spelling_variants_serve_locally_via_forward_decision` (Windows-only, mixed-slash/case collapse),
+`is_bundled_kenaz_core_detects_the_release_exe_path`,
+`bundled_current_and_source_root_requested_serve_locally`,
+`bundled_current_but_genuinely_different_project_forwards`, `genuinely_different_project_still_forwards`
+(Defect 2) — all in `t1999_project_identity_tests`.
+
+### 10.8 A4 live evidence (dev, 2026-09-22, default ON + short TTLs via env)
+
+Env: `KENAZ_SYNAPSE_WORKER_LEASED_IDLE_SECS=60 KENAZ_SYNAPSE_WORKER_IDLE_SECS=40 KENAZ_SYNAPSE_WORKER_REAP_TICK_SECS=10 KENAZ_SYNAPSE_WORKERS_TOTAL_RSS_MB=260`; a lease file for esp32 was placed in `.kenaz/specialist/sessions/leases/` (heartbeat fresh, 8 h liveness) so esp32 counted as leased. Calls were direct `synapse_recall` per project (see the probe note below).
+
+| step | observed |
+|---|---|
+| recall esp32, CognitiveLabs, trip | workers 18100 (154 MB), 18101 (120 MB), 18102 (145 MB); status rows carry `idle_secs` / `leased` / `rss_mb` |
+| next reaper ticks (total 419 MB > 260 MB cap) | `reaped worker ... project=esp32 reason="rss"`, then `... cognitivelabs reason="rss"`; trip (145 MB) survives under the cap |
+| trip idle > 40 s | `reaped worker ... project=trip reason="idle_leaseless"` |
+| esp32 touched again, then idle > 60 s with a live lease | `reaped worker ... project=esp32 reason="idle_leased"` |
+| recall esp32 after the reap | fresh worker on port 18104 (respawn on demand) |
+
+Two lessons for anyone re-running this: (1) the RSS cap is a hard cap and reaps leased workers too (idlest first) — intended; (2) `synapse_status` is main-only since R6, so a probe that alternates status/recall with one call per project spawns nothing — `dev/scripts/probe/pool-multi-project.js --tool-set worker` now uses `synapse_search` + `synapse_recall`.