@animalabs/connectome-host 0.7.3 → 0.7.4

package/CHANGELOG.md

@@ -2,6 +2,162 @@
 
 ## Unreleased
 
+## 0.7.4 — 2026-08-03
+
+### Changed
+
+- **Frontdesk agents ride the adaptive path.** `frontdesk` strategies now
+  default to adaptive resolution + kv-stable folding, same as
+  `autobiographical` (a recipe can pin `adaptiveResolution: false` to keep the
+  old hierarchical renderer). The hierarchical renderer reserves nothing for
+  the raw tail and cannot shed summary mass, so a long-lived frontdesk agent
+  eventually saturates its fixed context budget into a terminal
+  `UncoveredDropError` refusal loop — the 2026-08-03 boter clerk outage.
+  Details and deltas:
+  - Topic-aware chunking now rides context-manager's `chunkBoundaryHint` seam
+    (requires CM ≥0.6.3) instead of a fork of `rebuildChunks` that silently
+    bypassed chunk-record persistence and the fail-closed orphan guard.
+  - Existing frontdesk stores carry no chunk records (the fork never wrote
+    them); context-manager's `migrateChunkRecords` backfills them from L1
+    `sourceIds` on first load, so upgraded stores do not re-compress lived
+    history. First boot re-plans folds (one-time KV churn, possibly a burst of
+    L1 production for the un-summarized frontier).
+  - The salience-biased L1 emission order is retired (it was a hierarchical-
+    renderer concept); unanswered questions/@mentions are still preserved
+    verbatim through the compression prompt.
+  - Witnessed chunks now get the base witnessed compression prompt; the fork
+    predated witnessed prompts and overrode them.
+
+### Added
+
+- **Every WebUI inspection panel now works per fleet child.** One persistent
+  scope dropdown in the sidebar header ("inspecting: …") replaces the
+  per-tab pill rows Lessons/Files carried — MCPL, Context, Settings, Pins,
+  Health, and the main-pane Context document all follow it, instead of the
+  previous split where only Lessons/Files could switch (statefully, via
+  duplicated pickers), Context 404'd by mis-sending the child name as an
+  `?agent=` param, and MCPL/Settings/Pins/Health were silently locked to the
+  fleetmaster. Backed by one generic fleet IPC verb pair
+  (`panel-request`/`panel-response`) dispatching into a shared panel layer
+  (`src/web/panel-data.ts`) that both the WebUI host and headless children
+  run — a new panel op needs no protocol change to work fleet-wide.
+  Details:
+  - `/debug/context/{,makeup,coverage,curve,preview,maintenance}` and
+    `/healthz` accept `?scope=<child>` — the host proxies to the child over
+    the fleet IPC and answers with its JSON verbatim (still curl-able;
+    connectome-doctor / fleet hub can now watch children through the host).
+    `/curve?scope=<child>` passes through to the scoped JSON.
+  - The MCPL tab shows the scoped process's **live** loaded servers
+    (connection status, tool counts — the long-missing fleet mcpl snapshot)
+    above the shared registry file; registry edits stay host-scope (the
+    file is one cwd-shared registry, so a "child-local edit" would be a
+    lie) and the panel says so instead of hiding the fact.
+  - Settings mutations, dry-run previews (single-flight guard now lives in
+    the target process), and pin add/remove run inside the scoped child;
+    child pins snapshots ship picker candidates (real store ids) since the
+    SPA has no window into a child's message store.
+  - Scoped WS responses (`lessons-list`, `workspace-*`, `mcpl-list`,
+    `settings-state`, `pins-list`) now echo their `scope`, and the SPA
+    drops replies that arrive after the operator switched — fixing a
+    pre-existing race where a slow child's lessons/files could render under
+    another child's header.
+  - Child health snapshots include the child's recent provider-call ledger.
+- **TUI: context budget gauge.** The status bar's `ctx:` readout and the fleet
+  tree's per-agent readouts show `142k/180k` against the *live* runtime budget
+  (runtime overrides win over the recipe), and the status segment goes yellow at
+  75% / red at 90% — "how close to compression/trouble" at a glance instead of a
+  bare number.
+- **TUI: fleet view viewport.** The tree now scrolls with the cursor
+  (`┈ N lines above/below ┈` markers) instead of clipping past the bottom of the
+  terminal — previously a large fleet let the cursor walk below the fold and
+  Del:stop targeted rows the operator couldn't see.
+- **TUI: fleet view opens with a summary header** — agent counts
+  (running/done/failed/cancelled across local subagents *and* fleet children),
+  children up/crashed, session cost — plus the active ops alerts in full (the
+  status bar only has room for a count).
+- **TUI: event timestamps.** Alerts, tool batches, subagent results, wake
+  triggers, branch switches, errors and user messages get an `HH:MM` prefix, so
+  scrollback read an hour later still answers "when".
+- **TUI: root-agent tool completions are visible.** Verbose shows every
+  `✓ tool (1.2s)`; terse shows the slow ones (≥2s). Slow *running* tools show a
+  live elapsed in the status bar after 5s — "still executing" and "stuck" no
+  longer look identical.
+- **TUI: the status bar names the worst active alert** (`⚠ 2 ·
+  compression-quarantine`), with quarantine and inference-exhausted outranking
+  the merely-recent.
+
+### Changed
+
+- **TUI: thinking honors the Ctrl+V verbose toggle.** Terse mode collapses live
+  thinking to a counting one-liner (`💭 thinking… ~1.2k tok`) and replayed
+  history thinking to one truncated line per block — the toggle's label always
+  claimed this.
+- **TUI: session-history replay caps at the last 50 messages** (marker points at
+  the web UI for the rest) instead of flooding scrollback with the whole session.
+- **TUI: elapsed times are humane everywhere** — `5m48s`, not `348s`; the fleet
+  tree and both peek views now agree.
+- **TUI: peek-proc renders child `ops:alert` events properly** (red `⚠ kind:
+  message`, cyan for `-clear`) instead of a dim `· ops:alert` dot line, and no
+  longer prints dot lines for per-block/per-round bookkeeping events.
+- **TUI: the status-left segment truncates to fit** the terminal width instead of
+  shoving the tokens/mem segment off the row.
+
+### Fixed
+
+- `cancel-subagent-result` was missing from the headless runtime's
+  subscription-filter exemptions: a parent that narrowed the event stream
+  could never see its own cancel confirmations.
+- The Vite dev server proxies `/debug`, `/healthz`, `/curve`, and `/files`
+  to the running host — previously every HTTP panel fetch 404'd under
+  `bun run dev`.
+- **TUI: "Branch switched" announcements survive.** The line was printed *before*
+  `refreshFromStore()` cleared the scrollbox, so it was destroyed unread.
+
+### Upgrade notes
+
+- **subagents/lessons/retrieval are now opt-in** (they were opt-out, and
+  DEFAULT_RECIPE enabled all three). A recipe that omits them ran them under
+  v0.7.2 and stops running them on this upgrade — that is the fix for
+  "lessons injected despite following the onboarding guide" (Discord issue
+  #32) working as intended. A recipe that *explicitly* enables them keeps
+  them, deliberately: a defaults change cannot tell old boilerplate from a
+  real choice. Before upgrading an existing deployment, run
+
+      bun scripts/audit-module-optins.ts <recipes-and-data-dirs...>
+
+  It reports every explicit enable, every omission that changes behavior,
+  and every retrieval-without-lessons combination that would go silently
+  inert — and modifies nothing; the decisions stay with the operator.
+  Persisted `data/.recipe.json` files are launch-time snapshots, not
+  authoritative sources — the audit lists them separately as pointers back
+  to the source recipe. Retrieved-lesson injection also moved from the
+  system prompt to after the last user message, which keeps the stable
+  prefix KV-cacheable.
+
+### Added
+
+- **Operator retrieval traces.** The Web UI now exposes operator-only,
+  process-memory retrieval traces at `/debug/retrieval` and a readable
+  lesson-selection viewer at `/debug/retrieval/view`, including invoking-agent
+  attribution, mechanical candidates, relevance decisions, cache provenance,
+  and the exact injected lesson block. Exact conversation/model inputs remain
+  opt-in via literal `includeInputs=1`.
+
+### Fixed
+
+- **OpenAI retrieval reasoning effort.** Recipes using `openai-responses` or
+  `openai-codex` can set `modules.retrieval.reasoningEffort` independently of
+  the primary agent. Unsupported providers fail recipe validation instead of
+  receiving an invalid OpenAI-shaped request, and reasoning-enabled retrieval
+  requires an explicit model instead of falling through to the Claude default.
+  
+- **`mcpl_list` reports the live MCPL policy boundary.** Each server now shows
+  connected/retrying state, whether its initial policy was established, its
+  effective grant, host-masked and deny-by-default capability paths, and the
+  separate host-owned `host/command` authority. During a rolling upgrade,
+  fields unavailable from an older agent-framework render as `unknown` rather
+  than as a misleading empty grant.
+
 ## 0.7.3 — 2026-08-01
 
 ### Changed
@@ -68,8 +224,6 @@
   write, output, breakpoints, duration and verdict, with refusals and errors
   highlighted. Cumulative totals for the session remain in the Usage panel.
 
-## Unreleased
-
 ## 0.7.0 — 2026-07-26
 
 ### Added
@@ -89,8 +243,6 @@
   `/debug/context/makeup` this costs nothing and makes no `count_tokens` network
   call, so it is safe on the 15s health poll.
 
-## Unreleased
-
 ## 0.6.1 — 2026-07-26
 
 ### Fixed
@@ -105,8 +257,6 @@
   text/id filter. Caught by checking the endpoint against a real store before
   anyone used the panel.
 
-## Unreleased
-
 ## 0.6.0 — 2026-07-26
 
 ### Added
@@ -170,8 +320,6 @@
   measured cost, and notes runs are serialized so a second click is refused
   rather than queueing another pause.
 
-## Unreleased
-
 ## 0.5.3 — 2026-07-26
 
 ### Added
@@ -294,8 +442,6 @@
 - `compressionMaxTokens` recipe passthrough — cap compression output for
   models with low output ceilings (2c78936).
 
-## Unreleased
-
 ### Fixed
 
 - **TUI bug sweep** (#64): operator-safety and observability fixes.

package/HEADLESS-FLEET-PLAN.md

@@ -223,12 +223,34 @@ One JSON object per line. All events include `type`; most include `ts` (epoch ms
 {"type":"text","content":"Please resummarize last week"}
 {"type":"command","command":"/status"}
 {"type":"shutdown","graceful":true}
+{"type":"panel-request","op":"settings","params":{"agent":"clerk"},"corrId":"panel-1"}
 ```
 
 - `subscribe` is idempotent. Typically sent once right after connection. Supports simple glob (`tool:*`, `inference:*`).
 - `text` produces an `external-message` event with `source: 'headless'` — same effect as user typing into the child's own TUI.
 - `command` routes through the child's `commands.ts` handler exactly as if typed locally.
 - `shutdown` sets `graceful`; child completes in-flight inference, then exits. `graceful: false` is equivalent to SIGTERM.
+- `panel-request` runs one operator-panel operation in the child via the shared
+  `runPanelOp` dispatcher (`src/web/panel-data.ts`) — the same code the WebUI
+  host runs for its own process, so parent and child views of any panel can
+  never drift. Ops: `mcpl`, `settings`, `settings-update`, `settings-reset`,
+  `settings-cancel-transition`, `pins`, `pin-add`, `pin-remove`, `health`,
+  `context-makeup`, `context-coverage`, `context-curve`, `context-preview`,
+  `context-maintenance`, `debug-context`. The child answers with a single
+  `panel-response`:
+
+  ```json
+  {"type":"panel-response","op":"settings","corrId":"panel-1","ok":true,"data":{...}}
+  {"type":"panel-response","op":"settings","corrId":"panel-1","ok":false,"error":"Agent not found: x","status":404}
+  ```
+
+  `status` is HTTP-ish so the parent's `?scope=<child>` proxy routes
+  (`/debug/context/*`, `/healthz`) can answer faithfully. `panel-response`
+  (like the other request/response snapshots) bypasses the subscription
+  filter. Parent-side, `FleetModule.requestPanel()` wraps the corrId
+  bookkeeping in a promise and never rejects — unreachable children resolve
+  `{ok:false,status:502}`, silence resolves `{ok:false,status:504}` after the
+  timeout (default 30s).
 
 ### Connection lifecycle
 

package/README.md

@@ -123,12 +123,22 @@ subscription credits at a higher rate when applied.
 - **Web UI**: browser operator console (`modules.webui`) — live chat with full interiority (thinking, tool calls, streaming), agent/fleet tree, context makeup + compression coverage, call ledger with cache verdicts and billing-grade costs, health/ops alerts, Chronicle branch tree, lessons, MCPL config, workspace files; scoped read-only observer access via device keys
 - **TUI + readline modes**: OpenTUI interactive terminal or `--no-tui` for pipes/CI
 - **Subagent forking** (opt-in, `modules.subagents`): Spawn/fork parallel agents with fleet tree view (Tab to toggle)
-- **Persistent lessons** (opt-in, `modules.lessons`): Knowledge store with confidence scores and tags. Automatic retrieval-injection of lessons into context (`modules.retrieval`) is a separate opt-in — it adds per-turn context churn and Haiku calls, so enable it only for agents that actually curate a lesson library
+- **Persistent lessons** (opt-in, `modules.lessons`): Knowledge store with confidence scores and tags. Automatic retrieval-injection of lessons into context (`modules.retrieval`) is a separate opt-in — it adds per-turn context churn and retrieval-model calls, so enable it only for agents that actually curate a lesson library
 - **Time-travel**: Chronicle-backed undo/redo, named checkpoints, branch exploration
 - **Session management**: Isolated sessions with auto-naming
 - **MCPL support**: Connect any MCP/MCPL server; wake subscriptions for selective event triggering
 - **File products**: Write reports and documents, materialize to disk
 
+For `openai-responses` and `openai-codex`, an object-valued
+`modules.retrieval` can set `reasoningEffort` (`none`, `minimal`, `low`,
+`medium`, `high`, `xhigh`, or `max`) independently of the primary agent.
+Retrieval calls are independent one-shot requests, so there is no separate
+retrieval reasoning-context setting. When `reasoningEffort` is configured,
+`model` must also be set explicitly: the historical retrieval default is a
+Claude model and cannot be sent through an OpenAI adapter. Anthropic/Claude
+uses different native thinking controls and does not accept this OpenAI-shaped
+option.
+
 ## Prerequisites
 
 - [Node.js](https://nodejs.org/) 20+ and [Bun](https://bun.sh/) runtime
@@ -175,6 +185,7 @@ non-loopback binds require basic-auth credentials. Build the SPA bundle once wit
 - Ops alerts (compression quarantine, refusal streaks, inference-exhausted) render as persistent banner rows
 - Usage panel: per-agent costs and a billing-grade call ledger with cache verdicts
 - `/curve` — compression-curve visualization; `/healthz` — liveness JSON for doctor/fleet tooling
+- `/debug/retrieval/view` — operator-only per-run lesson selection viewer (see `docs/retrieval-traces.md`)
 - Read-only observer access via Ed25519 device keys with per-grant scopes (see `docs/webui-deployment.md`)
 
 For SPA development: `cd web && bun run dev` proxies the Vite dev server onto a

package/docs/AGENT-ONBOARDING.md

@@ -274,7 +274,7 @@ workloads whose cache is normally reused within five minutes.
 
 **`subagents`/`lessons`/`retrieval` are NOT part of the standard recipe.**
 All three are opt-in. RetrievalModule in particular injects context-dependent
-content into every compile (and spends two Haiku calls per turn), so it must
+content into every compile (and spends up to two configured retrieval-model calls), so it must
 be an explicit opt-in for agents that actually curate a lesson library.
 Current host code defaults all three to off, but keep the explicit `false`
 entries in the recipe anyway — older host checkouts treated these as opt-out,

package/docs/debug-context-api.md

@@ -75,8 +75,8 @@ The trade-off is fidelity: the default response **omits the dynamically
 gathered injections** (lessons, retrieval results, MCPL `beforeInference`
 context), because gathering those is *not* free or transparent:
 
-- module `gatherContext` can run inference — e.g. the retrieval module makes
-  Haiku calls, which cost tokens and add latency;
+- module `gatherContext` can run inference — e.g. the retrieval module makes configured model calls, which cost tokens and add
+  latency;
 - MCPL `beforeInference` hooks are arbitrary RPCs to external servers with
   side effects, and a preview never sends the paired `afterInference`, which
   can leave a stateful server half-open.

package/docs/retrieval-traces.md

@@ -0,0 +1,173 @@
+# Retrieval Traces
+
+The retrieval module can record a bounded, per-run explanation of automatic
+lesson selection. The trace shows which concepts the selector returned, which
+lessons matched mechanically, which candidates survived relevance filtering,
+and the exact lesson block injected into the next compile.
+
+Tracing is diagnostic. It does not change selection, trigger an extra model
+call, or claim access to hidden chain-of-thought.
+
+## Prerequisites
+
+Enable lessons, retrieval, and the Web UI in the recipe:
+
+```json
+{
+  "modules": {
+    "lessons": true,
+    "retrieval": {
+      "model": "gpt-5.4-mini",
+      "maxInjected": 5,
+      "reasoningEffort": "high"
+    },
+    "webui": {
+      "host": "127.0.0.1",
+      "port": 7340,
+      "basicAuth": {
+        "username": "${WEBUI_USER}",
+        "password": "${WEBUI_PASS}"
+      }
+    }
+  }
+}
+```
+
+`reasoningEffort` accepts `none`, `minimal`, `low`, `medium`, `high`, `xhigh`,
+or `max`. It is optional, applies to both retrieval calls, and does not inherit
+the primary agent's reasoning setting. This field is supported only when
+`agent.provider` is `openai-responses` or `openai-codex`; Anthropic/Claude uses
+separate native thinking controls and is rejected here rather than receiving an
+invalid OpenAI-shaped request.
+
+Retrieval remains opt-in and requires the lessons module. Depending on the
+candidate count, one turn can use one selector call plus an optional relevance
+call.
+
+## Operator viewer
+
+Open:
+
+```text
+http://127.0.0.1:7340/debug/retrieval/view
+```
+
+The viewer highlights selected lessons, lists all mechanically matched
+candidates and their match provenance, summarizes the relevance decision, and
+keeps the retained trace JSON behind a diagnostic disclosure. Each run is
+labeled with the invoking agent name. Separate Host processes retain separate
+trace stores and viewers; the endpoint does not aggregate fleet children.
+
+## JSON endpoint
+
+```text
+GET /debug/retrieval[?limit=20][&includeInputs=1]
+```
+
+| Parameter | Default | Meaning |
+|---|---:|---|
+| `limit` | `20` | Newest traces to return, clamped to `1..100`. |
+| `includeInputs` | off | Exact recent conversation and model-stage inputs are included only when the value is the literal `1`. |
+
+Example envelope:
+
+```json
+{
+  "schemaVersion": 1,
+  "enabled": true,
+  "includeInputs": false,
+  "traces": []
+}
+```
+
+Each trace can include:
+
+- configured model and request-level reasoning parameters;
+- recent-context hash, message count, and available message IDs;
+- selector prompt, raw returned text, normalized provider blocks, parsed
+  concepts, and parse mode;
+- every mechanical candidate, lesson snapshot, and content/tag match reason;
+- relevance-call output or the reason validation was skipped;
+- final relevant and injected lesson IDs, exact lesson snapshots, injection
+  namespace/position, and rendered `## Retrieved Knowledge` block;
+- cache-hit provenance, outcome, duration, and errors.
+
+Opaque or redacted reasoning blocks are retained only as provider-returned
+content blocks. The trace does not decrypt them or present them as hidden
+chain-of-thought. Each lesson snapshot is a point-in-time copy of every
+`Lesson` field: `id`, `content`, `confidence`, `tags`, `evidence`, `created`,
+`updated`, `deprecated`, and optional `deprecationReason`.
+
+If retrieval is disabled, the endpoint returns `enabled: false` and an empty
+trace list.
+
+## Authentication and privacy
+
+Both routes require password-derived operator authentication, including on a
+loopback bind. Web UI Basic Auth must be configured; the routes accept either
+direct valid Basic Auth or a full session created by password sign-in. A
+loopback-only Web UI without configured credentials returns `401` for these two
+routes, while other Web UI routes retain their historical loopback behavior.
+Read-only observer cookies are rejected even when they carry the `debug` scope.
+
+Exact recent conversation and stage inputs are omitted by default because they
+can contain private text. Use `includeInputs=1` deliberately; values such as
+`true` or `yes` do not enable disclosure. Lesson contents, selector output,
+candidate provenance, and the final injected block remain visible in the
+default trace because they are the subject of the diagnostic.
+
+Every response from either retrieval route uses `Cache-Control: no-store`,
+including authentication failures, not-yet-bound responses, and internal
+errors.
+
+## Retention and failure behavior
+
+The retrieval module retains at most the newest 100 traces and, by default, at
+most 8 MiB of UTF-8 encoded trace JSON in process memory. The limits are
+enforced after every active trace mutation. When the byte budget is reached,
+the oldest traces are evicted until both limits are satisfied. Traces are not
+written to Chronicle and disappear when the Host restarts. Cache provenance is
+rewritten when a referenced source is evicted or truncated, while each
+cache-hit trace keeps its own complete selected-lesson snapshots when they fit
+within the budget.
+
+If one trace alone exceeds the byte budget, it is replaced with a small record
+whose `truncation.kind` is `tombstone` and whose reason and original encoded
+size make clear that the detailed payload is unavailable. Tombstones are never
+presented as exact traces. Payloads that fit remain complete.
+
+Trace recording is fail-open: metadata collection and serialization failures
+must not block retrieval or the primary inference. Arbitrary provider blocks
+and request-level provider parameters are converted to JSON-safe snapshots
+before they are exposed. Their retained representations bound recursion depth,
+total nodes, array items, object keys,
+individual strings, and aggregate string bytes. When a bound applies, the
+stage's `responseContentTruncation` records the explicit reason and active
+limits. Cycles, BigInts, nonfinite numbers, and unreadable properties remain
+safe to serialize using explicit unavailable-value records.
+
+Viewing an existing trace is read-only. Retrieval itself still performs its
+normal model calls and lesson lookup when the agent gathers context.
+
+## Response codes
+
+| Status | Meaning |
+|---:|---|
+| `200` | Viewer or trace response returned. |
+| `401` | Missing operator authentication, including observer-only sessions. |
+| `500` | Retrieval Trace listing failed; the response remains noncacheable. |
+| `503` | Host application has not bound to the Web UI yet. |
+
+## Troubleshooting
+
+- `enabled: false`: enable both `modules.lessons` and `modules.retrieval`, then
+  restart with the updated recipe.
+- Empty `traces`: no retrieval run has completed or started since this Host
+  process began.
+- Many candidates but few selected lessons: the relevance stage is filtering
+  mechanical keyword matches. `maxInjected` is a ceiling, not a quota.
+- `sourceTraceEvicted: true`: a cache hit refers to a run older than the
+  in-memory retention window; the exact selected lesson snapshots remain on the
+  cache-hit trace.
+- `sourceTraceTruncated: true`: the referenced source is retained only as a
+  tombstone; inspect the complete snapshots on the cache-hit trace itself.

package/docs/webui-deployment.md

@@ -160,7 +160,8 @@ Query params:
 - `agent=<name>` — defaults to the recipe's root agent
 - `injections=1` — opt into full fidelity: gather the dynamic injections too.
   **Not transparent** — this can run inference (e.g. the retrieval module's
-  Haiku calls cost tokens) and fires MCPL `beforeInference` hooks (whose paired
+  configured model calls cost tokens) and fires MCPL `beforeInference` hooks
+  (whose paired
   `afterInference` is never sent, so a stateful server may be left half-open).
 - `pretty=1` — pretty-print the JSON
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@animalabs/connectome-host",
-  "version": "0.7.3",
+  "version": "0.7.4",
   "description": "General-purpose agent TUI host with recipe-based configuration",
   "type": "module",
   "scripts": {
@@ -16,7 +16,7 @@
   "dependencies": {
     "@animalabs/agent-framework": "^0.7.2",
     "@animalabs/chronicle": "^0.2.0",
-    "@animalabs/context-manager": "^0.6.0",
+    "@animalabs/context-manager": "^0.6.3",
     "@animalabs/membrane": "^0.5.77",
     "@opentui/core": "^0.1.82"
   },