pi-crew 0.11.55 → 0.11.57

package/CHANGELOG.md

@@ -1,5 +1,48 @@
 # Changelog
 
+## [0.11.56] — hướng A broker-signed respond + FINDING D runner-hosted broker + R6-5 + steer.force + spawn-trace (2026-11-01)
+
+Wave theme: **the broker-host (detached runner) becomes the trust anchor of async runs** — the dispatcher dies, the broker lives; the signer is foreign, the broker signs. Four commits, every layer live-proven.
+
+## [Unreleased] — hướng A: broker-signed respond delivery (ADR Option 3)
+
+**Added (product, HƯỚNG A — broker-appended respond rows).** The 0.11.55 dispatcher-key distribution (hướng C) still left every responder that is NOT the
+dispatch process undeliverable: a restarted leader, a foreign session, a scheduler host sign rows with keypairs no worker holds, so their rows drop as
+noise at the w1a verify. Hướng A closes the class by having the BROKER PROCESS append the row — the embedded broker's keypair IS the worker's spawner
+keypair in both dispatch modes (`lifecycle-handlers.ts` getOrStartBroker), so a broker-appended row verifies in every worker with zero new key
+distribution. Three surfaces:
+
+- `respond.challenge` + `respond.append` (broker, challenge-response): the broker binds the DISPATCHER'S PUBLIC verify key heap-only at construct
+(`CrewBrokerOptions.dispatcherVerifyKey` — async dispatch threads the spawner's key over the detached runner's stdin start-options into the new
+`dispatcher-key-registry.ts`; a root dispatcher host binds its own key; NEVER the manifest file, which is unauthenticated and worker-writable — w1a
+NO-GO). `respond.challenge` issues a 32-byte single-use nonce bound to (connection, runId) — burned on the first append attempt regardless of outcome,
+rate-bounded 16/connection, TTL 30s upper-bounded by the broker's 1s hello deadline; `respond.append` verifies an Ed25519 signature over the frozen
+`respondAppendSignInput` export (`${nonce}${canonical}` — no separator, no runId in the canonical; run binding rides the nonce record; client signers
+must use the export, never re-implement) and appends the row itself: `from:"leader"`, `kind:"response"`, questionId-tagged, `requestId = questionId`
+idempotency, signed with the broker host keypair. Worker-token connections are rejected on both respond paths (role derives from token type —
+defense-in-depth; the real bound is the private key they cannot hold).
+- `respond.force` (broker, one-shot live park): token-less human-responsibility path, called directly over the broker socket (NOT wired into the respond
+tool's `force` param — its semantics are unchanged). The gate is the park itself (fresh reload under the bounded busy-retry run lock must find
+`status:"waiting"` with the exact `waiting.questionId`), exactly-one per question (durable requestId dedup + heap fast-fail; force after a CR append
+returns `already-answered`), row `from:"leader-forced"` + `data.tag:"forced"`, append OUTSIDE the run lock (lock-ordering safety; exactly-once comes
+from the idempotent append, not the lock). This is the path that finally DELIVERS a foreign-session force respond — the gap hướng C could not close.
+- `deliverResponseRow` (`src/extension/team-tool/respond-delivery.ts`, wired into `handleRespond`'s ALIVE path): broker-first delivery — precheck =
+socket derivable from `manifest.sessionId` AND this process IS the dispatcher (`manifest.dispatcherVerifyKey === getMailboxVerifyKeyPublic()`;
+foreign/restarted responders skip straight to the fallback instead of burning an RPC that cannot verify), one short-lived pre-hello NDJSON connection
+(challenge→append on one socket — the nonce is per-connection; 2.5s connect / 5s per-request deadlines; no retries — the fallback IS the retry), ANY
+failure → local signed idempotent append (hướng C behavior + `respond.broker-delivery-fallback` warn). `handleRespond` now stages ALIVE rows inside the
+run lock (atomic with the exactly-one-dispatch scan) and delivers them after it releases — the broker handler takes its own run lock, so the RPC window
+must never sit under `withRunLockSync`. `requestId = questionId` keeps delivery exactly-once across broker + local + concurrent responds (a lost ack
+frame dedupes against the broker's own row). New additive mailbox.ts export `signCoordinationChallenge` (raw Ed25519 over arbitrary challenge bytes
+with the in-process coordination keypair — no sha256 wrap, unlike the row signers).
+
+New event `respond.broker_append` (registered in contracts.ts): audit for every broker-appended response row — data {taskId, questionId, auth:
+"challenge-response"|"forced", requestId, messageId, deduped} (+ tag/brokerSocket on the forced path); diagnostic only, never transitions run/task
+state. Known shape note: broker wire rows carry `replyTo` but not `replyFrom`/`replyDeadline` (informational fields — the local fallback row keeps
+them; documented in respond-delivery.ts). Residuals documented in trust-model v1.5: leader restart mid-run still open (needs a dispatcher re-bind
+flow), broker restart mid-run mints a new heap keypair (documented, not fixed), force forge-channel consciously accepted (one live park, one-shot,
+tagged, audited), CR closes the no-leader gap only while the dispatcher process lives.
+
 ## [0.11.55] — ask-park clobber fix + w1a dispatcher key distribution (2026-10-31)
 
 **Fixed (product, FINDING A — ask-park clobber).** Cross-process coalesce race on tasks.json: the HOST process's 50ms coalesced
@@ -9505,3 +9548,96 @@ the repo. Two genuine wins, plus an informational CI job that never blocks.
 
 ### Hardening (W10 batch A)
 - `check-slop-rules` no-double-cast baseline ratchet (41, 2026-10-22 — burn down only) + real-CLI e2e pin; W10-7 reconcileUsage missing-finalUsage guard; W10-8 READ-ONLY lane mutation one-line stderr visibility; W10-9 structuredOutput JSON round-trip pin; W10-2 verify-gate memo time-dependent-skip intent pin.
+
+### Leader post-review additions (live probe8/9, 2026-11-01)
+
+- **Discovery fix**: the background runner now stamps the ACTUAL broker socket path onto
+  `TeamRunManifest.brokerSocketPath` at stdin-creds accept; `deliverResponseRow` prefers it over
+  `getBrokerSocketPath(manifest.sessionId)` derivation — live probe proved the derivation hashes the
+  dispatcher-side synthetic session (7e4df036) while the run's broker listens on a different path
+  (9529a08a/de784999), so the RPC path was unreachable in async production topology without the stamp.
+- **P3 comment fixes** (reviewer FIX_THEN_SHIP): CR sign-input contract doc (`${nonce}${canonical}`,
+  no separator, no runId) + lost-ack dedup topology condition.
+- **KNOWN BLOCKER (Finding D — next wave)**: the run's broker socket file vanishes from the filesystem
+  within ~seconds of runner start (listener fd stays valid for existing connections — park works — but
+  fresh connects get ENOENT; live evidence: stamped path valid at park time, absent seconds later,
+  runner alive+heartbeating, /proc/net/unix shows no bind for it while older sockets remain). Until
+  Finding D is fixed, `deliverResponseRow`'s RPC path fails fast to the hướng-C local append in async
+  runs — delivery remains correct via 0.11.55 semantics; `respond.force`/`respond.append` activate
+  fully once the socket lifetime is fixed. Suspected vectors: sibling lifecycle stale-sweep /
+  libuv close-unlink-by-name (socket-stop.ts documents the primitive) / sessionId inheritance
+  collision between the runner and its children.
+
+### FINDING D closed + R6-5 waiting-preserve (2026-11-01, probes 10-13)
+
+- **FINDING D root cause**: async runs handed every worker the DISPATCHER's broker
+  socket; a `pi -p` dispatcher exits seconds after dispatch and session_shutdown
+  unlinks that socket while the run lives on — every later broker op (parks,
+  messages, respond RPC) hit ENOENT. Fix: the detached runner now HOSTS the run's
+  broker (`runner-broker.ts`) — fresh `runner-<runId>` socket, per-task compound
+  tokens minted in the runner heap (same one-pipe trust domain as the stdin
+  pre-mints), dispatcher's public verify key still the respond.append CR anchor,
+  construction mirrors the lifecycle host config-for-config, stopped in runCleanup.
+  Soft-fail to the legacy dispatcher-socket issuer when hosting fails.
+- **R6-5 waiting-preserve** (FINDING A residual closed): the runner's
+  persistHeartbeat ~1s whole-record persist carried a dispatch-time snapshot that
+  never learned the broker parked the task — the first post-park heartbeat ERASED
+  the park from tasks.json (probe10 "no-park" on a verifiably parked worker; stack
+  pinned via temporary saveRunTasksCoalesced instrumentation, probe12 run
+  team_20261101040510). mergeTaskUpdateWithDisk now preserves a disk park against
+  park-unaware non-terminal snapshots (mirror of R6-1 terminal-preserve);
+  legitimate clears (respond requeue / broker wait-resolve victim flip) write full
+  arrays via saveRunTasks directly and are unaffected.
+- **Live-proven end-to-end (probe13, run team_20261101041053)**: leader dispatched
+  async and exited → runner-hosted broker (manifest-stamped socket) → worker parked
+  → park SURVIVED heartbeats (R6-5) → FOREIGN session (no token/key) responded via
+  `respond.force` (`auth:"forced"`) → worker resumed with the answer and completed
+  the task — the full hướng A acceptance that probes 8/9 could not reach.
+
+### HƯỚNG A follow-up: respond.force surfaced through the product respond tool (2026-11-01, probes 14-18)
+
+- Probes 14-17 exposed a dead end: a RESTARTED/foreign responder (the heap-only
+  dispatcher keypair died with the original `pi -p` process — which structurally
+  exits after async dispatch) could only reach the LOCAL append, whose row is
+  signed with a key the worker does not hold → dropped as noise → the ask timed
+  out with NO product surface able to answer it. deliverResponseRow now has a
+  three-tier chain: broker CR (dispatcher only) → **respond.force** (token-less
+  one-shot for a LIVE park, broker-SIGNED row — probe13/18 live-proven, incl.
+  `task.respond_delivered` worker ack) → local signed append (last-resort trail).
+  Same-uid trust domain unchanged (any local process could already speak the
+  protocol); this surfaces the existing capability through the product tool.
+- probe18 product-level proof: a foreign session's `team respond` on a live park
+  of a dead-dispatcher run delivered via force end-to-end (respond.broker_append
+  auth:forced → worker ack → task resumed).
+
+### steer.force — token-less live-worker steer for foreign/restarted sessions (2026-11-01, probes 19-20)
+
+- probe19 exposed the steer twin of the respond dead end: a foreign/restarted session's
+  `team steer` writes pendingSteers (next-spawn only) + a steering-file row signed with ITS
+  key — outside the live worker's verify set → pollSteering drops it → in-context delivery
+  NEVER happens (probe19: file row present, worker conversation received NONE).
+- New pre-hello broker method `steer.force` (mirror of respond.force's lane): the run's
+  broker-host (detached runner — its keypair IS in every worker's verify set) performs the
+  delivery itself: mailbox row (kind:"steer", interrupt) + BROKER-SIGNED steering line;
+  live-task gate (terminal rejected), requestId dedup (R2-L-A4 mirror), worker-role
+  connections rejected, `steer.broker_force` audit event.
+- `handleSteer` now auto-routes through steer.force (fire-and-forget) when the caller is not
+  the trusted signer — same-uid trust domain unchanged, this only surfaces VERIFIED delivery
+  through the product tool.
+- probe20 live-proven end-to-end: foreign session `team steer` mid-run → worker's conversation
+  received the message verbatim ("exactly ONE injected steering message observed verbatim").
+
+### spawn-phase-trace — observability for log-silent spawn hangs (2026-11-01, incident team_20261101074334)
+
+- Live incident: 02_execute's first worker spawn hung ~7 minutes between task.started
+  (07:50:05) and the idle-retry respawn (07:57:05) with ZERO trace — no worker.spawned,
+  no checkpoint childPid, no background.log line in the window (the runner's whole log is
+  17 lines). Forensics could not bisect WHERE it hung (candidates: worker-slot semaphore
+  wait, model-registry resolution chain, child spawn env build — all log-silent).
+- `createSpawnPhaseTracer`: a 30s unref'd interval per spawn attempt printing
+  `[spawn.trace] task=… attempt=… phase=… elapsedMs=…` to background.log — silent for
+  healthy sub-30s spawns, a breadcrumb every 30s while stuck. Wired in child-executor
+  (per-attempt, ended on onSpawn + attempt-finally).
+- `runWorker` slot-wait timing: acquire→fn-start >5s logs
+  `[spawn.trace] worker-slot wait=…ms` — separates slot starvation from in-child phases
+  in the next hang's trail.