instar 1.3.508 → 1.3.509

package/upgrades/side-effects/multi-machine-seamlessness-ws12b-drain.md

@@ -0,0 +1,205 @@
+# Side-Effects Review — WS1.2b drain verb: active-topic transfers COMPLETE
+
+**Version / slug:** `multi-machine-seamlessness-ws12b-drain`
+**Date:** `2026-06-12`
+**Author:** `Instar Agent (echo)`
+**Second-pass reviewer:** `independent reviewer subagent (MANDATORY — ownership + session lifecycle; verdict appended below)`
+
+## Summary of the change
+
+Implements the WS1.2 drain semantics from the merged MULTI-MACHINE-SEAMLESSNESS-SPEC
+(§WS1.2): a transfer of an ACTIVELY-USED conversation now COMPLETES — the owner
+finishes the in-flight turn (bounded), suspends any autonomous run (the remote WS1.4
+arm), closes its local session, and lands the target's claim, which is exactly the
+moment the durable queue's ownership-contention barrier releases inbound to the new
+owner. Files: `src/core/SessionOwnership.ts` (FSM `abort-transfer` action),
+`src/core/SessionDrainRunner.ts` (new owner-side bounded sequence, all I/O injected),
+`src/core/MeshRpc.ts` (`drain` verb + router-only RBAC + `drain-unauthorized` 403),
+`src/core/OwnershipReconciler.ts` (drain-claim grace — the reconciler backstops a
+dead-mid-drain owner without front-running a live one), `src/core/seamlessnessConfig.ts`
+(SEAMLESSNESS_PROTOCOL_VERSION 1→2 per spec), `src/core/AutonomousSessions.ts`
+(`markAutonomousInterruptedMidTask`), `src/core/types.ts`
+(`seamlessnessFlags.ws12DrainReceive`), `src/commands/server.ts` (runner construction,
+mesh handler, heartbeat advertisement, `_sendDrain` local/remote sender),
+`src/server/AgentServer.ts` + `src/server/routes.ts` (ctx plumbing + the
+`/pool/transfer` drain leg).
+
+## Decision-point inventory
+
+- `SessionOwnership` FSM — **modify** — adds `abort-transfer` (transferring →
+  active(self), owner-only, epoch+1). The emergency-stop escape: an aborted transfer
+  leaves NOTHING split, and the epoch bump fences a stale target claim.
+- `MeshRpc` RBAC — **modify** — `drain` is router-only with its own refusal reason.
+  Reach ≠ authority: the receiver's CAS re-validates ownership + epoch regardless.
+- `SessionDrainRunner` — **add** — the owner-side drain decision sequence (refuse
+  not-owner / stale-epoch / cas-lost; abort on emergency stop EVERY poll; forced
+  close only at the bound, with the honest marker + ONE notice).
+- `/pool/transfer` drain leg — **modify** — orders the drain BEFORE the pin when the
+  owner can drain; 409 failed-needs-retry ONLY on the emergency-stop abort; every
+  other failure degrades to today's pin path (recorded in the response, never blocking).
+- `OwnershipReconciler` — **modify** — transferring-to-me claim held back by the
+  drain grace window (45s = bound + slack) so it backstops without front-running.
+
+---
+
+## 1. Over-block
+
+**What legitimate inputs does this reject that it shouldn't?**
+
+- A transfer ordered while ownership raced (stale epoch) is refused by the drain and
+  DEGRADES to today's pin path — the transfer still happens, just without the drain.
+  No new rejection surface for the caller except the deliberate emergency-stop 409.
+- The 409 abort path requires the operator's own emergency stop within the last 120s
+  — the only input it "rejects" is the move the operator just halted. A STALE stop
+  flag (>120s) never vetoes (the freshness bound exists precisely for this).
+- The drain bound (30s default) force-closes a session mid-turn rather than blocking
+  the transfer forever — the spec's explicit choice; the honest notice + the
+  `interrupted_mid_task` marker are the compensations. No issue identified beyond
+  the spec-accepted forced close.
+
+## 2. Under-block
+
+**What failure modes does this still miss?**
+
+- The spec's full Track-H release gate ("final context flush durably replicated
+  before barrier release") is implemented as drain-completion release with the
+  target's spawn-time history fetch + working-set pull carrying context — the
+  TransferOrchestrator's ledger-flush dependency does not exist yet. Honest bound,
+  named in the runner header. <!-- tracked: CMT-1416 -->
+- `sessionQuiet` reads tmux activity (isSessionActivelyWorking); a session that
+  *looks* idle mid-thought (long tool call quiescing the pane) can be closed at a
+  non-ideal boundary. Mitigated by the resume-side history fetch; same signal the
+  reaper already trusts.
+- A drain that lands the claim but whose pin-set then fails on the holder leaves
+  ownership@target with no pin — the WS1.3 reconciler converges it (this is the
+  exact divergence class it ships for).
+
+## 3. Level-of-abstraction fit
+
+The runner is a pure injected-deps sequence in core (unit-testable, no transport);
+transport lives in MeshRpc (where every other verb lives); the sender leg lives in
+the route that already owns transfer semantics. The drain BARRIER reuses the queue's
+existing ownership-contention hold — no parallel queueing machinery was built (the
+level-of-abstraction note in the runner header records why TransferOrchestrator was
+NOT used: its ledger deps don't exist; the runner can later become its drain dep).
+
+## 4. Signal vs authority compliance
+
+The drain verb carries real authority (it closes a session) — so authority is
+LAYERED, never brittle-single-check: RBAC proves the sender may ask (router-only);
+the runner's CAS fence proves the ask still matches reality (owner + exact epoch);
+the FSM proves the transition is legal; the emergency stop preempts at every poll;
+and the forced path is bounded + marked + noticed. A failed/refused drain never
+blocks the transfer — it degrades to the pre-existing path (failure of the new
+machinery cannot deny the old capability).
+
+## 5. Interactions
+
+- **WS1.3 reconciler:** the transferring-to-me claim now waits out the drain grace
+  (45s) so a LIVE drain is never front-run; a dead-mid-drain owner is still
+  backstopped (grace expiry → reconciler claim → transfer completes). Tested.
+- **P19 closeout breaker (#1092):** after a drained transfer the local session is
+  already closed — the closeout finds nothing; no double-close (terminateSession is
+  CAS'd on live status).
+- **Durable queue:** route() queues inbound while status='transferring'
+  (ownership-contention) — the drain window IS the barrier; the claim is the release
+  point. No new queue states introduced.
+- **Emergency stop:** the abort CAS can lose to an already-landed claim (grace
+  raced) — reported honestly as `abort-cas-lost`; the stop's own machinery still
+  halts local work.
+- **Double-drain:** a re-delivered drain for the SAME transfer resumes the wait loop
+  idempotently (no second CAS); a drain for a DIFFERENT transfer dies at the epoch
+  fence.
+
+## 6. External surfaces
+
+- New mesh verb `drain` (signed, router-only). An OLD peer answers 501 no-handler →
+  the sender degrades to today's path; an old SENDER never emits the verb. The
+  capability flag (`ws12DrainReceive`, heartbeat-advertised from runner presence)
+  means a doomed order is normally never sent at all. SEAMLESSNESS_PROTOCOL_VERSION
+  bumped 1→2 per the spec's skew table.
+- `/pool/transfer` response grows `drain` + may now 409 with `failedNeedsRetry` on
+  an emergency-stop abort; existing fields unchanged (additive).
+- Timing: the route awaits the drain (≤ bound + slack; remote call capped at 50s) —
+  a transfer of an active conversation now takes up to ~30-50s instead of instant
+  paperwork. That is the feature: the paperwork-instant transfer was the half-move.
+  The route carries its own request-timeout budget (`POOL_TRANSFER_TIMEOUT_MS`
+  = 75s, above the 50s remote cap) so the middleware never 408s mid-drain
+  (second-pass concern #1).
+
+## 7. Multi-machine posture (Cross-Machine Coherence)
+
+**Replicated/coordinated by design — this IS the multi-machine path.** Ownership
+moves through the epoch-fenced CAS registry (single-router topology); placement
+history journals on every CAS (call-site pairing); the capability travels in the
+authenticated heartbeat (fixed-size boolean — Phase C: no per-topic inventory, works
+for N cloud VMs); the drain order is a signed mesh verb; takeover-without-consent
+remains impossible (claim requires the FSM's named-target rule or the reconciler's
+evidence gate). Notices: the forced-close notice goes to the TOPIC (the one voice
+for it); no new URL surfaces.
+
+## 8. Rollback cost
+
+- The sender leg self-disables when no owner advertises the capability — reverting
+  the PR (or running mixed-version) restores today's pin path exactly.
+- No durable format changes: the FSM action is additive; protocol version is
+  advisory metadata; the autonomous-file marker is an additive frontmatter line.
+- No config flag was added for the drain itself BY DESIGN: it activates only along
+  capability advertisement, which exists only where this code runs; the pool's
+  master staging (`multiMachine.sessionPool.stage`) still gates the surrounding
+  machinery. Worst-case back-out = revert + release (no data migration).
+
+---
+
+## Post-push CI fix (no-silent-fallbacks ratchet)
+
+CI shard 3 failed the no-silent-fallbacks ratchet (469 > 468). Investigation:
+every catch this PR ADDS is tagged `@silent-fallback-ok` (drain emergency-stop
+flag read, runner-not-wired guard, `_sendDrain` RPC catch, /pool/transfer
+drain-leg degrade catch, forced-close notice `.catch`) or is non-matching
+(returns a populated object). The flagged-list set-diff shows NO genuine new
+fallback — the +1 is a detection-window artifact: line shifts in the two edited
+files reshape the heuristic's 20-line catch window so a pre-existing,
+previously-uncounted catch now matches (the exact fragility the file's own
+174→186 and 437→447 bump notes document). Baseline bumped 468→469 with that
+justification (the sanctioned escape valve); the count only decreases from here.
+
+## Second-pass review
+
+**MANDATORY (ownership + session lifecycle) — independent reviewer, run on Opus 4.8.**
+(The first attempt died on the Fable 5 outage — the reviewer subagent inherited
+the escalated claude-fable-5 model, which Anthropic disabled mid-run under a
+government directive; re-run cleanly on Opus.)
+
+**Verdict: CONCERN RAISED (one real defect), everything else CONCURS.**
+
+- **Concern #1 (real, was shippable-blocking) — `/pool/transfer` had no
+  request-timeout override for the synchronous drain.** The route awaits the
+  owner-side drain (up to the 30s bound + the 50s remote-call cap) but inherited
+  the 30s middleware default, so a clean drain would 408 mid-handler while the
+  handler kept running to completion (landing the claim + pin) — the exact
+  "408 while the handler keeps running" class the outbound/parity overrides
+  already exist to prevent. The integration tests passed only because they
+  inject a synchronous `sendDrain` mock that returns instantly, never exercising
+  the bound — a genuine coverage gap.
+  **FIXED:** added `POOL_TRANSFER_TIMEOUT_MS = 75_000` and wired `/pool/transfer`
+  into `buildRequestTimeoutOverrides()` (clears the 50s remote cap with margin);
+  added two assertions to `tests/unit/AgentServer-outbound-timeout.test.ts`
+  (the route resolves to the drain budget; the fast sibling `/pool/placement`
+  stays on the default). The fix mirrors the exact pattern the reviewer pointed
+  to and is verified against the production map + matcher.
+
+- **CONCURRED, verified in source (reviewer notes 1–10):** the kill path (topic→
+  tmux→record resolution can't touch another topic's session; force-bypass only
+  at the bound; missing session = non-fatal skip); nothing-split-on-abort (owner-
+  only abort, epoch bump fences a stale target claim); no-theft (runner fences +
+  reconciler 45s drain grace + degrade-path never steals a live owner); barrier
+  correctness (the owner's target-named claim is FSM-legal via the confirmClaim
+  precedent; queue holds while transferring; claim is the release); emergency
+  stop (every-poll check, 120s flag freshness bound, 409 + no-pin only on abort);
+  fail-degrade (every non-abort failure shape → today's pin path); version skew
+  (501 degrade, verb never sent by old senders, honest capability advertisement,
+  registry stores + serves seamlessnessFlags); markAutonomousInterruptedMidTask
+  (atomic, idempotent, missing-file no-op); double-drain idempotency; RBAC
+  router-only with its own 403. 100 tests green across the 6 drain files (+2 in
+  the timeout test = 102).