instar 1.3.500 → 1.3.501

package/upgrades/1.3.501.md

@@ -0,0 +1,51 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: minor -->
+
+## What Changed
+
+Implements `docs/specs/reap-notify-per-topic-and-midwork-resume-queue.md` (converged 7 iterations with GPT-5.5 + Gemini 2.5 Pro cross-review, operator-approved 2026-06-12, topic 24662 — "reaped sessions should ALWAYS notify the user in the corresponding topic, and mid-work reaps need a persistent resume mechanism").
+
+**Part A — per-topic durable reap notices (default ON):**
+- `ReapNotifier` v2: per-topic coalescing — every topic that lost a session gets ONE notice in THAT topic (plain-English reasons, mid-work tag); the lifeline gets unbound sessions + a cross-topic index. Affected-set tracking is separate from the detail buffer, so storms still produce correct per-topic counts. Rollback: `monitoring.reapNotify.perTopic: false`.
+- Durable delivery: notices are `reap-notify:`-prefixed rows in the shared PendingRelayStore (release holds ride the existing `next_attempt_at` column — zero DDL), drained by a NEW always-on `ReapNoticeDrain` (30s tick, CAS claims, per-pass cap 15, backoff to 8 attempts, terminal escalation into ONE aggregated attention item). Independent of the default-OFF DeliveryFailureSentinel. Rollback: `monitoring.reapNotify.drainEnabled: false`.
+- **Store-level fix for everyone (R1.6):** the pending-relay restore-purge cutoff is now `max(attempted_at, next_attempt_at)` with a 7-day corruption clamp — a message HELD for future release no longer gets eaten at boot (the 2026-06-05 silent-deletion class, fixed for ALL rows, not just reap notices).
+- Outcome records: every notice attempt lands in the reap-log as `type:'notify'` pairs (`enqueued` → `sent`/`send-failed-escalated`/`no-topic`/`enqueue-failed`) — "did the user get told?" is now auditable.
+
+**Part B — mid-work tagging + resume queue (ships enabled + dry-run observe-only):**
+- Killers supply work evidence at THEIR decision point (the quota-shed migrator snapshots it BEFORE its Ctrl+C grace round tears the work down); the single kill chokepoint clamps it to an exact enum and stamps `midWork` on the event, the reap-log, and the session record.
+- `ResumeQueue`: durable per-machine queue (fsync persist discipline, single-writer lockfile with same-host stale reclaim + foreign-host loud-disable, corrupt-file sidecar, boot reconciliation from the reap-log). Eligibility is stricter than midWork: ≥1 strong signal (or topic-bound + 2 distinct weak); jobs opt in via `resumeOnReap: true` (default false); operator kills excluded by default.
+- `ResumeQueueDrainer`: revives AT MOST ONE entry per minute, only after 3 consecutive calm pressure ticks + quota headroom + session-cap + no migration in flight. Seven drain-time reality validations re-check the world before any spawn. Failure ladder: 3 attempts with backoff → gave-up; resurrection cap (2 per identity per 24h) kills kill-resume-kill loops; circuit breaker on consecutive failures. ALL give-up classes fold into ONE rolling attention item.
+- Emergency stops reach the queue: stop-all/MessageSentinel stops pause it (TTLs frozen, mutation routes 409); per-topic stops cancel that topic's entries; explicit unpause via `POST /sessions/resume-queue/resume`.
+- New spawn-path capability: sessions can now be spawned with an explicit per-spawn `cwd`, so interrupted worktree work resumes in ITS tree.
+- Tier 1 supervision (observe-only during the soak): each about-to-resume decision gets an audited fast-tier LLM sanity check that never defers.
+
+**Foundation fixes pulled in-scope (no deferrals):**
+- `SessionMigrator` no longer records a ReapGuard REFUSAL as "halted" — refused-but-alive sessions are no longer double-respawned after a quota migration.
+- The reap-log normalizer no longer strips `launchLane` on read.
+
+**Migration parity:** ConfigDefaults gains `reapNotify.perTopic` + `maxImmediatePerFlush` (resumeQueue keys deliberately code-defaulted for the later fleet flip); new content-sniffed `migrateClaudeMd()` section + framework-shadow marker; `state/resume-queue.json` registered machine-local (excluded from backup/restore).
+
+## What to Tell Your User
+
+- "When the system shuts down your sessions (low memory, usage limits), each affected conversation now gets its own clear notice — and the notice can't be silently lost anymore, even across restarts."
+- "Coming online in watch-mode: sessions killed mid-work get queued and automatically restarted one at a time once your machine recovers, picking the work back up."
+
+## Summary of New Capabilities
+
+| Capability | How to Use |
+|-----------|-----------|
+| Per-topic shutdown notices | Automatic — each affected topic is told, in plain English |
+| Durable notice delivery | Automatic — `reap-notify` rows in the pending-relay store, always-on drain |
+| Notice audit trail | `GET /sessions/reap-log` — `type:'notify'` outcome pairs |
+| Mid-work tagging | reap-log + session records carry `midWork` + `workEvidence` |
+| Resume queue state | `GET /sessions/resume-queue` (entries, paused, breaker, lastTickAt) |
+| Queue levers | `POST /sessions/resume-queue/:id/cancel` · `/:id/requeue` · `/resume` · `/drain` |
+| Job opt-in | `resumeOnReap: true` on a job definition |
+
+## Evidence
+
+- Tier-1: `tests/unit/pending-relay-store.test.ts` (R1.6 + origin scoping + CAS), `reap-log.test.ts`, `work-evidence.test.ts`, `session-migrator.test.ts` (pre-grace evidence + refusal recording), `reap-notifier.test.ts` (v2 matrix + legacy modes), `reap-notice-drain.test.ts`, `resume-queue.test.ts`, `resume-queue-drainer.test.ts`.
+- Tier-2/3: resume-queue route integration + reap→durable-notify lifecycle + feature-alive E2E (same PR).
+- Side-effects artifact: `upgrades/side-effects/reap-notify-resume-queue.md`.

package/upgrades/side-effects/reap-notify-resume-queue.md

@@ -0,0 +1,89 @@
+# Side-Effects Review — Per-Topic Reap Notification + Mid-Work Resume Queue
+
+**Version / slug:** `reap-notify-resume-queue`
+**Date:** `2026-06-12`
+**Author:** `echo (instar-dev agent)`
+**Second-pass reviewer:** `[pending — required: session lifecycle + sentinel surface]`
+
+> STATUS: IN-FLIGHT — this artifact accompanies the per-step commits of the
+> build and is completed (all seven questions answered, second-pass appended)
+> before the PR opens. Driven by the converged + approved spec
+> `docs/specs/reap-notify-per-topic-and-midwork-resume-queue.md` (r7).
+
+## Summary of the change
+
+Implements the reap-notify + resume-queue spec: Part A makes every non-silent
+session reap produce a durable per-topic notice (PendingRelayStore rows with a
+`reap-notify:` PK prefix, drained by a new always-on ReapNoticeDrain; the
+store's restore-purge gains the R1.6 held-row exemption + 7-day corruption
+clamp), and Part B tags mid-work reaps with killer-supplied evidence at the
+terminateSession chokepoint and revives them in order via a durable
+ResumeQueue + gated ResumeQueueDrainer (observe-only Tier 1 LLM check during
+soak). Files: src/messaging/{pending-relay-store,reap-notice-delivery-id}.ts,
+src/monitoring/{delivery-failure-sentinel,ReapNotifier,ReapNoticeDrain,ReapLog,
+ResumeQueue,ResumeQueueDrainer,PressureGauge,SessionMigrator,SessionReaper}.ts,
+src/core/SessionManager.ts, server wiring + routes, ConfigDefaults,
+PostUpdateMigrator, templates, three test tiers.
+
+## Decision-point inventory
+
+- `PendingRelayStore.purgeStaleClaimable` — modify — restore-purge staleness predicate (R1.6); brittle by design (transport-layer mechanics, not judgment).
+- `PendingRelayStore claim queries (selectClaimable / selectClaimableReapNotices / claimCas)` — modify/add — origin-scoped single-owner contract between two drains; transport-layer mechanics.
+- `ReapNotifier flush` — modify — per-topic grouping + release-tier selection (IMMEDIATE vs SUMMARY vs quiet-hours); deterministic template authoring, no judgment blocking.
+- `ReapNoticeDrain` — add — tier0 deterministic delivery state machine (claim → send → backoff → terminal escalation).
+- `terminateSession evidence clamp` — modify — enum whitelist on killer-supplied evidence (hard-invariant validation, brittle-blocker exemption).
+- `ResumeQueue eligibility classifier` — add — deterministic eligibility rules (strong/weak evidence, job opt-in).
+- `ResumeQueueDrainer gates` — add — deterministic spawn-eligibility checks delegating to EXISTING authorities (PressureGauge, QuotaManager.canSpawnSession, session cap, migration-in-flight); plus observe-only Tier 1 LLM check.
+- `Dequeue hard invariants` — add — UUID/enum/charset/length clamps protecting `claude --resume` argv (brittle-blocker exemption).
+- `Emergency-stop → queue pause` — add — pass-through consumer of the existing MessageSentinel/stop-all authority.
+
+---
+
+## 1. Over-block
+
+[IN-FLIGHT — completed at Phase 4 after build.]
+
+## 2. Under-block
+
+[IN-FLIGHT — completed at Phase 4 after build.]
+
+## 3. Level-of-abstraction fit
+
+[IN-FLIGHT — completed at Phase 4 after build.]
+
+## 4. Signal vs authority compliance
+
+[IN-FLIGHT — completed at Phase 4 after build. Phase 1 written check recorded
+in build plan: no new brittle judgment blockers; drainer gates delegate to
+existing authorities; Tier 1 check observe-only; hard invariants under the
+documented exemption.]
+
+## 5. Interactions
+
+[IN-FLIGHT — completed at Phase 4 after build.]
+
+## 6. External surfaces
+
+[IN-FLIGHT — completed at Phase 4 after build.]
+
+## 7. Rollback cost
+
+[IN-FLIGHT — completed at Phase 4 after build. Levers: reapNotify.perTopic=false,
+reapNotify.drainEnabled=false, resumeQueue.enabled=false / dryRun=true; no DDL.]
+
+## Build progress notes (per-step, folded into the final review)
+
+- Step 1 (relay-store foundation): R1.6 purge fix + origin scoping + CAS claim; DFS spec §3h updated.
+- Step 2 (ReapLog): notify record pairs + midWork/workEvidence through the normalizer; fixed pre-existing launchLane drop-on-read (Rule-1 deviation, noted for SUMMARY).
+- Step 3 (evidence chokepoint): WorkEvidence vocabulary module + terminateSession opts.workEvidence with enum clamp + ReapGuard.workEvidence() observe-only fallback (closure-error → nothing; critical-tier marker) + midWork stamped on event/reap-log/session record.
+- Step 4 (killer stamps): SessionMigrator pre-grace evidence snapshot + halt-refusal recording (refusals ≠ halted, no double-respawn); SessionReaper asserts authoritative-empty evidence on proven-idle reaps; chokepoint fallback excludes active-process under bypassActiveProcessKeep.
+- Step 5 (ReapNotifier v2): per-topic grouping with separate affected-set (cap 500 + overflow), plain-English reason map, IMMEDIATE/SUMMARY release tiers with quiet-hours holds + per-flush cap, durable enqueue with outcome records, loud enqueue-failure fallback, legacy modes preserved (perTopic:false byte-compatible; drainEnabled:false direct-send).
+- Step 6 (ReapNoticeDrain): always-on 30s drain over the reap-notify lane (CAS claim, lease + prior-boot reclaim, per-pass cap 15, backoff to maxAttempts 8, terminal escalation into ONE stable-id attention item, bounded terminal-row cleanup [Rule-2 deviation: added because DFS's retention pass is default-OFF, so the always-on lane needed its own growth bound]); boot wiring with lazy telegram refs; NotificationBatcher gains quietHoursEndAt/nextSummaryReleaseAt (one quiet-hours clock).
+- Step 7 (PressureGauge): the spec's "shared PressureGauge extraction" already exists as HostPressureSampler (one shared definition delegating to SessionReaper.computePressure, parity-tested in host-pressure-sampler.test.ts) — plan deviation Rule 4: reused instead of duplicating. Wired sessionManager.setPressureTierProvider to it with the reaper's configured thresholds.
+- Step 8 (ResumeQueue): durable JSON queue (fsync discipline temp→rename→dir), single-writer lockfile (same-host stale reclaim; foreign-host NEVER probed — loud disable with documented recovery), R2.2 eligibility classifier, stable-key dedupe, resurrection ledger with 24h window + requeue-as-the-one-override, incident-age TTL with pause-freeze + pressure-starved marker, overflow drop-to-aggregate, corrupt-file sidecar, boot reconciliation (starting→failed-attempt + reap-log re-enqueue seam). Dry-run interpretation recorded: entries ARE durably enqueued (soak observable); only spawning + user-facing claims are gated.
+
+- Step 9 (ResumeQueueDrainer + spawn cwd): gated one-per-tick drainer (calm-ticks on the shared gauge, quota/cap/migration gates never bypassable, manual drain skips calm-ticks ONLY), 7 reality validations + hard invariants (fail-safe on dep errors), failure ladder + breaker + aggregated give-ups, Tier1 observe-only with 5s deadline + off-lever, dry-run inertness (would-resume once per entry, no attention), R2.11 honest notice + literal-data continuation prompt; spawnInteractiveSession + spawnSessionForTopic gain the explicit per-spawn cwd (L13 named extension).
+- Step 10 (boot wiring + emergency stop + routes): queue+drainer constructed at boot (lock-disable logged, lazy telegram refs, 30s-deferred reap-log boot reconciliation — topic-bound candidates only; job entries rely on cron recurrence since opt-in is not reconstructible from the reap-log); enqueue hook at the sessionReaped listener; emergency-stop wiring (stop-all → pause; per-topic stop + MessageSentinel stop → cancelByTopic + operator-stop record); R2.10 routes with clamps; QuotaManager.isMigrationInFlight added (canSpawnSession permits mid-migration spawns absent pressure — the drainer needs the stricter probe); Session.cwd recorded at spawn; JobDefinition.resumeOnReap; Tier1 check wired via shared LlmQueue with attribution.
+- Step 11 (config + migration parity): ConfigDefaults gains reapNotify.perTopic + maxImmediatePerFlush ONLY (drainEnabled + resumeQueue.* deliberately code-defaulted for the fleet flip — registration-discipline test pins this); resumeQueue classified in DARK_GATE_EXCLUSIONS (cost-bearing); resume-queue state files registered machine-local in the state-coherence registry; generateClaudeMd Reap-Log section updated + NEW content-sniffed migrateClaudeMd section + framework-shadow marker; release fragment with maturity-tagged announcements (Part A stable, Part B preview/watch-mode).
+- Step 12 (test tier): integration lifecycle (reap → durable notify pair, quota-shed end-to-end, P17/P19 burst invariants incl. K-entries-vs-rejecting-spawn-target) + R2.10 routes/emergency-stop integration + Tier-3 feature-alive E2E on the production init path (real AgentServer + ReapLog + PendingRelayStore + ReapNotifier + ReapNoticeDrain + ResumeQueue + ResumeQueueDrainer; cwd round-trip through the L13 spawn-path parameter; wiring integrity for pressure gauge / spawn gate / relay store / LlmQueue tier-1).
+- Step 13 (full-suite triage): ratchet/parity updates for the new code (dark-gate golden map +10 shift re-verified by hand; ResumeQueueDrainer registered in componentCategories; migrator CLAUDE.md section tracked in feature-delivery-completeness; session-reaper exact-match assertions gain workEvidence). Behavior-relevant: THREE lenient-side catches REMOVED so a throwing dep resolves to the drainer's strict side per R2.6 — QuotaManager.isMigrationInFlight no longer swallows probe errors (sole caller is the drainer gate, which blocks on throw), and the server's migrationInFlight/topicOwnerElsewhere dep wrappers no longer catch (drainer gate/validateReality resolve to blocked/invalidated). Every remaining new catch carries an in-brace @silent-fallback-ok justification; no-silent-fallbacks baseline untouched (net −1 vs main). localhost-link-guard-route test made hermetic (own tmp stateDir — '/tmp' shared outbound-dedup.db across runs and suppressed sends).