instar 1.3.552 → 1.3.554

package/upgrades/1.3.554.md

@@ -0,0 +1,79 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: patch -->
+
+## What Changed
+
+- **Scheduled-job claims can upgrade from the best-effort AgentBus broadcast to a
+  durable, epoch-fenced lease over the replicated journal.** The decision is made by
+  a single pure gate (`JobLeaseCutoverGate.decideClaimPath`): the journal-lease path
+  engages ONLY when the flag is on AND the pool is flag-coherent — every ONLINE peer
+  advertises the `ws43JournalLease` capability in its heartbeat. An older peer (no
+  flags field) or a peer with the flag off keeps the WHOLE pool on the legacy bus.
+- **The cutover guarantees the two claim mechanisms are NEVER both live for the same
+  job set** — the named migration hazard the spec's "Cutover discipline" rule closes
+  (one machine leasing via the journal while a peer broadcasts via the bus). Each
+  scheduler evaluation returns exactly one path; the release seam idempotently closes
+  whichever store took the claim.
+- **Epoch fencing** (`JobLeaseClaimStore`): a claim carries the coordinator's lease
+  epoch; a stale-epoch peer record is rejected on apply and a same/older-epoch peer
+  lease fences our claim out — so a demoted machine's late claim can't steal a job
+  from the current lease-holder, and a partition double-run is structurally prevented
+  when the journal is reachable. Claim records store METADATA ONLY (machine, slug,
+  epoch, timestamps) — never job payloads.
+- Ships **dark** behind `multiMachine.seamlessness.ws43JournalLease` (default false)
+  with `ws43JournalLeaseDryRun` default true — the first rollout rung logs intended
+  journal claims while the legacy bus path still runs, so a dry-run pool never
+  half-migrates. Flag-off / mixed-pool / single-machine = byte-for-byte today's
+  behavior (the bus path). The advert is only `true` when the flag is on AND not
+  dry-run. Plain seamlessness booleans (read live at the claim boundary), so absence
+  yields the safe default without a migration — no migrateConfig entry needed.
+
+This completes the WS4.3 workstream alongside the merged jobs read-side (#1104) and
+role-guard-at-spawn (#1147). <!-- tracked: CMT-1416 -->
+
+## What to Tell Your User
+
+This is internal multi-machine plumbing that ships turned off, so you will not
+see a change. When it is eventually enabled on a fleet where all your machines
+support it, your scheduled jobs get a more reliable way to avoid two machines
+running the same job at once — a durable, fenced claim instead of a best-effort
+broadcast. Until then, and on any single-machine setup or a mixed fleet, job
+scheduling behaves exactly as it does today. There is nothing to configure and no
+action needed.
+
+## Summary of New Capabilities
+
+- WS4.3 journal-lease cutover: scheduled-job claims can upgrade from the AgentBus
+  broadcast to a durable, epoch-fenced lease over the replicated journal, engaged
+  only when every online peer advertises the capability (flag coherence). A single
+  gate guarantees the two claim mechanisms are never both live for one job set.
+  Ships dark behind multiMachine.seamlessness.ws43JournalLease (dry-run default);
+  single-machine and mixed-pool installs are a strict no-op.
+
+## Evidence
+
+- `tests/unit/job-lease-cutover-gate.test.ts` (9): the full decision matrix —
+  flag-off → bus; single-machine / all-peers-offline → no-op bus; coherent →
+  journal; ONE non-advertising peer → whole pool on bus; older peer (absent flags)
+  counts as non-advertising; an offline incoherent peer does NOT block coherence;
+  dry-run → bus with journalDryRun flagged; and the explicit "NEVER returns both
+  mechanisms" invariant.
+- `tests/unit/job-lease-claim-store.test.ts` (8): take/idempotent-reclaim; a peer's
+  live lease blocks our same/older-epoch claim; our strictly-newer epoch supersedes;
+  applyRemote rejects a stale-epoch record; no self-spoof; completion releases;
+  expired peer lease no longer blocks; durable round-trip across restart.
+- `tests/integration/scheduler-journal-lease-cutover.test.ts` (6): the scheduler
+  routes through the gate — coherent→journal (bus NOT called), mixed→bus (journal
+  NOT taken), dry-run→bus, single-machine no-op, remote-journal-lease→skip, and a
+  live flag flip at the boundary changing the path immediately.
+- `tests/e2e/scheduler-journal-lease-cutover-alive.test.ts` (4): the EXACT server.ts
+  provider closure (real config + real MultiMachineCoordinator epoch + real
+  MachinePoolRegistry fed by a real heartbeat) drives a real triggerJob — coherent
+  takes the journal lease and never the bus; mixed takes the bus and never the
+  journal; flag-off no-op; single-machine no-op. Proves the wiring is live, not a stub.
+- `tsc --noEmit` clean; no-silent-fallbacks, the dev-agent dark-gate line-map (+18
+  recompute via the attributor), and feature-delivery-completeness all green;
+  docs-coverage class floor improved (new symbols documented in under-the-hood.md +
+  multi-machine.md).

package/upgrades/side-effects/multi-machine-seamlessness-ws44-pool-cache.md

@@ -0,0 +1,115 @@
+# Side-Effects Review — WS4.4(f): global pool-cache unification
+
+**Version / slug:** `multi-machine-seamlessness-ws44-pool-cache`
+**Date:** `2026-06-13`
+**Author:** `Instar Agent (echo)`
+**Spec:** `docs/specs/MULTI-MACHINE-SEAMLESSNESS-SPEC.md` §WS4.4 clause (f) (converged + approved; deferred second half of WS4.4, tracked CMT-1416)
+**Second-pass reviewer:** not-required — pure read-side efficiency/observability; no new authority, no credential-bearing code, no destructive op, no third-party spend.
+
+## Summary of the change
+
+WS4.4 clause (f), the deferred second half of WS4.4: every pool-scope dashboard
+surface (`/sessions?scope=pool`, `/jobs?scope=pool`, `/attention?scope=pool`,
+`/guards?scope=pool`, …) used to fan out to every peer machine INDEPENDENTLY — once
+per surface, per client, per poll interval. With a dashboard polling several tabs at
+once, the same peer is hit N times per interval. This unifies all of them onto ONE
+shared per-peer poll cache: one fan-out per interval feeds every pool-scope surface,
+and under CPU load-shed the cache serves last-cached peer data tagged `stale: true`
+instead of re-fanning (load-shed, honestly labeled).
+
+New module (pure + dependency-injected, unit-testable):
+- `src/server/PoolPollCache.ts` — `fetchPeer(peerMachineId, routePath, fetcher)`:
+  within-TTL cache hit (one fan-out feeds two surfaces), single-flight coalescing of
+  concurrent callers, CPU load-shed stale-serve (never from an empty cache — a first
+  read always fetches), a failed fetch is NEVER cached (transient errors don't stick),
+  per-(peer, route) keying, and a read-only `snapshot()` of live load + counters.
+
+Wiring (all behind the dark flag):
+- `routes.ts` — the `/jobs?scope=pool` per-peer fetch routes through `ctx.poolPollCache`
+  when wired; `null` (flag dark / single-machine) ⇒ today's direct per-peer fetch
+  byte-for-byte. New read-only `GET /pool/poll-cache` observability route (503 when
+  dark — the ships-dark contract). A stale-served peer body surfaces an honest
+  `pool.stale: true` tag on the merged response.
+- `commands/server.ts` + `AgentServer.ts` — dev-gated construction (resolveDevAgentGate)
+  in the mesh-setup block; `seamlessnessFlags.ws44PoolCache` capability advert.
+- `types.ts` / `ConfigDefaults.ts` / `devGatedFeatures.ts` — flag (`ws44PoolCache`,
+  DELIBERATELY OMITTED so the dev-gate decides) + `ws44PoolCacheTtlMs` tunable +
+  `MachineCapacity.ws44PoolCache` advert field.
+- `PostUpdateMigrator.ts` + `templates.ts` — Migration Parity (config strip-default +
+  CLAUDE.md section + shadow-capability markers) + Agent Awareness.
+
+## Decision-point inventory
+
+- `/jobs?scope=pool` per-peer fetch through the shared cache — **add** — wired only
+  when the dark flag is on; else the existing direct fetch is unchanged.
+- `GET /pool/poll-cache` observability route — **add** — 503 while dark (ships-dark
+  contract, like `/pool/queue`); 200 with the snapshot when wired.
+- CPU load-shed stale-serve — **add** — over the load-per-core threshold, serve
+  last-cached tagged `stale: true` rather than re-fanning; an empty cache still fetches.
+
+---
+
+## 1. Over-block
+Flag-off (default fleet): zero new behavior — every pool-scope surface keeps its direct
+per-peer fetch byte-for-byte (verified by the "NOT WIRED (cache null)" integration
+test). Flag-on: the 3s default TTL matches the per-route pool caches already in use, so
+no legitimate fresh read is starved. Load-shed only triggers at/above the load-per-core
+threshold (default 1.5, the SessionReaper critical default) AND only when something is
+already cached — a first read is never load-shed.
+
+## 2. Under-block
+Not an authority — this gates nothing. It only decides whether to reuse a recently
+fetched peer body or re-fetch. The worst case is serving a body up to one TTL (or, under
+load-shed, the last-cached) older than a fresh fan-out would produce — and load-shed is
+HONESTLY tagged (`stale: true` on the body, `pool.stale: true` on the merge) so a stale
+serve is never silent. A failed fetch is never cached, so a transient peer error cannot
+stick for the whole TTL.
+
+## 3. Level-of-abstraction fit
+Right layer. `PoolPollCache` is a pure primitive (injected clock + load reader); the
+route change is a thin branch that calls the fetcher directly when the cache is null.
+It rides the existing pool fan-out pattern (resolvePeerUrls + per-peer timeout + failed
+markers) rather than inventing new transport.
+
+## 4. Signal vs authority compliance
+Everything here is a SIGNAL, never an authority: the cache hit / load-shed staleness is
+observability + an efficiency choice; it never gates serving anything, never mutates
+state, and never caches private end-user content (the pool-scope surfaces are
+operator-Bearer reads of mesh METADATA — `/view/:id` private bodies are handled by
+WS4.4 pool-links, which explicitly never caches them).
+
+## 5. Interactions
+- Flag-off path leaves every pool-scope surface identical (no shadowing of the direct
+  fetch).
+- Coexists with the per-route payload caches (`jobsPoolCache`, etc.) — those dampen two
+  rapid calls to the SAME surface; this shared cache dampens the cross-surface,
+  cross-client per-peer fan-out. The integration test proves the shared cache is
+  genuinely in the path by asserting on ITS OWN counters (`stats.fetches`,
+  `cachedKeys`), not the maskable per-route "1 peer hit".
+- Single-machine installs have no peers ⇒ a strict no-op (the cache is never
+  constructed, surfaces never call it).
+
+## 6. Rollback
+Set/omit `multiMachine.seamlessness.ws44PoolCache` (dev-gated; dark on the fleet by
+default). When off, the cache is never constructed and `/pool/poll-cache` 503s — fully
+reverting to the direct per-peer fetch. No data migration, no persisted state to undo.
+
+## Evidence
+
+- `tests/unit/pool-poll-cache.test.ts` (9): both sides of every decision boundary —
+  within-TTL hit vs expired re-fetch; single-flight coalescing; load-shed over/under
+  threshold; load-shed with an EMPTY cache still fetches; a failed fetch is never
+  cached; per-(peer, route) keying; snapshot wiring + boundary flip at the threshold.
+- `tests/integration/ws44-pool-cache-route.test.ts` (4): real wiring (real second-server
+  peer that COUNTS its `/jobs` hits) — the fan-out routes THROUGH the shared cache
+  (asserted via the cache's OWN `stats.fetches` + `cachedKeys`, not the maskable
+  per-route "1 hit"); a second poll past the per-route window is served from the shared
+  cache WITHOUT re-hitting the peer; the merged body is byte-identical wired vs unwired;
+  `GET /pool/poll-cache` 503-when-dark / 200-when-wired.
+- `tests/e2e/pool-poll-cache-alive.test.ts` (3): the feature is ALIVE (200, not 503) on
+  the real AgentServer stack; sits behind auth (no Bearer → 401/403); the ships-dark
+  default 503s with `{ enabled: false }`.
+- Gate checks green: `tsc --noEmit`; `docs-coverage --check`; `no-silent-fallbacks`
+  (the one new fail-closed construction-catch is tagged `@silent-fallback-ok`);
+  `feature-delivery-completeness` (new featureSection + both shadow-marker variants);
+  `lint-dev-agent-dark-gate` (line-map recomputed for the +17 ConfigDefaults shift).

package/upgrades/side-effects/ws43-journal-lease-cutover.md

@@ -0,0 +1,132 @@
+# Side-Effects Review — WS4.3 journal-lease cutover
+
+**Spec:** `docs/specs/MULTI-MACHINE-SEAMLESSNESS-SPEC.md` §WS4.3 ("Cutover discipline")
+**Slug:** ws43-journal-lease-cutover
+**Tier:** 2 (multi-machine correctness invariant; safety-invariant risk floor)
+**Flag:** `multiMachine.seamlessness.ws43JournalLease` (default false), `ws43JournalLeaseDryRun` (default true)
+
+## What changed
+
+Scheduled-job claim deduplication can now upgrade from the legacy best-effort
+AgentBus broadcast (`JobClaimManager`) to a durable, epoch-fenced lease over the
+replicated journal (`JobLeaseClaimStore`). A single pure decision point
+(`JobLeaseCutoverGate.decideClaimPath`) selects which mechanism a job uses, and
+the scheduler routes its three claim seams (remote-claim check, claim-before-spawn,
+release-on-complete) through that decision. New files: `src/scheduler/
+JobLeaseCutoverGate.ts`, `src/scheduler/JobLeaseClaimStore.ts`. Wired into
+`JobScheduler` (new `setJournalLeaseCutover` injector + `resolveClaimPath`/
+`releaseClaim` helpers) and `server.ts` (the production provider closure reading
+the flag + coordinator lease epoch + pool peers' advertised capability). Config
+defaults + types + the `seamlessnessFlags` heartbeat advert added.
+
+## Phase 1 — Principle check (signal vs authority)
+
+This IS a decision point (it gates which claim mechanism runs). It is correctly
+an **authority** because dedup is a correctness mechanism, not a signal — but the
+authority is NARROW and STRUCTURAL: the gate keys ONLY on objective facts (flag
+state, online-peer presence, the boolean capability each peer advertises in its
+authenticated heartbeat) — never on interpreting message content or any brittle
+heuristic. It cannot block a user action; the worst case is "stay on the legacy
+bus path," which is byte-for-byte today's behavior. The gate fails toward the
+conservative side (bus) on any ambiguity (older peer, offline peer, throwing gate
+read). This complies with `docs/signal-vs-authority.md`: a structural ownership/
+coherence check, not a brittle content gate with blocking authority.
+
+## Phase 4 — Review questions
+
+1. **Over-block** — The gate never blocks a job; it only chooses a dedup
+   mechanism. The journal lease's epoch fence could in principle refuse OUR claim
+   if a peer holds a strictly-newer-epoch lease — but that is the correct
+   behavior (the newer-epoch holder is the current lease-holder; we must yield).
+   A same/older-epoch peer claim is fenced OUT, so we are never wrongly blocked by
+   a stale peer. No legitimate run is wrongly rejected.
+
+2. **Under-block** — The journal-lease path relies on the replicated journal
+   actually carrying claim records between machines (`applyRemote`). This PR ships
+   the gate + the local epoch-fenced store + the cutover wiring; the journal
+   EMISSION/APPLY transport for the claim kind is the existing replication
+   substrate's responsibility and is gated behind the same flag-coherence advert
+   so an older peer that can't apply the kind keeps the WHOLE pool on the bus (it
+   never advertises `ws43JournalLease`). Until the flag is flipped on a fully
+   coherent pool, dedup behavior is IDENTICAL to today (bus). The dry-run default
+   means even a flag-on coherent pool stays on the bus while logging intended
+   claims — so under-block risk during rollout is zero (the bus is the baseline).
+
+3. **Level-of-abstraction fit** — Correct layer. The cutover decision lives in
+   the scheduler (which owns the claim seams), the gate is a pure function (no IO,
+   independently testable), and the durable lease is its own store mirroring the
+   existing `JobClaimManager` shape. The flag-coherence read reuses the existing
+   `seamlessnessFlags` heartbeat advert + registry the WS1.1/WS4.4 gates already
+   use — no parallel coherence machinery.
+
+4. **Signal vs authority compliance** — Compliant (see Phase 1). Structural
+   authority keyed on objective facts; fails conservative; no content heuristics;
+   cannot block a user action.
+
+5. **Interactions** — The gate guarantees the journal lease and the bus broadcast
+   are NEVER both consulted for the same job in the same evaluation (exactly one
+   `path` per decision) — the named migration hazard. The release seam calls
+   BOTH stores' `completeClaim` idempotently (each is a no-op when this machine
+   doesn't own the slug's record), so a mid-run flag flip between claim and
+   complete cannot strand a lease. It composes cleanly with the WS4.3 role-guard
+   (a separate spawn-boundary refusal that runs BEFORE the claim seam) — the two
+   are orthogonal. No double-fire: `resolveClaimPath` is evaluated once at the
+   check seam and the result reused at the claim seam.
+
+6. **External surfaces** — Adds one boolean field (`ws43JournalLease`) to the
+   `seamlessnessFlags` heartbeat advert. Additive — older peers omit it (read as
+   non-participant, the conservative side). No new HTTP route. No user-visible
+   message. The advert is only `true` when the flag is on AND not dry-run, so a
+   dry-run machine never advertises participation it isn't actually performing.
+
+7. **Multi-machine posture** — This feature IS multi-machine machinery.
+   Posture: **machine-local decision fed by a proxied-on-read coherence view.**
+   Each machine independently runs the gate over the peers' advertised capability
+   (read live from the registry's heartbeat data), and the journal lease store is
+   per-machine but its records replicate via the journal. Invariant-5 flag
+   coherence is enforced: the cutover engages pool-wide ONLY when every online
+   peer advertises the capability — never a window where one machine leases while
+   a peer broadcasts. Single-machine (no peers) is a strict no-op (the gate
+   returns `bus`/`single-machine` and the journal store is never touched). The
+   epoch fence (coordinator lease epoch) carries ownership across a demotion so a
+   demoted machine's stale claim can't steal a job. Designed for N machines, no
+   2-peer assumption.
+
+8. **Rollback cost** — Trivial. The feature is dark (flag default false). Flip
+   `multiMachine.seamlessness.ws43JournalLease` off (or leave it absent) and the
+   scheduler reverts to the legacy bus path with no migration, no data repair —
+   the journal lease store is just an inert local JSON ledger. Dry-run (default)
+   is an additional safe rung: flag on but no real cutover. The off-switch is read
+   live at each claim boundary, so a config edit + session restart fully reverts.
+
+## Phase 5 — Second-pass review (high-risk: touches scheduler claim/dedup + a "gate")
+
+Concern lens applied: could the cutover ever let TWO machines run the same job?
+- Within ONE machine's evaluation, the gate returns exactly one path — verified by
+  the `NEVER returns both mechanisms` unit test and the integration/e2e
+  `never-both` assertions (coherent → journal only; mixed → bus only).
+- ACROSS machines: the cutover engages pool-wide only under flag coherence (every
+  online peer advertises it), so the pool is either all-journal or all-bus for a
+  job set — never split. A peer that can't participate keeps everyone on the bus.
+- A demotion race is fenced by the lease epoch (stale-epoch applyRemote rejected;
+  same/older-epoch tryClaim refused) — covered by the epoch-fence unit tests.
+
+Concur with the review. The load-bearing invariant (no double-run across the
+cutover; no stranded timers on a demoted machine) is structurally enforced by the
+single-decision gate + epoch fence and is covered at all three test tiers.
+
+## Tests
+
+- Unit: `tests/unit/job-lease-cutover-gate.test.ts` (9), `tests/unit/
+  job-lease-claim-store.test.ts` (8) — gate decision matrix incl. never-both +
+  flag coherence + single-machine no-op; epoch fencing + durability.
+- Integration: `tests/integration/scheduler-journal-lease-cutover.test.ts` (6) —
+  the scheduler routes through the gate; coherent→journal, mixed→bus, dry-run,
+  single-machine, remote-lease skip, live flag flip.
+- E2E "feature is alive": `tests/e2e/scheduler-journal-lease-cutover-alive.test.ts`
+  (4) — the EXACT server.ts provider closure (real config + real coordinator
+  epoch + real MachinePoolRegistry heartbeat) drives a real triggerJob to take
+  the journal lease vs the bus; proves the wiring is live, not a stub.
+
+All 27 new tests pass; tsc clean; no-silent-fallbacks, dark-gate line-map, and
+feature-delivery-completeness gates pass; docs-coverage class floor improved.