instar 1.3.633 → 1.3.635
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/commands/server.d.ts.map +1 -1
- package/dist/commands/server.js +59 -4
- package/dist/commands/server.js.map +1 -1
- package/dist/config/ConfigDefaults.d.ts.map +1 -1
- package/dist/config/ConfigDefaults.js +18 -1
- package/dist/config/ConfigDefaults.js.map +1 -1
- package/dist/core/LeaseCoordinator.d.ts +7 -0
- package/dist/core/LeaseCoordinator.d.ts.map +1 -1
- package/dist/core/LeaseCoordinator.js +9 -0
- package/dist/core/LeaseCoordinator.js.map +1 -1
- package/dist/core/MachinePoolRegistry.d.ts +8 -0
- package/dist/core/MachinePoolRegistry.d.ts.map +1 -1
- package/dist/core/MachinePoolRegistry.js +1 -0
- package/dist/core/MachinePoolRegistry.js.map +1 -1
- package/dist/core/MultiMachineCoordinator.d.ts +69 -0
- package/dist/core/MultiMachineCoordinator.d.ts.map +1 -1
- package/dist/core/MultiMachineCoordinator.js +183 -0
- package/dist/core/MultiMachineCoordinator.js.map +1 -1
- package/dist/core/churnBreaker.d.ts +68 -0
- package/dist/core/churnBreaker.d.ts.map +1 -0
- package/dist/core/churnBreaker.js +98 -0
- package/dist/core/churnBreaker.js.map +1 -0
- package/dist/core/clockSkewAlarm.d.ts +61 -0
- package/dist/core/clockSkewAlarm.d.ts.map +1 -0
- package/dist/core/clockSkewAlarm.js +68 -0
- package/dist/core/clockSkewAlarm.js.map +1 -0
- package/dist/core/configCoherence.d.ts +39 -0
- package/dist/core/configCoherence.d.ts.map +1 -0
- package/dist/core/configCoherence.js +52 -0
- package/dist/core/configCoherence.js.map +1 -0
- package/dist/core/leaseLiveness.d.ts +42 -0
- package/dist/core/leaseLiveness.d.ts.map +1 -0
- package/dist/core/leaseLiveness.js +34 -0
- package/dist/core/leaseLiveness.js.map +1 -0
- package/dist/core/pollIntent.d.ts +65 -0
- package/dist/core/pollIntent.d.ts.map +1 -0
- package/dist/core/pollIntent.js +112 -0
- package/dist/core/pollIntent.js.map +1 -0
- package/dist/core/pollerCount.d.ts +61 -0
- package/dist/core/pollerCount.d.ts.map +1 -0
- package/dist/core/pollerCount.js +62 -0
- package/dist/core/pollerCount.js.map +1 -0
- package/dist/core/types.d.ts +57 -1
- package/dist/core/types.d.ts.map +1 -1
- package/dist/core/types.js.map +1 -1
- package/dist/lifeline/TelegramLifeline.d.ts +18 -0
- package/dist/lifeline/TelegramLifeline.d.ts.map +1 -1
- package/dist/lifeline/TelegramLifeline.js +99 -0
- package/dist/lifeline/TelegramLifeline.js.map +1 -1
- package/dist/lifeline/pollDecision.d.ts +47 -0
- package/dist/lifeline/pollDecision.d.ts.map +1 -0
- package/dist/lifeline/pollDecision.js +50 -0
- package/dist/lifeline/pollDecision.js.map +1 -0
- package/dist/monitoring/CommitmentTracker.d.ts +15 -0
- package/dist/monitoring/CommitmentTracker.d.ts.map +1 -1
- package/dist/monitoring/CommitmentTracker.js +59 -22
- package/dist/monitoring/CommitmentTracker.js.map +1 -1
- package/dist/monitoring/DegradationReporter.d.ts +10 -0
- package/dist/monitoring/DegradationReporter.d.ts.map +1 -1
- package/dist/monitoring/DegradationReporter.js +45 -17
- package/dist/monitoring/DegradationReporter.js.map +1 -1
- package/dist/server/routes.d.ts.map +1 -1
- package/dist/server/routes.js +17 -0
- package/dist/server/routes.js.map +1 -1
- package/package.json +1 -1
- package/src/data/builtin-manifest.json +49 -49
- package/upgrades/1.3.634.md +59 -0
- package/upgrades/1.3.635.md +53 -0
- package/upgrades/side-effects/commitment-tracker-verify-batches-saves.md +109 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-b1-lifeline-consumer.md +43 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-b1-poll-decision.md +38 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-b1-poll-intent-producer.md +37 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-b2-churn-breaker.md +42 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-b3-renew-timer.md +43 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-b4-clockskew-alarm-core.md +25 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-b4-d9-freeze-vs-skew.md +71 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-b4-skew-immune-liveness.md +41 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-b5-poller-count-route.md +24 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-b5-poller-count.md +38 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-b5-pollingactive-heartbeat.md +38 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-ci-housekeeping.md +17 -0
- package/upgrades/side-effects/mm-lease-poll-robustness-config-coherence.md +23 -0
- package/upgrades/1.3.633.md +0 -37
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Side-Effects Review — B1 lifeline consumer (multimachine-lease-poll-robustness, Decisions 4/5/6)
|
|
2
|
+
|
|
3
|
+
**Change:** The lifeline now reconciles its Telegram poll to the fenced-lease intent the server publishes — `reconcilePolling()` runs on the existing 15s loop (+ once at boot): reads the poll-intent file (freshness/dead-writer gated via `effectivePollIntent`), gathers the operator override + the local 409 signal + a start-debounce, calls `decidePollAction`, and writes `lifeline-poll-active.json` (the B5 truth source). **Ships DRY-RUN by default even on a dev agent** (`pollFollowsLease.dryRun:true`): it LOGS the would-start/would-stop and writes the truth file, but does NOT start/stop the real poll. The live flip (`dryRun:false`) is gated on the Phase-4 two-host proof + B2/B5 live (Decision 12).
|
|
4
|
+
|
|
5
|
+
**Files:** `src/lifeline/TelegramLifeline.ts` (reconcilePolling + wiring), `src/core/pollIntent.ts` (poll-active helpers + `pidAlive`), `src/core/types.ts` + `src/config/ConfigDefaults.ts` (`pollFollowsLease`/`pollOverride`), `tests/unit/pollIntent.test.ts`.
|
|
6
|
+
|
|
7
|
+
## THE LOAD-BEARING SAFETY CLAIM
|
|
8
|
+
This touches live Telegram ingress, and Phase-0's manual pins are keeping the live agent healthy RIGHT NOW. The guarantee: **in dry-run (the default, including on this dev agent) `reconcilePolling()` NEVER sets `this.polling` and NEVER calls `this.poll()`/stop** — it only reads, logs, and writes the advisory `lifeline-poll-active.json`. The two `this.polling` mutations + `this.poll()` call live exclusively inside the `if (!dryRun)` branch. So enabling the dev gate cannot change which machine polls; only a deliberate `dryRun:false` does. The static `shouldOwnTelegramPoll` boot decision is completely untouched and still controls real polling.
|
|
9
|
+
|
|
10
|
+
## Phase 1 — Principle check (signal vs authority)
|
|
11
|
+
When live, this IS authority (start/stop ingress), so the principle governs. It fails toward HOLD on every uncertainty: a stale/dead-writer/missing intent → `effectivePollIntent` null → `decidePollAction` returns hold; STOP is immediate on a real lease loss (losing the slot is the safe harm); START never spawns a 2nd poller (409/peer gate) and rides a debounce. In dry-run it holds NO authority (changes nothing).
|
|
12
|
+
|
|
13
|
+
## 1. Over-block / 2. Under-block
|
|
14
|
+
Decision logic is the unit-tested `decidePollAction` (12 tests) + the freshness gate (11 tests). Over-block (silence): guarded by STOP-only-on-real-loss + hold-on-stale. Under-block (dual-poll): guarded by the no-2nd-poller gate (local 409 = partition-immune; peer-pollingActive cross-check is the B5-surface follow-up) + debounce.
|
|
15
|
+
|
|
16
|
+
## 3. Level-of-abstraction fit
|
|
17
|
+
Right layer — the reconcile lives in the lifeline (which owns the socket), consuming the server's intent. Pure decision (`decidePollAction`) + pure freshness (`effectivePollIntent`) are isolated and tested; the lifeline just gathers inputs + applies.
|
|
18
|
+
|
|
19
|
+
## 4. Signal vs authority compliance
|
|
20
|
+
In dry-run: pure signal (logs). When live: authority over only THIS machine's own poll, fail-safe toward hold. (Ref `docs/signal-vs-authority.md`.)
|
|
21
|
+
|
|
22
|
+
## 5. Interactions
|
|
23
|
+
- **Static boot decision (`shouldOwnTelegramPoll`):** untouched; in dry-run it still drives real polling, reconcile only observes. When live, the live branch overrides it.
|
|
24
|
+
- **409 backoff:** `consecutive409s` is reused read-only as the recentLocal409 signal; reconcile never resets it.
|
|
25
|
+
- **replayInterval (15s):** reconcile is appended; pure/guarded, cannot wedge replay.
|
|
26
|
+
- **Phase-0 telegramPolling pin:** mapped to `force-mute` (Decision 7), so a Phase-0-pinned standby stays muted; `true`/absent defers to the lease (NOT force-poll) so two pinned machines don't become a permanent dual-poll.
|
|
27
|
+
|
|
28
|
+
## 6. External surfaces
|
|
29
|
+
A new advisory file `state/lifeline-poll-active.json` (local same-uid IPC). Dry-run log lines on a would-transition (infrequent). No route, no message. Live ingress unchanged in dry-run.
|
|
30
|
+
|
|
31
|
+
## 7. Multi-machine posture (Cross-Machine Coherence)
|
|
32
|
+
Per-machine: each lifeline reconciles its own poll from its own server's intent (machine-local IPC). The poll-active file is the per-machine truth B5 reads pool-wide. Single-machine / gate-off: `enabled` false (non-dev) → reconcile returns immediately; with no peer the lease drives a clean single poller when eventually live.
|
|
33
|
+
|
|
34
|
+
## 8. Rollback cost
|
|
35
|
+
Trivial — `pollFollowsLease.enabled:false` → reconcile no-ops (exact static behavior). Even enabled, `dryRun:true` (default) changes no ingress. The live flip is a deliberate, reversible config change. No state/migration.
|
|
36
|
+
|
|
37
|
+
## Verification
|
|
38
|
+
- `npx tsc --noEmit` clean.
|
|
39
|
+
- `tests/unit/pollIntent.test.ts` 11/11 (incl. poll-active round-trip + pidAlive) + `pollDecision.test.ts` 12/12.
|
|
40
|
+
- Dry-run safety is structural: the only `this.polling`/`this.poll()` writes are inside `if (!dryRun)`.
|
|
41
|
+
|
|
42
|
+
## Phase 5 — Second-pass review (INGRESS-CRITICAL)
|
|
43
|
+
Independent reviewer verdict: **Concur with the review.** The load-bearing dry-run-safety claim was verified BOTH structurally (every `this.polling=`/`this.poll()` is strictly inside `if (!dryRun)`; hold/dry-run return before any mutation) AND against the LIVE agent's real config (`/Users/justin/.instar/agents/echo/.instar/config.json`: `pollFollowsLease` absent → inherits `{dryRun:true}` → dry-run on this agent; `telegramPolling:true` does NOT map to force-mute). So the change CANNOT change which machine polls on the live agent — only a deliberate `dryRun:false` does. 409 backoff untouched (read-only), writePollActive atomic + unconsumed, pidAlive correct (EPERM=alive/ESRCH=dead). NON-BLOCKING note for the EVENTUAL live flip (out of scope now): re-verify boot static-decision vs first-reconcile ORDERING against the B2 intent producer — the "static decision overridden when live" is convergence-eventual, not boot-synchronous; the 409 gate is the dual-poll backstop regardless. Recorded for the live increment.
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Side-Effects Review — B1 poll-ownership decision core (multimachine-lease-poll-robustness, Decisions 4/5/7)
|
|
2
|
+
|
|
3
|
+
**Change:** Add the pure poll-ownership decision (`decidePollAction`) — should this lifeline START / STOP / HOLD its Telegram poll, given the server-written lease intent, the operator override, observed peer poll state, a recent-409 flag, the debounce timer, and a failover signal. Asymmetric hysteresis: STOP immediate on lease loss; START guarded (never a 2nd poller; debounce a flap; immediate on genuine failover); stale/corrupt intent → HOLD (no surprise silence, no blind start); operator force-poll/force-mute is the local floor (Phase-0 pin survives as force-mute). Pure, deterministic, observe-only.
|
|
4
|
+
|
|
5
|
+
**Files:** `src/lifeline/pollDecision.ts` (new pure helper), `tests/unit/pollDecision.test.ts`.
|
|
6
|
+
|
|
7
|
+
**Rollout sequencing (deliberate, not deferral):** this commit lands the decision CORE — the riskiest part of B1 (the logic that prevents both double-handling AND silence) is now pure and exhaustively unit-tested in isolation. The cross-process PLUMBING that supplies its inputs and applies its action — the server writing `state/telegram-poll-intent.json` (PID/bootId/ts integrity, on every role transition incl. the 5s pull path), the lifeline `reconcilePolling()` loop on its 15s interval, `state/lifeline-poll-active.json` (the B5 guard's pollingActive source), the Phase-0 `telegramPolling`-pin migration, and the B5 `/guards` surface — lands in the next increment(s) behind `multiMachine.pollFollowsLease` (dev-gate, gated on B2+B5 live per Decision 12), with the partition fault-injection integration test the spec names + the full second-pass review for an ingress-critical change. Landing the decision first keeps each PR small and the highest-risk logic proven before it has a live ingress consumer. Tracked: spec Decisions 4–7/11/12 + CMT-1710.
|
|
8
|
+
|
|
9
|
+
## Phase 1 — Principle check (signal vs authority)
|
|
10
|
+
Pure decision function — no authority, no I/O, no consumer in this commit. When wired, its only "authority" is start/stop of THIS machine's own poll, and it fails toward HOLD on every uncertainty (stale intent, can't confirm peer state) — never a surprise stop, never a blind start. That is the signal-vs-authority-safe direction (silence > nothing; dual-poll is the harm it refuses).
|
|
11
|
+
|
|
12
|
+
## 1. Over-block / 2. Under-block
|
|
13
|
+
Gates nothing in this commit. Correctness axes (unit-tested): never start a 2nd poller (anotherMachinePolling / 409 → hold, even on a failover signal); never surprise-stop on a stale intent (→ hold); STOP is immediate on a real lease loss; failover starts immediately (no silence gap); a flap is ridden out by the debounce.
|
|
14
|
+
|
|
15
|
+
## 3. Level-of-abstraction fit
|
|
16
|
+
Right layer — a pure decision in `src/lifeline/` (where the poll loop lives), isolated from the file-I/O plumbing that will supply its inputs. Matches the B3/B4/B2/B5 pure-helper pattern.
|
|
17
|
+
|
|
18
|
+
## 4. Signal vs authority compliance
|
|
19
|
+
Compliant — pure function, no blocking authority. (Ref `docs/signal-vs-authority.md`.)
|
|
20
|
+
|
|
21
|
+
## 5. Interactions
|
|
22
|
+
None in this commit (no consumer). When wired: reads the poll-intent file (B1 plumbing) + peer pollingActive (B5 source) + the local 409 signal; co-gated with B2 (flap breaker) + B5 (poller guard) per the spec's enforced rollout order so B1 can't ship ahead of its safety nets.
|
|
23
|
+
|
|
24
|
+
## 6. External surfaces
|
|
25
|
+
None in this commit (pure module). The live feature switches real Telegram ingress on/off — hence the deliberate decision-first sequencing and the full second-pass + partition integration test at wiring time.
|
|
26
|
+
|
|
27
|
+
## 7. Multi-machine posture (Cross-Machine Coherence)
|
|
28
|
+
The decision is per-machine (each lifeline decides its OWN poll), but it explicitly reasons about the POOL (peer poll state, failover) so two machines reach a single-poller outcome. The cross-process intent file is machine-local-by-design (each machine's own server↔lifeline). Single-machine: with no peer, `anotherMachinePolling`/`peerPresumedGone` are trivially false and the lease intent drives a clean single poller.
|
|
29
|
+
|
|
30
|
+
## 8. Rollback cost
|
|
31
|
+
Trivial — a pure module with no consumer; deleting it removes a tested helper. The live feature is dev-gated (`multiMachine.pollFollowsLease`) with a clean kill-switch returning the exact current static-flag behavior.
|
|
32
|
+
|
|
33
|
+
## Verification
|
|
34
|
+
- `npx tsc --noEmit` clean.
|
|
35
|
+
- `tests/unit/pollDecision.test.ts` 12/12: STOP-immediate; standby/awake holds; no-2nd-poller (another polling / 409); failover-immediate-start; debounce-hold vs debounce-elapsed-start; stale-intent→hold (both directions); force-mute / force-poll floor; and the safety case (a failover signal does NOT override the no-2nd-poller gate).
|
|
36
|
+
|
|
37
|
+
## Phase 5 — Second-pass review
|
|
38
|
+
**Not required for this commit** — a PURE decision module with NO consumer and NO live authority (gates nothing, touches no session/lifeline/messaging path yet). The full second-pass (ingress-critical) runs on the wiring increment that connects it to the live poll loop. Declared `not-required` with this reasoning (consistent with B5's decision-core commit).
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# Side-Effects Review — B1 poll-intent file + server producer (multimachine-lease-poll-robustness, Decisions 5/6)
|
|
2
|
+
|
|
3
|
+
**Change:** Add the cross-process poll-intent file module (`pollIntent.ts` — atomic write/read + a pure freshness/integrity gate `effectivePollIntent`) and wire the SERVER producer: `MultiMachineCoordinator` writes its lease-derived poll intent (`{shouldPoll, leaseEpoch, role, serverPid, bootId, ts}`) in `reconcileRoleToLease` (on every real role transition) and a safe default (`shouldPoll:false`) at lease boot. **Observe-only / producer-only: NOTHING reads the intent yet** (the lifeline `reconcilePolling` consumer is the next increment), so this cannot change Telegram ingress or role. Dev-gated (`multiMachine.pollFollowsLease`, `enabled` omitted → developmentAgent gate).
|
|
4
|
+
|
|
5
|
+
**Files:** `src/core/pollIntent.ts` (new), `src/core/MultiMachineCoordinator.ts` (bootId + resolver + writer + 2 call sites), `tests/unit/pollIntent.test.ts`.
|
|
6
|
+
|
|
7
|
+
## Phase 1 — Principle check (signal vs authority)
|
|
8
|
+
Pure producer of an advisory file + a pure trust gate. No authority — it writes a file no code reads in this commit. The freshness gate (`effectivePollIntent`) returns `null` (= "no opinion" = the lifeline will HOLD) on any uncertainty (stale/dead-writer/corrupt), the safe direction.
|
|
9
|
+
|
|
10
|
+
## 1./2. Over/Under-block
|
|
11
|
+
Gates nothing. Integrity (unit-tested): a stale `shouldPoll:true` (old ts, or dead writer pid) → null, so it can NOT resurrect a poller after a crash; a stale `shouldPoll:false` likewise → null, so it can NOT wrongly silence a live machine; corrupt/partial JSON → null (never trusted). Boot writes the safe default (mute).
|
|
12
|
+
|
|
13
|
+
## 3. Level-of-abstraction fit
|
|
14
|
+
Right layer — `reconcileRoleToLease` is the single role-transition chokepoint (the correct producer point); `pollIntent.ts` isolates the file protocol + the pure trust gate.
|
|
15
|
+
|
|
16
|
+
## 4. Signal vs authority compliance
|
|
17
|
+
Compliant — advisory file, pure gate, no blocking authority. (Ref `docs/signal-vs-authority.md`.)
|
|
18
|
+
|
|
19
|
+
## 5. Interactions
|
|
20
|
+
- **reconcileRoleToLease:** the write is appended AFTER the transition + the existing log, alongside the (observe-only) B2 breaker call. Guarded by try/catch → a write failure logs once, never throws into the role-transition path. Cannot alter the transition.
|
|
21
|
+
- **No consumer:** grep confirms nothing reads `telegram-poll-intent.json` in this commit; the lifeline consumer + the Phase-0 pin migration land next behind the same flag.
|
|
22
|
+
|
|
23
|
+
## 6. External surfaces
|
|
24
|
+
A new file `state/telegram-poll-intent.json` (local same-uid IPC, never network-reachable; parity with `TelegramPollOwnerLease`). One log line only on a write failure. No route, no message.
|
|
25
|
+
|
|
26
|
+
## 7. Multi-machine posture (Cross-Machine Coherence)
|
|
27
|
+
**Machine-local BY DESIGN** — each machine's own server writes its own intent for its own lifeline; nothing replicated/proxied. Single-machine / gate-off: `pollFollowsLeaseEnabled()` false or no leaseCoordinator → the writer is a no-op.
|
|
28
|
+
|
|
29
|
+
## 8. Rollback cost
|
|
30
|
+
Trivial — `pollFollowsLease.enabled:false` (read live) → no writes. The file is advisory; deleting it = no opinion. No state/migration.
|
|
31
|
+
|
|
32
|
+
## Verification
|
|
33
|
+
- `npx tsc --noEmit` clean.
|
|
34
|
+
- `tests/unit/pollIntent.test.ts` 8/8: atomic write/read round-trip; missing → null; corrupt/partial → null; fresh+live → shouldPoll; stale-ts → null; dead-pid → null; fresh mute honored.
|
|
35
|
+
|
|
36
|
+
## Phase 5 — Second-pass review (touches the lease role-transition path)
|
|
37
|
+
Independent reviewer verdict: **Concur with the review.** Verified genuinely observe-only (grep confirms NO production consumer of the intent — only the writer + tests, so it cannot change ingress/role), the write is try/caught + atomic (cannot throw into the role transition), the gate is correct (dev-gate; no-op single-machine/off), the integrity logic never trusts a stale shouldPoll:true (stale-ts/dead-pid/corrupt → null), and the boot default-mute overwrites a prior-boot stale record. One doc-accuracy nit (the comment claimed a graceful-shutdown mute that this diff doesn't implement, non-load-bearing since the consumer's dead-pid gate covers it) — **corrected** the comment. Everything that ships fails safe.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Side-Effects Review — B2 churn breaker (multimachine-lease-poll-robustness, Decision 8)
|
|
2
|
+
|
|
3
|
+
**Change:** Implement the consumer for the previously-DEAD `leaseSelfHeal.churnDetector` config — a lease flap circuit-breaker (`ChurnBreaker`). On >`maxFlipsPerWindow` real role transitions within `windowMs`, it LATCHES; the latched role is DETERMINISTIC (preferred-awake machine → awake, others → standby) so the resting state is exactly-one-awake, never a mid-flap coin-flip snapshot. Auto-resets after a calm window; EXHAUSTS (stays latched) above `maxLatchesPerHour` (guard-bypass-carries-its-own-cap). Wired observe-only/**dry-run**: `reconcileRoleToLease` records each true flip and LOGS the would-latch verdict; applying the deterministic role is the live graduation (`dryRun:false`). Dev-gated (`churnDetector.enabled` omitted → developmentAgent gate).
|
|
4
|
+
|
|
5
|
+
**Files:** `src/core/churnBreaker.ts` (new pure breaker), `src/core/MultiMachineCoordinator.ts` (field + getter + recordFlip in reconcileRoleToLease + tick in tickLease), `src/core/types.ts` + `src/config/ConfigDefaults.ts` (`enabled`/`dryRun`/`maxLatchesPerHour`), `tests/unit/churnBreaker.test.ts`.
|
|
6
|
+
|
|
7
|
+
## Phase 1 — Principle check (signal vs authority)
|
|
8
|
+
It feeds a decision (latch → hold a role), so the principle applies. As shipped here it is **signal-only** (dry-run: records + logs, applies nothing) — the role-application authority is gated behind a deliberate `dryRun:false`. The breaker logic is deterministic (a flip counter over a window + a deterministic latch target from the existing `preferredAwakeMachineId`), not brittle. When it does graduate to live, it stops a machine CONTENDING (the safe direction — it never force-promotes a peer), and the deterministic target guarantees exactly-one-awake.
|
|
9
|
+
|
|
10
|
+
## 1. Over-block
|
|
11
|
+
When live, the risk is latching the WRONG machine. Designed out: the latch target is deterministic (preferred→awake / other→standby), not a snapshot — so two machines latch to consistent, complementary roles (the spec's core B2 fix). In dry-run (this ship) it blocks nothing.
|
|
12
|
+
|
|
13
|
+
## 2. Under-block
|
|
14
|
+
A flap slower than `maxFlipsPerWindow / windowMs` won't trip it — by design (that's not a flap). The window pruning is unit-tested (a slow drip never trips).
|
|
15
|
+
|
|
16
|
+
## 3. Level-of-abstraction fit
|
|
17
|
+
Right layer. `reconcileRoleToLease` is the single chokepoint for real role transitions (it's past the `desired === this._role` early-return), so it's the correct place to count flips. The pure `ChurnBreaker` isolates the lifecycle for testing.
|
|
18
|
+
|
|
19
|
+
## 4. Signal vs authority compliance
|
|
20
|
+
Compliant. As shipped: pure signal (dry-run log). The breaker class holds no authority; the coordinator decides. (Ref `docs/signal-vs-authority.md`.)
|
|
21
|
+
|
|
22
|
+
## 5. Interactions
|
|
23
|
+
- **reconcileRoleToLease:** the recordFlip is appended AFTER the transition is fully applied + emitted; it cannot alter the transition. A throw is impossible (pure in-memory).
|
|
24
|
+
- **tickLease:** a `getChurnBreaker()?.tick()` at the top advances auto-reset; no-op when the gate is off; pure, cannot wedge the tick.
|
|
25
|
+
- **preferredAwakeMachineId (Phase-0 / F4):** the breaker REUSES it as the deterministic latch target — consistent with the live stabilization, never fights it.
|
|
26
|
+
- **Monotonic clock:** the breaker is driven by `monoNowMs()` so a wall-clock step can't fake/mask a flap.
|
|
27
|
+
|
|
28
|
+
## 6. External surfaces
|
|
29
|
+
One new log line, only when the breaker latches (`[MultiMachine] [churn] breaker LATCHED …`). No route, no message, no Attention item yet (the Attention raise + /guards row are the live-graduation increment). Dry-run by default.
|
|
30
|
+
|
|
31
|
+
## 7. Multi-machine posture (Cross-Machine Coherence)
|
|
32
|
+
**Machine-local BY DESIGN.** Each machine runs its own breaker over its OWN role transitions; the deterministic latch target is computed from the (replicated-by-config) `preferredAwakeMachineId`, so two machines independently reach complementary roles WITHOUT coordination — that's the point. Nothing replicated/proxied. Single-machine no-op: a single-machine agent never flips role, so the breaker never trips; the gate is also dark on the fleet.
|
|
33
|
+
|
|
34
|
+
## 8. Rollback cost
|
|
35
|
+
Trivial. `churnDetector.enabled:false` (read live each transition/tick) → the breaker is not built, zero effect. The role-application is additionally behind `dryRun` (default true) so even enabled it only logs until a deliberate flip. No state, no migration.
|
|
36
|
+
|
|
37
|
+
## Verification
|
|
38
|
+
- `npx tsc --noEmit` clean.
|
|
39
|
+
- `tests/unit/churnBreaker.test.ts` 6/6: no-latch at-or-below threshold; latch above threshold; DETERMINISTIC role (preferred→awake / other→standby); auto-reset after a calm window; EXHAUSTION above maxLatchesPerHour (stays latched, no auto-reset); window pruning (a slow drip never trips).
|
|
40
|
+
|
|
41
|
+
## Phase 5 — Second-pass review (high-risk: lease role-transition path)
|
|
42
|
+
Independent reviewer verdict: **Concur with the review.** Verified: recordFlip counts ONLY true flips (after the `desired === this._role` early-return); the wiring is pure in-memory and cannot throw/wedge the transition path; genuine no-op gate-off AND dry-run (NOTHING reads the verdict to override the role — cannot regress the live Phase-0 stabilization); lifecycle correctness (`>` trip, deterministic latch, calm-window auto-reset, exhaustion stays-latched, window/hour pruning, no unintended never-reset); monotonic clock is the right skew-proof source. Recommendation (added): an explicit two-machine complementary-resting-state test before the `dryRun:false` graduation — **added** (now 7/7 green). Noted for the live increment: getChurnBreaker rebuilds (loses history) on an off→on runtime toggle (harmless/conservative).
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Side-Effects Review — B3 renew timer (multimachine-lease-poll-robustness)
|
|
2
|
+
|
|
3
|
+
**Change:** Add a dedicated lease-renew timer (`clamp(leaseTtlMs × 0.5, [5s, 60s])`) decoupled from the 120s heartbeat-check timer, so a held lease is renewed (SAME epoch) before it lapses — stopping the epoch-climb (default TTL 60s < tick 120s → re-acquire at epoch+1 every tick, observed live 2026-06-20). Dev-gated (`leaseSelfHeal.resilientRenew`, `enabled` omitted → live-on-dev / dark-on-fleet).
|
|
4
|
+
|
|
5
|
+
**Files:** `src/core/MultiMachineCoordinator.ts` (timer + gate + cleanup), `src/core/LeaseCoordinator.ts` (`ttlMs` getter), `src/core/types.ts` (`resilientRenew` config), `src/config/ConfigDefaults.ts` (default block, no `enabled`), `src/commands/server.ts` (thread `developmentAgent` into the coordinator), `tests/unit/LeaseCoordinator-resilientRenew.test.ts`.
|
|
6
|
+
|
|
7
|
+
## Phase 1 — Principle check (signal vs authority)
|
|
8
|
+
Does this involve a decision point that gates information flow / blocks actions / constrains agent behavior? **No.** It is a pure-timing change: a timer that calls `renew()` (a same-epoch refresh) on a lease THIS machine already holds. It never acquires, never demotes, never gates a message, never changes a role decision. `holdsLease()`, the monotonic self-fence, and `acquireIfEligible()`'s authority are all untouched. The renew tick early-returns unless `holdsLease()` is already true. So signal-vs-authority does not apply — this is a behavior-neutral timing correction.
|
|
9
|
+
|
|
10
|
+
## 1. Over-block
|
|
11
|
+
N/A — blocks nothing. Worst "over" case: it renews more often than strictly necessary (every TTL/2 ≈ 30s vs the old 120s). That is the intent, and the cost is one signed same-epoch refresh per 30s (trivial mesh/file traffic).
|
|
12
|
+
|
|
13
|
+
## 2. Under-block
|
|
14
|
+
N/A — the change does not gate. The remaining gap it does NOT close: a lease that lapses for a reason OTHER than the slow tick (e.g. a genuine partition where renew can't confirm) still self-suspends correctly (unchanged). Decision 3's confirmed-same-epoch-renew-on-lapse (the residual edge where a lapse still happens) is a tracked continuation of this spec <!-- tracked: CMT-1710 --> — it is additive safety, not required for the renew timer to fully fix the normal-case climb.
|
|
15
|
+
|
|
16
|
+
## 3. Level-of-abstraction fit
|
|
17
|
+
Right layer. The renew cadence belongs in the coordinator that owns the lease lifecycle timers (it already owns the heartbeat-check, lease-pull, and tick-watchdog timers). The `ttlMs` getter on `LeaseCoordinator` is the minimal exposure needed to size the timer from the authoritative TTL. No higher/lower layer is a better owner.
|
|
18
|
+
|
|
19
|
+
## 4. Signal vs authority compliance
|
|
20
|
+
Compliant — it adds NO blocking authority and NO brittle decision logic. It is a timer that calls an existing same-epoch renew. (Ref: `docs/signal-vs-authority.md`.)
|
|
21
|
+
|
|
22
|
+
## 5. Interactions
|
|
23
|
+
- **tickLease (120s heartbeat):** unchanged. It still renews-if-holds / acquires-if-not. With the renew timer keeping the lease fresh, tickLease's `holdsLease()` branch now reliably takes the renew path (same epoch) instead of falling to acquire — which is the fix. No double-renew harm: a redundant same-epoch refresh is idempotent.
|
|
24
|
+
- **Re-entrancy:** `leaseRenewing` guard prevents overlapping ticks; `withTickTimeout` bounds a hung broadcast so the timer can't wedge.
|
|
25
|
+
- **soloCaptainHold / staleHolderTakeover / preferredAwakeMachineId:** untouched — those gate ACQUISITION/role, not renewal cadence.
|
|
26
|
+
- **Tick watchdog (F1):** unaffected — the renew timer is a separate `setInterval`; it doesn't touch `lastTickRunMonoMs`.
|
|
27
|
+
- Does NOT race with cleanup: `stop()` clears `leaseRenewTimer`.
|
|
28
|
+
|
|
29
|
+
## 6. External surfaces
|
|
30
|
+
One new log line (`[MultiMachine] lease renew timer armed …`), once per boot when enabled. No new route, no user-facing surface, no message. The reduced epoch-advance rate is visible only in `logs/server.log` (fewer `acquired lease at epoch N` lines) — a strict improvement.
|
|
31
|
+
|
|
32
|
+
## 7. Multi-machine posture (Cross-Machine Coherence)
|
|
33
|
+
**Machine-local BY DESIGN.** The renew timer is each machine renewing ITS OWN held lease on its own clock. Nothing is replicated or proxied — a standby (non-holder) renew tick early-returns (`!holdsLease()`), so only the actual holder renews. Single-machine no-op: a single-machine / no-leaseCoordinator agent never attaches a coordinator with a lease, so `startLeaseRenewTimer()` returns immediately. Dev-gated, so dark on the fleet until graduated.
|
|
34
|
+
|
|
35
|
+
## 8. Rollback cost
|
|
36
|
+
Trivial. Flip `leaseSelfHeal.resilientRenew.enabled: false` (read live — applies on the next renew tick / restart) → the timer no-ops and behavior reverts to exactly today's (epoch climbs, harmlessly, with the role stable as Phase-0 already ensures). No data migration, no state repair. The change is purely additive (a new timer); removing it cannot corrupt lease state.
|
|
37
|
+
|
|
38
|
+
## Verification
|
|
39
|
+
- `npx tsc --noEmit` clean on the changed files.
|
|
40
|
+
- `tests/unit/LeaseCoordinator-resilientRenew.test.ts` — 2/2 pass: (a) renewing every TTL/2 over 50 cycles keeps the SAME epoch (no climb); (b) contrast — letting the lease lapse re-acquires at epoch+1 (the bug, RED-proving the fix's necessity).
|
|
41
|
+
|
|
42
|
+
## Phase 5 — Second-pass review (high-risk: lease)
|
|
43
|
+
An independent reviewer audited the diff + artifact. Verdict: **Concern raised** — the renew tick lacked the `isLeaseObserveOnly` guard that `tickLease`'s observe-only branch enforces, so a machine that booted observe-only/muted while still NAMED in a persisted prior lease (the F3 silent-standby zombie) could have that lease renewed/re-broadcast for up to ~TTL, fighting the silent-standby-relinquish self-heal. **Resolved:** added `if (this.isLeaseObserveOnly) return;` at the top of `leaseRenewTick` (parity with `tickLease`). Reviewer concurred on everything else (fence integrity untouched, dev-gate correct + threaded, lifecycle/cleanup clean, single-machine no-op genuine, test non-vacuous). Re-verified post-fix: tsc clean, 2/2 tests pass.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Side-Effects Review — B4 clock-skew alarm decision core (multimachine-lease-poll-robustness, Decision 9)
|
|
2
|
+
|
|
3
|
+
**Change:** Add the pure clock-skew early-warning decision (`evaluateClockSkew`): given a measured cross-machine offset (max(ewma,last)) + this machine's own NTP-sync status, decide whether to raise an EARLY-WARNING below the 30s MeshRpc reject cliff, with hysteresis and N=2 self-blame attribution (blame self when own clock is unsynced; the peer when ours is verified synced; unknown when unprobed — never a confident finger-point). Pure, observe-only.
|
|
4
|
+
|
|
5
|
+
**Files:** `src/core/clockSkewAlarm.ts` (new), `tests/unit/clockSkewAlarm.test.ts`.
|
|
6
|
+
|
|
7
|
+
**Rollout sequencing (deliberate):** decision CORE first (fully unit-proven). The measurement wiring — a round-trip offset on the 5s lease-pull from the SIGNED `env.timestamp` (read pre-reject so it can report even past 30s), the own-NTP probe, and surfacing in `getSyncStatus`/a read route + a deduped Attention — is the follow-up. Tracked: spec Decision 9 + CMT-1710.
|
|
8
|
+
|
|
9
|
+
## Phase 1 — Principle check
|
|
10
|
+
Pure decision, no authority. It NEVER widens the MeshRpc 30s reject (replay-safety) — it only decides an advisory alarm. Fails toward not-alarming on uncertainty (within tolerance / unprobed NTP → no confident blame).
|
|
11
|
+
|
|
12
|
+
## 1-8
|
|
13
|
+
- Over/under-block: N/A (advisory). Correctness (unit-tested): early threshold (alarm at 20s, below the 30s break), hysteresis (clear at 12s), abs (step skew), self/peer/unknown blame.
|
|
14
|
+
- Abstraction: pure decision isolated from the (later) mesh measurement.
|
|
15
|
+
- Signal vs authority: pure signal — never the reject decision. (Ref docs/signal-vs-authority.md.)
|
|
16
|
+
- Interactions: none yet (no consumer). When wired: reads the measured offset + own-NTP, raises one deduped Attention.
|
|
17
|
+
- External: none (pure module).
|
|
18
|
+
- Multi-machine: inherently the N=2 attribution problem — encodes self-blame so two machines don't both finger-point. Single-machine: no peer → never measured → no alarm.
|
|
19
|
+
- Rollback: trivial (pure module, no consumer).
|
|
20
|
+
|
|
21
|
+
## Verification
|
|
22
|
+
- `tsc --noEmit` clean. `tests/unit/clockSkewAlarm.test.ts` 6/6.
|
|
23
|
+
|
|
24
|
+
## Phase 5 — Second-pass review
|
|
25
|
+
Not required — pure decision module, no consumer/authority, no session/messaging/recovery path. Full second-pass at the measurement-wiring increment.
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# Side-Effects Review — B4-D9 clock-skew alarm: freeze-vs-skew disambiguation
|
|
2
|
+
|
|
3
|
+
**Spec:** `docs/specs/multimachine-lease-poll-robustness.md` (approved) — B4, Decision 9.
|
|
4
|
+
**Files:** `src/core/clockSkewAlarm.ts`, `tests/unit/clockSkewAlarm.test.ts`.
|
|
5
|
+
|
|
6
|
+
## What changed
|
|
7
|
+
|
|
8
|
+
`evaluateClockSkew` gains an optional `recentLocalStarvationAgeMs` input (+ a
|
|
9
|
+
`starvationFreshnessMs` window, default 30s). When THIS machine's event loop has
|
|
10
|
+
recently starved, a large apparent peer offset is attributed to the local freeze
|
|
11
|
+
(`blame: 'local-freeze'`) and the peer-skew alarm is **suppressed** rather than
|
|
12
|
+
raised. Pure function; no new I/O, no consumer wired yet (the measurement wiring
|
|
13
|
+
that feeds `recentLocalStarvationAgeMs` from the SleepWakeDetector drift signal is
|
|
14
|
+
the next step under the same spec).
|
|
15
|
+
|
|
16
|
+
## Why (live evidence, 2026-06-20, topic 13481)
|
|
17
|
+
|
|
18
|
+
The agent's own server briefly wedged: a live-tail+mesh-rpc flap storm starved the
|
|
19
|
+
event loop ~171s (SleepWakeDetector drift burst). Mesh timestamps set *before* the
|
|
20
|
+
freeze were verified *after* it and were rejected as `stale-timestamp` — even though
|
|
21
|
+
the two machines' clocks were measured **1s apart** (Laptop vs Mini), i.e. perfectly
|
|
22
|
+
synced. The prior alarm logic would read that large apparent offset and raise a
|
|
23
|
+
"peer clock skew" alarm — a false positive that would point the operator at a
|
|
24
|
+
non-existent peer-clock problem. This guard makes the alarm honest under the exact
|
|
25
|
+
condition that produced it.
|
|
26
|
+
|
|
27
|
+
## The 8 questions
|
|
28
|
+
|
|
29
|
+
1. **Over-block.** It suppresses a *real* skew alarm while a local starvation is
|
|
30
|
+
fresh (default 30s window). Honest worst case (per 2nd-pass): a *repeating* freeze
|
|
31
|
+
STORM — like the 171s flap storm that motivated this — keeps re-stamping a small
|
|
32
|
+
`recentLocalStarvationAgeMs`, so suppression persists for the **full duration of
|
|
33
|
+
ongoing local freezing**, not a single window. That is still the correct safe
|
|
34
|
+
direction (you do not want to finger-point a peer's clock mid-storm), and it
|
|
35
|
+
self-heals the moment the storm ends and the offset re-alarms. Never permanently
|
|
36
|
+
hidden.
|
|
37
|
+
2. **Under-block.** It does not detect skew that *coincides* with a freeze (both true
|
|
38
|
+
at once). By design we cannot distinguish them during the freeze, so we defer; the
|
|
39
|
+
persistent-offset re-alarm after the window covers the real-skew case.
|
|
40
|
+
3. **Level-of-abstraction fit.** Correct layer: a pure decision helper. The freeze
|
|
41
|
+
signal is owned by SleepWakeDetector (its own surface); this helper merely *reads*
|
|
42
|
+
its age and refuses to mis-attribute. It does not duplicate freeze detection.
|
|
43
|
+
4. **Signal vs authority.** SIGNAL-ONLY, unchanged. It never gates, never widens the
|
|
44
|
+
MeshRpc reject (replay-safety preserved), never pushes a notification. Per the
|
|
45
|
+
operator constraint (topic 13481, this session) the alarm surfaces only on
|
|
46
|
+
`/guards` + logs — a pull surface — never a Telegram topic.
|
|
47
|
+
5. **Interactions.** Complements, does not shadow: the SleepWakeDetector still owns
|
|
48
|
+
freeze detection; this only prevents the skew alarm from double-counting a freeze
|
|
49
|
+
as skew. No race (pure function of its inputs).
|
|
50
|
+
6. **External surfaces.** None new. Optional input is back-compatible (undefined →
|
|
51
|
+
byte-identical prior behavior, proven by a test).
|
|
52
|
+
7. **Multi-machine posture.** Machine-local BY DESIGN: the freeze it guards against is
|
|
53
|
+
a property of THE LOCAL event loop on the machine running the check; the input is
|
|
54
|
+
read from that machine's own SleepWakeDetector. Each machine evaluates its own
|
|
55
|
+
skew alarm; no replication needed (the verdict is about *this* machine's clock vs a
|
|
56
|
+
peer, computed locally on both ends).
|
|
57
|
+
8. **Rollback cost.** Trivial: omit the new input and behavior reverts exactly (the
|
|
58
|
+
default `?? 30_000` and the `undefined` short-circuit make it inert when unfed).
|
|
59
|
+
No migration, no state.
|
|
60
|
+
|
|
61
|
+
## Second-pass
|
|
62
|
+
|
|
63
|
+
Required (touches a "guard"/alarm decision on the lease/recovery path).
|
|
64
|
+
|
|
65
|
+
**Reviewer verdict (independent):** *Concur with the review.* The freeze branch is
|
|
66
|
+
correctly gated behind the `!alarming` early-return (clearing a genuine alarm during
|
|
67
|
+
a fresh local freeze is the intended safe-direction behavior, self-healing once the
|
|
68
|
+
freeze ages out); back-compat is byte-identical on undefined input; the
|
|
69
|
+
in-tolerance-no-spurious-verdict case is covered. The reviewer's one doc nit — that a
|
|
70
|
+
repeating freeze storm suppresses for the storm's full duration, not a single window —
|
|
71
|
+
is folded into Q1 above. No code fix required to ship.
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# Side-Effects Review — B4 skew-immune lease liveness (multimachine-lease-poll-robustness, Decision 10)
|
|
2
|
+
|
|
3
|
+
**Change:** Derive the lease's `presumedDeadHolders` / `allPeersPresumedGone` peer-liveness from the SKEW-IMMUNE router-observed clock (`MachinePoolRegistry.getCapacity().online`, keyed on `routerReceivedAt`) instead of the peer's skew-contaminated `lastSeen`. Closes the flap's ROOT trigger: under clock skew a slow peer's `lastSeen` looks stale → false failover (the flap); a fast peer's looks fresh → delayed failover. Pure decision extracted to a unit-tested helper `isPeerPresumedDead`; the two server closures call it with a forward-ref to the (later-built) registry. Dev-gated (`leaseSelfHeal.skewImmuneLiveness`, `enabled` omitted → developmentAgent gate); flag off ⇒ byte-for-byte legacy `lastSeen` behavior.
|
|
4
|
+
|
|
5
|
+
**Files:** `src/core/leaseLiveness.ts` (new pure helper), `src/commands/server.ts` (forward-ref + 2 closures + flag), `src/core/types.ts` + `src/config/ConfigDefaults.ts` (`skewImmuneLiveness`), `tests/unit/leaseLiveness.test.ts`.
|
|
6
|
+
|
|
7
|
+
## Phase 1 — Principle check (signal vs authority)
|
|
8
|
+
Does it gate/block/constrain? It feeds a DECISION (presumed-dead → eligible-to-take-over), so signal-vs-authority applies. The decision logic is NOT brittle: it is a deterministic comparison over a router-observed timestamp (the same skew-immune source `MachinePoolRegistry` already uses for placement). It REPLACES an existing skew-contaminated input to the SAME authority (the lease's `canAcquire`); it adds no NEW blocking authority. The conservative direction is preserved (only positive staleness evidence ⇒ dead; unknown ⇒ alive), so a wrong reading fails toward NOT taking over (the safe direction — a wrongful takeover is the split-brain).
|
|
9
|
+
|
|
10
|
+
## 1. Over-block
|
|
11
|
+
The risk axis here is "wrongly presume a LIVE peer dead" (→ takeover → split-brain). Guarded: skew-immune path is used ONLY when the registry has actually observed the peer this incarnation (`routerObserved`); a known-on-disk-but-not-yet-observed peer (fresh boot) falls back to `lastSeen` rather than being wrongly marked dead (the convergence-review edge, unit-tested).
|
|
12
|
+
|
|
13
|
+
## 2. Under-block
|
|
14
|
+
"Wrongly presume a DEAD peer alive" (→ delayed failover). Under skew this is strictly BETTER than today (a fast-clock peer no longer looks alive forever). Flag off ⇒ unchanged.
|
|
15
|
+
|
|
16
|
+
## 3. Level-of-abstraction fit
|
|
17
|
+
Right layer. The skew-immune source already exists in `MachinePoolRegistry` (used for placement). This routes the SAME source to the lease layer, which previously had its own skew-blind copy — removing a dual-source disagreement rather than adding a layer. The pure helper isolates the decision for testing.
|
|
18
|
+
|
|
19
|
+
## 4. Signal vs authority compliance
|
|
20
|
+
Compliant. The helper is a pure function (no I/O, no authority); it returns a boolean the existing lease authority consumes. No brittle blocking added. (Ref `docs/signal-vs-authority.md`.)
|
|
21
|
+
|
|
22
|
+
## 5. Interactions
|
|
23
|
+
- **soloCaptainHold / staleHolderTakeover:** both consume `allPeersPresumedGone` / `presumedDeadHolders` — they get a more-accurate (skew-immune) input; their own gates (preferred-awake, no-higher-epoch) are untouched. soloCaptainHold ships dark regardless.
|
|
24
|
+
- **TDZ safety:** the forward-ref `leaseLivenessRegistry` is a function-body `let`, declared BEFORE the lease block and assigned AFTER the registry is built. During `initializeLease()` (which can call the closures) the ref is still `undefined` → `routerObserved:false` → lastSeen fallback. No throw, safe degrade.
|
|
25
|
+
- **No double-source race:** when the registry IS set, both closures read the same live `getCapacity` view; consistent within a tick.
|
|
26
|
+
|
|
27
|
+
## 6. External surfaces
|
|
28
|
+
None new. No route, no message, no log added (the existing lease logs are unchanged). The only observable effect is more-correct failover timing under skew.
|
|
29
|
+
|
|
30
|
+
## 7. Multi-machine posture (Cross-Machine Coherence)
|
|
31
|
+
**Machine-local read of a per-machine observation.** Each machine decides peer-liveness from ITS OWN router-observed clock (`MachinePoolRegistry` is per-machine, in-memory). Nothing replicated/proxied. Single-machine no-op: `peerIds.length === 0` returns false (a solo machine never presumes a peer gone); `presumedDeadHolders` returns an empty set. Dev-gated → dark on the fleet until graduated.
|
|
32
|
+
|
|
33
|
+
## 8. Rollback cost
|
|
34
|
+
Trivial. `leaseSelfHeal.skewImmuneLiveness.enabled:false` (read live each call) → the closures revert to exact legacy `lastSeen` behavior immediately. No state, no migration. The helper + forward-ref are inert when the flag is off (the skew-immune branch is simply not taken).
|
|
35
|
+
|
|
36
|
+
## Verification
|
|
37
|
+
- `npx tsc --noEmit` clean.
|
|
38
|
+
- `tests/unit/leaseLiveness.test.ts` 4/4: flag-off legacy threshold; flag-on skew-immune wins over a fast-clock (future) lastSeen AND over a slow-clock (stale-looking) lastSeen (the false-failover that CAUSED the flap); not-yet-observed → lastSeen fallback (no wrongful dead); conservative unknown → not-dead.
|
|
39
|
+
|
|
40
|
+
## Phase 5 — Second-pass review (high-risk: lease/recovery)
|
|
41
|
+
Independent reviewer verdict: **Concur with the review.** Verified TDZ/forward-ref safety (function-scope let, runtime-only invocation, undefined→lastSeen fallback), split-brain safety (the skew-immune path is STRICTLY more conservative in the takeover direction than legacy lastSeen — a skew-quarantined-but-reachable peer stays online:true and is never presumed dead, the flap fix), getCapacity/threshold consistency, live flag resolution, and non-vacuous tests. One optional coverage suggestion (override-to-dead with a fresh lastSeen) — **added** as a 5th test (now 5/5 green). No code change required to ship.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Side-Effects Review — B5 /pool/poller-count read route (multimachine-lease-poll-robustness, Decision 11)
|
|
2
|
+
|
|
3
|
+
**Change:** Add `GET /pool/poller-count` — exposes the exactly-one-Telegram-listener verdict (`poolPollerVerdict` over the pool's `MachineCapacity` rows: ok / dual / silence / indeterminate). Observe-only read route; the `pollingActive` source is the heartbeat field (B5), already propagated. The deduped Attention item on dual/silence is the follow-up; this surfaces the verdict on demand now.
|
|
4
|
+
|
|
5
|
+
**Files:** `src/server/routes.ts` (route + import), `tests/integration/pool-routes.test.ts` (2 tests).
|
|
6
|
+
|
|
7
|
+
## Phase 1 — Principle check
|
|
8
|
+
Read-only route, no authority. Returns a computed verdict; gates nothing.
|
|
9
|
+
|
|
10
|
+
## 1-8
|
|
11
|
+
- Over/under-block: N/A (read route). The verdict's correctness (dark-peer→indeterminate, never a false silence/dual) is the unit-tested `poolPollerVerdict` (11 unit tests) + 2 integration tests here.
|
|
12
|
+
- Abstraction: thin route over the tested pure adapter, mirroring the existing `/pool` route.
|
|
13
|
+
- Signal vs authority: pure read signal.
|
|
14
|
+
- Interactions: reads `machinePoolRegistry.getCapacities()` (same as `/pool`); no mutation.
|
|
15
|
+
- External surface: a new authed read route (returns the verdict + per-machine pollingActive). No message.
|
|
16
|
+
- Multi-machine: pool-scoped read, dark-peer-tolerant by construction. Single-machine → one capacity row → ok when self polls.
|
|
17
|
+
- Rollback: trivial (delete the route handler).
|
|
18
|
+
|
|
19
|
+
## Verification
|
|
20
|
+
- `tsc --noEmit` clean.
|
|
21
|
+
- `tests/integration/pool-routes.test.ts` 8/8 (incl. the 2 new: exactly-one→ok; dark-peer→indeterminate).
|
|
22
|
+
|
|
23
|
+
## Phase 5 — Second-pass review
|
|
24
|
+
Not required — a read-only observability route over the already-unit-tested `poolPollerVerdict`; no authority, no gate, no session/messaging/recovery path.
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Side-Effects Review — B5 poller-count decision (multimachine-lease-poll-robustness, Decision 11)
|
|
2
|
+
|
|
3
|
+
**Change:** Add the pure three-valued exactly-one-listener decision (`evaluatePollerCount`). Given per-machine poll-active observations (some possibly dark/unknown) + a local Telegram-409 signal, it returns `ok` (exactly one fresh poller) / `dual` (≥2 positively observed, OR a 409 = partition-immune evidence of a 2nd poller even when a peer is dark) / `silence` (a real zero — everyone fresh+known) / `indeterminate` (a dark/unknown peer → cannot confirm; NEVER a false silence/ok). Pure, deterministic, observe-only.
|
|
4
|
+
|
|
5
|
+
**Files:** `src/core/pollerCount.ts` (new pure helper), `tests/unit/pollerCount.test.ts`.
|
|
6
|
+
|
|
7
|
+
**Rollout sequencing (deliberate, not deferral):** this commit lands the decision CORE. Its guard surface — the `pollingActive` field on the capacity heartbeat, the `/guards` row, the pool fan-out, and the 409 source — wires in the B1 increment, because the authoritative `pollingActive` source is the lifeline's actual poll state (`lifeline-poll-active.json`) that B1 introduces (spec Decision 6). Landing the tested decision module first keeps each PR small and the wiring honest (the helper is fully unit-proven before it has a live consumer). Tracked: spec Decision 11/12 + CMT-1710.
|
|
8
|
+
|
|
9
|
+
## Phase 1 — Principle check (signal vs authority)
|
|
10
|
+
Pure decision function, no authority, no I/O. It will feed a SIGNAL-only guard (observe → Attention item), never a block/gate. Signal-vs-authority satisfied by construction.
|
|
11
|
+
|
|
12
|
+
## 1. Over-block / 2. Under-block
|
|
13
|
+
N/A — gates nothing. The decision's correctness axis is "never false-alarm": a dark/unknown peer yields `indeterminate`, not a false `silence`/`ok` (the adversarial-review requirement) — unit-tested. The 409 path prevents the under-detection of a partition-induced dual-poll that heartbeat-counting alone misses.
|
|
14
|
+
|
|
15
|
+
## 3. Level-of-abstraction fit
|
|
16
|
+
Right layer — a pure decision isolated from data-gathering, so the guard wiring (B1) just supplies observations. Matches the B3/B4/B2 pure-helper pattern.
|
|
17
|
+
|
|
18
|
+
## 4. Signal vs authority compliance
|
|
19
|
+
Compliant — pure function, no blocking authority. (Ref `docs/signal-vs-authority.md`.)
|
|
20
|
+
|
|
21
|
+
## 5. Interactions
|
|
22
|
+
None yet — no consumer in this commit. When wired (B1), it reads the capacity heartbeat (fail-open `pollingActive`) + the local 409 signal; it produces an Attention item with a stable dedupKey (`poll-ownership:<state>`) + a `/guards` row (both in the B1 increment).
|
|
23
|
+
|
|
24
|
+
## 6. External surfaces
|
|
25
|
+
None in this commit (pure module). The future guard surface is observe-only (an Attention item + a `/guards` row), never a message gate.
|
|
26
|
+
|
|
27
|
+
## 7. Multi-machine posture (Cross-Machine Coherence)
|
|
28
|
+
The decision is inherently POOL-scoped (it reasons over all machines' observations) and dark-peer-tolerant BY DESIGN (`indeterminate` on any visibility gap). The future guard reads via the existing `?scope=pool` fan-out. Single-machine: one observation (self), `ok` — unit-tested.
|
|
29
|
+
|
|
30
|
+
## 8. Rollback cost
|
|
31
|
+
Trivial — a pure module with no consumer yet; deleting it removes a tested helper. When wired, the guard is dev-gated (`multiMachine.pollOwnershipGuard`) with a clean kill-switch.
|
|
32
|
+
|
|
33
|
+
## Verification
|
|
34
|
+
- `npx tsc --noEmit` clean.
|
|
35
|
+
- `tests/unit/pollerCount.test.ts` 8/8: ok; dual (positive ≥2); silence (real zero); indeterminate (dark peer — NOT false silence); indeterminate (older-peer unknown field); dual-via-409 even when the peer is dark (partition-immune); positive-≥2 wins over an unknown third; single-machine → ok.
|
|
36
|
+
|
|
37
|
+
## Phase 5 — Second-pass review
|
|
38
|
+
**Not required.** This commit is a PURE decision module with NO consumer and NO authority (it gates nothing, touches no session lifecycle / messaging / recovery path). The second-pass review will run on the B1 increment that WIRES it into the guard surface (where it gains a live surface). Declared `not-required` with this reasoning.
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Side-Effects Review — B5 pollingActive heartbeat + pool adapter (multimachine-lease-poll-robustness, Decision 11)
|
|
2
|
+
|
|
3
|
+
**Change:** Propagate each machine's REAL Telegram poll state pool-wide so the exactly-one-listener guard can count actual pollers. Adds `pollingActive?: boolean` to `HeartbeatObservation` + `MachineCapacity` (+ the `assemble` passthrough); the server reads its own `lifeline-poll-active.json` (the lifeline's truth file from B1) and advertises `pollingActive` in its heartbeat — `undefined` (unknown) when the file is absent / stale (>90s) / the lifeline pid is dead (fail-open). Adds `poolPollerVerdict()` — the adapter that maps the pool's `MachineCapacity` rows → the unit-tested `evaluatePollerCount` (online = freshness; absent pollingActive = unknown).
|
|
4
|
+
|
|
5
|
+
**Files:** `src/core/MachinePoolRegistry.ts` (obs field + assemble passthrough), `src/core/types.ts` (`MachineCapacity.pollingActive`), `src/commands/server.ts` (advertise self pollingActive from the truth file), `src/core/pollerCount.ts` (`poolPollerVerdict` adapter), `tests/unit/pollerCount.test.ts`.
|
|
6
|
+
|
|
7
|
+
## Phase 1 — Principle check (signal vs authority)
|
|
8
|
+
Additive, observe-only. `pollingActive` is a new advisory heartbeat field consumed by NOTHING that makes an authority decision — placement does NOT read it (it gates on quota/clock/load only). The adapter is a pure function. No authority added.
|
|
9
|
+
|
|
10
|
+
## 1./2. Over/Under-block
|
|
11
|
+
Gates nothing. The field is the INPUT to the (later) guard surface; the verdict logic (three-valued, dark-peer→indeterminate, 409 ground truth) is the already-tested `evaluatePollerCount`. Fail-open: an absent/stale/dead-lifeline poll-active → `undefined` → unknown → the guard reports `indeterminate`, never a false silence/dual alarm.
|
|
12
|
+
|
|
13
|
+
## 3. Level-of-abstraction fit
|
|
14
|
+
Right layer — poll-active truth lives in the lifeline's file (B1); the server (which assembles the heartbeat) reads it and advertises it; the registry carries it like every other capacity field (quotaState, servesChannels). The adapter sits with the pure verdict helper.
|
|
15
|
+
|
|
16
|
+
## 4. Signal vs authority compliance
|
|
17
|
+
Compliant — advisory field + pure adapter, no blocking authority. (Ref `docs/signal-vs-authority.md`.)
|
|
18
|
+
|
|
19
|
+
## 5. Interactions
|
|
20
|
+
- **Heartbeat assembly / placement:** purely additive — placement reads `online`/quota/load, not `pollingActive`, so placement behavior is unchanged. The field rides the same fail-open passthrough as `quotaState`.
|
|
21
|
+
- **B1 lifeline-poll-active.json:** the source. The server discounts a dead-lifeline/stale file → unknown (so a crashed lifeline can't be miscounted as polling).
|
|
22
|
+
- **Older peers (mid-rollout):** emit no `pollingActive` → `undefined` → unknown → `indeterminate`, the safe direction (wire-compat, Decision 11).
|
|
23
|
+
|
|
24
|
+
## 6. External surfaces
|
|
25
|
+
A new optional field on `/pool` capacity rows (already-authed). No new route yet (the `/guards` poll-ownership row + the Attention item are the completing increment, now trivially computable via `poolPollerVerdict`). No message.
|
|
26
|
+
|
|
27
|
+
## 7. Multi-machine posture (Cross-Machine Coherence)
|
|
28
|
+
Pool-scoped BY DESIGN: each machine advertises its own `pollingActive`; `poolPollerVerdict` reasons over the merged `?scope=pool` view and is dark-peer-tolerant (`online:false` → not fresh → indeterminate). Single-machine: one capacity row (self) → `ok` when polling.
|
|
29
|
+
|
|
30
|
+
## 8. Rollback cost
|
|
31
|
+
Trivial — an additive optional field; absent everywhere = exact current behavior (the verdict reports indeterminate/unknown). No state/migration. The `poolPollerVerdict` adapter is pure with no live consumer yet.
|
|
32
|
+
|
|
33
|
+
## Verification
|
|
34
|
+
- `npx tsc --noEmit` clean.
|
|
35
|
+
- `tests/unit/pollerCount.test.ts` 11/11 (8 verdict cases + 3 adapter cases: online+active→ok, dark-peer→indeterminate, two-online→dual).
|
|
36
|
+
|
|
37
|
+
## Phase 5 — Second-pass review
|
|
38
|
+
**Not required** — additive, observe-only heartbeat field (fail-open, not read by any placement/authority decision) + a pure tested adapter with no live consumer. No session-lifecycle / messaging-gate / recovery path is touched. The full second-pass runs on the `/guards` row + Attention-item increment that gives it a live surface. Declared `not-required` with this reasoning (consistent with the B5/B1 decision-core commits).
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
# Side-Effects Review — CI housekeeping: dev-gate canonicalization + decision-audit + lint
|
|
2
|
+
|
|
3
|
+
**Change:** Three CI-correctness fixes the now-wired husky gate (npm run prepare) surfaced on the B1-B5 commits (the build worktree was created with raw `git worktree add`, so the gate hadn't run):
|
|
4
|
+
1. **Dev-gate canonicalization (real source fix):** route every dev-gated resolver I added through the canonical `resolveDevAgentGate(explicit, config)` instead of hand-rolling `explicit ?? !!developmentAgent` — per DEV-AGENT-DARK-GATE-ENFORCEMENT-SPEC. 6 sites: MultiMachineCoordinator (resilientRenew, churnDetector, pollFollowsLease), server.ts (skewImmuneLiveness ×2 closures), TelegramLifeline (pollFollowsLease). Behavior-identical (the helper IS `explicit ?? !!config.developmentAgent`), now standard-conformant + lint-clean.
|
|
5
|
+
2. **Decision-audit backfill:** honest Tier-2 records in `.instar/instar-dev-decisions.jsonl` for the B1-B5 commits (gate evidence the un-wired husky shim didn't write).
|
|
6
|
+
3. **Test lint-allow:** `safe-git-allow`/`safe-fs-allow` comments for pollIntent.test.ts's per-test tmpdir rmSync (the SafeFsExecutor-containment lint's documented test convention).
|
|
7
|
+
|
|
8
|
+
**Files:** `src/core/MultiMachineCoordinator.ts`, `src/commands/server.ts`, `src/lifeline/TelegramLifeline.ts`, `tests/unit/pollIntent.test.ts`, `.instar/instar-dev-decisions.jsonl`.
|
|
9
|
+
|
|
10
|
+
## Phase 1 — Principle check
|
|
11
|
+
The dev-gate canonicalization is behavior-preserving (identical resolution, now via the shared helper). No new decision/authority. Items 2-3 are pure process/lint.
|
|
12
|
+
|
|
13
|
+
## 1-8 (over/under-block, abstraction, signal-vs-authority, interactions, external, multi-machine, rollback)
|
|
14
|
+
N/A behavior change — `resolveDevAgentGate` returns exactly the prior expression, so every gate resolves identically (live-on-dev / dark-on-fleet). No external surface, no multi-machine posture change. Rollback: trivial (the helper call ≡ the inline expression). 48 unit tests pass unchanged; dev-gate-dark lint clean.
|
|
15
|
+
|
|
16
|
+
## Phase 5 — Second-pass review
|
|
17
|
+
Not required — a behavior-preserving canonicalization to the standard helper + audit/lint housekeeping; no new runtime behavior, decision, or authority. (The underlying B1-B5 features each had their own second-pass already.)
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# Side-Effects Review — Phase 2 #7 startup config-coherence WARNINGS
|
|
2
|
+
|
|
3
|
+
**Change:** Add `checkMultiMachineConfigCoherence` (pure) + wire it to LOG warnings at server boot (multi-machine only). Flags the audit's worst-of-both state — `meshTransport.enabled:false` while `sessionPool.stage:live-transfer` (transfer live but single-rope, reintroducing the lease flap) — and duplicate/non-positive mesh rope priorities. WARN-only — NEVER a boot reject.
|
|
4
|
+
|
|
5
|
+
**Files:** `src/core/configCoherence.ts` (new pure checker), `src/commands/server.ts` (boot-time log), `tests/unit/configCoherence.test.ts`.
|
|
6
|
+
|
|
7
|
+
## Phase 1 — Principle check
|
|
8
|
+
Signal only — it logs warnings; it NEVER throws or blocks boot. Per the audit (F-CLAMP1), a hard-reject invariant would refuse fleet boot on a default-config combination, so this is deliberately advisory.
|
|
9
|
+
|
|
10
|
+
## 1-8
|
|
11
|
+
- Over/under-block: N/A (logs only). Single-machine → no warnings (harmless no-op there). Undefined config → no warnings (never throws).
|
|
12
|
+
- Abstraction: pure checker + a thin boot-time log call. Right layer (boot, where config resolves).
|
|
13
|
+
- Signal vs authority: pure signal; boot is never blocked. (Ref docs/signal-vs-authority.md.)
|
|
14
|
+
- Interactions: reads `config.multiMachine` at boot, after the coordinator starts; no mutation.
|
|
15
|
+
- External: yellow `⚠ config-coherence [...]` boot log lines only.
|
|
16
|
+
- Multi-machine: only runs when `coordinator.enabled` (a real multi-machine agent); the exact combination it flags is the one my 2026-06-20 morning band-aid created.
|
|
17
|
+
- Rollback: trivial (remove the boot call; the pure checker is inert).
|
|
18
|
+
|
|
19
|
+
## Verification
|
|
20
|
+
- `tsc --noEmit` clean. `tests/unit/configCoherence.test.ts` 7/7 (mesh-off-while-live-transfer; single-machine no-flag; mesh-on coherent; duplicate priorities; non-positive priorities; well-formed no-warn; undefined no-throw).
|
|
21
|
+
|
|
22
|
+
## Phase 5 — Second-pass review
|
|
23
|
+
Not required — a pure WARN-only config check that logs at boot; no authority, no gate, never blocks boot, no session/messaging/recovery path touched.
|
package/upgrades/1.3.633.md
DELETED
|
@@ -1,37 +0,0 @@
|
|
|
1
|
-
# Upgrade Guide — vNEXT
|
|
2
|
-
|
|
3
|
-
<!-- assembled-by: assemble-next-md -->
|
|
4
|
-
<!-- bump: patch -->
|
|
5
|
-
|
|
6
|
-
## What Changed
|
|
7
|
-
|
|
8
|
-
`FeedbackManager` now treats a `429`/`503` from the canonical feedback endpoint as a
|
|
9
|
-
stop-signal: it HALTS the current retry batch and waits (honoring `Retry-After`, else
|
|
10
|
-
exponential 60s→1h) before POSTing again. Previously `retryUnforwarded()` re-POSTed the
|
|
11
|
-
ENTIRE un-forwarded backlog every scheduled cycle regardless of rate-limiting — against
|
|
12
|
-
a 429-ing endpoint that re-sent the whole backlog forever (observed live: 661 backlog
|
|
13
|
-
items → 2,384+ failed POSTs). A pure `decideFeedbackRetry()` helper makes the cadence
|
|
14
|
-
decision; the local feedback record is always written (nothing is lost or dropped).
|
|
15
|
-
|
|
16
|
-
## What to Tell Your User
|
|
17
|
-
|
|
18
|
-
If your agent had been quietly retrying feedback against a busy endpoint, it now backs
|
|
19
|
-
off politely instead of hammering it — fewer wasted network calls and less background
|
|
20
|
-
churn. Nothing you do changes; no feedback is lost (notes are still saved locally and
|
|
21
|
-
sent once the endpoint is ready).
|
|
22
|
-
|
|
23
|
-
## Summary of New Capabilities
|
|
24
|
-
|
|
25
|
-
- Rate-limit-aware feedback forwarding: halts on 429/503, honors `Retry-After`, backs
|
|
26
|
-
off exponentially (capped), and never re-sends the whole backlog against a limited
|
|
27
|
-
endpoint. Back-compatible on a healthy endpoint (forwards immediately as before).
|
|
28
|
-
|
|
29
|
-
## Evidence
|
|
30
|
-
|
|
31
|
-
- New `tests/unit/feedbackBackoff.test.ts` (11 cases): 2xx clears backoff; 429/503 halt
|
|
32
|
-
+ exponential backoff capped at 1h; `Retry-After` honored + capped; non-429 errors
|
|
33
|
-
retry-next-cycle and PRESERVE the 429 streak (flap protection); junk/negative
|
|
34
|
-
`Retry-After` rejected.
|
|
35
|
-
- `tests/unit/FeedbackManager.test.ts` (+2 cases): a 429 halts the batch after ONE POST
|
|
36
|
-
(not the whole backlog) and the next cycle POSTs zero times while backed off; no item
|
|
37
|
-
lost. Independent second-pass review concurred.
|