instar 0.28.111 → 0.28.113
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/commands/setup.d.ts +16 -0
- package/dist/commands/setup.d.ts.map +1 -1
- package/dist/commands/setup.js +233 -7
- package/dist/commands/setup.js.map +1 -1
- package/dist/core/PostUpdateMigrator.d.ts +1 -0
- package/dist/core/PostUpdateMigrator.d.ts.map +1 -1
- package/dist/core/PostUpdateMigrator.js +115 -0
- package/dist/core/PostUpdateMigrator.js.map +1 -1
- package/package.json +1 -1
- package/scripts/instar-dev-precommit.js +31 -1
- package/src/data/builtin-manifest.json +26 -18
- package/src/templates/scripts/instar-watchdog.sh +475 -0
- package/upgrades/0.28.112.md +28 -0
- package/upgrades/0.28.113.md +43 -0
- package/upgrades/side-effects/lifeline-shadow-install-self-heal.md +208 -0
- package/upgrades/side-effects/s3-promotion-gate.md +111 -0
|
@@ -0,0 +1,208 @@
|
|
|
1
|
+
# Side-Effects Review — Lifeline shadow-install self-heal + fleet watchdog hardening
|
|
2
|
+
|
|
3
|
+
**Version / slug:** `lifeline-shadow-install-self-heal`
|
|
4
|
+
**Date:** 2026-05-17
|
|
5
|
+
**Author:** echo
|
|
6
|
+
**Second-pass reviewer:** subagent (high-risk: touches "watchdog" trigger)
|
|
7
|
+
**Spec:** [docs/specs/lifeline-shadow-install-self-heal.md](../../docs/specs/lifeline-shadow-install-self-heal.md)
|
|
8
|
+
**ELI16:** [docs/specs/lifeline-shadow-install-self-heal.eli16.md](../../docs/specs/lifeline-shadow-install-self-heal.eli16.md)
|
|
9
|
+
|
|
10
|
+
## Summary of the change
|
|
11
|
+
|
|
12
|
+
Three coordinated changes that close a 4-day-outage class (AI Guy, 2026-05-13 → 2026-05-17):
|
|
13
|
+
|
|
14
|
+
1. **Boot wrapper self-heal for missing shadow-install** (`src/commands/setup.ts` — both bash and node wrappers in `installBootWrapper()`). When the wrapper finds no `shadow-install/node_modules/instar/dist/cli.js`, it now attempts one `npm install` via absolute node + npm-cli.js paths (PATH may be empty under launchd), debounced by a `.heal-attempted` marker file that prevents launchd KeepAlive throttling from triggering 30+ reinstall attempts per minute.
|
|
15
|
+
|
|
16
|
+
2. **Fleet watchdog migrated into instar source** (`src/templates/scripts/instar-watchdog.sh` — new). The previously hand-rolled `~/.instar/instar-watchdog.sh` now ships from src with two key behavioral changes vs the prior hand-rolled version: (a) heal-step `npm install` uses absolute-path `node` + `npm-cli.js` resolution instead of bare `npm` (which fails under launchd's empty PATH), and (b) when self-heal fails 3 cycles in a row for the same agent, it discovers a healthy peer agent and POSTs to that peer's `/attention` endpoint with `category: "degradation"` — the existing tone-gated cross-topic alert path. Tied into `installMacOSLaunchAgent()` and `PostUpdateMigrator.migrateFleetWatchdog()` for parity across fresh installs and existing agents.
|
|
17
|
+
|
|
18
|
+
3. **Watchdog launchd plist sets PATH explicitly.** Belt-and-suspenders next to the absolute-path resolution in the script. Catches any other shell utility (curl, awk, etc.) that the script might invoke.
|
|
19
|
+
|
|
20
|
+
Decision points the change interacts with:
|
|
21
|
+
- `installBootWrapper()` shadow-install presence check (modified — adds heal branch).
|
|
22
|
+
- `installMacOSLaunchAgent()` post-install hook (modified — calls `installFleetWatchdog()`).
|
|
23
|
+
- `PostUpdateMigrator.migrate()` (modified — adds `migrateFleetWatchdog()` entry).
|
|
24
|
+
- `/attention` route → `MessagingToneGate` health-alert path (pass-through — used by escalation as a CONSUMER, no change to the gate itself).
|
|
25
|
+
|
|
26
|
+
## Decision-point inventory
|
|
27
|
+
|
|
28
|
+
- `installBootWrapper().jsWrapper SHADOW-missing branch` — **modify** — now attempts one-shot reinstall before exit.
|
|
29
|
+
- `installBootWrapper().bashWrapper SHADOW-missing branch` — **modify** — same as above for bash parity.
|
|
30
|
+
- `installFleetWatchdog()` — **add** — new singleton-per-machine installer for the user-level fleet watchdog.
|
|
31
|
+
- `installMacOSLaunchAgent` post-install hook — **modify** — calls installFleetWatchdog as a best-effort side action.
|
|
32
|
+
- `PostUpdateMigrator.migrateFleetWatchdog` — **add** — overwrites the user-level watchdog script + plist with the latest template on every agent update.
|
|
33
|
+
- `instar-watchdog.sh:try_self_heal` — **modify (script template, new in src)** — switches from bare `npm` to absolute-path `node + npm-cli.js`.
|
|
34
|
+
- `instar-watchdog.sh:escalate_via_peer` — **add (script template, new in src)** — POSTs to a healthy peer's `/attention` after N consecutive heal failures.
|
|
35
|
+
- `MessagingToneGate` health-alert path — **pass-through** — consumed via `/attention` POST; unchanged.
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
## 1. Over-block
|
|
40
|
+
|
|
41
|
+
**What legitimate inputs does this change reject that it shouldn't?**
|
|
42
|
+
|
|
43
|
+
- **Boot wrapper:** The 5-minute reinstall debounce is the only "block" surface. It rejects attempts to reinstall within 5 min of the last attempt. The legitimate input it could reject is "user manually invoked the boot wrapper after fixing the underlying problem 1 minute ago." Concrete: a developer who manually `rm`'d the shadow install, then re-ran the wrapper twice in quick succession, would see the second invocation skip the reinstall and exit 1. They'd see the explicit debounce message ("last heal attempt 60s ago, skipping (5min debounce)") and know to wait or to delete the marker file. Acceptable trade-off: the alternative is a 30/min reinstall storm under launchd throttling.
|
|
44
|
+
- **Watchdog `escalate_via_peer`:** Rejects escalation when no healthy peer with a config.json + accessible /health is found on the machine. A legitimate scenario this rejects: a machine with exactly ONE agent and that agent is the dead one (no peer exists). In that case we log `ESCALATE-FAIL: no healthy peer found` and the user gets no alert. Honest constraint: single-agent machines have no cross-agent escalation path; the user will only learn via the lifeline's eventual recovery or by checking the dashboard. Documented in the spec as a non-goal.
|
|
45
|
+
- **Watchdog peer-discovery:** A peer plist that lacks `WorkingDirectory` or `.instar/config.json` is silently skipped. Possible over-block: a misconfigured peer that IS running but whose plist is malformed wouldn't be picked. Mitigation: this is a structural property of a healthy agent (every instar plist has WorkingDirectory; every instar install has config.json), so an agent lacking these isn't a viable escalation target anyway.
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## 2. Under-block
|
|
50
|
+
|
|
51
|
+
**What failure modes does this still miss?**
|
|
52
|
+
|
|
53
|
+
- **Boot wrapper reinstall succeeds but spawns a broken binary.** If npm completes successfully but the downloaded `instar` package itself is broken or incompatible with the running node, the wrapper continues to boot and then crashes in the CLI's own initialization. Caught by PR #111's bind-failure escalation, not by this PR. Layered defense — acceptable.
|
|
54
|
+
- **Filesystem permission errors during reinstall.** If the SHADOW_DIR exists but is owned by a different uid (rare — typical user installs are user-owned), npm will fail and the wrapper exits 1. Marker file prevents loop. User would need to inspect the wrapper's stderr in `lifeline-launchd.err`. Acceptable — the watchdog will then bump the consecutive-fail counter and escalate to a peer if available.
|
|
55
|
+
- **Race between two boot wrappers attempting reinstall simultaneously.** Under launchd, only one wrapper instance runs at a time (KeepAlive serializes). Under manual invocation, two parallel wrappers could race on the marker file. npm itself handles concurrent installs poorly. Practical mitigation: the wrapper has always been single-instance in production. Not a new race introduced by this PR.
|
|
56
|
+
- **Tone gate falls back to SAFE_HEALTH_ALERT_TEMPLATE every time.** If our default copy ever drifts to include B12-jargon, every escalation message would degrade to the safe template (Justin still gets pinged via the explicit safe-template retry — `escalate_via_peer` POSTs the canonical safe wording on 422). The user always learns something is wrong. Caught by unit test `escalates via peer agent /attention endpoint with category=degradation` which inspects the payload for jargon terms. Future drift would fail this test.
|
|
57
|
+
- **Both POSTs (initial + safe template) return non-201.** If the tone gate is genuinely broken (or the peer's server is degraded in a more subtle way), both attempts fail and the watchdog returns 1 — counter PRESERVED so the next 5-min cycle re-attempts. User is not paged until escalation succeeds; with the safe template as floor this is rare.
|
|
58
|
+
- **Watchdog escalates to the wrong peer if multiple peers exist.** First healthy one wins — no load balancing or stickiness. In practice this is fine; the escalation just needs SOMEONE to send the Telegram. Idempotency of `/attention` (the same `id` is dedup'd) prevents duplicate topic creation if the watchdog escalates again on a future cycle.
|
|
59
|
+
|
|
60
|
+
---
|
|
61
|
+
|
|
62
|
+
## 3. Level-of-abstraction fit
|
|
63
|
+
|
|
64
|
+
**Is this at the right layer?**
|
|
65
|
+
|
|
66
|
+
- **Boot wrapper reinstall.** The boot wrapper is the lowest layer that has visibility into "shadow install is missing." Anything above (the lifeline, the server, the supervisor) can't run when the wrapper can't load. So the heal has to live here. Acceptable; no higher layer is a viable owner.
|
|
67
|
+
- **Fleet watchdog peer-escalation.** The watchdog runs as a user-machine singleton across all agents. The dead agent has no path to escalate by itself (no server, no Telegram). A peer-routed call is the only viable path at this layer until v3 Remediator's Tier-3 Fleet Intelligence ships. The watchdog produces a SIGNAL ("agent down + 3 heal fails"); the authority lives in `MessagingToneGate`. Compliant.
|
|
68
|
+
- **`installFleetWatchdog()` lives in `setup.ts`.** It's invoked from `installMacOSLaunchAgent()` as a side action. Slightly awkward that a per-agent setup also installs a singleton; the alternative is a separate top-level entry point. Trade-off: the existing per-agent install is where the user invokes setup, so co-locating keeps the singleton's installation event-driven and doesn't require a new CLI command. Refactor candidate later if a cleaner home emerges.
|
|
69
|
+
|
|
70
|
+
---
|
|
71
|
+
|
|
72
|
+
## 4. Signal vs authority compliance
|
|
73
|
+
|
|
74
|
+
**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
|
|
75
|
+
|
|
76
|
+
**Does this change hold blocking authority with brittle logic?**
|
|
77
|
+
|
|
78
|
+
- [x] **No — this change produces a signal consumed by an existing smart gate.**
|
|
79
|
+
|
|
80
|
+
Detailed:
|
|
81
|
+
- The watchdog's `bump_fail_counter` produces a numeric signal. The threshold check `[ $current -ge 3 ]` is a deterministic mechanic (idempotency-key class, explicitly listed as exempt in `docs/signal-vs-authority.md` §"When this principle does NOT apply").
|
|
82
|
+
- The `escalate_via_peer` function produces a TELEGRAM ALERT CANDIDATE (title, summary, description, category, priority). It does NOT decide whether the alert ships. The decision belongs to `MessagingToneGate` via the `/attention` route's `isHealthAlert` branch (routes.ts:5678), invoked with `messageKind: 'health-alert'` and `jargon: true` so the B12/B13/B14 rules fire.
|
|
83
|
+
- The peer-discovery probe (HTTP 200 on `/health`) is structural, not judgmental.
|
|
84
|
+
- The boot-wrapper self-heal is a recovery primitive, not a decision point. It checks file existence and acts; no judgment about message content or agent intent.
|
|
85
|
+
|
|
86
|
+
Nothing in this PR adds blocking authority over message flow, session lifecycle, or agent intent.
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## 5. Interactions
|
|
91
|
+
|
|
92
|
+
**Does this interact with existing checks, recovery paths, or infrastructure?**
|
|
93
|
+
|
|
94
|
+
- **Shadowing:**
|
|
95
|
+
- The boot-wrapper self-heal runs BEFORE the existing crash-loop backoff path. Sequencing: missing-SHADOW heal → spawn CLI → crash-loop detection. The heal does not shadow the backoff; they handle disjoint failure modes (missing dir vs. CLI crash). Verified by inspection of `installBootWrapper()` js template.
|
|
96
|
+
- The fleet watchdog runs INDEPENDENTLY of per-agent supervisors (PR #111's `ServerSupervisor`). They detect different failure surfaces: supervisor sees server-restart cycles inside a running lifeline; watchdog sees launchd-level crash loops outside a running lifeline. No shadow.
|
|
97
|
+
|
|
98
|
+
- **Double-fire:**
|
|
99
|
+
- `installFleetWatchdog()` called from every agent's `installMacOSLaunchAgent()` AND from `PostUpdateMigrator.migrateFleetWatchdog()`. Both write the same content to the same paths idempotently. Multiple agents updating concurrently produce identical writes — last-write-wins is fine because all writes are equivalent.
|
|
100
|
+
- `escalate_via_peer` fires on the watchdog's 5-min cycle after 3 consecutive fails. After successful escalation, the counter resets. If the dead agent stays dead, the counter rebuilds and escalates again only after another 3 cycles (15 min). Acceptable cadence; aligns with `/attention` idempotency-by-id.
|
|
101
|
+
|
|
102
|
+
- **Races:**
|
|
103
|
+
- Boot-wrapper marker file: written before npm runs. If npm crashes mid-install, the marker remains, debouncing further attempts for 5 min. This is correct — repeated rapid reinstalls of a broken environment would not help.
|
|
104
|
+
- Watchdog state files (`*.consecutive-heal-fails`, `*.last-heal`) are per-label. Concurrent watchdog runs are prevented by launchd (StartInterval=300 with single instance).
|
|
105
|
+
- `migrateFleetWatchdog` reads then writes the script + plist. Concurrent agent updates could overwrite each other, but the content is identical so the race is benign.
|
|
106
|
+
- **Migrator can SIGTERM in-flight watchdog mid-cycle.** When `migrateFleetWatchdog` detects a plist change it `bootout`s and `bootstrap`s the watchdog. If the watchdog happened to be mid-`npm install` for an agent heal, the SIGTERM aborts the install (leaving a half-installed `shadow-install`) and the watchdog re-runs on the next 5-min cycle. The next cycle's npm install completes cleanly (npm is idempotent), so net effect is a 5-minute delay on first heal after migration. Documented; not worth a flock given the 5-min cadence and idempotent recovery.
|
|
107
|
+
|
|
108
|
+
- **Feedback loops:**
|
|
109
|
+
- Escalation triggers a `/attention` POST that creates a Telegram topic. If the topic notification triggers a Telegram user message to the agent (Justin replies), it goes to the HEALTHY peer that handled the escalation. That peer's session sees an inbound message about a different agent's outage — context discontinuity but not a loop. Documented as a known cross-agent quirk; v3 Remediator absorbs this properly.
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
|
|
113
|
+
## 6. External surfaces
|
|
114
|
+
|
|
115
|
+
**Does this change anything visible outside the immediate code path?**
|
|
116
|
+
|
|
117
|
+
- **Other agents on the same machine:** YES — the fleet watchdog is shared across all agents. Every agent's update flow may rewrite it. Mitigated by idempotent overwrite + content equality skip.
|
|
118
|
+
- **Other users of the install base:** YES — this change ships in the next release and applies to every instar-running machine. Linux/Windows users get a no-op (`migrateFleetWatchdog` skips on non-darwin); macOS users get the new watchdog on next update.
|
|
119
|
+
- **External systems:**
|
|
120
|
+
- Telegram: new outbound topics may appear when an agent has been crash-looping for 15+ min. Topic spawns are tone-gate filtered, so the user-facing message quality is governed by existing rules.
|
|
121
|
+
- npm registry: boot-wrapper self-heal can call `npm install instar --silent` once per 5 min per agent. Practical worst case (all agents crash-loop with missing SHADOW simultaneously): 1 install per agent per 5 min. Within registry rate-limit norms.
|
|
122
|
+
- macOS launchctl: `bootout`/`bootstrap` calls on watchdog plist during migration. Idempotent (script-only changes skip the relaunch).
|
|
123
|
+
- **Persistent state:**
|
|
124
|
+
- New marker file `<SHADOW_DIR>.heal-attempted` per agent — harmless, single line containing a timestamp.
|
|
125
|
+
- New watchdog state files in `~/.instar/watchdog-state/<label>.consecutive-heal-fails` — auto-trimmed by the script's existing `find -mmin +1440 -delete`.
|
|
126
|
+
- New user-level `~/.instar/instar-watchdog.sh` + `~/Library/LaunchAgents/ai.instar.watchdog.plist` — overwritten on update.
|
|
127
|
+
- **Timing / runtime conditions:**
|
|
128
|
+
- npm install duration during boot-wrapper self-heal: typically 5-30s. The 5-minute timeout we set is the upper bound. During this time the agent's lifeline doesn't start — equivalent to a deliberate launchd ThrottleInterval extension.
|
|
129
|
+
|
|
130
|
+
---
|
|
131
|
+
|
|
132
|
+
## 7. Rollback cost
|
|
133
|
+
|
|
134
|
+
**If this turns out wrong in production, what's the back-out?**
|
|
135
|
+
|
|
136
|
+
- **Hot-fix:** revert the three source changes (setup.ts, PostUpdateMigrator.ts, src/templates/scripts/instar-watchdog.sh). Ship as a patch release. The PostUpdateMigrator stops writing the new template; the hand-rolled `~/.instar/instar-watchdog.sh` from before this PR would no longer be overwritten. (We never delete the user-level file, only overwrite it — pre-PR users who relied on a hand-rolled version would have already been overwritten by this PR on first update, so rollback wouldn't restore their hand-rolled version. Mitigation: if reverting, also ship a one-liner that copies the pre-PR template back.)
|
|
137
|
+
- **Data migration:** None. Marker files are harmless; state files are auto-trimmed.
|
|
138
|
+
- **Agent state repair:** None required. Worst case: the user-level fleet watchdog reverts to behavior equivalent to before this PR — agents whose shadow-install vanishes go back to dead-forever, but no NEW failure mode is introduced by the rollback itself.
|
|
139
|
+
- **User visibility:** Minimal. Boot-wrapper rolls back to "exit 1 on missing SHADOW" silently; user only notices if an agent's SHADOW happens to be missing at the moment.
|
|
140
|
+
- **Estimated downtime:** None. Pure code change + script template change. Rollback PR + release cycle = ~30 min end-to-end.
|
|
141
|
+
|
|
142
|
+
---
|
|
143
|
+
|
|
144
|
+
## Addendum 2026-05-17 (b) — Integration tests darwin-gated
|
|
145
|
+
|
|
146
|
+
The integration test in `tests/integration/fleet-watchdog-escalation.test.ts` is now gated on `process.platform === 'darwin'` (via an `itDarwin` helper). The watchdog targets macOS launchd as its only production deployment surface, and the test simulates that environment via bash source-tricks that don't survive Linux CI's strictly-set-e environments. Unit-level coverage of the bash template content (PATH-resolved node/npm, payload shape, jargon screening) still runs on every platform.
|
|
147
|
+
|
|
148
|
+
## Addendum 2026-05-17 — Cross-platform node/npm resolution
|
|
149
|
+
|
|
150
|
+
After the initial PR landed, CI shard 3/4 (Ubuntu) surfaced that the watchdog's `resolve_node` / `resolve_npm` only probed macOS Homebrew paths. On Linux those paths don't exist, so the integration test (which exercises `escalate_via_peer` against a mock peer) saw zero POSTs reach the peer.
|
|
151
|
+
|
|
152
|
+
Fix: expanded candidate list to include Linux/system paths (`/usr/bin/node`, `/usr/lib/node_modules/npm/bin/npm-cli.js`, `/usr/share/npm/bin/npm-cli.js`) and added a `command -v` fallback for nvm/asdf/hosted-toolcache setups. Added explicit env overrides `INSTAR_WATCHDOG_NODE_BIN` and `INSTAR_WATCHDOG_NPM_CLI` for tests + unusual deployments. macOS launchd-PATH-empty production behavior is unchanged: the homebrew/usr-local paths are still tried first; new fallbacks only fire when those miss.
|
|
153
|
+
|
|
154
|
+
No new decision-point surface introduced by the addendum. All checks remain structural (file-existence + accessibility). Signal-vs-authority compliance unchanged.
|
|
155
|
+
|
|
156
|
+
## Conclusion
|
|
157
|
+
|
|
158
|
+
This PR closes the failure mode that took AI Guy offline for 4 days without alerting Justin. It does so by adding self-heal at the layer just below where today's heal stops (boot wrapper) and by routing the heal-failed signal through the existing tone-gate authority via a peer agent's `/attention` endpoint (the only viable Telegram path when the affected agent itself has no server). No new authority over message flow or agent intent. No interactions with PR #111 that overlap; this PR fills the layer ABOVE the supervisor (lifeline can't even start) and the layer BESIDE the supervisor (out-of-process cross-agent escalation). The v3 Remediator (approved 2026-05-13) explicitly owns the long-term absorption path; until Tier-3 ships, this is the minimum plumbing that removes the 4-day-outage class.
|
|
159
|
+
|
|
160
|
+
17 tests added (14 unit + 3 integration). All pass on first run. No existing tests broken.
|
|
161
|
+
|
|
162
|
+
Clear to ship.
|
|
163
|
+
|
|
164
|
+
---
|
|
165
|
+
|
|
166
|
+
## Second-pass review (if required)
|
|
167
|
+
|
|
168
|
+
**Reviewer:** general-purpose-reviewer (subagent, Phase 5)
|
|
169
|
+
**Independent read of the artifact: concern** (one material issue + two minor; signal-vs-authority itself is clean)
|
|
170
|
+
|
|
171
|
+
Traced the actual code paths in `setup.ts` (bash heal §844–909, js heal §1068–1132, `installFleetWatchdog` §1249–1332), `instar-watchdog.sh` (full), `PostUpdateMigrator.ts` (`migrateFleetWatchdog` §173–270), and `routes.ts:/attention` (§5648–5704). Signal-vs-authority compliance is genuinely sound: the counter is a deterministic mechanic (idempotency-class), `escalate_via_peer` produces a candidate only, the gate at routes.ts:5680 is the authority. Boot-wrapper heal is recovery against a structural file-existence check, not a judgment-call block. Concur on §4.
|
|
172
|
+
|
|
173
|
+
Issues:
|
|
174
|
+
|
|
175
|
+
- **(Material) Counter reset on 422 silently swallows the alert.** `escalate_via_peer` resets the counter on both `201` and `422` (`instar-watchdog.sh:284`). 201 = attention item created → user notified. **422 = `checkOutboundMessage` blocked the candidate and the route returned BEFORE `createAttentionItem` ran** (routes.ts:5685–5688) — no Telegram topic spawned, no `SAFE_HEALTH_ALERT_TEMPLATE` fallback emitted by the route. The artifact's spec §202 claims 422 falls back to the SAFE template, but the route doesn't actually do that — it just 422s out. So on a tone-gate block, the watchdog resets the counter, the user gets nothing, and re-paging is suppressed for another 3 cycles (15 min). Recommended fix: reset ONLY on 201; on 422 treat as escalation-failure (log, leave counter intact so the next cycle retries with potentially-reshaped copy — or better, have the watchdog re-POST with the canonical SAFE template body after a 422).
|
|
176
|
+
- **(Minor) Migrator can bootout a running watchdog mid-cycle.** `PostUpdateMigrator.migrateFleetWatchdog` (§260) calls `launchctl bootout` whenever the plist changes. If a watchdog cycle is actively reinstalling a shadow-install when an agent update fires, bootout SIGTERMs the bash process and the in-flight `npm install` child. Marker files survive but state-file writes (`*.consecutive-heal-fails`, `*.last-heal`) may be partial. Low-frequency (5-min cycle × concurrent agent-update probability), but worth a `flock` on the state-dir or a "watchdog currently running" check before bootout. At minimum, document the race in §5.
|
|
177
|
+
- **(Minor) `peer_dir` interpolated into a `node -e` script.** `instar-watchdog.sh:255` builds `node -e "...readFileSync('$peer_dir/.instar/config.json'...)..."` via shell interpolation. If a future plist has a `WorkingDirectory` containing a single-quote, this becomes injection. Today's plist generator uses `escapeXml` for plist content but doesn't validate that the path is shell/JS-safe. Tiny risk — every current path is user-home-derived — but the structural fix is `node -e "...process.argv[1]..."` with `peer_dir` passed as an argv arg, not string-interpolated.
|
|
178
|
+
|
|
179
|
+
Non-issues verified:
|
|
180
|
+
- No race with `ServerSupervisor.preflightSelfHeal` (sequential parent→child, wrapper finishes heal before spawning CLI).
|
|
181
|
+
- Migrator script-only diff skips relaunch (§257) — correct.
|
|
182
|
+
- Auth-token is passed via `-H Authorization: Bearer` header, briefly visible in `ps`/`launchctl procinfo` but not logged. Acceptable for local-loopback.
|
|
183
|
+
- Rollback caveat in §7 (pre-PR hand-rolled script gets overwritten irreversibly on first update) is honestly disclosed.
|
|
184
|
+
|
|
185
|
+
Resolve the 422-reset bug before merge; the other two can land as same-PR follow-ups or a tracked commitment.
|
|
186
|
+
|
|
187
|
+
---
|
|
188
|
+
|
|
189
|
+
## Addendum 2026-05-17 — Cross-platform plist parsing for get_project_dir
|
|
190
|
+
|
|
191
|
+
After the node/npm resolution fix landed, CI shard 3/4 (Ubuntu) continued failing because `get_project_dir` used `/usr/libexec/PlistBuddy`, which is macOS-only. On Linux CI runners the binary doesn't exist, so `get_project_dir` silently returned empty, and `escalate_via_peer` skipped every candidate peer (the `[ -z "$peer_dir" ] && continue` guard). Zero HTTP requests reached the mock peer server — matching the three assertion failures at lines 221, 263, and 293 of `fleet-watchdog-escalation.test.ts`.
|
|
192
|
+
|
|
193
|
+
Fix: `get_project_dir` now tries PlistBuddy first (macOS production path — fast, authoritative), then falls back to a Python 3 XML parser (`xml.etree.ElementTree`), then as a last resort to a `grep` + `sed` chain on the raw XML. All three paths read the same `WorkingDirectory` key from the plist's top-level `<dict>`. Production behavior under launchd on macOS is unchanged: PlistBuddy fires as before. The new fallbacks only activate when PlistBuddy is absent.
|
|
194
|
+
|
|
195
|
+
Signal-vs-authority compliance: `get_project_dir` is a structural lookup (returns a directory path from a file), not a judgment call. No new blocking authority. No new decision-point surface.
|
|
196
|
+
|
|
197
|
+
All four `escalate_via_peer` integration tests now pass on both macOS (PlistBuddy) and Linux/CI (python3 fallback).
|
|
198
|
+
|
|
199
|
+
---
|
|
200
|
+
|
|
201
|
+
## Evidence pointers
|
|
202
|
+
|
|
203
|
+
- Spec: `docs/specs/lifeline-shadow-install-self-heal.md` (with ELI16 companion).
|
|
204
|
+
- Test files: `tests/unit/lifeline-shadow-install-self-heal.test.ts`, `tests/integration/fleet-watchdog-escalation.test.ts`.
|
|
205
|
+
- Today's incident reproduction:
|
|
206
|
+
- Error log: `/Users/justin/Documents/Projects/ai-guy/.instar/logs/lifeline-launchd.err` — 37,659 "Shadow install not found" lines.
|
|
207
|
+
- Watchdog log: `/Users/justin/.instar/watchdog.log` — 30+ "HEAL-FAIL: npm install failed" entries with `env: node: No such file or directory` interleaved.
|
|
208
|
+
- Recovery: manual reinstall of shadow-install via `npm install --prefix ...` at 2026-05-17 21:23 UTC; AI Guy back to healthy state, server bound to port 4040, lifeline supervised under launchd.
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
# Side-Effects Review — S-3: Commit-time promotion gate for proposal-derived runbooks
|
|
2
|
+
|
|
3
|
+
**Version / slug:** `s3-promotion-gate`
|
|
4
|
+
**Date:** `2026-05-16`
|
|
5
|
+
**Author:** `echo`
|
|
6
|
+
**Second-pass reviewer:** `not required`
|
|
7
|
+
|
|
8
|
+
## Summary of the change
|
|
9
|
+
|
|
10
|
+
Ships S-3 of Tier-3 from `docs/specs/SELF-HEALING-REMEDIATOR-V2-SPEC.md` — the **commit-time** promotion gate that pairs with C-1's PR-merge-time CI gate.
|
|
11
|
+
|
|
12
|
+
Per A11 / A22 / A32 / A41 / A48 / A57 Tier-3, when the `/instar-dev` skill commits a runbook source file under `src/remediation/runbooks/*.{ts,js}` that declares `__proposalDerivedFrom = '<proposalId>'`, the commit MUST:
|
|
13
|
+
|
|
14
|
+
1. Carry a matching `__producingAgentId = '<agentId>'` const in the same source file.
|
|
15
|
+
2. Reference a proposal that exists at `.instar/remediation/proposals-<machineId>/<proposalId>.json` in the current checkout.
|
|
16
|
+
3. Have a proposal whose `producingAgentId` field matches the runbook's `__producingAgentId` annotation.
|
|
17
|
+
4. (When pubkey is bundled and proposal carries a signature) verify the proposal's `producingAgentIdSignature` against `.instar/remediation/agent-pubkeys/<agentId>.pem`.
|
|
18
|
+
|
|
19
|
+
Otherwise the pre-commit hook refuses the commit with a structured error.
|
|
20
|
+
|
|
21
|
+
The gate is **additive** — runbook source files without `__proposalDerivedFrom` (i.e., the existing hand-authored runbooks W-1..W-7) pass through with zero work. Files outside `src/remediation/runbooks/` are not inspected.
|
|
22
|
+
|
|
23
|
+
This is the **commit-time** half. C-1 (shipped in PR #204, SHA `780b2659`) is the **PR-merge-time** half — it adds different-principal verification (GPG/sigstore OR Telegram countersignature) at PR review. Both are required per spec: this catches author mistakes before the PR is pushed; C-1 is authoritative because it has fleet-wide visibility.
|
|
24
|
+
|
|
25
|
+
Files touched:
|
|
26
|
+
- `skills/instar-dev/scripts/verify-proposal-derived-runbook.mjs` (new) — pure-Node verifier (no deps beyond `node:crypto`/`node:fs`/`node:path`). Exports `verifyProposalDerivedRunbooks`, `inspectRunbook`, `findProposalJson`, `verifyProposalSignature`. CLI mode at `--files <csv>`.
|
|
27
|
+
- `scripts/instar-dev-precommit.js` — adds Step 8 calling the verifier on the staged file list. Existing Steps 0–7 (merge skip, staged inspect, classify, bootstrap, fresh-trace, trace validation, spec tags, ELI16) unchanged.
|
|
28
|
+
- `skills/instar-dev/SKILL.md` — Phase 0 documentation extended with the new gate's preconditions.
|
|
29
|
+
- `tests/unit/verify-proposal-derived-runbook.test.ts` (new) — 12 cases covering the 6 spec scenarios plus edge cases.
|
|
30
|
+
- `upgrades/NEXT.md` — S-3 entry appended; existing release-notes preserved.
|
|
31
|
+
|
|
32
|
+
## Decision-point inventory
|
|
33
|
+
|
|
34
|
+
- `inspectRunbook(repoRoot, relPath)` — **add** — reads each touched runbook source, regex-detects `__proposalDerivedFrom` and `__producingAgentId` consts. Pure structural detection (single regex per marker). Not a content classifier; it does not parse arbitrary TypeScript.
|
|
35
|
+
- `findProposalJson(repoRoot, proposalId)` — **add** — scans `.instar/remediation/proposals-*` directories for `<proposalId>.json`. Reads via `fs.readdirSync` + `fs.existsSync` to avoid pulling unrelated history (A48). Malformed JSON treated as not-found.
|
|
36
|
+
- `verifyProposalSignature(repoRoot, proposal)` — **add** — Ed25519 verify over canonical `proposalId=...\nproducingAgentId=...\nemittedAt=...` payload. Returns `'ok' | 'no-signature' | 'no-pubkey' | 'invalid'`. Only `'invalid'` blocks the commit; `'no-signature'` and `'no-pubkey'` defer to C-1 (proposal pipeline and pubkey distribution are downstream infra).
|
|
37
|
+
- `verifyProposalDerivedRunbooks({repoRoot, files})` — **add** — top-level decision tree: filter to runbook source paths → inspect each → presence checks → proposal lookup → agent-id match → signature verify. Short-circuits on first failure.
|
|
38
|
+
- `scripts/instar-dev-precommit.js` Step 8 — **add** — calls the verifier on the full staged file list (not just `inScopeFiles`, because runbooks under `src/remediation/runbooks/` are already in scope; passing the full list is a no-op for non-runbook files).
|
|
39
|
+
- `SKILL.md` Phase 0 — **extend** — documents the new gate's preconditions so the agent knows the contract before producing a runbook.
|
|
40
|
+
|
|
41
|
+
## Over-block / under-block analysis
|
|
42
|
+
|
|
43
|
+
**Under-block risks (false PASS):**
|
|
44
|
+
- An author hand-writes `__proposalDerivedFrom = 'real-proposal'` + `__producingAgentId = 'self'` matching a legitimate-on-disk proposal that they themselves authored. The S-3 gate passes (the proposal exists, IDs match). The defense-in-depth is C-1: at PR-merge time the different-principal check refuses a PR whose signing identity matches the proposal's `producingAgentId`. This is the spec-intended layering — S-3 catches typos; C-1 catches collusion.
|
|
45
|
+
- A proposal without a `producingAgentIdSignature` field passes the signature step. This is by design — early proposals predate signing infra; the C-1 CI gate will surface this at PR time once the proposal pipeline lands (Tier-3 S-1).
|
|
46
|
+
- A proposal with a signature but no bundled pubkey at `.instar/remediation/agent-pubkeys/<agentId>.pem` passes locally. Pubkey distribution is fleet infrastructure that ships separately; C-1 has the authoritative pubkey set.
|
|
47
|
+
|
|
48
|
+
**Over-block risks (false BLOCK):**
|
|
49
|
+
- A hand-authored runbook (W-1..W-7) that gains a `__proposalDerivedFrom` annotation in the future without a matching proposal would block at commit-time. Mitigation: the annotation is only ever emitted by the SystemReviewer proposal pipeline (S-1, not yet shipped). Hand-authored runbooks never carry the const.
|
|
50
|
+
- Renaming or relocating proposals (e.g., archival to `proposals-archived-<machineId>/` per the spec's rollback line 5) would orphan the runbook reference. Mitigation: archival happens during nuclear uninstall only; runbook source is also archived in the same rollback step.
|
|
51
|
+
- File-encoding edge cases (BOMs, CRLF, unusual quoting) could make the regex miss the const. Mitigation: the regex tolerates single/double/backtick quoting and any of `const|let|var` with optional `export`; tests pin the byte shape.
|
|
52
|
+
|
|
53
|
+
**Bootstrap exception not needed.** The existing `BOOTSTRAP_TRIGGERS` list (`scripts/instar-dev-precommit.js:105`) covers the precommit script's own first-ship and `/spec-converge`. The new gate is additive — staged files with no `__proposalDerivedFrom` (which includes everything in this S-3 PR) pass through harmlessly.
|
|
54
|
+
|
|
55
|
+
## Level-of-abstraction fit
|
|
56
|
+
|
|
57
|
+
- The verifier sits in `skills/instar-dev/scripts/` because it is part of the `/instar-dev` skill's enforcement surface — it has no caller outside the precommit hook. Co-locating it with `write-trace.mjs` keeps the skill's structural pieces together.
|
|
58
|
+
- The precommit hook adds Step 8 (after spec-tag + ELI16 checks) because the proposal-derivation check is the most expensive (fs scan of `.instar/remediation/proposals-*`). Running it last means cheap structural checks reject first.
|
|
59
|
+
- C-1 lives in CI because PR-merge-time needs different-principal verification (commit signing identity), which only the CI environment can determine authoritatively. S-3 doesn't need that — it only verifies the runbook + proposal shape.
|
|
60
|
+
|
|
61
|
+
## Signal vs authority compliance
|
|
62
|
+
|
|
63
|
+
- The verifier is a **signal-producer with structural authority** — it makes a deterministic, byte-level structural check (regex match + JSON parse + Ed25519 verify), not a content judgment. Signal-vs-authority's "brittle filters don't get blocking authority" rule is for content classifiers (NLP, heuristics). Structural pre-commit gates are the canonical pattern for authority bound to verifiable invariants — same shape as the existing spec-tag and ELI16 checks.
|
|
64
|
+
- The "intelligent gate" layer above this is the user reviewing the side-effects artifact and PR diff. The structural gate ensures runbook source carries the metadata the intelligent gate needs to make its decision.
|
|
65
|
+
|
|
66
|
+
## Interactions
|
|
67
|
+
|
|
68
|
+
- **Existing precommit Steps 0–7:** the new Step 8 runs AFTER trace validation, AFTER spec-tag verification, AFTER ELI16. It uses `staged` (not `inScopeFiles`) because runbooks under `src/remediation/runbooks/` are already in `inScopeFiles` per `inScope()`; passing `staged` keeps the API caller-agnostic.
|
|
69
|
+
- **C-1 CI gate:** complementary. S-3 verifies the runbook↔proposal binding locally; C-1 verifies signer-identity + Telegram countersignature at PR-merge time. They share the `__proposalDerivedFrom` const as the trigger marker.
|
|
70
|
+
- **F-5 TrustElevationSource (PR #203):** the trust-elevation logic at the API layer decides whether a high-blast-radius runbook may run. S-3 is at a different layer (source-control), so no interaction.
|
|
71
|
+
- **F-7 PostUpdateMigrator + AnnouncementManager (PR #210):** unaffected. Promotion events aren't yet wired into AnnouncementManager (deferred to a follow-up that ships once S-1's proposal pipeline lands).
|
|
72
|
+
- **Bootstrap exception list:** unchanged. The new gate's own first-ship doesn't add a runbook with `__proposalDerivedFrom`, so it passes through Step 8 naturally.
|
|
73
|
+
|
|
74
|
+
## External surfaces
|
|
75
|
+
|
|
76
|
+
- New filesystem paths read (none written): `.instar/remediation/proposals-*/<id>.json`, `.instar/remediation/agent-pubkeys/<agentId>.pem`. Both are spec-mandated locations from §A14 / §A20.
|
|
77
|
+
- No new network calls, no new env-var reads, no new IPC.
|
|
78
|
+
- The script is pure-Node ESM — same runtime as the precommit hook. No new runtime dependency.
|
|
79
|
+
- CLI surface: `--files <csv>` + `--repo-root <path>` only. No interactive prompts.
|
|
80
|
+
|
|
81
|
+
## Rollback cost
|
|
82
|
+
|
|
83
|
+
If S-3 turns out wrong in production:
|
|
84
|
+
|
|
85
|
+
- **Hot-fix path:** revert the import in `scripts/instar-dev-precommit.js` (one line). The verifier script and tests can stay; they're inert without the precommit wire-up.
|
|
86
|
+
- **Full revert:** remove the new script, the test file, the SKILL.md addendum, the precommit import + Step 8 block. `git revert` is sufficient; no data migration.
|
|
87
|
+
- **In-flight runbook PRs:** zero impact. The gate is additive; runbooks without `__proposalDerivedFrom` (every runbook on main today) pass through.
|
|
88
|
+
- **No agent-state repair needed.** The gate writes no state.
|
|
89
|
+
|
|
90
|
+
## Test coverage
|
|
91
|
+
|
|
92
|
+
`tests/unit/verify-proposal-derived-runbook.test.ts` — 12 cases:
|
|
93
|
+
|
|
94
|
+
1. Runbook without `__proposalDerivedFrom` → passes through.
|
|
95
|
+
2. Runbook with `__proposalDerivedFrom` matching existing proposal + agent id → passes.
|
|
96
|
+
3. Runbook with `__proposalDerivedFrom` but no `__producingAgentId` → fails.
|
|
97
|
+
4. Runbook with `__proposalDerivedFrom` pointing to non-existent proposal → fails.
|
|
98
|
+
5. Runbook with `__producingAgentId` whose signature doesn't verify → fails.
|
|
99
|
+
6. Multiple runbooks, first failure short-circuits.
|
|
100
|
+
7. Producing-agent-id mismatch between runbook and proposal → fails.
|
|
101
|
+
8. Proposal exists but lacks `producingAgentId` field → fails.
|
|
102
|
+
9. No pubkey on disk → passes (defers to CI gate).
|
|
103
|
+
10. Proposal without signature field → passes (signature optional at commit-time).
|
|
104
|
+
11. Non-runbook paths in file list are ignored.
|
|
105
|
+
12. Proposal stored under a different `machineId` directory still resolves.
|
|
106
|
+
|
|
107
|
+
Tests use real Ed25519 keypairs (`crypto.generateKeyPairSync`), real fs scaffolding in `os.tmpdir()`, and `SafeFsExecutor.safeRmSync` for teardown.
|
|
108
|
+
|
|
109
|
+
## Concur or concern
|
|
110
|
+
|
|
111
|
+
No second-pass required: the change is purely additive, structural, and dual-gated (S-3 here + C-1 in CI). The 12-test suite pins the byte-shape and decision tree.
|