instar 1.1.2 → 1.1.4

package/upgrades/side-effects/lifeline-version-skew-recovery.md

@@ -0,0 +1,180 @@
+# Side-Effects Review — Lifeline version-skew recovery
+
+**Version / slug:** `lifeline-version-skew-recovery`
+**Date:** `2026-05-20`
+**Author:** Echo (instar developer agent)
+**Trigger:** 2026-05-19 → 2026-05-20 b2lead-insights field incident (21h silent
+Telegram ingress drop after server auto-update from v1.0.13 → v1.1.0;
+lifeline stayed on v1.0.13; HTTP 426 from `/internal/telegram-forward`
+went to 3-attempts-then-drop replay policy; restart attempts blocked by
+cooldown; manual operator recovery required).
+
+## Summary of the change
+
+Five interlocking fixes that together close the silent-drop failure
+class. All independently revertable.
+
+1. **Rate-limit cooldown bypass for `versionSkew` bucket.** A
+   no-op-against-mismatch wait is replaced with allow-but-rate-limit
+   (daily cap still applies). One-line condition change in
+   `decide()`.
+
+2. **Drop policy guards on `versionSkewActive` instance flag.** Three
+   new state operations:
+   - `handleVersionSkew` sets `versionSkewActive = true` and fires
+     one user-visible alert (dedupe via `versionSkewAlertSentAt`).
+   - `forwardToServer` success clears the flag + dedupe timestamp.
+   - `replayQueue` re-queues without incrementing failures while
+     the flag is set.
+
+3. **CLI service label fix.** `com.instar.<name>.lifeline` →
+   `ai.instar.<name>`. Matches the plist label written by
+   `installMacOSLaunchAgent`. Stops the always-fall-back-to-pkill path.
+
+4. **pkill fallback escalates + lock-acquire recognizes wedged-S.**
+   CLI: SIGTERM, 3s grace, SIGKILL by pattern. Lock-acquire: detects
+   sleeping ('S') state with lock-write age > 5 min as a wedged
+   lifeline, sends its own SIGTERM → 3s poll → SIGKILL sequence.
+
+5. **`--build-from-source` + `--ignore-scripts` on every native
+   rebuild.** Two call sites: ServerSupervisor preflight rebuild
+   (line 751) and NativeModuleHealer.healBetterSqlite3Sync
+   (line 397). The Remediator-orchestrated path already had these
+   flags (NativeModuleHealer.healBetterSqlite3FromRemediator).
+
+Files touched (excluding tests / docs):
+- `src/lifeline/rateLimitState.ts` — `decide()` cooldown bypass
+- `src/lifeline/TelegramLifeline.ts` — instance state, `handleVersionSkew`,
+  `forwardToServer`, `replayQueue`, `acquireLockFile`
+- `src/cli.ts` — label fix + pkill escalation
+- `src/lifeline/ServerSupervisor.ts` — rebuild flags
+- `src/memory/NativeModuleHealer.ts` — rebuild flags
+
+## Decision matrix
+
+### Over-block
+
+| Scenario | Outcome |
+|----------|---------|
+| Lifeline at SAME version as server (no skew) | No change — `versionSkewActive` stays false |
+| Lifeline at OLD version, server unchanged (no auto-update) | No change — no 426, no skew |
+| Genuine transient 5xx | Drop-after-3 still fires (versionSkewActive is false) |
+| `ForwardServerBootError` (server starting) | Drop-after-3 still fires — known limitation; this PR doesn't try to also branch on serverBoot. Out of scope. Justified: serverBoot is short-lived (seconds), version skew is steady-state until restart. |
+| Lifeline restart loop after fix | Daily cap (3-per-24h) plus storm detection bound the loop |
+
+No over-block. The new authority only fires when ALL of:
+1. The forward threw `ForwardVersionSkewError`,
+2. The server version in the body differs from the lifeline's,
+3. The body's `upgradeRequired === true`.
+
+That's the same gate that already triggers the restart request, so we
+haven't widened the surface — just made the consequences during the
+window correct.
+
+### Under-block
+
+| Scenario | Outcome |
+|----------|---------|
+| Lifeline version "skews" because of a malformed 426 body | Guarded by existing `body.upgradeRequired !== true` and `serverVersion === this.lifelineVersion` checks — they were already in `handleVersionSkew` |
+| Lifeline tries to forge a skew to bypass cooldown | The 426 must come from the SERVER (loopback addr only on a privileged port); the lifeline cannot self-trigger the body shape |
+| Two version-skew episodes in <24h (daily cap reached) | Falls through to existing `version-skew-daily-cap` denial — observability path remains intact |
+| Stuck-lock detection: live healthy lifeline misidentified as wedged | Requires lock-age > 5 min AND 'S' state — a healthy lifeline writes the lock, immediately starts polling Telegram (lots of 'R' time), and answers heartbeats. Misidentification window is narrow but possible during long Telegram long-poll waits. Acceptable: false positive triggers SIGTERM → 3s grace → SIGKILL of a healthy lifeline → launchd respawns it within seconds. Worst case: ~5s downtime + a fresh boot. Far better than the 21h+ silent drop the current code permits. |
+| Forced rebuild with `--build-from-source` fails because node-gyp toolchain absent | Already-existing failure path: spawnSync returns non-zero, healer logs error, falls back to existing escalation. No new silent failure. |
+
+### Level of abstraction fit
+
+- Rate-limit bypass — at the decision-point that already knows about
+  buckets. Right layer.
+- Drop policy — at the replay loop that already has the failure-counter
+  state. Right layer. Could have lived in `forwardToServer` instead,
+  but the queue is what owns "should this message be dropped" — the
+  forward path only owns "did this single attempt succeed".
+- CLI label — at the kickstart callsite. One source of truth.
+- pkill escalation — at the CLI fallback. The CLI is the place where
+  "user typed `instar lifeline restart`" gets translated to a system
+  action.
+- Lock-acquire — at the lock-acquire function. The detector and the
+  recovery primitive are both bounded mechanics in the same scope.
+- Rebuild flags — at the rebuild callsites. No new abstraction.
+
+No over-engineering. Each change is at the layer that already owns
+the concern.
+
+### Signal-vs-authority compliance
+
+- `rateLimitState.decide` — returns signal. No authority change.
+- `versionSkewActive` flag — pure state signal. Replay loop is the
+  authority that decides re-queue vs drop.
+- `handleVersionSkew` — already had authority over "request a restart".
+  Now also: sets the flag and sends an alert. Both are bounded
+  mechanics with clear preconditions, not judgmental gates.
+- CLI label — mechanical change.
+- pkill escalation — bounded recovery (fixed 3s grace, single SIGKILL).
+- Lock-acquire stuck-S — bounded recovery (fixed 5min threshold for
+  age, 3s SIGTERM grace, single SIGKILL).
+- Rebuild flags — mechanical change.
+
+Compliant. No new judgmental gates; new authorities are bounded
+recovery primitives.
+
+### Interactions with adjacent systems
+
+| System | Interaction | Risk |
+|--------|-------------|------|
+| `MessageQueue` | Replay loop re-queues messages during skew (same enqueue path used for transient failures) | None — queue persistence semantics unchanged |
+| `notifyMessageDropped` | NOT called during version-skew episodes (drop is suppressed) | Strictly improves — these messages will deliver |
+| `DegradationReporter` | Still fires for non-skew drops; version-skew episode itself emits a degradation via the existing path | None |
+| `RestartOrchestrator` | Receives a `versionSkew` request that's now more likely to be granted (cooldown bypass) | Storm detection + daily cap remain as backstops |
+| `ServerSupervisor.preflightSelfHeal` | Rebuild flag change is a strict improvement | None |
+| Remediator (Tier 1+) | Not yet shipped; in-line healer is the steady-state path. When Remediator ships, its rebuild already used `--build-from-source` — no conflict | None |
+| `NativeModuleHealer.invokeFromRemediator` | Unchanged (already had the flag) | None |
+| `LifelineHealthWatchdog` | Tracks `consecutiveForwardFailures`; this PR clears it on success (existing behavior). No change to the watchdog's trip conditions | None |
+| `WatchdogTriggers.versionSkew` | The flag-set + alert-send happens BEFORE `initiateRestart`. If the restart is rate-limited, the alert still fires — good (user knows ingress is paused even while the lifeline is being patient about restarts) | None |
+| `acquireLockFile` callers | The added SIGTERM-then-SIGKILL path runs in the boot-startup window. New lifelines that find a wedged old one will spend up to 3s waiting for graceful exit. Acceptable | None |
+| Existing tests | All 18 `rateLimitState.test.ts` + 14 `lifeline-shadow-install-self-heal.test.ts` + 9 `NativeModuleHealer.test.ts` tests pass unchanged | Verified |
+
+### Rollback cost
+
+- Each fix is independently revertable.
+- No persistent state schema changes; the new lifeline instance
+  fields (`versionSkewActive`, `versionSkewAlertSentAt`) are not
+  serialized to disk.
+- No migration entry; existing in-the-wild agents pick up the fix on
+  their next `npx instar` update like any other patch.
+- Reverting the rebuild flag returns to the pre-existing flaky
+  rebuild behavior — agents that had drifted to "rebuild succeeded
+  but still fails" on the old code path continue exactly as they
+  did.
+
+## Acceptance criteria (test-mapped)
+
+1. `tests/unit/lifeline/rateLimitState.test.ts > versionSkew bucket
+   BYPASSES the cooldown` — asserts the new bypass.
+2. `tests/unit/lifeline/version-skew-recovery.test.ts > CLI service
+   label` — asserts `ai.instar.${...}` is in source and
+   `com.instar.${...}.lifeline` is gone.
+3. `tests/unit/lifeline/version-skew-recovery.test.ts > pkill
+   fallback escalates to SIGKILL` — asserts both signal calls
+   present in the restart command source.
+4. `tests/unit/lifeline/version-skew-recovery.test.ts >
+   ServerSupervisor preflight rebuild passes --build-from-source` —
+   asserts the rebuild argv on the supervisor side.
+5. `tests/unit/lifeline/version-skew-recovery.test.ts >
+   NativeModuleHealer in-line rebuild passes --build-from-source` —
+   asserts EVERY rebuild call in the healer source has the flag.
+6. `tests/unit/lifeline/version-skew-recovery.test.ts > lifeline
+   source guards drop with versionSkewActive` — asserts the source
+   order (skew bypass before drop check).
+7. `tests/unit/lifeline/version-skew-recovery.test.ts >
+   handleVersionSkew sets the active flag + alert dedupe`.
+8. `tests/unit/lifeline/version-skew-recovery.test.ts >
+   forwardToServer success clears the version-skew episode flag`.
+9. `tests/unit/lifeline/version-skew-recovery.test.ts > lock-acquire
+   treats sleeping (S) state > 5 min as recoverable`.
+
+## Forward note
+
+If the auto-update flow eventually grows a "tell the lifeline to
+restart" step, the failure class this PR addresses becomes
+impossible-by-construction. Until then, this is the steady-state
+recovery for the window between server-update and lifeline-restart.

package/upgrades/side-effects/test-env-isolation.md

@@ -0,0 +1,84 @@
+# Side-effects review — test-env-isolation
+
+Spec: `docs/specs/test-env-isolation.md`
+ELI16: `docs/specs/test-env-isolation.eli16.md`
+
+## Surface map
+
+| Layer | File(s) added / changed | Surface |
+|------|------------------------|---------|
+| 1 | `tests/vitest-setup.ts` | Process env at vitest startup. Every test inherits the strip. |
+| 2 | `tests/helpers/git-test-env.ts` (new); refactors to `tests/unit/worktree-monitor.test.ts`, `tests/unit/SafeGitExecutor.test.ts`, `tests/integration/rich-profile-integration.test.ts` | Three fixtures' git-spawn calls now pass an explicit sanitized env. |
+| 3 | `scripts/pre-push-fixture-guard.mjs` (new); `.husky/pre-push` (wire-in) | Pre-push hook gains a guard step that runs before the test suite. |
+| 4 | `scripts/check-repo-invariants.mjs` (new); `.github/workflows/ci.yml` (new `invariants` job) | Standalone CI job, no dependencies, runs in parallel with type check. |
+| 5 | Deletions: `file-0.txt`, `seed` | Repo root no longer carries fixture stowaways. |
+
+Tests: 14 new across `tests/unit/vitest-setup-git-env-strip.test.ts`, `tests/unit/scripts/pre-push-fixture-guard.test.ts`, `tests/unit/scripts/check-repo-invariants.test.ts`.
+
+## Over-block analysis
+
+**Layer 1 strip.** Could a legitimate test *need* the inherited `GIT_DIR`? In our codebase, no — every existing fixture that touched git either operates on a tmpdir (which doesn't need it) or already strips the vars locally. We grepped `tests/` for any consumer of `process.env.GIT_DIR` and found zero direct readers. The strip cannot regress a test that depends on the override, because no such test exists.
+
+If a future test needs to operate against the real repo (e.g., a meta-test that inspects the project's own git state), it can re-set the variable explicitly inside its own scope — the global strip removes the *inherited* value, not the *ability to set* one.
+
+**Layer 3 fixture-author guard.** Could it block a legitimate commit? Possible failure cases:
+
+- Author email of `test@test.com` is in the bad-author set. A real human who happens to use that email would be blocked. Mitigation: the email is in our internal fixture set; real contributors use their own emails. If a real case ever arises, `INSTAR_PRE_PUSH_FIXTURE_GUARD_SKIP=1` lets them through.
+- Subject line `init` matches the pattern. A real "init" commit from a new repo bootstrap would trip the guard. Mitigation: bootstrap commits live on their own branch and aren't pushed against an established `upstream/main`, so the guard's `<base>..HEAD` range returns empty for them. Tested in `pre-push-fixture-guard.test.ts`.
+
+**Layer 4 README line floor.** Could a legitimate README shrink below 100 lines? The current README is 271 lines and the project is growing, not shrinking. A deliberate slim-down would need `INSTAR_README_MIN_LINES` override, which is exactly the escape hatch.
+
+## Under-block analysis
+
+**Layers 1+2 don't catch process-spawning that re-injects the env.** If a future utility spawns `git` with an explicit `env: { GIT_DIR: '...' }`, layers 1+2 don't help — they only strip *inherited* env. Layer 3 catches the *consequence* (a pollution commit on the branch), so the failure class is still closed end-to-end, but layers 1+2 alone are not sufficient. This is acknowledged in the spec.
+
+**Layer 3 doesn't catch destructive changes that don't match the historical signature.** A future test fixture that wrote `# Different Stub` to README would land it if the commit message and author look normal. Layer 4 catches the resulting line-count regression; together, layers 3 and 4 cover both the "known-pattern pollution" and "novel pollution that drops content" failure modes.
+
+**Layer 4 doesn't enforce content correctness.** A README with 100+ lines of garbage would pass. We deliberately scoped Layer 4 to structural invariants only; content review is the job of human PR review.
+
+## Level-of-abstraction fit
+
+The fix lives at the right layer:
+
+- The root cause is in *test setup*, so the fix is in `vitest-setup.ts` — the central, single point that runs before every test. We did not push the strip down into individual fixtures (would require N changes and could be missed); we did not push it up into the test runner config (the var would still leak into test files that import their own helpers that ran first).
+- The push-time guard lives in `.husky/pre-push`, which is where push-time checks already live.
+- The CI invariant is a standalone job, not a step bolted onto an existing job that might be skipped under failure-routing rules.
+
+## Signal-vs-authority compliance
+
+Layer 3 is a brittle pattern matcher (authors + subjects) — it's a **signal**. It cannot block a sufficiently-disguised commit, and it doesn't try to. Authority for refusing a commit rests with the deterministic layers (1, 2, 4) and human review. The signal exists to make the *common* failure mode loud, not to be the load-bearing protection.
+
+The deterministic strip (Layer 1) is the load-bearing protection. The CI invariant (Layer 4) is the structural assertion. The signal (Layer 3) makes the failure visible early but is not the only line of defense.
+
+## Interactions with existing systems
+
+- **`pre-push-gate.js`** (NEXT.md / version check) runs first and remains unchanged.
+- **Working-tree integrity check** (post-test step in `ci.yml`) catches *uncommitted* mutations; the new invariants job catches *committed* state. Complementary, not duplicate.
+- **`lint-no-direct-destructive.js`** already requires `safe-git-allow:` on raw git use. Our new scripts carry it; the test files for those scripts carry it. No conflict.
+- **`PostUpdateMigrator`** is not affected — none of these changes touch agent-installed files. The Migration Parity Standard does not apply.
+
+## Rollback cost
+
+All five layers are net-additive. Each can be reverted independently:
+
+- Layer 1: revert five `delete` lines.
+- Layer 2: revert the helper file and three fixture edits.
+- Layer 3: comment out two lines in `.husky/pre-push` or delete the script.
+- Layer 4: comment out the `invariants` job in `ci.yml`.
+- Layer 5: re-create `file-0.txt` / `seed` — but they had no legitimate function, so no consumer breaks.
+
+No data migration. No schema change. No agent-state mutation. Reverting any layer leaves the repo in a valid state and the test suite green.
+
+## Risk summary
+
+- **Low risk of regression.** Layer 1 is a single-line process.env mutation in test setup that affects nothing in production code. Layers 2–5 are net-additive guards.
+- **Moderate risk of friction.** Layer 3 will be loud when triggered, including against legitimate edge cases (the documented bypass exists for this). Layer 4 will fail builds on cosmetic README slim-downs unless `INSTAR_README_MIN_LINES` is overridden.
+- **No risk of data loss.** Nothing in this change mutates state outside the worktree.
+
+## Verification done before commit
+
+- All targeted test files pass locally (45 in worktree-monitor + SafeGitExecutor, 14 in new tests).
+- `npm run lint` passes with the `safe-git-allow:` headers in place.
+- `node scripts/check-repo-invariants.mjs` against the prepared worktree returns clean.
+- `node scripts/pre-push-fixture-guard.mjs` against the prepared worktree returns clean.
+- Spec carries `approved: true` per direct principal authorization in Telegram topic 11235.