instar 1.3.102 → 1.3.104

package/upgrades/1.3.103.md

@@ -0,0 +1,77 @@
+---
+review-convergence: complete
+approved: true
+approved-by: justin (verbal, topic 2169: "yes! then we need to do a full, robust audit to fix this whole class of issues since it keeps coming up")
+---
+
+# Upgrade Guide — vNEXT
+
+<!-- bump: patch -->
+
+## What Changed
+
+**Fleet-wide fix for the silent-403 class.** Every shipped hook, script, Node
+helper, and migrator-emitted template that resolves the agent's `authToken`
+now survives the secret-externalization refactor. Two related bugs landed
+together because both produced the same user-visible symptom (a hook that
+stops emitting any output):
+
+- **Auth-token resolution: env-first with a string-type guard.**
+  `INSTAR_AUTH_TOKEN` (set by `SessionManager` per spawned session and by
+  `JobScheduler` per scheduled job) is checked first. The disk fallback now
+  guards `cfg.authToken` against non-string values, so the literal
+  `{ "secret": true }` placeholder produced by `SecretMigrator` after
+  multi-machine pairing can never leak through as a Bearer token. (Previously,
+  scripts sent `[object Object]` or `{'secret': True}` as the Authorization
+  header, the server 403'd, and the script's downstream Python parser
+  silently exited on the error JSON — no output at all.)
+
+- **Port-parse tolerates whitespace.** The `grep -o '"port":[0-9]*'` pattern
+  used in every hook required no whitespace between the colon and the
+  number, but our prettified `config.json` writes `"port": 4042` (with
+  space). The pattern silently produced empty `$PORT`, the hook exited
+  early, and not a single HTTP request reached the server. Replaced with
+  `grep -oE '"port"[[:space:]]*:[[:space:]]*[0-9]+'` + digit-only extraction.
+
+The structural cure includes a unit-tier lint that fails any future re-
+introduction of the broken pattern (positive + negative-verified) and a new
+migration pass (`migrateSecretExternalizationSurvivability`) that upgrades
+deployed auxiliary scripts (`imessage-reply.sh`, `serendipity-capture.sh`,
+`slack-channel-context.sh`) without touching custom forks. The canonical
+always-overwrite hooks (`session-start.sh`, `compaction-recovery.sh`,
+`telegram-topic-context.sh`) heal on next auto-update by the existing
+`migrateHooks` path.
+
+## What to Tell Your User
+
+If your agent recently went silent after compaction — emitting only the
+wall-clock-time block and then nothing else, despite a healthy server — this
+is the fix. Topic-history injection is back. No configuration needed.
+
+After auto-update, send yourself a Telegram message in any forum topic.
+The agent's response should reference what you actually said — not greet
+you like it's the first message. If it doesn't, something else is wrong;
+please flag.
+
+## Summary of New Capabilities
+
+| Capability | How to Use |
+|-----------|-----------|
+| Auth-token resolution survives secret-externalization | Automatic. Every shipped script + hook now reads `INSTAR_AUTH_TOKEN` env first, then falls back to `config.json` with a string-type guard. |
+| Port-parse tolerates JSON whitespace | Automatic. Hooks no longer exit-early when `config.json` is prettified. |
+| Lint blocks future regressions | `tests/unit/secret-externalization-hook-resolver-lint.test.ts` greps every shell + Node read for the broken pattern and fails on any reintroduction. |
+| Auxiliary-script migration for deployed agents | Existing agents with the old `imessage-reply.sh` / `serendipity-capture.sh` / `slack-channel-context.sh` get the fix on next auto-update via `migrateSecretExternalizationSurvivability`. Custom forks (no shipped marker) are untouched. |
+
+## Evidence
+
+- 13 new tests across 3 tiers (unit + integration-grade + e2e). All green on a
+  full `vitest` run.
+- 35 PostUpdateMigrator-* unit test files (232 tests) verified clean (no
+  regression from migrator extensions).
+- Manual smoke test against a real externalized agent: the fixed
+  `getTelegramTopicContextHook()` content correctly fetches topic history
+  with INSTAR_AUTH_TOKEN env alone (config.json holds the placeholder), and
+  correctly degrades to the unauth branch without leaking the placeholder
+  when env is missing.
+- Side-effects review:
+  `upgrades/side-effects/secret-externalization-hook-resolver-audit.md`.

package/upgrades/1.3.104.md

@@ -0,0 +1,58 @@
+# Upgrade Guide — vNEXT
+
+<!-- bump: patch -->
+
+## What Changed
+
+**Two fixes that make running one agent across multiple machines actually work
+end-to-end.**
+
+First, a real bug: the private channel machines use to send each other commands
+was being blocked by the normal login check before it could even verify the
+message's own signature. Because each machine has its own login token, one
+machine could never present a token the other would accept — so every
+cross-machine action (checking if a peer is alive, handing a conversation to
+another machine, moving one) was silently refused. That channel is already
+protected by a strong per-machine cryptographic signature, so it now correctly
+skips the token check and relies on the signature. This is the missing piece
+that made a second machine show as offline even when both machines could reach
+each other perfectly.
+
+Second, the headline "move this to the mini" feature is now wired all the way
+through. Saying "move this to <machine nickname>" (or "run this on <nickname>")
+in a conversation now pins that conversation to the named machine and hands it
+over, so it continues there. Before, the phrase was understood but nothing acted
+on it.
+
+## What to Tell Your User
+
+If you run on one machine, nothing changes. If you run across two machines: a
+second machine will now correctly show as online and become eligible to receive
+conversations, and you can say "move this to the mini" (using whatever nickname
+shows on your Machines list) to hand the current conversation to that machine —
+it picks up there. Both machines need to update for this to work.
+
+## Summary of New Capabilities
+
+- The machine-to-machine command channel is exempt from the API token check and
+  authed solely by its per-machine cryptographic signature — so cross-machine
+  presence, delivery, and transfer work over the network, not just in tests.
+- New `TopicPlacementPinStore`: durably pins a conversation to a chosen machine.
+- "move this to <nickname>" / "run this on <nickname>" is wired on inbound:
+  recognize the command, pin the conversation, release local ownership, and let
+  it re-place onto the named machine via the existing place-and-resume path.
+
+## Evidence
+
+- `tests/unit/mesh-rpc-auth-exemption.test.ts` — the machine command route
+  reaches its handler with no token (and ignores a wrong one — the signature is
+  the auth); a normal protected route still requires the token.
+- `tests/unit/topic-placement-pin-store.test.ts` — pin set/get/clear, durable
+  across restarts, tolerant of a corrupt file.
+- `tests/unit/transfer-activation-wiring.test.ts` — the recognizer/planner are
+  wired on inbound before routing, a transfer sets the pin + releases ownership,
+  the pin is passed into placement, and the whole path is dark-gated.
+- Found on real hardware (laptop + Mac mini): the mini stayed offline because its
+  `/mesh/rpc` returned 401 before the signed envelope was checked; with the
+  exemption the cross-machine calls authenticate off the envelope as designed.
+- Side-effects: `upgrades/side-effects/mesh-rpc-auth-and-transfer-activation.md`.

package/upgrades/NEXT.eli16.md

@@ -0,0 +1,71 @@
+# vNEXT — plain English overview
+
+## What this change is
+
+Your agent has a bunch of helper scripts that need to talk to its own local
+server. They authenticate using an "auth token" — basically a password the
+server expects in every request.
+
+Recently we added a feature that **moves that password out of the plain
+config file into an encrypted store**, so the password no longer sits in
+plain text on disk. That's a good security improvement.
+
+But when we did that, we forgot to update *every* place the helpers read
+the password from. So a bunch of helpers kept reading the plain config
+file, found the placeholder marker (a tiny note that says "the real
+password lives in the encrypted store now"), and used **that marker** as
+the password. The server rejected every request. The helpers didn't
+notice the rejection — they just quietly emitted nothing.
+
+The user-visible result: your agent comes back from a memory compaction
+and has no idea what conversation you were in. So it greets you like a
+stranger.
+
+## What already exists
+
+- The `INSTAR_AUTH_TOKEN` environment variable. Every time the agent
+  spawns a new session, it sets this env var with the real password.
+- The encrypted secret store at `.instar/secrets/config.secrets.enc`,
+  managed by `SecretMigrator` / `SecretStore`.
+- A migration system (`PostUpdateMigrator`) that rewrites helper scripts
+  on every auto-update. Some helpers were already on this track; others
+  weren't.
+
+## What's new
+
+- **Every helper now reads `INSTAR_AUTH_TOKEN` env first.** The env var
+  is always present in a session, so it's the canonical path.
+- **The disk-fallback now has a "is this actually a password?" check.**
+  If the config file says `{ "secret": true }` (the placeholder), the
+  guard rejects it and falls through to no-auth instead of sending the
+  placeholder as a password.
+- **A second, unrelated bug** also got fixed in the same change: the
+  helpers parsed the server port from the config file with a regex that
+  refused to allow whitespace, so `"port": 4042` (the format we
+  actually ship) was unparseable. The helpers exited silently with no
+  port, hitting nothing. Replaced with a whitespace-tolerant pattern.
+- **A new automated check** scans every helper for this broken pattern
+  and refuses to commit any future change that reintroduces it.
+- **An update routine** that fixes the helpers on existing deployed
+  agents (without overwriting any custom scripts the user wrote
+  themselves).
+
+## What you need to decide
+
+Nothing. This is a bug fix, no configuration involved. Auto-update will
+heal every deployed agent on next version bump.
+
+## How to verify it worked after deploy
+
+After your agent updates past the version that includes this fix, send
+yourself a Telegram message in any forum topic. The agent's response
+should reference what you actually said — not greet you like it's the
+first message. If it doesn't, something else is wrong; please flag.
+
+## Why this matters more than it might look
+
+This is a "silent failure" class — the helpers were broken but emitted
+no error, so the bug only surfaced through the symptom (incoherent
+agent replies after compaction). That makes the failure invisible until
+a user notices the symptom. The structural lint added in this change
+makes the class itself impossible to reintroduce silently.

package/upgrades/side-effects/mesh-rpc-auth-and-transfer-activation.md

@@ -0,0 +1,72 @@
+# Side-effects review — /mesh/rpc bearer-auth exemption + transfer-by-nickname activation
+
+## What was happening (real-hardware, 2026-05-29)
+
+After the HTTP presence transport (B, v1.3.102) deployed to the laptop, the Mac
+mini STILL showed offline in `GET /pool`. Root cause, found by probing the mini's
+`/mesh/rpc` over its tunnel: **401 "Missing or invalid Authorization header"**.
+
+`/mesh/rpc` — the §L0 machine-to-machine command transport — was sitting behind
+the general API bearer-token middleware, but `MeshRpcClient.send` sends only the
+signed envelope (no `Authorization` header), and it *can't* send a useful one:
+each machine install holds its own `authToken`, so the sender has no token the
+receiver would accept. So EVERY cross-machine call — the new presence pull AND
+`deliverMessage` / `place` / `claim` / `transfer` — was rejected 401 before the
+dispatcher's envelope check ever ran. The pool passed all in-process dispatcher
+tests yet was completely dead over a real tunnel.
+
+Separately, the headline "move this to <nickname>" trigger was unbuilt: the
+recognizer + planner existed as pure tested units but had no caller, nothing
+persisted the pin, and `SessionRouter.route()` was called with no `topicMetadata`
+— so placement always fell back to least-loaded and a relocation command did
+nothing.
+
+## The fix
+
+1. **`src/server/middleware.ts`** — exempt `/mesh/rpc` from the bearer gate, next
+   to the other signature/HMAC-authed m2m endpoints (`/a2a/inbox`, relay). The
+   route is authed SOLELY by its Ed25519 signed, recipient-bound envelope
+   (verify → RBAC → nonce, in the dispatcher). This single change is what makes
+   the cross-machine pool function over the wire at all.
+
+2. **`src/core/TopicPlacementPinStore.ts`** (new) — durable `{topicId → {preferredMachine,
+   pinned, updatedAt}}`. `asTopicMetadata()` shapes the pin for
+   `PlacementExecutor.decide`; `lastUpdatedAtMs()` feeds the rate-limit guard.
+
+3. **`src/commands/server.ts`** — wire the §L4 transfer-by-nickname activation
+   (gated on stage ≠ 'dark', BEFORE the route() interception): recognize the
+   command → plan → on transfer/noop set the pin AND release local ownership so
+   the topic's next message re-places onto the pinned machine via the
+   already-wired placeAndClaim → spawnOnMachine → owner-side `onAccepted` resume
+   path; confirm-required/reject reply to the user; the command message is
+   consumed. And pass `topicMetadata: pinStore.asTopicMetadata(sessionKey)` into
+   `route()` so placement honors the pin.
+
+## Blast radius
+
+- **The auth exemption is the only change to a live, always-on code path.** It
+  REMOVES a gate from `/mesh/rpc`; the dispatcher's envelope verification (which
+  already ran on every accepted call) becomes the sole auth — strictly stronger
+  than a shared bearer for cross-machine (per-peer Ed25519 keys, recipient
+  binding, nonce replay protection). A non-mesh / single-machine agent never
+  receives a `/mesh/rpc` POST, so the exemption is inert for it.
+- **The activation is dark-gated.** The recognizer/pin/route-metadata wiring only
+  runs when `_sessionPoolStage() !== 'dark'` (default dark), so it ships inert and
+  is exercised only in the controlled rollout/proof. The pin store is a new file
+  under `stateDir/session-pool/`; nothing reads it unless the activation runs.
+- **No new config / route / schema / hook / skill → no migration.** Server +
+  middleware + one new core module; existing agents pick it up on the next
+  release. Both machines need it (the receiver's `/mesh/rpc` must accept the
+  envelope; the router needs the recognizer + pin).
+
+## Tests
+
+- `tests/unit/mesh-rpc-auth-exemption.test.ts` — `/mesh/rpc` reaches its handler
+  with NO Authorization header (and even with a wrong bearer — the envelope is
+  the auth); a normal protected path still 401s without a token.
+- `tests/unit/topic-placement-pin-store.test.ts` — set/get/asTopicMetadata/
+  lastUpdatedAtMs/clear, durable across instances, corrupt-file tolerance.
+- `tests/unit/transfer-activation-wiring.test.ts` — server boot constructs the
+  pin store + recognizer/planner imports, recognizes on inbound before route(),
+  applies a transfer (sets pin + releases ownership), returns when handled, passes
+  the pin as `topicMetadata`, and is dark-gated.

package/upgrades/side-effects/secret-externalization-hook-resolver-audit.md

@@ -0,0 +1,48 @@
+# Side-effects review — secret-externalization hook resolver audit
+
+## What changed
+
+Two related fleet-wide classes are fixed in one PR.
+
+1. **Auth-token reads survive secret-externalization.** Every shipped shell hook, shell script, Node helper, and migrator-emitted template string that resolves the instar server's `authToken` now:
+   - Prefers `INSTAR_AUTH_TOKEN` env (set by `SessionManager` per spawned session and by `JobScheduler` per scheduled job — always available inside the Claude Code session).
+   - Falls back to reading `config.json` with a string-type guard. When `SecretMigrator` has externalized the token, on-disk `authToken` is the literal placeholder `{ "secret": true }`; the guard rejects it and yields empty so the placeholder cannot leak as a Bearer.
+2. **Port-parse regex tolerates whitespace.** `grep -o '"port":[0-9]*'` is replaced by `grep -oE '"port"[[:space:]]*:[[:space:]]*[0-9]+'` + digit-only extraction. The previous form silently produced empty PORT whenever `config.json` had `"port": 4042` (with space) — and our prettified configs all do — making every history-injection hook exit early on a healthy install. Both bugs together produced the 2026-05-29 silent-history failure.
+
+## Why
+
+The 2026-05-29 incident: Justin's `telegram-topic-context.sh` stopped injecting topic history after the agent compacted. The hook silently 403'd because `authToken` had been moved to the encrypted secret store and the hook was still reading the literal `{ secret: true }` placeholder out of `config.json` as a Bearer token. The agent came back with no topic context, sent an incoherent reply, and Justin had to manually trace the root cause back. This is the SECOND time a script in this class has bitten a deployed agent (the 2026-05-28 `telegram-reply.sh` 403 was the first), so the fix is structural: env-first everywhere + string-type guard + lint that catches the broken pattern + migrator that upgrades stale-on-disk copies.
+
+## Risk surface
+
+- **Hook scripts (always-overwritten):** `session-start.sh`, `compaction-recovery.sh`, `telegram-topic-context.sh`. Existing agents get the fix on first auto-update. The canonical content lives inside `PostUpdateMigrator.getHookContent()`; the parallel `src/templates/hooks/*.sh` files are kept in sync (the manifest tracks their SHA256). Risk: low — these are already overwritten on every migration tick.
+- **Reply scripts (SHA-based migration):** `telegram-reply.sh` upgrades via `migrateReplyScriptToPortConfig` (pre-PR SHA `371d7e8f…` is already in the prior-shipped allowlist; agents get the fix on first auto-update). `slack-reply.sh` and `whatsapp-reply.sh` upgrade via `migrateReplyScriptTo408`, extended to also fire when `INSTAR_AUTH_TOKEN` is missing — previously the function skipped any script with HTTP 408 handling, which would have stranded a 408-aware-but-auth-broken copy forever. Risk: low — the marker-string check still protects custom forks.
+- **Auxiliary scripts (new migration):** `imessage-reply.sh`, `serendipity-capture.sh`, `slack-channel-context.sh` get a new `migrateSecretExternalizationSurvivability` pass that upgrades shipped copies and skips custom forks (shipped-marker check). Risk: low — guarded by both the marker check and the negative `INSTAR_AUTH_TOKEN` check.
+- **CLI/in-process callers:** `gate.ts` and `nuke.ts` did raw `JSON.parse(fs.readFileSync('config.json'))` then `cfg.authToken` directly. Patched to env-first + string-type-guard. The other CLI commands go through `loadConfig()` (which calls `mergeConfigWithSecrets`) and are already safe. Risk: low — the change is the same env-first/string-guard pattern.
+- **Daemons:** `listener-daemon.ts` (HMAC-derives a signing key from `authToken`), `mcp-stdio-entry.ts` (uses `authToken` as the inbound Bearer). Both now env-first + string-type-guard. Risk: low — the env-first path is preferred when the daemon is launched by the spawn pipeline (which exports the var); the disk fallback still works for plaintext-config installs.
+- **CI / lint:** new unit test `secret-externalization-hook-resolver-lint.test.ts` greps every shell + Node read of `authToken`. The lint MUST go green on the fixed tree (verified) and MUST fail on any re-introduction of the broken pattern (verified via destructive negative test). Risk: low — the lint is allowlisted for in-process `loadConfig` callers; future authors won't trip on the safe path.
+
+## Bug surfaces eliminated
+
+- Topic-history injection silently 403s after secret externalization. (Direct cause of the 2026-05-29 incident.)
+- `telegram-reply.sh` 403s when invoked from a session whose env is missing — partially mitigated in the 2026-05-28 fix but the audit confirmed the disk-fallback path was still broken; this PR closes both halves.
+- `session-start.sh` could not fetch working-memory or shared-state after externalization (same root cause as the topic-history bug).
+- `compaction-recovery.sh` lost its post-compaction memory query after externalization.
+- A standby-machine daemon whose key derivation went through `authToken` would silently derive a key from `{ secret: true }` and lose peer connectivity. (Speculative — no incident observed, but the path was broken.)
+
+## Migration footprint
+
+- `migrateHooks()` already always-overwrites the three canonical hooks → existing agents heal automatically on next auto-update.
+- `migrateScripts()` already SHA-migrates `telegram-reply.sh` and now (with the extended INSTAR_AUTH_TOKEN check) auto-migrates `slack-reply.sh` + `whatsapp-reply.sh` when the previously-shipped marker is present.
+- `migrateSecretExternalizationSurvivability()` (NEW) auto-migrates `imessage-reply.sh`, `serendipity-capture.sh`, `slack-channel-context.sh`. Idempotent on the post-fix shape.
+
+## Testing
+
+- Unit (lint): `tests/unit/secret-externalization-hook-resolver-lint.test.ts` — 3 tests. Greps `src/templates/`, `src/`, `scripts/` for the broken pattern; verified negative (broken pattern caught) and positive (clean pattern passes).
+- Unit (migrator): `tests/unit/secret-externalization-survivability-migrator.test.ts` — 5 tests. Seeds broken state, runs `migrate()`, asserts upgrade. Custom-fork test verifies the marker check protects user-modified scripts.
+- Integration-grade (in `tests/unit/`): `tests/unit/secret-externalization-hook-injection.test.ts` — 2 tests. Spins an in-process stub server (bound to all interfaces; ASYNC `spawn` is required because `execFileSync` blocks Node's event loop and freezes the stub). Seeds `config.json` with the placeholder, runs the canonical migrator-emitted hook, asserts injection works with env-first auth and never leaks the placeholder as a Bearer.
+- E2E: `tests/e2e/secret-externalization-hook-lifecycle.test.ts` — 3 tests. Simulates pre-fix agent → `migrate()` → asserts every shipped script carries the env-first canary; verifies idempotency on a second pass; verifies the canonical migrator-emitted hooks always carry the canary.
+
+## Test count
+
+13 new tests across 3 tiers (per Testing Integrity Standard). 29 existing related tests (`PostUpdateMigrator-time-injection.test.ts`, `compaction-telegram-context.test.ts`) remain green.