instar 1.3.94 → 1.3.95

package/upgrades/side-effects/burn-attribution-wiring.md

@@ -0,0 +1,103 @@
+# Side-Effects Review — wire AttributionResolver into ingest + exempt the pre-attribution sentinel
+
+## What changed
+
+Completes **Phase 2** of `docs/specs/token-burn-detection-and-self-heal.md`, which
+was specced but never wired. Three source files + their tests:
+
+1. **`src/monitoring/AttributionResolver.ts`** — exports a new
+   `PRE_ATTRIBUTION_KEY = 'unknown::pre-attribution'` constant: the sentinel
+   meaning "the resolver has never run on this event" (the column default and
+   the historical hardcoded ingest value). `resolveAttribution()` itself is
+   unchanged and never returns this sentinel.
+
+2. **`src/monitoring/TokenLedger.ts`**
+   - `ingestLine()` now calls `resolveAttribution({sessionId, projectPath: cwd,
+     prompt: null, model})` instead of hardcoding `'unknown::pre-attribution'`.
+     The Claude-CLI JSONL assistant line carries no user prompt, so this is
+     cwd/job/hook/session-level attribution; prompt-shape attribution remains a
+     separate follow-up (needs user+assistant line correlation at ingest).
+   - New `ledger_meta` key/value table + `getMeta`/`setMeta` helpers.
+   - New `backfillAttributionOnce()` — one-shot, idempotent, marker-guarded
+     (`attribution-backfill-v1`) re-resolution of legacy sentinel rows, run
+     non-fatally from the constructor after `prepareStatements()`.
+
+3. **`src/monitoring/BurnDetector.ts`** — `tick()` exempts the
+   `PRE_ATTRIBUTION_KEY` sentinel from the **absolute-share** trigger only. A
+   bucket at 100% under the sentinel is a coverage gap, not a burn. The
+   genuinely-residual `unknown::<sessionId>` key (resolver ran, no match) is
+   NOT exempt and still alerts (the spec's "alert on unattributable spend").
+   Baseline-divergence is unaffected.
+
+## Why
+
+The recurring `unknown::pre-attribution consumed 100.0% of 24h spend
+(threshold 25%)` alert was a **fleet-wide false positive**. Because the
+resolver was never wired, every JSONL-sourced event defaulted to the single
+sentinel bucket, whose share was always 100%, so the absolute-share trigger
+fired every hour forever — on every agent with burn-detection enabled (default)
+and any daily spend. Real spend on the reporting agent was ~1.5M tokens/24h
+(the bleed this system targets was ~3B/day).
+
+## Behavioral side effects
+
+- **BurnDetector no longer emits absolute-share on the sentinel.** Intended.
+  This is the false-positive fix. Real burns are unaffected: (a) any resolved
+  key (component/job/hook/`unknown::<sid>`) still trips at >25%; (b) the
+  sentinel only exists transiently until backfill clears it; (c) a real burner
+  sharing the 24h window with sentinel rows still trips (the sentinel is
+  skipped, not counted against the burner's share — verified by test
+  "does not let the sentinel mask a real burner sharing the window").
+- **Existing attribution distribution changes shape.** Instead of one 100%
+  bucket, spend now splits per origin (mostly `unknown::<sessionId>` for the
+  dominant Claude-CLI path, plus `user-job:*` / `user-hook:*`). The
+  `/tokens/*` summary/sessions/by-project routes are unchanged in schema; only
+  the `byAttributionKey` grouping is now meaningful. No route signature changed.
+- **One-time backfill UPDATE on first boot after upgrade.** On the reporting
+  agent that's ~384k rows across a few hundred distinct (session, project,
+  model) triples — resolved once per triple inside a single transaction, so a
+  handful of bulk UPDATEs, not 384k. Marker-guarded → runs exactly once per DB.
+  Wrapped in try/catch in the constructor: a backfill failure logs and is
+  swallowed (worst case: sentinel rows stay unattributed and thus exempt from
+  burn alerts — i.e. degrades to the old-but-safe behavior, never a crash).
+
+## Migration parity
+
+No `PostUpdateMigrator` change required. The backfill self-heals on the
+constructor's next run, which happens on every server start. Existing agents
+get it on their next restart onto a version carrying this change. The new
+`ledger_meta` table is created idempotently in the SCHEMA list alongside the
+existing tables.
+
+## Blast radius / isolation
+
+- Scope is the three monitoring files. **No HTTP route, server-boot, or
+  CapabilityIndex change** → no agent-awareness/CLAUDE.md-template update owed.
+- Codex token sessions live in a separate table (`codex_token_sessions`) and
+  are untouched — burn detection reads only `token_events`.
+- Read-only-observability invariant of the TokenLedger is preserved: the only
+  new write is to the ledger's own SQLite DB (the backfill UPDATE + the meta
+  marker). No source JSONL file is touched.
+
+## Rollback
+
+Revert the three `src/monitoring/*` files. Backfilled `attribution_key` values
+are harmless if the revert lands (they're just better labels); the
+`ledger_meta` table and its marker row are inert without the new code. No data
+loss, no schema-incompatible state. (Per-phase revert matches the umbrella
+spec's rollback model.)
+
+## Tests
+
+- New `tests/unit/burn-attribution-wiring.test.ts` (11 tests): resolver never
+  returns the sentinel; ingest now resolves a real key; two sessions split into
+  two keys; job-cwd → `user-job:*`; backfill converts legacy sentinel rows on
+  reopen + is idempotent; BurnDetector regression — sentinel at 100% emits
+  nothing, residual `unknown::<sid>` at 100% still emits, sentinel doesn't mask
+  a co-occurring real burner.
+- Updated `tests/unit/token-ledger.test.ts` (#512 schema-order migration test):
+  now asserts the migrated legacy row is backfilled to `unknown::sess-old` and
+  the sentinel is gone, instead of asserting the sentinel persists.
+- Full sweep green: `vitest run` over burn-detection-phase-1..6 + token-ledger +
+  TokenLedger-codex + TokenLedgerPoller-codex + burn-attribution-wiring =
+  **149 passed, 0 failed**. `tsc --noEmit` clean.

package/upgrades/side-effects/noninteractive-pool-join.md

@@ -0,0 +1,85 @@
+# Side-effects review — Non-interactive, code-authenticated pool join (trust model A)
+
+## What changed & why
+
+Bringing up a second real machine for the Multi-Machine Session Pool surfaced
+that the awake machine never learned about a machine that joined headlessly: the
+existing `pair`/`join` flow is **interactive by design** — `/api/pair` was
+"signal-only" and real registration completed only via a human confirming
+matching SAS visual symbols on both screens. An active-active session pool can't
+require a human to eyeball symbols per machine, so (operator decision: "Proceed
+with A") pairing becomes **code-authenticated and non-interactive**: the
+time-limited, single-use pairing code (carried over the TLS tunnel) is the
+shared secret that authorizes the join; Ed25519-signed mesh RPCs handle trust
+thereafter.
+
+Root cause of the gap: `instar pair` created the `PairingSession` into an
+**unused** `_pairingSession` variable and discarded it, so the running server
+had no code to validate against.
+
+Changes:
+- **`src/core/PairingSessionStore.ts`** (new): persists the validation-relevant
+  fields of the active `PairingSession` to `.instar/machine/pairing-session.json`
+  (0600), and loads it. The ephemeral X25519 private key is deliberately NOT
+  persisted (it is a `KeyObject`, not JSON-serializable, and unused for code auth).
+- **`src/commands/machine.ts`** (`pairMachine`): persists the session via
+  `PairingSessionStore` (10-min window) instead of discarding it; (`joinMesh`):
+  sends the joiner's own advertised URL in the pair body when it already has one.
+- **`src/server/machineRoutes.ts`** (`/api/pair`): validates the submitted code
+  against the persisted session (`validatePairingCode` — single-use,
+  attempt-capped, TTL'd), and on success registers the joiner as **standby**,
+  stores its public keys (`storeRemoteIdentity`, so MeshRpc can verify it),
+  records its advertised URL, and burns the code. Rejections are audited.
+- **`src/core/MachineIdentity.ts`**: adds a `baseDir` getter so the route can
+  co-locate its `PairingSessionStore` with the registry.
+
+## Security analysis
+
+- **The code IS the auth.** It is a `WORD-WORD-NNNN` secret, single-use
+  (`consumed`), attempt-capped (default 3 — failed attempts persist across
+  requests so the cap actually throttles brute force), and TTL'd (10 min as set
+  by `instar pair`). It travels over the Cloudflare TLS tunnel. Brute force over
+  a large code space within 3 attempts is infeasible.
+- **A joiner can only ever become `standby`.** The role is hard-coded in the
+  handler; a join request can never claim the awake role or move the lease.
+- **`/api/pair` remains unauthenticated** (it is the bootstrap, before the joiner
+  is known) — but it is now *gated*: with no active session it rejects (403)
+  rather than the old behavior of accepting any request. Malformed identities are
+  rejected (400) before anything is persisted.
+- **What we trade away vs. SAS:** the human visual-symbol MITM check. Mitigated
+  by code-over-TLS + short TTL + single-use + attempt cap. This is the operator's
+  explicit trust-model choice for automated pool formation.
+
+## Blast radius
+
+- **Dark-feature-adjacent.** This is the multi-machine pairing path; a
+  single-machine agent never calls it. No behavior change unless an operator runs
+  `instar pair` + `instar join`.
+- **No new config / schema.** `/api/pair` is server code → existing agents get
+  the new handler on update; no `PostUpdateMigrator` entry needed. The
+  `pairing-session.json` file is created on demand by `instar pair`.
+- **Backward-compatible join:** `joinMesh` still registers the awake machine from
+  the response and records its URL (the #531 fix); the only response change is
+  `status: 'paired'` instead of `'pending'`, which `joinMesh` does not branch on.
+
+## Not closed here (scoped follow-up)
+
+For a **quick-tunnel** standby, its reachable URL is only known after its server
+starts (post-join), so it cannot be sent in the pair body. The awake machine
+therefore still needs a way to learn a quick-tunnel standby's URL — either the
+standby uses a named tunnel (deterministic URL, exchanged at pair time), or a
+dedicated authenticated URL-announce path is added (the standby is now registered
+with verifiable keys, so it can sign such a call). Raised explicitly; the
+heartbeat path is awake→standby (lease arbitration), so it is not a clean carrier.
+
+## Tests
+
+- `tests/unit/pairing-session-store.test.ts` — `PairingSessionStore` roundtrip,
+  null-on-absent, no-private-key-persisted, 0600 perms, overwrite, malformed-file
+  tolerance.
+- `tests/integration/pool-noninteractive-pairing.test.ts` — valid code registers
+  joiner as standby + records URL + stores keys + consumes code; wrong code 403 +
+  attempt increment; brute-force lockout; no-session 403; consumed 403; malformed
+  identity 400.
+- `tests/integration/machine-routes.test.ts` — updated the legacy signal-only
+  assertion to the new session-gated contract.