instar 1.3.604 → 1.3.606

package/upgrades/1.3.605.md

@@ -0,0 +1,29 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: patch -->
+
+## What Changed
+
+The feedback-factory's clustering/triage pass (`processUnprocessed`) was built and parity-tested but only ever invoked by tests — so reports were ingested into the canonical store and never sorted. This wires it into a real, dev-gated capability per the approved migration spec (`docs/specs/feedback-factory-migration.md`, §191: "the processor job is actually constructed and scheduled, not dead code"):
+
+- New `FeedbackProcessingService` exposing `stats()` (read) and `processNow()` (one clustering pass); it reloads the canonical store from disk at the start of every call so it sees rows the separate-process inbox drainer appended since boot.
+- New routes `GET /feedback-factory/stats` (counts) and `POST /feedback-factory/process` (trigger), both `503` when the feature is dark.
+- New built-in `supervision: tier1` job `feedback-factory-process` (recurring) that calls the trigger and validates the pass against the post-pass stats; ships `enabled: false` (fleet-dark) and installs for every agent on update.
+- Dev-gated via `resolveDevAgentGate(feedbackFactory.processing.enabled)`: live on development agents, dark on the fleet (`enabled` omitted in defaults so the gate decides).
+
+## What to Tell Your User
+
+Nothing changes for fleet agents — this capability ships dark, so its routes stay off and its background job stays disabled until it is deliberately turned on. On a development agent it runs live, so the feedback-processing pipeline gets exercised and matured before any fleet rollout. If you operate the feedback factory and want to turn it on, enable feedback processing in your instar config.
+
+## Summary of New Capabilities
+
+- `GET /feedback-factory/stats` — read-only canonical feedback store counts (total, byStatus, clusters, dispatches, lastWriteAt).
+- `POST /feedback-factory/process` — run one clustering/triage pass against the canonical store.
+- `feedback-factory-process` job — recurring tier-1 processing pass, dark on the fleet.
+
+## Evidence
+
+- `npx tsc --noEmit` clean; all 16 instar-dev lints clean (incl. `lint-dev-agent-dark-gate`).
+- Tests: 14 unit (`tests/unit/feedback-factory/processing-service.test.ts`), 5 integration (`tests/integration/feedback-factory-stats-route.test.ts`), 6 e2e + wiring-integrity (`tests/e2e/feedback-factory-process-lifecycle.test.ts`) — all green.
+- Side-effects review: `upgrades/side-effects/feedback-processor-job.md` (second-pass review caught + fixed a stale-boot-store defect; reviewer re-concurred).

package/upgrades/1.3.606.md

@@ -0,0 +1,28 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: patch -->
+
+## What Changed
+
+A new background watcher (sibling to the silently-stopped trio) that posts ONE honest, observational line when an autonomous run has gone quiet to the user past a threshold **AND** the terminal shows genuinely fresh output. It covers the *"busy-but-silent"* failure mode — a heads-down build emitting no heartbeat for an hour, indistinguishable to the user from a stall.
+
+- Hedged, movement-gated wording — never a bare timer, never a false "still working" claim on a frozen screen.
+- Real per-run cooldown/backoff (not the inert duplicate-suppressor); content scrubbed for secrets/paths; cross-machine-move guard.
+- Read-only status at `GET /autonomous-heartbeat`.
+- Ships **dark on the fleet, dryRun-first on dev** (the dev-agent gate). Off everywhere by config until a deliberate flip; signal-only, never blocks or rewrites a message.
+
+## What to Tell Your User
+
+If you're running a long autonomous job and the agent goes quiet, it will now occasionally drop a brief, honest note about what it was last doing — so you can tell "busy" from "stuck." It never claims to be working when it isn't, never spams (it backs off), and never talks over a real reply. It's off by default and only runs in observe-mode on a development agent first.
+
+## Summary of New Capabilities
+
+- A timer-backed safety net that makes a long autonomous run *legible*: if the agent goes heads-down and silent past a threshold while still genuinely working, it surfaces one hedged "here's where I was" line instead of leaving the user staring at silence.
+- `GET /autonomous-heartbeat` — read-only status (active runs, per-run cooldown/backoff, last-emitted).
+
+## Evidence
+
+- **Reproduction (observed, 2026-06-16):** during a 24-hour autonomous run (topic 12476), the agent went heads-down fixing a multi-step CI failure for ~60 minutes and emitted zero user-facing messages. From the user's side this was indistinguishable from a stall; the user asked "did the session stall?" The existing silent-freeze watchdog correctly stayed quiet (the terminal WAS changing), and the promise-beacon's liveness line is tuned to once-per-60m — too slow.
+- **Before:** "report every ~30m" was a prompt instruction (willpower); a heads-down task blew past it with no structural backstop → an hour of silence.
+- **After:** the heartbeat fires on genuine fresh output + user-silence past the threshold, capped by cooldown/backoff → a long run produces a handful of honest liveness lines, not silence and not spam. Verified by 6 e2e "feature is alive" cases (emits when silent+moving; suppressed when mid-move / frozen / dryRun) + 25 unit + 2 integration tests (33 green).

package/upgrades/side-effects/autonomous-progress-heartbeat-ratchet-fix.md

@@ -0,0 +1,36 @@
+# Side-effects review — heartbeat no-silent-fallbacks ratchet fix
+
+## Change
+
+Comment-only. Adds `@silent-fallback-ok` exemption markers (with written
+justifications) to three fail-safe `catch` blocks introduced by the
+autonomous-progress-heartbeat feature:
+
+- `src/monitoring/AutonomousProgressHeartbeat.ts` — heartbeat send-failure catch
+  (already surfaced via `recordSuppressed` + `send-error` event).
+- `src/monitoring/AutonomousProgressHeartbeat.ts` — `mostRecentOutboundAt`
+  history-read catch (fail-closed sentinel return).
+- `src/core/AutonomousSessions.ts` — `readAutonomousRunMarkers` state-file read
+  (a missing `.local.md` is the expected no-run case; null is normal control flow).
+
+## Side effects
+
+- **Runtime behavior:** NONE. The edits are inside comments only; no executable
+  line changed. `tsc` is unaffected (comments).
+- **Tests:** `tests/unit/no-silent-fallbacks.test.ts` count returns from 479 to
+  476 (== baseline), so the ratchet passes. No other test is touched.
+- **Config / migrations:** none. No agent-installed file changes; no
+  CLAUDE.md/template/hook/skill changes; no PostUpdateMigrator change needed.
+- **External operations:** none. No network, no fs writes, no session spawns.
+- **Security:** none. The exemptions document why each catch is fail-safe; they
+  do not weaken any error path — the errors were already surfaced or fail-closed.
+
+## Signal vs authority
+
+N/A — no decision/gating surface changes. The exemption markers are a static
+lint annotation read only by the ratchet test.
+
+## Rollback
+
+Revert the commit; the three comments disappear and the ratchet returns to 479
+(which is exactly the pre-fix CI-red state). No data or state to unwind.

package/upgrades/side-effects/autonomous-progress-heartbeat.md

@@ -0,0 +1,40 @@
+# Side-Effects Review — AutonomousProgressHeartbeat (liveness backstop for a silent-to-user autonomous run)
+
+**Spec:** docs/specs/autonomous-progress-heartbeat.md (converged — 3 rounds, 6 internal reviewers + GPT-5.5 + Gemini-2.5-pro, review report at docs/specs/reports/autonomous-progress-heartbeat-convergence.md). **Constitutional principle served:** Structure > Willpower (converts "report every ~30m" from a prompt wish into a timer-driven backstop).
+**Ships DARK** behind `monitoring.autonomousProgressHeartbeat.enabled` (omitted in defaults → the dev-agent gate resolves it live-on-dev / dark-on-fleet) **and dryRun-first**. Off everywhere by config until a deliberate flip. Single change of behavior even when live: it may ADD one hedged line to a topic; it never blocks, delays, or rewrites anything.
+**Files:** src/monitoring/AutonomousProgressHeartbeat.ts (new), src/monitoring/autonomousHeartbeatScrub.ts (new), src/commands/server.ts, src/monitoring/sentinelWiring.ts, src/server/routes.ts, src/server/CapabilityIndex.ts, src/config/ConfigDefaults.ts, src/core/devGatedFeatures.ts, src/core/types.ts, src/core/AutonomousSessions.ts, src/core/PostUpdateMigrator.ts, src/scaffold/templates.ts, src/monitoring/ProxyCoordinator.ts, plus 3 test files.
+
+## What changed
+
+1. **AutonomousProgressHeartbeat.ts (new):** the watcher. On a tick it evaluates a cheap-first predicate chain for each active autonomous run: (#1) user-silence elapsed past `userSilenceThresholdMs`; (#2) NOT mid-machine-move (reads `movedTo`/`move_suspended_at` markers via AutonomousSessions); (#3) the run has been active on THIS machine ≥ one window (destination-warmup guard); (#8) the screen frame shows genuinely fresh output (reuses the silence sentinel's existing snapshot — no extra capture). Only when ALL hold does it emit ONE hedged, observational line. Per-run cooldown + widening backoff cap the volume (~handful over a 24h run).
+2. **autonomousHeartbeatScrub.ts (new):** scrubs the "last observed activity" focus text for secrets/paths before it can appear in a user-facing line. Fail-closed: an unscrubable/empty focus suppresses the line rather than risk a leak.
+3. **server.ts / sentinelWiring.ts:** constructs + starts the watcher behind the gate, mirroring the ActiveWorkSilenceSentinel wiring; shares the existing SubagentTracker snapshot rather than taking its own.
+4. **routes.ts / CapabilityIndex.ts:** `GET /autonomous-heartbeat` read-only status (current runs, per-run cooldown/backoff, last-emitted), 503 when dark.
+5. **ConfigDefaults.ts / devGatedFeatures.ts / types.ts:** the `monitoring.autonomousProgressHeartbeat` config block (enabled OMITTED → dev-gate; dryRun default true; thresholds), and the dev-gate registration.
+6. **PostUpdateMigrator.ts / templates.ts:** migrateConfig adds the config block (existence-checked); the CLAUDE.md template gains the agent-awareness section + the proactive trigger.
+
+## Blast radius
+
+- **Config-gated, not wiring-gated.** With `enabled` resolving false (fleet) the watcher is constructed-but-inert — `start()` returns before scheduling a tick, so zero behavior change. On a dev agent it resolves live but `dryRun:true` makes it LOG "would emit" without sending — observe-mode before it speaks.
+- **No new outbound path.** When it does emit, it routes through the EXISTING one-voice/dedupe send plumbing (the same path the other sentinels use), so it cannot spam or talk over another of my messages.
+- **No new capture cost.** Predicate #8 reuses the ActiveWorkSilenceSentinel's existing screen snapshot (a convergence-review fix to an earlier design that took its own).
+- **Reads only run-state files + the existing frame.** No whole-tree walk; cost is O(active runs) per tick, same class as the reaper.
+
+## Risk + mitigation
+
+- **Risk:** re-creating the "still working" filler the honest-progress work deliberately deleted. **Mitigation:** the wording is hedged + observational ("haven't posted in a while — last observed activity was X") and movement-gated (predicate #8) — never a bare-timer "still working" claim. All 8 reviewers flagged the first design; the shipped one is the rebuilt honest version.
+- **Risk:** leaking internal text/secrets into a user line. **Mitigation:** autonomousHeartbeatScrub fail-closed — an unscrubable focus suppresses the line.
+- **Risk:** a duplicate line during a cross-machine handoff (one-voice lock is machine-local). **Mitigation:** accepted bounded low-harm risk, written into the spec; predicate #2/#3 already suppress on a mid-move/just-warmed run, shrinking the window.
+- **Risk:** flooding a long run. **Mitigation:** per-run cooldown + widening backoff cap → a handful of lines over 24h, not fifty.
+
+## Migration parity
+
+- `migrateConfig` adds the `monitoring.autonomousProgressHeartbeat` block (existence-checked; only missing fields). CLAUDE.md template gains the awareness section. No hook/skill/settings change. New agents get it via init; existing agents via the update path.
+
+## Dark-gate line-map
+
+- `ConfigDefaults.ts` adds the `monitoring.autonomousProgressHeartbeat` block with `enabled` OMITTED (resolved by the dev-agent gate) + `dryRun:true`. `node scripts/lint-dev-agent-dark-gate.js` stays clean (no hardcoded `enabled:false`).
+
+## Rollback
+
+- Revert the PR, or set `monitoring.autonomousProgressHeartbeat.enabled:false` (or leave dryRun:true). The watcher goes inert; no durable state to unwind (cooldown/backoff is in-memory). Byte-identical to pre-PR behavior when off.

package/upgrades/side-effects/feedback-processor-job.md

@@ -0,0 +1,60 @@
+# Side-Effects Review — Wire feedback-factory processor into a scheduled job + stats route
+
+**Version / slug:** `feedback-processor-job`
+**Date:** 2026-06-16
+**Author:** Echo (instar-dev agent)
+**Second-pass reviewer:** Echo reviewer subagent (see Phase 5 appendix)
+
+## Summary of the change
+
+The feedback-factory's clustering/triage pass (`processUnprocessed`, already built + parity-tested at 100% over the recorded corpus) was invoked ONLY by tests — there was no production trigger, so reports were ingested into the canonical `JsonlFeedbackStore` but never clustered. This change wires that existing signal-producer into a real, dev-gated capability: (1) a new `FeedbackProcessingService` that owns a `JsonlFeedbackStore` at the canonical dir and exposes `stats()` (pure read) + `processNow()` (one clustering pass); (2) `GET /feedback-factory/stats` (read-only counts) and `POST /feedback-factory/process` (server-side trigger) in routes.ts, both 503 when dark; (3) a built-in `supervision: tier1` agentmd job (`feedback-factory-process.md`, cron `*/30 * * * *`) that curls the trigger and validates the pass against post-pass stats. Gated through `resolveDevAgentGate(config.feedbackFactory?.processing?.enabled, config)` — live on dev agents, dark (503 / job `enabled:false`) on the fleet. Files: `src/feedback-factory/processing/FeedbackProcessingService.ts`, `src/feedback-factory/store/JsonlFeedbackStore.ts` (added `stats()`), `src/server/routes.ts`, `src/server/AgentServer.ts`, `src/server/CapabilityIndex.ts`, `src/core/types.ts`, `src/config/ConfigDefaults.ts`, `src/scaffold/templates.ts`, `src/scaffold/templates/jobs/instar/feedback-factory-process.md`, `src/core/PostUpdateMigrator.ts`, + 3 test tiers. Driven by the approved spec `docs/specs/feedback-factory-migration.md` (Phase 1 / §191 "the processor job is actually constructed and scheduled, not dead code").
+
+## Decision-point inventory
+
+- `feedbackFactory.processing` dev-gate (AgentServer construction + route guards) — **add** — standard `resolveDevAgentGate` config gate; live-on-dev / dark-on-fleet. Not a content filter.
+- `processUnprocessed` clustering (existing) — **pass-through** — unchanged; this change only wires it to a trigger. It remains a SIGNAL producer (similarity grouping); it never force-closes a cluster and terminal transitions stay evidence-gated (spec §252).
+- `GET /feedback-factory/stats` / `POST /feedback-factory/process` — **add** — stats is read-only; process runs the existing pass. Neither gates message flow or agent behavior.
+
+## 1. Over-block
+
+Nothing is blocked. The stats route is a read; the process route runs an idempotent clustering pass. The only "block" surface is the dev-gate returning 503 when the feature is dark — which is correct (the capability genuinely isn't active on the fleet yet) and mirrors `/feedback-inbox/status`. No legitimate input is rejected.
+
+## 2. Under-block
+
+N/A — this change holds no blocking authority. The real interaction concern with the InboxDrainer is **staleness, not corruption** (an earlier draft of this artifact mis-framed it as "mitigated because both use the same append-only store" — that was WRONG: the drainer is a SEPARATE PROCESS with its OWN in-memory store instance; sharing one in-memory Map is impossible across processes). `JsonlFeedbackStore` loads `feedback.jsonl` ONLY in its constructor, and `FeedbackProcessingService` builds its store ONCE at boot — so rows the drainer appends AFTER boot would never enter the processor's in-memory Map, making every pass after the initial backlog a permanent no-op over newly-ingested reports (re-introducing the exact "ingested but never clustered" defect §191 closes; the tier-1 supervisor reads the same stale store, sees 0 unprocessed, and falsely reports healthy). **Mitigation (the actual fix): `JsonlFeedbackStore.reload()` re-folds all three JSONL files from disk, and `FeedbackProcessingService` calls `store.reload()` at the START of BOTH `stats()` and `processNow()`** so every read/pass sees what other processes appended since boot. No concurrent-write hazard: reload is a read-only re-fold and appends are atomic single-line writes (appendFileSync) — a reload racing a mid-append at worst skips one torn trailing line (already handled by the load path's torn-line skip), which the next reload picks up. `processUnprocessed` remains forward-only/idempotent (an item already flipped to `processing` is never re-picked). A regression test (`tests/unit/feedback-factory/processing-service.test.ts`) reproduces the production ordering — service constructed FIRST, then a NEW row appended by a SEPARATE store instance pointed at the same dir — and asserts the next pass clusters it; it fails without the reload and passes with it.
+
+## 3. Level-of-abstraction fit
+
+Correct per the approved spec §162/§251: receiver/dispatch are HTTP at the edge (Vercel), the heavy/bursty processor runs at the JOB layer on a capable machine (here echo), curated state is data. The processor runs server-side via the trigger route (single authority over the canonical store) rather than a duplicate store in a CLI process — matching the established agentmd-job pattern (correction-analyzer, release-readiness-check). No layer inversion.
+
+## 4. Signal vs authority compliance
+
+Compliant (`docs/signal-vs-authority.md`). The processor is a DETECTOR/signal producer: similarity/Jaccard grouping emits grouping signals. It holds NO terminal authority — it never force-closes a cluster; terminal lifecycle transitions remain evidence-gated by the existing API hard gate (scar a, spec §252). The new dev-gate is a config enable/disable, not a brittle content blocker with its own block path. No new brittle authority is introduced.
+
+## 5. Interactions
+
+- Shares the canonical store dir with the InboxDrainer (ingest writer), which is a **SEPARATE PROCESS** (the launchd `feedback-inbox-drain.mjs` job) holding its OWN `JsonlFeedbackStore` instance. There is no shared in-memory state and there cannot be (different processes). Coordination is purely on-disk via the append-only last-write-wins JSONL format (each mutation appends a full row; torn lines skipped on load): no double-write corruption. The cross-process **staleness** risk (the processor's boot-time in-memory snapshot never seeing post-boot ingest) is closed by the reload-per-pass fix in §2 — `FeedbackProcessingService` re-folds from disk at the start of every `stats()` and `processNow()`, so it always observes the drainer's appended rows.
+- `applyDefaults` backfills `feedbackFactory.processing` for existing agents; `migrateBuiltinJobs` installs the job (ships `enabled:false`); `migrateClaudeMd` adds the awareness section. No shadowing of an existing check; no double-fire (single cron schedule, server-side single-flight by virtue of the route).
+- Does not touch outbound/inbound messaging, session lifecycle, or any sentinel/watchdog.
+
+## 6. External surfaces
+
+Two new authenticated routes (503 when dark) and one new built-in job (off by default on the fleet). No change visible to other agents/users/systems while dark. On a dev agent the job runs every 30 min and appends to local `clusters.jsonl` + flips local feedback items `unprocessed→processing` — no network egress, no external API, no messages, no GitHub. The processing pass's only effect is local JSONL appends (re-verified: zero external side effects). Timing dependence is limited to the cron cadence; no conversation-state dependence.
+
+## 7. Multi-machine posture (Cross-Machine Coherence)
+
+**Machine-local BY DESIGN.** The canonical `JsonlFeedbackStore` lives on the machine running the feedback-factory processor (per spec §162, the heavy processor runs on ONE capable machine, NOT replicated — the operated instance owns its store). The stats route reports THAT machine's store; the job runs on the machine where it's scheduled. This is the spec's intended single-canonical-store posture (split-brain prevention, §253 "ONE shared canonical DB"). No silent single-machine assumption defect — it is the deliberate, spec-mandated design. No user-facing notice surface (so no one-voice concern); no generated URLs; durable state (the JSONL store) is machine-owned and does not transfer on topic move.
+
+## 8. Rollback cost
+
+Cheap and safe. PR-revertible. Dark on fleet → flag-off is byte-identical to today (route 503s, job `enabled:false`). If wrong on a dev agent: set `feedbackFactory.processing.enabled:false` (no restart needed for the route gate read; job disable via the manifest). The processing pass only appends to `clusters.jsonl` and flips item status forward — no destructive mutation, no data loss, fully forward-only. Worst case: delete the derived `clusters.jsonl` and the source `feedback.jsonl` is untouched (items just revert to being re-clusterable since status flips are additive rows).
+
+## Phase 5 — Second-pass review appendix
+
+**Concern raised** (independent reviewer): the long-lived `FeedbackProcessingService` constructed its `JsonlFeedbackStore` once at boot and never reloaded, while the InboxDrainer (a SEPARATE process) appends rows continuously after boot — so after the initial backlog the 30-min job would be a permanent no-op over newly-ingested reports, reintroducing the exact §191 "ingested but never clustered" defect, with the tier-1 supervisor masking it (same stale store → 0 unprocessed → false healthy).
+
+**Resolution** (iterated before commit, per Phase 5): added `JsonlFeedbackStore.reload()` (re-folds all three JSONL files from disk using the same load path as the constructor) and call it at the START of both `stats()` and `processNow()` in `FeedbackProcessingService`. Added a regression test reproducing the live production ordering (service constructed FIRST, a NEW row then appended by a SEPARATE store instance on the same dir) — independently verified to FAIL without the reload and PASS with it. tsc clean; 14 unit + 5 integration + 6 e2e green. Artifact §2/§5 corrected (the original "same append-only store mitigates it" framing was wrong — the risk was staleness across separate-process in-memory instances, not corruption).
+
+**Verdict after resolution:** concern resolved; the design now reads fresh canonical state every pass. Re-grounded against the code by the driving agent (Echo), not just asserted.
+
+**Independent reviewer re-confirmation:** "Concern resolved — concur." Verified: `reload()` re-folds all durable state via the same constructor load path and assigns fresh Maps wholesale (no stale leak / compacted rows don't linger); `reload()` precedes every read in both `stats()` and `processNow()`; exactly one reload per pass (no intra-pass double-processing); the regression test authentically models separate-process-append-after-boot and fails if the reload is removed from either surface. Non-blocking note (performance, not correctness): the reload re-parses the full JSONL per call — acceptable at the 30-min cadence + low stats-call rate; cost scales with total durable rows. If the canonical store grows very large, a future optimization could track a byte offset (like TokenLedgerPoller) instead of full re-fold. <!-- tracked: topic-12476 -->