instar 1.3.324 → 1.3.326

package/src/scaffold/templates.ts

@@ -527,7 +527,7 @@ This routes feedback to the Instar maintainers automatically. Valid types: \`bug
 - Create an instance: \`curl -X POST -H "Authorization: Bearer $AUTH" http://localhost:${port}/apprenticeship/instances -H 'Content-Type: application/json' -d '{"id":"codey-to-gemini","instanceType":"mentorship","overseer":"echo","mentor":"codey","mentee":"gemini","framework":"gemini-cli","priorInstanceId":null}'\` (id/overseer/mentor/mentee/framework are charset-clamped to \`^[a-z0-9-]+$\`; dup id rejected; harvestFrom=mentor / harvestTo=mentee).
 - Transition status (the ONLY way it changes — runs the gate): \`curl -X POST -H "Authorization: Bearer $AUTH" http://localhost:${port}/apprenticeship/instances/:id/transition -H 'Content-Type: application/json' -d '{"to":"active"}'\` (refused with a reason + 409 on a failed gate or illegal transition; \`complete\` is terminal).
 - Preview a gate without mutating: \`POST /apprenticeship/instances/:id/can-start\` · \`.../can-complete\`.
-- Record a manual cycle: \`POST /apprenticeship/cycles\` with \`instanceId\`, positive \`cycleNumber\`, \`task\`, \`menteeOutput\`, optional \`mentorFlagged\` / \`overseerDifferential\` / \`coaching\` / \`infraItems\`, \`kind\` (\`mentor-mentee-differential\`, \`overseer-apprentice-devreview\`, \`overseer-mentee-direct\`), and \`channel\` (\`telegram-playwright\`, \`threadline-backup\`, \`direct-shortcut\`, \`unknown\`). Use this when the overseer or manual loop found a differential outside the automated mentor tick.
+- Record a manual cycle: \`POST /apprenticeship/cycles\` with \`instanceId\`, positive \`cycleNumber\`, \`task\`, \`menteeOutput\`, optional \`mentorFlagged\` / \`overseerDifferential\` / \`coaching\` / \`infraItems\`, \`kind\` (\`mentor-mentee-differential\`, \`overseer-apprentice-devreview\`, \`overseer-mentee-direct\`), and \`channel\` (\`telegram-playwright\`, \`threadline-backup\`, \`direct-shortcut\`, \`unknown\`). A \`telegram-playwright\` cycle additionally REQUIRES a \`transcriptAudit\` block — \`{ topicIds, window: {start,end}, summary, findingDedupKeys, generatedAt, ledger: 'local'|'remote'|'dry-run'|'failed' }\` — built from \`instar dev:post-drive-transcript-audit\` run over the drive window (use \`--history-base-url\` when the transcript lives on the mentee's server; \`ledger:'local'\` claims are cross-checked against the real framework ledger). Use this when the overseer or manual loop found a differential outside the automated mentor tick.
 - **When to use** (PROACTIVE): when starting or closing a mentorship/apprenticeship instance, drive it through the registry + transitions so the retro-harvest is reviewed before the next instance starts and the lessons are captured before this one closes — never track an instance's lifecycle by memory.
 
 **Failure-Learning Loop** — Dev-process failure forensics (instar self-hosting). When something you built breaks later, it's captured and traced back to the spec/initiative/project AND the dev toolchain that produced it; the analyzer surfaces process-gap patterns and opens human-approved tracked fixes, then verifies whether each fix actually reduced that failure class. Ships OFF (\`monitoring.failureLearning.enabled\`); registers itself on the initiative board.

package/upgrades/1.3.325.md

@@ -0,0 +1,59 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: patch -->
+
+## What Changed
+
+The multi-machine live-tail streamer (the lease holder pushing recent conversation
+content to the standby every 5s) rebuilt EVERY known topic's content on EVERY tick —
+and the Telegram content provider resolved each rebuild by synchronously reading the
+entire JSONL message log (up to 75,000 lines) per topic. On 2026-06-05 this blocked
+the live echo server's event loop for 5–40s at a stretch (151 measured gaps >5s in
+10 minutes), which made outgoing mesh RPC timestamps stale, which made the standby
+reject flushes (403), which the source hot-retried every tick — a self-amplifying
+storm that read as "machine under extremely heavy load" and stuck Telegram delivery.
+Four bounded mechanisms close it: (1) a per-topic monotonic content version in
+`TelegramAdapter` lets `LiveTailSource` skip unchanged topics without building their
+content; (2) `getTopicHistory` is served from an in-memory per-topic tail cache
+(single-pass batch seed for all live topics, maintained on append, byte-equivalent
+to a file scan — the handoff hash depends on that and it is test-pinned); (3) a
+failed flush backs off exponentially (5s base doubling to a 5min cap) instead of
+retrying at tick rate, while the handoff path explicitly forces past gate+backoff;
+(4) a single flush's content is capped at 256KiB (matching the standby buffer's
+own per-topic ceiling).
+
+## What to Tell Your User
+
+If you run me on more than one machine: my server no longer freezes for seconds at
+a time while keeping the other machine's copy of our conversations fresh, and the
+two machines no longer talk themselves into retry storms when one of them stalls.
+Messages stop getting stuck, and false "the other machine looks dead" machine-swaps
+stop being triggered. Nothing about what the standby machine knows changes — it
+just costs almost nothing to keep it current now.
+
+## Summary of New Capabilities
+
+- Live-tail version gate: idle topics cost one Map lookup per tick instead of a
+  full message-log read (TelegramAdapter.getTopicContentVersion → LiveTailSource).
+- Topic tail cache: getTopicHistory serves from memory after a one-time single-pass
+  seed — no per-call file reads (also speeds respawn history + handoff hashing).
+- Flush failure backoff: a rejecting/unreachable standby is retried on an
+  exponential schedule (5s→5min), never hammered at tick rate; handoffs bypass it.
+- Flush content cap: one flush ≤256KiB (freshest suffix kept) — matching what the
+  standby retains anyway.
+
+## Evidence
+
+Live incident 2026-06-05 (topic "Resource Limitation Mitigation"): event loop gaps
+measured via server.log timestamp deltas (151 gaps >5s in ~10min); `sample` traces
+showed timer callbacks in giant live-tail payload serialization; Mini rejected
+flushes `stale-timestamp` / live-tail flush 403s with "flush seq=1 not acknowledged
+— will retry" + "content diverged — resending full tail" hot loops. Root cause in
+code: `LiveTailSource.pushTick` → `getTopicContent` per topic per tick →
+`TelegramAdapter.getTopicHistory` full `readFileSync` of the 75k-line JSONL per
+call. Spec: `docs/specs/live-tail-event-loop-guards.md`. Tests: 32 green across
+`LiveTailSource.test.ts` (gate/backoff/cap/force + all pre-existing delta pins),
+`TelegramAdapter-topicTailCache.test.ts` (read-count spy pins + cache-vs-file byte
+parity), `live-tail-version-gate-wiring.test.ts` (wiring integrity), updated
+`handoff-sentinel-boot-wiring.test.ts` (force pin); tsc + full lint chain clean.

package/upgrades/1.3.326.md

@@ -0,0 +1,58 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: minor -->
+
+## What Changed
+
+Closes the loop on the UX-blindspot arc: the post-drive transcript auditor
+(PR #864) is no longer optional on the channel it was built for. An
+apprenticeship cycle recorded with `channel: 'telegram-playwright'` — the
+dogfooded drive through the mentee's real Telegram UX — now structurally
+REFUSES to exist without a `transcriptAudit` block proving the objective
+audit ran:
+
+- `POST /apprenticeship/cycles` requires `transcriptAudit: { topicIds, window:
+  {start,end}, summary, findingDedupKeys, generatedAt, ledger:
+  'local'|'remote'|'dry-run'|'failed', notes? }` on telegram-playwright cycles
+  (optional-but-validated on other channels). The refusal error teaches the
+  exact CLI that produces the artifact.
+- **Anti-fabrication tooth**: a block declaring `ledger:'local'` with filed
+  findings is cross-checked against the REAL framework ledger — if none of the
+  claimed dedup keys exist, the record is refused. Honest declarations
+  (`remote`/`dry-run`/`failed`) are accepted as declared and stay queryable.
+- `instar dev:post-drive-transcript-audit` gains `--history-base-url` (+
+  `INSTAR_HISTORY_AUTH_TOKEN`): read the drive transcript from the MENTEE's
+  server (where the Playwright drive actually lands) while filing findings
+  into your OWN ledger. The local token is never sent to the remote server.
+- Mentor standing orders, the CLAUDE.md template, and a targeted
+  PostUpdateMigrator line-rewrite all teach the new shape — deployed agents
+  learn it on update instead of hitting mystery 400s.
+
+Why: "Observation Needs Structure" (PR #861) — an observation tool nobody is
+forced to run is the same prose-wish the article bans, one level up. #856
+gated the subjective half (mentor seat-counts); this gates the objective half.
+
+## What to Tell Your User
+
+Nothing user-visible changes day to day. Under the hood, the apprenticeship
+program's drive records now require proof that the objective transcript audit
+ran — so UX friction caught on a drive can no longer be silently skipped, and
+fabricated audit claims are refused against the real findings ledger.
+
+## Summary of New Capabilities
+
+- `transcriptAudit` block on `POST /apprenticeship/cycles` — required for
+  telegram-playwright cycles, validated everywhere, ledger-verified for
+  `local` claims.
+- `FrameworkIssueLedger.hasDedupKey(key)` — read-only existence probe.
+- `instar dev:post-drive-transcript-audit --history-base-url <url>` — split
+  read/write servers for the cross-agent mentor flow.
+
+## Evidence
+
+All three test tiers green: store unit (21), auditor CLI unit (8, incl. the
+token-isolation case), routes integration (21, incl. both sides of the
+anti-fabrication boundary), e2e lifecycle (7, incl. the feature-alive gate
+case through AgentServer). Migration test (4) covers rewrite, idempotency,
+and no-double-fire. `tsc --noEmit` clean.

package/upgrades/side-effects/apprenticeship-transcript-audit-gate.md

@@ -0,0 +1,53 @@
+# Side-Effects Review - apprenticeship transcript-audit gate
+
+**Version / slug:** `apprenticeship-transcript-audit-gate`
+**Date:** `2026-06-05`
+**Author:** `echo`
+**Second-pass reviewer:** `not required (Tier 1)`
+
+## Summary of the change
+
+Makes the post-drive transcript audit (PR #864) structurally unskippable on the channel it was built for: `telegram-playwright` apprenticeship cycles now refuse to record without a validated `transcriptAudit` block. Adds a route-level anti-fabrication cross-check for `ledger:'local'` claims, a `--history-base-url` split on the auditor CLI so the cross-agent flow (read mentee's transcript, file to own ledger) actually works, and updates every teaching surface (mentor standing orders, CLAUDE.md template, targeted migrator line-rewrite).
+
+## Decision-point inventory
+
+- `validateTranscriptAudit` (store) - add - refuses telegram-playwright cycles without the artifact; validates shape on any channel when supplied.
+- `POST /apprenticeship/cycles` (route) - modify - cross-checks `ledger:'local'` claimed dedup keys against the real FrameworkIssueLedger; refuses fabricated claims.
+- `FrameworkIssueLedger.hasDedupKey` - add - read-only existence probe, no new write authority.
+- `runPostDriveTranscriptAuditCli` - modify - splits history reads (possibly remote) from ledger writes (always local).
+- `PostUpdateMigrator` cycle-line rewrite - add - idempotent CLAUDE.md text patch.
+
+## 1. Over-block
+
+The gate refuses telegram-playwright cycle records lacking the audit block. Risk: a mentor who genuinely CANNOT run the audit (mentee server down, history route unavailable) is blocked from recording the cycle. Mitigation: the `ledger:'failed'` declaration exists precisely for this — run the auditor, let it fail, declare `failed` with the reason in `notes`, and the record is accepted honestly. The refusal message teaches the producing CLI including the cross-agent flag. The cross-check can also over-block if an agent files findings locally and the ledger write genuinely succeeded but into a DIFFERENT framework value — the check is framework-agnostic (any framework's dedup key counts), which minimizes this.
+
+## 2. Under-block
+
+Fabrication remains possible for `remote` / `dry-run` / `failed` declarations — the route cannot verify another server's ledger without cross-agent auth machinery (out of scope for this slice). Accepted deliberately: the declaration is durably recorded and queryable, so a pattern of suspicious declarations is itself visible to meta-analysis. The operatorSeatUx block has the same trust model. A future slice can verify `remote` claims over Threadline.
+
+## 3. Level-of-abstraction fit
+
+The shape gate lives in the store (single writer, same place as the operatorSeatUx gate — the two halves of the same observation record). The ledger cross-check lives in the route because only the route has the FrameworkIssueLedger dependency; the store stays persistence-only. The CLI split is read-side only; the write path is untouched.
+
+## 4. Blast radius
+
+- New column `transcript_audit_json` via the established idempotent ALTER pattern; legacy rows read as null. No data rewrite.
+- One existing caller class affected: mentor sessions recording telegram-playwright cycles over HTTP. They get a teaching 400 until they supply the block — this is the designed behavior (same rollout shape as #856, which the mentor loop absorbed same-day). Standing orders updated in the same PR so newly-spawned mentor sessions know before they hit it.
+- Non-telegram-playwright callers: zero behavior change unless they supply a malformed block (which previously would have been silently ignored — now validated).
+- The CLI defaults preserve #864 single-server behavior byte-for-byte when `--history-base-url` is omitted.
+
+## 5. Failure modes
+
+- Remote history server unreachable → auditor CLI errors per-topic read; mentor declares `ledger:'failed'`. No silent skip.
+- No FrameworkIssueLedger on the recording server → cross-check skipped (declaration recorded); shape still validated. Documented in tests.
+- Corrupt stored audit JSON (hand-edited row) → reads as null via the lenient parse path with `@silent-fallback-ok` justification, mirroring operatorSeatUx.
+
+## 6. Security
+
+The local auth token is never sent to a remote history server: remote reads use `--history-auth-token` / `INSTAR_HISTORY_AUTH_TOKEN` or no header at all. Token-via-flag is discouraged in the help text (ps visibility); the env var is the documented path.
+
+## 7. Migration parity
+
+- New agents: template teaches the new POST shape.
+- Existing agents: targeted PostUpdateMigrator line-rewrite (idempotent, content-sniffed on old-line-present + new-marker-absent) updates the apprenticeship section they already carry.
+- Mentor loop: standing orders (buildAutoloopGoal step 5) teach audit-then-record including the cross-agent read.

package/upgrades/side-effects/live-tail-event-loop-guards.md

@@ -0,0 +1,77 @@
+# Side-Effects Review — Live-Tail Event-Loop Guards
+
+**Version / slug:** `live-tail-event-loop-guards`
+**Date:** `2026-06-05`
+**Author:** `Echo (instar-dev agent)`
+**Second-pass reviewer:** `independent reviewer subagent — CONCUR (one non-blocking memory-retention fix + two doc-precision notes, all applied)`
+
+## Summary of the change
+
+The multi-machine live-tail streamer (`LiveTailSource.pushTick`, every `liveTailPushRateMs`=5s while holding the lease) rebuilt EVERY known topic's content EVERY tick, and the Telegram content provider (`TelegramAdapter.getTopicHistory`) resolved each rebuild with a synchronous full read of the JSONL message log (≤75k lines) — measured on 2026-06-05 as 5–40s event-loop blocks that went on to stale mesh timestamps → standby 403s → tick-rate hot retries (a self-amplifying storm). This change adds: a per-topic monotonic content version (`TelegramAdapter.appendToLog` bump → `getTopicContentVersion`) consumed by `LiveTailSource` as an optional `getTopicVersion` dep so unchanged topics are skipped WITHOUT building content; an in-memory per-topic tail cache behind `getTopicHistory` (single-pass batch seed of all live topics, maintained on append, lazy per-topic fallback, reclaimed on `unregisterTopic`); exponential per-topic failure backoff (5s base ×2 → 5min cap) replacing tick-rate retries; a 256KiB per-flush content cap (freshest suffix); and a `{ force: true }` handoff path that bypasses gate+backoff (`handoffSentinelBootWiring`). Files: `LiveTailSource.ts`, `TelegramAdapter.ts`, `handoffSentinelBootWiring.ts`, one wiring line in `server.ts`, four test files.
+
+## Decision-point inventory
+
+- `LiveTailSource.flushTopic` — **modify** — adds skip conditions (unchanged version / inside backoff window) and a size cap; the delta-accounting model (never advance `seq`/`streamed` on failure) is unchanged.
+- `TelegramAdapter.getTopicHistory` — **modify (transparent)** — same results, served from memory; byte-parity with a fresh file scan is test-pinned (the handoff hash depends on it).
+- `TelegramAdapter.appendToLog` — **modify (additive)** — bumps the version counter + maintains the cache before either persistence path; persistence behavior unchanged.
+- `handoffSentinelBootWiring.pushTick` — **modify** — forces (gate/backoff bypass) so a handoff manifest can never silently drop a mid-backoff topic.
+- `LiveTailSource` backoff ledger — **add** — a pure cadence suppressor; holds no authority over WHAT is eventually streamed.
+
+## 1. Over-block
+
+**What legitimate inputs does this change reject that it shouldn't?**
+
+Nothing is rejected — only deferred or trimmed, all bounded: (a) a topic mid-backoff delays its retry ≤5min (cleared instantly on any success; bypassed entirely by the handoff force path, so a planned handoff can never be starved by the window); (b) the version gate skips only topics whose version — bumped by the single funnel every logged message passes through — is unchanged, and an identical-content no-op records the version only after byte-comparing the real content; (c) the 256KiB cap trims a flush to its freshest suffix — the standby's `LiveTailBuffer` enforces its own independent per-topic byte cap, and the reviewer verified `getTail()` currently has no consumer (post-failover history is reconstructed from each machine's own message log, not the buffer), so trimmed middle bytes have no downstream reader today.
+
+## 2. Under-block
+
+**What failure modes does this still miss?**
+
+(a) The mesh-RPC clock-tolerance rejection itself is untouched (correct behavior; the defect was upstream). (b) Sync child-process spawns in OTHER timer ticks (reaper/backstop `execSync` paths) remain — separate subsystem, explicitly tracked as follow-up in the multi-machine loop-safety audit <!-- tracked: CMT-1109 --> (topic "Resource Limitation Mitigation"). (c) The backoff is per-topic, not per-peer: with many topics all failing against one dead peer, first-attempt cost is still N serializations once per window — bounded by the cap and far below tick-rate, acceptable. (d) `maxFlushBytes` counts UTF-16 chars, not UTF-8 bytes (≈3× under-count for heavily non-ASCII content) — sender-side cost bound only; the standby caps independently (doc-noted per reviewer).
+
+## 3. Level-of-abstraction fit
+
+**Is this at the right layer?**
+
+Yes. Each guard sits exactly where its cost is generated: the version signal lives in the adapter that owns message logging (the only component that KNOWS when content changes); the gate/backoff/cap live in the source that generates the serialization work; the cache lives behind the existing `getTopicHistory` contract (every consumer — respawn history, handoff hash, live-tail — speeds up without knowing). No new authority, no new config surface, no parallel mechanism duplicating the standby buffer's own caps. The backoff mirrors the established suppressor shape (`AgeKillBackoff`, `AttentionTopicGuard`): pure logic, injectable clock, bounded state.
+
+## 4. Signal vs authority compliance
+
+**Required reference:** `docs/signal-vs-authority.md`
+
+- [x] No — this change produces/consumes signals (a change counter, a "skip until" window, a size bound) inside an existing data-freshness pipeline; it holds NO authority over sessions, leases, kills, or message delivery.
+
+The version counter cannot suppress a real change (every logged message bumps it; the reviewer traced all ingress/egress paths through `appendToLog`). The backoff cannot drop content (state never advances on failure; the owed delta sends when the window opens or a handoff forces). The cap cannot starve the standby (its own buffer cap is lower-or-equal in practice). Lease/handoff authority is untouched.
+
+## 5. Interactions
+
+- **Handoff hash parity (the critical one):** both machines hash their OWN `getTopicHistory(topic, 500)`. The cache stores the same entry objects the file receives, in the same order (append updates both in one synchronous call; the seed reads file order). Reviewer confirmed `hashTopicHistory` reads only `timestamp` + `text` (primitive strings in every shape) and ran the real two-server `planned-handoff-e2e` — including the hash-mismatch abort path — green through the cache.
+- **Version capture timing:** `version` is captured in `flushTopic` together with the content build (no `await` between them) and recorded only on success — a message arriving during the `await broadcast` bumps the live counter ABOVE the recorded snapshot, so the next tick correctly re-opens. (Reviewer probed this specifically.)
+- **Double-fire / races:** the cache+counter are plain in-process Maps touched only from synchronous adapter calls; `seedTailCacheFromLog` is fully synchronous (no event-loop yield between read and `tailCacheSeeded=true`), so no append can fall between seed-read and cache-live. Appends during a broadcast `await` land in the seeded cache and bump the version consistently.
+- **Feedback loops:** the change only REMOVES a feedback loop (frozen loop → stale timestamps → rejects → hot retries → more freeze). Skipping/deferring work cannot trigger more work.
+- **Shadowing:** force on the handoff path ensures the new skip conditions can never shadow the handoff manifest's freshness requirement (test-pinned `pushTick:force`).
+
+## 6. External surfaces
+
+- **Other agents / users:** behavior ships fleet-wide on update; observable change is fewer event-loop stalls, fewer mesh 403 storms, fewer log lines ("not acknowledged — will retry" now carries a backoff horizon, plus capped/truncation lines). No API, schema, or message-shape change.
+- **External systems:** none beyond the existing machine-to-machine live-tail channel (same wire format, same encryption; only cadence + max size change).
+- **Persistent state:** none. Counter + cache + backoff ledger are in-memory; nothing written to disk; no migration. (Migration Parity: nothing to migrate — defaults live in code.)
+- **Timing:** standby copy freshness for a FAILING peer degrades from "retry every 5s" to "retry ≤5min" — the deliberate trade; a HEALTHY peer's freshness is unchanged (new content still flushes on the next tick).
+
+## 7. Rollback cost
+
+Pure in-process code change. Revert the commit and ship a patch. No persistent state to clean, no config to unwind, no schema. Per-component soft-disable exists naturally for tests (omit `getTopicVersion` → pre-fix gating behavior); no operator dial is exposed deliberately — the pre-fix behavior is a measured pathology with no legitimate operating point.
+
+## Conclusion
+
+The review confirmed the three invariants that matter: (a) the standby converges to the SAME content as before — only the cost/cadence of getting there changes (handoff-hash parity test-pinned and e2e-verified); (b) no authority moved — every new mechanism is a bounded cadence/size signal inside an existing pipeline; (c) rollback is a trivial revert with zero persistent state. One reviewer finding changed code: `unregisterTopic` now reclaims the topic's tail cache (long-lived servers churning topics must not retain ≤500 entries per topic forever) — the version counter is deliberately retained (~8 bytes) to avoid a re-registration version-collision against `lastSeenVersion` snapshots.
+
+---
+
+## Phase 5 — Second-pass review (cross-machine continuity change → performed)
+
+An independent reviewer subagent audited the staged diff, the final files, the spec, and the wiring at line level, specifically probing: handoff-hash divergence through the cache (including entry order, JSON round-trip shape, and the `channelId` compat path), version-gate miss scenarios (other writers, rotation, multi-instance), cache coherence windows, backoff-vs-gate retry semantics and version-capture timing, content-cap interaction with the delta model and `LiveTailBuffer` semantics, memory bounds, the force chain, and ran the unit + neighbor + real two-server handoff e2e suites (all green, tsc clean).
+
+**Verdict: CONCUR.** No blocking defects. Three non-blocking findings, all addressed: (1) `unregisterTopic` did not reclaim the tail cache → fixed (cache deleted on unregister; counter deliberately kept, with the collision rationale documented inline); (2) the `maxFlushBytes` chars-vs-bytes approximation overstated "matching the standby ceiling" → comment and this artifact corrected to "sender-side cost bound; standby caps independently"; (3) the cache-vs-file parity test is stricter than the hash requires and would read as a false regression if the shared-MessageLogger flag ever flips → scope-noted in the test.
+
+**Post-CI note (same day):** shard-3 ratchet (no-silent-fallbacks) flagged the seed's per-line malformed-JSONL skip (460 > 459 baseline); annotated `@silent-fallback-ok` inside the catch per the report-or-exempt rule — identical deliberate behavior to the scan path's long-standing skip.