instar 1.3.739 → 1.3.741

package/upgrades/side-effects/prompt-parser-contract-standard.md

@@ -0,0 +1,67 @@
+# Side-Effects Review — Prompt↔Parser Contract Standard (defect class 1, dark increment)
+
+**Version / slug:** `prompt-parser-contract-standard`
+**Date:** `2026-07-03`
+**Author:** `echo`
+**Tier:** `1 (dark increment — additive library + classification + pinned ratchet; no runtime wiring, no live prompt/parser change, no operator-gated text)`
+
+## Summary of the change
+
+The mechanical arm of the "The Prompt and the Parser Are One Contract" standard (defect class 1 closure; `docs/specs/prompt-parser-contract-standard.md`), shipped as a self-contained DARK increment. It ships:
+
+1. **`src/core/promptContract.ts`** — the shared prompt↔parser contract library. The `PromptContract` manifest type (a co-located, machine-readable promise carried next to a prose-shaped prompt), the `ContractForm` election type, and `deriveRejectedForms(vocabulary, extras, options)` — a PURE generator for the fail-closed counter-examples a per-callsite contract test feeds the REAL parser (case-mutation, prefix-truncation at each separator — the exact B15 shape — separator-stripping, plus hand-picked extras, minus any form that collides with a promised token). No runtime caller in this increment.
+2. **`src/data/llmBenchCoverage.ts`** (additive) — `LLM_PARSER_CONTRACT`, the `contract` axis of the program's shared per-callsite metadata record. Required-explicit for every `COMPONENT_CATEGORY` key; the four spec-named highest-stakes callsites seed `contract-wave-1`, other enumerated-verdict callsites are `contract-wave-2`, and a no-closed-vocabulary callsite is `{ false: '<reason>' }`.
+3. **`tests/unit/parser-contract-classification-ratchet.test.ts`** — required-explicit + no-dangling + valid-wave + wave-1 seed floor + shrink-only pending + shrink-only argued-false + real-reason floor + the gate/sentinel cross-check lint.
+4. **`tests/unit/promptContract.test.ts`** — the derive-rejected-forms mutation logic (incl. the B15 prefix shape, collision-exclusion, purity) and the manifest type surface.
+5. **`docs/specs/prompt-parser-contract-standard.md`** + **`.eli16.md`** — the converged spec and its plain-English overview.
+
+## What is DELIBERATELY out of scope (not orphan deferrals)
+
+- **The per-callsite contract tests + the live-builder render refactor (spec rollout §0/§1).** A contract test renders the REAL production prompt through an exported pure render function; several production builders (e.g. `MessagingToneGate`'s prompt builder) are private instance methods with live deps and need that render refactor, which TOUCHES live parsing code. Per the CAREFUL constraint on this defect class (live, load-bearing parsers), that refactor is deferred to its own A/B-gated increments, one callsite at a time, so live parse behavior is UNCHANGED on this merge. The four siblings deferred their behavioral/render arms for exactly this reason; this increment mirrors that.
+- **The runtime contract-drift warning (spec Frontloaded Decision #3 / "Decision points touched" (a)).** A signal-only prompt-build-time assertion that also touches the live builders; it lands with the by-construction single-sourcing migrations, not this inventory increment.
+- **The registry / constitution text (spec §1).** Operator-gated — ships ONLY with Justin's explicit sign-off (spec front-matter `operator-gate`). The registry entry is DRAFTED in the spec; this run does not write `docs/STANDARDS-REGISTRY.md`.
+
+## Decision-point inventory
+
+- `src/core/promptContract.ts` — **add** — a pure library (types + string/array helpers). NO runtime caller in this increment (dark by construction); it changes NO prompt text and NO parser until a per-callsite contract test + render refactor lands through its own gate. No allow/deny surface.
+- `LLM_PARSER_CONTRACT` classification (`src/data/llmBenchCoverage.ts`) — **add** — build-time metadata only. Read by the new pinned ratchet; no runtime consumer. A SIGNAL producer (which callsites parse a taught output vocabulary), never a runtime authority.
+- The two new vitest tests — **add** — CI-only pinned baselines (same family as `llm-bench-coverage-ratchet`). The ratchet gates the build (adding an unclassified callsite / growing the pending or false set silently is red CI), never a runtime path.
+
+---
+
+## 1. Over-block
+
+**What legitimate inputs does this change reject that it shouldn't?**
+
+None at runtime — nothing is wired to a runtime allow/deny path, and no live prompt or parser changes, so no model answer that was accepted before is rejected now. At CI/build time the only new red-CI surfaces are: (a) adding a new LLM component to `COMPONENT_CATEGORY` without an explicit `contract` classification, (b) growing the pending or argued-false set without editing the pinned baseline, and (c) marking a gate/sentinel `false` without the reviewed allowlist. All three are intentional, self-describing failures with fix-instructions in the assertion message — the "visible, reviewed act" the standard exists to force, not an over-block of legitimate work.
+
+## 2. Under-block
+
+**What failure modes does this still miss?**
+
+- This increment adds NO test that a callsite's prompt and parser are ACTUALLY coherent — that is the per-callsite contract test, deferred with its render refactor. So a classified-`pending` callsite can still ship a drifted prompt↔parser until its contract test graduates. This is the known staged-rollout gap, tracked in the spec's rollout §0→§3, not a silent miss — the pending set IS the honest inventory of that remaining work.
+- Classification accuracy is author-judged against a stated criterion (a CLOSED taught verdict vocabulary → needs a contract; free-text / open-set / no-live-prompt → `false`). A mislabel of a gate/sentinel is partly caught by the cross-check (the highest-risk category); a reflector/job mislabeled `false` would pass. The argued-false shrink-only pin makes every such call visible and reviewed at PR time; this seeding PR IS that review.
+
+## 3. Level-of-abstraction fit
+
+**Is this at the right layer?**
+
+Yes. The contract library is a pure `src/core/` module (the layer where shared prompt/parser scaffolding belongs — sibling to `promptClauses.ts` from the authority-clause standard); the classification extends the ONE shared metadata record the program mandates (`src/data/llmBenchCoverage.ts`), not a new parallel registry; the ratchet is in the established pinned-baseline vitest family. No higher gate already owns "does the prompt's taught vocabulary match the parser"; no lower primitive is duplicated.
+
+## 4. Signal vs authority compliance
+
+The classification and the ratchet are SIGNALS (which callsites parse a taught vocabulary; is the pending/false inventory shrink-only) — they inform the build and the reviewer, never grant or deny a runtime action. `deriveRejectedForms` returns an array; it is inert until a contract test consumes it. No model-produced field is wired to satisfy any authorization check. The library changes no live parser's accept/reject decision.
+
+## 5. Interactions with existing systems
+
+- **`llm-bench-coverage-ratchet` / `untrusted-input-classification-ratchet`** — unaffected: `LLM_PARSER_CONTRACT` is a NEW additive export; the existing `LLM_BENCH_COVERAGE` and `LLM_UNTRUSTED_INPUT` records and their ratchets are untouched (all green post-change). The argued-false membership is consistent with the sibling axes' "no live judging/parsing callsite" reasoning where they overlap (PromiseBeacon, InteractivePoolCanaryJudge, AutoApprover, IntegrationGate, CoherenceGate, InputDetector), and adds the free-text-content reflectors/summarizers on top (which parse no closed verdict vocabulary).
+- **`lint-scrape-fixture-realness`** — the new helper is named `deriveRejectedForms` (not `parse*`/`scrape*`), so it is deliberately outside that lint's `parse*`/`scrape*` surface; it consumes no untrusted real-world text (it operates on a callsite's own declared vocabulary constant).
+- **green-PR auto-merge protected paths** — `src/data/llmBenchCoverage.ts` is already in the class-closure agent-authored-artifact predicate; the pinned ratchet baselines route every future classification/pending/false edit to operator review while a fully-conforming graduation (pending→contractTest) keeps auto-merge (program-shared machinery).
+
+## 6. Failure modes / rollback
+
+Pure additive TypeScript + tests + docs. Rollback = revert the commit; nothing persists state, nothing runs at runtime, no migration, no config key (no agent-side config key exists or is wanted — repo posture only, so no Migration Parity work). `tsc --noEmit` clean, `npm run lint` exit 0, all new + adjacent ratchets green under bounded single-file vitest runs (the machine has intermittent external CPU pressure; verification used bounded runs per the CI-as-gate pattern, full suite gated by CI).
+
+## 7. Second-pass reviewer
+
+Self-review (bounded machine-pressure build). The change is additive, dark, and test-pinned; no runtime surface and no live prompt/parser touched. Key self-checks: (1) confirmed `deriveRejectedForms` never emits a promised token as a "rejected form" (the collision-exclusion test), so a future contract test's accept/reject assertions cannot contradict; (2) confirmed the classification covers all 53 `COMPONENT_CATEGORY` keys exactly once (the required-explicit + no-dangling ratchet asserts it); (3) confirmed the wave-1 seed is exactly the four spec-named highest-stakes callsites; (4) confirmed no `src/` file outside tests imports `promptContract.ts`, so the increment is dark by construction.

package/upgrades/side-effects/reaper-log-flood-fix.eli16.md

@@ -0,0 +1,39 @@
+# Reaper log-flood fix — Plain-English Overview
+
+> The one-line version: the reaper was scribbling the same "couldn't clean this up" note into an audit file thousands of times a minute until the file grew to 142MB and froze the whole server when anything read it — this makes it write the note once and read the file in small bites.
+
+## The problem in one breath
+
+Every agent has a background "reaper" that tidies up finished sessions. When a session CAN'T be tidied yet (it has an open promise to the user, or it belongs to another machine), the reaper politely skips it — and writes a one-line "skipped, here's why" note to an audit log. The bug: it re-checks and re-writes that identical note on every single tick, forever. On a live agent that piled up 3,218 + 1,608 duplicate notes and grew the log to 142MB. Worse, the code that READS that log loaded the entire 142MB into memory at once, which pegged a CPU and froze the server for ~90 seconds each time.
+
+## What already exists
+
+- **The reaper** — the background tidier that decides which sessions to close. Its actual close/keep decisions are correct and are NOT changed by this work.
+- **`reap-log.jsonl`** — the "why did my session vanish?" audit trail. One line per close, one line per refused close. This is the file that ballooned.
+- **`reaper-audit.jsonl`** — a SEPARATE, sibling audit file that already does the right thing: it only writes when a decision CHANGES, not on every tick. It stayed tiny (under half a megabyte). We copy its proven habit.
+
+## What this adds
+
+The reaper's skip-note now follows the same "only write when something changed" rule its sibling already uses. If a session is skipped for the exact same reason as last time, no new note is written. The moment the reason changes — the promise closes, the session moves machines, or it finally gets reaped — a fresh note is written, so the audit trail still tells the true story with none of the spam.
+
+Two more safety belts:
+
+- **Small-bite reading** — reading the log now pulls only the last couple of megabytes off the end of the file instead of the whole thing, so even a giant log can never freeze the server again.
+- **Auto-rotation** — if the file ever crosses 16MB anyway, it's instantly rolled aside to a single backup and a fresh file is started, so it can never grow without bound. Reading transparently stitches the backup back in so recent history is never lost.
+
+## The new pieces
+
+- **Transition dedup (in-memory)** — a tiny table remembering each session's last-written skip reason. It's per-machine, capped at 2,000 entries (sessions are only ever a few dozen in practice), and forgotten the instant a session is reaped so it can't leak. It is NOT allowed to change any close/keep decision — it only decides whether a LOG LINE is worth writing.
+- **Bounded tail reader** — reads the end of the file only. It carefully drops a half-a-line at the read boundary so a torn record is never mis-parsed.
+- **Size-cap rotation** — an instant rename (no slow file rewrite) when the file gets too big.
+
+## The safeguards
+
+- The reaper's real authority — which sessions live or die — is untouched. This is purely about how much it writes and how the file is read.
+- Nothing is lost from the audit story: first-skip, every change-of-reason, and every actual reap are all still recorded. Only the identical-to-last-time repeats are dropped.
+- It's per-machine by design (each machine audits its own reaping), matching the sibling file. No cross-machine syncing to get wrong.
+- Rollback is a one-file revert with no data migration — old fat logs and new backups both read correctly through the new reader.
+
+## What you need to decide
+
+Nothing structural — this is a contained, low-risk hygiene fix with full unit-test coverage (16 focused tests; 115 reaper-family tests green). The one judgment call baked in: we accept losing the per-tick "still skipped at 12:00:01, still skipped at 12:00:02…" heartbeat in exchange for a log that doesn't self-destruct. That trade matches what the sibling audit file already chose, so it's a consistent, already-proven decision rather than a new gamble.

package/upgrades/side-effects/reaper-log-flood-fix.md

@@ -0,0 +1,90 @@
+# Side-Effects Review — Reaper self-inflicted log-flood fix (ReapLog transition-dedup + bounded read + rotation)
+
+**Version / slug:** `reaper-log-flood-fix`
+**Date:** `2026-07-03`
+**Author:** `Echo (instar-dev agent)`
+**Second-pass reviewer:** `reviewer subagent (see appended concurrence)`
+
+## Summary of the change
+
+Three self-limiting changes to `src/monitoring/ReapLog.ts`, the append-only audit trail behind `GET /sessions/reap-log`:
+
+1. **Write-side transition dedup (primary cure).** `recordSkipped()` now logs on TRANSITION only — it keeps an in-memory `session → last-logged skip signature (${reason}::${skipped})` map and drops a re-append when the signature is unchanged. The reaper re-evaluates a permanently-vetoed session (`open-commitment`, `not-lease-holder`, `protected`) on every tick and previously appended an identical `skipped` row each time — on a live agent this produced 3218 `open-commitment` + 1608 `not-lease-holder` repeat rows and grew the log to 142MB / 463k lines (observed 2026-07-03). The dedup state is cleared when the session is reaped (`recordReaped` → `forgetSkip`) so a same-named successor logs fresh and the map cannot leak. The map has a hard ceiling (`MAX_SKIP_STATE`, default 2000, oldest-pruned).
+2. **Read-side bounded tail (fixes the freeze-on-read).** `read()` previously did `fs.readFileSync(WHOLE file).split('\n')` just to return the last `limit` rows — a 142MB slurp + split that blocked the event loop when the route was queried. It now reads only the last `TAIL_READ_BYTES` (default 2MB) of the file via a positional `readSync`, drops a leading partial line, and merges the rotated `.1` tail if the live file doesn't yield enough rows.
+3. **Size-cap rotation (defense-in-depth).** `append()` rolls the file to `<path>.1` (O(1) `renameSync`, no data rewrite on the hot path) once it would cross `MAX_LOG_BYTES` (default 16MB), so the file can never grow unbounded even if a future caller floods it. One backup generation is retained; `read()` merges it.
+
+The caps are overridable via an optional `ReapLogOptions` constructor arg (defaults = module constants) purely so tests can trigger rotation cheaply. The production callsite (`src/commands/server.ts`) is unchanged.
+
+## Decision-point inventory
+
+This change touches NO block/allow/gate decision point. `ReapLog` is a read-only observability sink — the class docstring states it "never gates or mutates a session — it only records." The reaper's keep/kill authority (`SessionManager` / `SessionReaper`) is entirely untouched; only the LOGGING cadence and READ mechanics change.
+
+- `ReapLog.recordSkipped` — modify — logs on transition instead of every tick (observability cadence only).
+- `ReapLog.read` — modify — bounded tail read instead of whole-file slurp (I/O mechanics only).
+- `ReapLog.append` — modify — adds O(1) size-cap rotation (I/O mechanics only).
+
+---
+
+## 1. Over-block
+
+No block/allow surface — over-block not applicable. `ReapLog` never blocks a session. The only "suppression" is of DUPLICATE LOG ROWS for an unchanged skip state, which is the intended behavior (mirrors the existing `reaper-audit.jsonl` log-on-transition pattern).
+
+---
+
+## 2. Under-block
+
+No block/allow surface — under-block not applicable. In audit-fidelity terms, the analogue is "what does the log no longer record?": it no longer records a fresh row for an UNCHANGED skip state on each tick. It STILL records: the first skip (transition into skipped), every change of skip reason, and every reap. A reviewer asking "is this session still being skipped?" reads the most recent skip row plus the absence of a later reap — the same answer, without 5000 duplicate rows. Loss of the per-tick "still skipped at time T" heartbeat is intentional and matches reaper-audit.
+
+---
+
+## 3. Level-of-abstraction fit
+
+Correct layer. The dedup lives INSIDE `ReapLog` (the class that owns the log), so it covers every caller — not bolted onto the single `reapBlocked` event wiring in `server.ts`, which would leave other/future callers un-deduped. The read/rotation are pure file mechanics that belong on the class that owns the file. This is a detector/observability primitive, not an authority — it produces no signal consumed by any gate. It mirrors the already-shipped `reaper-audit.jsonl` transition pattern rather than inventing a new one.
+
+---
+
+## 4. Signal vs authority compliance
+
+Compliant. The change adds NO blocking authority and NO brittle decision logic in front of any gate. It is a data-sink optimization: fewer duplicate rows written, bounded bytes read. `docs/signal-vs-authority.md` governs decision points; this change has none.
+
+---
+
+## 5. Interactions
+
+- **Mirrors, does not shadow, `reaper-audit.jsonl`.** That sink already logs reaper decisions on transition; this brings `reap-log.jsonl`'s skip rows to the same discipline. They remain separate files with separate purposes.
+- **`recordReaped` → `forgetSkip` ordering.** Reap clears the skip-state so a same-named successor logs fresh. A reap for a session never previously skipped is a harmless no-op delete.
+- **No race with cleanup.** `ReapLog` is synchronous and single-process; the rotation `renameSync` is atomic. A concurrent reader mid-rotation either sees the old file (pre-rename) or the new empty file + `.1` (post-rename) — `read()` merges `.1`, so no window loses the newest rows.
+- **`normalizeEntry` untouched** — the notify/skipped/reaped type + field whitelist that other tests guard is unchanged; all 115 reaper-family unit tests pass.
+
+---
+
+## 6. External surfaces
+
+- **`GET /sessions/reap-log`** — same JSON shape, same newest-last ordering, same `limit` semantics. The only observable difference: far fewer duplicate `skipped` rows, and the route no longer blocks the event loop on a large file. No API contract change.
+- No change visible to other agents or users beyond the route above. No dependency on timing or conversation state.
+- **Pre-existing on-disk bloat is not rewritten by this change.** A file already at 142MB stays until the next rotation trims it (or an operator archives it, as was done live on 2026-07-03). The bounded READ means even a still-large file no longer freezes the loop; the write-side dedup means it stops growing.
+
+---
+
+## 7. Multi-machine posture (Cross-Machine Coherence)
+
+**Machine-local BY DESIGN.** `reap-log.jsonl` is a per-machine audit of that machine's own reap decisions (the log path is under the machine's own `logs/`, and `GET /sessions/reap-log` already reports only the local machine's entries — pool-scope reap history is not a feature). The in-memory `skipState` dedup map is likewise per-process and correctly scoped to the machine doing the reaping. No replication, no proxied read, no cross-machine state. This matches the existing `reaper-audit.jsonl` posture. No topic-transfer, one-voice, or URL-durability concerns (it is not user-facing and generates no links).
+
+---
+
+## 8. Rollback cost
+
+Low. The change is contained to one file (`ReapLog.ts`) plus its unit tests. Back-out is a straight revert — no data migration (the on-disk format is unchanged; older whole-file logs and `.1` backups both read correctly through the new tail reader). If the transition-dedup were ever suspected of hiding a needed row, reverting restores per-tick logging immediately. No agent-state repair, no hot-fix data surgery. The caps are constructor-overridable if a deployment needs different thresholds without a code change.
+
+---
+
+## Second-pass review (independent reviewer subagent)
+
+**Concur with the review.** The change is observability-only and touches no kill/keep decision — the reaper still evaluates every tick, so the transition-dedup can never strand an operator's "why did my session vanish / not get reaped?" answer.
+
+1. **Transition-dedup:** correctly keyed on `${reason}::${skipped}`; suppresses only a byte-identical re-evaluation of the same permanent veto. Any genuine change (veto lifts → reap row always appends since `recordReaped` has no dedup; reason changes → new signature → new row) is still logged. `forgetSkip` on reap prevents both the stale-successor bug and unbounded map growth, backed by the `MAX_SKIP_STATE` oldest-first ceiling. No operator-needed row is hidden.
+2. **Bounded read:** traced the merge/tail logic — older `.1` correctly prepended to newer live, `.slice(-limit)` keeps the newest, leading partial line dropped only when `start > 0`, trailing empty line filtered, `.1` read only when live yields `< wanted`. No off-by-one, no torn record, newest row never dropped, no cross-file duplication (rename→fresh-empty-live).
+3. **Rotation:** O(1) `renameSync` off the hot path; `approxSize` seeded once then advanced per-write (no per-append stat); a pre-existing 142MB file rotates on the first append. On-disk bounded to `live (≤cap) + .1 (≤cap)` since each rotation overwrites the prior `.1`. Concurrent reader sees a consistent pre/post-rename state, both handled by `read()`.
+4. **Signal-vs-authority:** no blocking authority added; `SessionReaper`/`SessionManager` kill logic untouched; audit-sink failures swallow silently rather than perturbing behavior.
+
+No correctness defect found. Tests exercise both sides of the dedup boundary, the rotation/merge boundary, and the absent-file path. Sound to merge.