instar 0.28.78 → 0.28.80

package/upgrades/side-effects/presence-proxy-ack-and-baseline.md

@@ -0,0 +1,260 @@
+---
+title: PresenceProxy — brief-ack tolerance + post-message baseline
+slug: presence-proxy-ack-and-baseline
+date: 2026-05-04
+author: echo
+second_pass_required: false
+---
+
+## Summary of the change
+
+PresenceProxy emits tiered standby updates (20s / 2m / 5m) when an agent
+hasn't replied to a user message yet. Two regressions had silently
+broken the user-facing behavior:
+
+1. **Brief acks were cancelling all tier timers.** Telegram-bridged agents
+   are now instructed to send an immediate ack ("Got it, looking into
+   this", "On it") on every inbound message. The proxy interpreted that
+   ack as the agent's response and cancelled every pending tier check.
+   Result: the user never saw a 20s/2m/5m progressive update again — the
+   feature looked broken even though the timer machinery was intact.
+
+2. **Tier-summary prompts described pre-message work.** The prompts
+   read whatever was visible in the agent's tmux pane right now, which
+   is the rolling window — so older work from BEFORE the user's latest
+   message often dominated the snapshot. The user got summaries of work
+   the agent was already doing, not work the agent was doing in response
+   to their message.
+
+This change adds two things:
+
+- `isBriefAck(text)` — a length-bounded, opener-only pattern matcher that
+  classifies short forward-looking acks ("On it", "Got it, looking into
+  this") as non-cancelling. `onMessageLogged` now skips the cancellation
+  branch for brief acks (still records them on conversation history so
+  subsequent prompts know an ack went out).
+- `userMessageBaselineSnapshot` on `PresenceState`, captured in
+  `handleUserMessage` at the moment the user message arrives. Plus an
+  `extractDeltaSinceBaseline()` helper and a `buildScopedSnapshotBlock()`
+  method that the four tier-prompt builders now use to feed the LLM only
+  the post-baseline delta.
+
+Files touched:
+- `src/monitoring/PresenceProxy.ts` — new helpers, baseline capture,
+  ack-aware message handling, scoped prompt blocks.
+- `tests/unit/presence-proxy-ack-and-baseline.test.ts` — 15 new tests
+  across `isBriefAck`, `extractDeltaSinceBaseline`, brief-ack handling,
+  and baseline capture.
+
+## Decision-point inventory
+
+The change touches one decision point: PresenceProxy's
+"is this agent message a real response?" predicate, which gates timer
+cancellation. `isSystemOrProxyMessage` was already shared with several
+other subsystems (compaction recovery, stall triage, log scans); we
+intentionally did NOT modify that helper. Brief acks ARE real agent
+messages — they just shouldn't end the standby cycle. So the new check
+lives inside PresenceProxy's `onMessageLogged` branch only and does not
+leak into other subsystems' definition of "real reply."
+
+---
+
+## 1. Over-block
+
+The brief-ack filter is the only thing that could over-block. The
+"block" here is "block cancellation" — i.e., a real substantive reply
+gets misclassified as an ack and tier timers keep running. The user
+sees one extra standby message after the agent has already answered.
+
+Mitigations:
+
+- **Length cap of 200 chars.** Substantive replies tend to be longer.
+  200 is generous enough to cover compound acks ("Got it — looking into
+  both: foo and bar. On it.") but tight enough to exclude short
+  substantive answers in practice.
+- **Opening-only match (first 60 chars).** Patterns like `\bi['']?ll\s+(?:dig|look|...)` only fire when the message STARTS with that
+  phrase — a 200-char substantive reply that mentions "I'll get to that
+  next" deep in the body won't match.
+- **Conservative pattern list.** Generic "I will" / "let me" alone is
+  not enough — must be followed by an action verb (`dig`, `look`,
+  `check`, etc.). This was tightened in response to a failing test that
+  caught an over-match on a 267-char substantive plan.
+
+Worst case: tier 1 fires after a real reply, the user sees one
+"🔭 the-agent is currently …" message immediately after the substantive
+answer. No data loss, no repeated tier 2/3 because tier 1 reads the
+post-message terminal pane (which now contains the substantive reply)
+and produces a brief, accurate snapshot summary. Then the timers re-arm
+on the next user message.
+
+---
+
+## 2. Under-block
+
+Under-blocking the cancellation = a real substantive reply incorrectly
+classified as ack → timers fire when they shouldn't. Covered above.
+
+The other direction is "ack misclassified as substantive" → timers
+cancel as before, user sees no progressive updates. This is the bug we
+were already living with; our change can't make it worse than the
+status quo.
+
+---
+
+## 3. Level-of-abstraction fit
+
+Both new helpers are pure, exported, and live next to the other
+detectors (`detectQuotaExhaustion`, `detectSessionIdle`,
+`isLongRunningProcess`) in PresenceProxy.ts — same level of abstraction
+the file already operates at. No new modules, no new framework, no new
+queue. The state field (`userMessageBaselineSnapshot`) is a sibling of
+existing snapshot fields. The prompt-scoping helper
+(`buildScopedSnapshotBlock`) is a private method on the proxy class,
+co-located with the four prompt builders that consume it.
+
+The baseline snapshot is intentionally NOT persisted to disk
+(consistent with the existing policy for `tier1Snapshot` /
+`tier2Snapshot`) — too large, contains potentially sensitive content,
+and a session restart loses the original user-message moment anyway.
+After restart, `recoverFromRestart` sets the baseline to null and
+prompts fall back to the legacy "full pane" path with a `[scope: full
+pane — baseline anchor scrolled off]` label.
+
+---
+
+## 4. Signal vs authority compliance
+
+**Required reference:** [docs/signal-vs-authority.md](../../docs/signal-vs-authority.md)
+
+- [x] **`isBriefAck` is a brittle pattern-matching filter — a SIGNAL,
+      not an authority.** It does not block cancellation outright; it
+      simply withholds the cancellation that the proxy was about to
+      perform. The "authority" in this flow remains the natural one:
+      either a substantive reply lands and cancels timers, or the
+      agent's tier message comes from the LLM-backed prompt builder
+      (which has full context of conversation history). No brittle
+      filter is making a final decision on user experience.
+- [x] **Baseline scoping is also a signal-shaping change**, not an
+      authority change. The LLM still decides what to say in the tier
+      message; we just narrow the input so the decision happens on the
+      right context. If the baseline anchor can't be located, we
+      conservatively widen back to the full pane and label the prompt
+      so the LLM knows scope is best-effort.
+
+The dangerous failure mode in this kind of work is "brittle filter
+silently determines the user-facing outcome." Both fixes are
+specifically scoped to AVOID that: false positives on `isBriefAck`
+produce a slightly redundant user message (recoverable in the next
+message); false positives on the baseline anchor produce slightly less
+focused tier summaries (no worse than the pre-change behavior).
+
+---
+
+## 5. Interactions with adjacent subsystems
+
+- **CompactionSentinel.recoverFn** uses `findLastRealMessage` /
+  `isSystemOrProxyMessage` to decide whether to re-inject after
+  compaction. We did NOT modify those helpers — brief acks are still
+  considered "real" by that subsystem (which is correct: an ack IS a
+  real outbound message that the user saw). Only PresenceProxy's
+  cancellation logic treats brief acks specially.
+- **PromiseBeacon / shared LLM queue** — unchanged. Tier messages still
+  go through the same `interactive` lane and respect the daily spend
+  cap.
+- **ProxyCoordinator mutex** — unchanged. The mutex acquisition order
+  in `sendProxyMessage` is the same; we only changed prompt content
+  and added a new pre-cancel branch.
+- **Persisted state files** — schema unchanged; the new
+  `userMessageBaselineSnapshot` is an in-memory-only field. Existing
+  state files still load correctly (the spread in `recoverFromRestart`
+  picks up undefined for the new field, then we explicitly set it to
+  null).
+- **Tier 1 fallback (no LLM, intelligence: null)** — unchanged. Falls
+  back to the same templated message; the new snapshot scoping only
+  affects the LLM prompt, never the fallback path.
+
+---
+
+## 6. Rollback cost
+
+Single-file revert + test deletion. No schema changes, no on-disk
+artifacts, no API contract changes. If the brief-ack filter
+misbehaves in production we can:
+
+1. Empty the `BRIEF_ACK_PATTERNS` array — every agent message is
+   substantive again, behavior matches v0.28.79.
+2. Or revert the entire commit — same outcome, plus the prompts go
+   back to full-pane scope.
+
+Both rollbacks are atomic and require no migration.
+
+---
+
+## 7. Test coverage
+
+New tests in `tests/unit/presence-proxy-ack-and-baseline.test.ts`:
+
+- `isBriefAck` (5 tests):
+  - very short messages always ack
+  - forward-looking phrases under 200 chars are acks
+  - substantive multi-sentence replies (267 chars) are NOT acks
+  - empty/null/whitespace not classified as ack
+  - 280-char substantive cap (boundary)
+
+- `extractDeltaSinceBaseline` (5 tests):
+  - null/empty baseline → full current
+  - null current → empty
+  - anchor found → returns post-anchor content, anchored=true
+  - identical baseline+current → hasNewActivity=false
+  - anchor missing (terminal scrolled) → falls back to full current,
+    anchored=false
+
+- `PresenceProxy brief-ack handling` (3 tests):
+  - tier 1 + tier 2 both fire after brief ack
+  - substantive reply DOES cancel tiers
+  - multiple acks in sequence don't cancel; substantive reply finally does
+
+- `PresenceProxy baseline capture` (2 tests):
+  - baseline captured at user-message arrival
+  - capture failure doesn't crash, baseline stays null
+
+All 64 pre-existing PresenceProxy tests still pass — no regression in
+cancel-race, build-heartbeat suppression, idle detection,
+context-exhaustion, quota detection, or long-tool-wait paths.
+
+---
+
+## 8. Evidence
+
+- Repro source: Justin's message in topic 8882 (2026-05-04, 03:52
+  UTC) reporting "this feature no longer seems to give progressive
+  updates" and "messages from standby mode often seem to be
+  summarizing what the agent was working on BEFORE the user's last
+  message."
+- Root cause for #1: `handleAgentMessage` is called from
+  `onMessageLogged` for any non-system, non-proxy outbound message.
+  Telegram bridge instructions added an "On it" ack as the first
+  outbound message on every inbound user message → cancellation
+  fires before tier 1 can run.
+- Root cause for #2: tier prompts at lines 1192/1211/1244/1271
+  passed `snapshot.slice(0, 3000)` directly. The snapshot is the
+  full visible pane, with no boundary marker for "what was here
+  when the user's message arrived."
+
+---
+
+## 9. What this does NOT change
+
+- Tier 1 / 2 / 3 timing (still 20s / 2m / 5m by default).
+- LLM cost or model selection.
+- Persistence schema or on-disk state files.
+- `isSystemOrProxyMessage` / `findLastRealMessage` (shared with
+  compaction + stall triage).
+- Conversation-history capping.
+- Proxy mutex acquisition / release semantics.
+- `triggerManualTriage`, unstick, restart, quiet, resume command flows.
+
+The change is intentionally surgical: two well-defined behaviors
+(timer cancellation predicate + prompt input scoping) modified in
+their natural locations, with new pure helpers exposed for direct
+testing.

package/upgrades/side-effects/semantic-memory-corruption-recovery.md

@@ -0,0 +1,98 @@
+# Side-Effects Review — SemanticMemory corruption detection and auto-recovery
+
+**Version / slug:** `semantic-memory-corruption-recovery`
+**Date:** 2026-04-27
+**Author:** gfrankgva (contributor)
+**Second-pass reviewer:** Echo (EchoOfDawn), 3 review rounds
+
+## Summary of the change
+
+Three files touched:
+
+1. `src/core/types.ts` — `SemanticMemoryConfig` gains an optional `autoRebuildMaxBytes?: number` field (default 50 MB). No existing code passes this field, so all callers keep current behavior.
+
+2. `src/memory/SemanticMemory.ts` — `open()` gains an integrity check block mirroring TopicMemory's pattern:
+   - After opening the DB, runs `PRAGMA integrity_check`. If result is not `'ok'`, or if the pragma itself throws (severely corrupt DB), triggers recovery.
+   - **Secondary probe read**: If `integrity_check` passes, reads 100 rows from each existing table. Catches torn interior pages that `integrity_check` misses (pages not reachable from the B-tree schema walk).
+   - Recovery: calls `quarantineCorruptDb()` which renames the DB to `.corrupt.<timestamp>`, removes WAL/SHM sidecars, writes a JSON marker file. Falls back to delete if rename fails.
+   - After schema creation and vector init, checks `_needsRebuild` flag. If JSONL exists and is within the size gate, rebuilds synchronously. If JSONL exceeds `autoRebuildMaxBytes`, logs warning, starts empty, and writes a `skipped-rebuild` marker file.
+
+3. `tests/unit/semantic-memory-corruption-recovery.test.ts` — Test file with 12 contract-style tests covering: open-without-throwing, quarantine preservation, marker shape, sidecar cleanup (strengthened WAL/SHM assertions), JSONL rebuild, no-JSONL fresh start, healthy-DB no-op, severe-corruption pragma-throws path, partial-corruption (valid header + 4KB corrupted data page in 5000-row DB), size-gate skip, skipped-rebuild marker file, and subsequent-open stability.
+
+## Decision-point inventory
+
+- `SemanticMemoryConfig.autoRebuildMaxBytes` — **add** (type: optional number, default 50 MB).
+- `SemanticMemory.open()` integrity check block — **add** (new code path between DB constructor and pragma setup).
+- `SemanticMemory.quarantineCorruptDb()` — **add** (new private method).
+- `SemanticMemory._needsRebuild` — **add** (new private field, transient between integrity check and rebuild).
+- Auto-rebuild size gate — **add** (checks `fs.statSync(jsonlPath).size` against config limit).
+- `SemanticMemory` probe-read block — **add** (secondary detection after integrity_check passes; reads 100 rows from each existing table).
+- `SemanticMemory.writeSkippedRebuildMarker()` — **add** (new private method; writes `.skipped-rebuild.<ts>.marker.json` when size gate triggers).
+
+---
+
+## 1. Over-block
+
+**What legitimate inputs does this change reject that it shouldn't?**
+
+When JSONL exceeds `autoRebuildMaxBytes` (default 50 MB), the DB starts empty after corruption recovery. This means an operator with a large knowledge graph (> ~500k entities) would need to trigger `importFromJsonl()` manually after startup. This is deliberate — blocking server startup for minutes on a synchronous import is worse than starting with empty memory. The operator can rebuild during a maintenance window.
+
+The integrity check itself runs on every `open()` call, adding measurable startup latency for large DBs. TopicMemory pays the same cost, so consistency wins. If semantic DBs grow very large, a `quick_check` pragma (subset of integrity_check) could be a future optimization.
+
+## 2. Under-block
+
+**What failure modes does this still miss?**
+
+- **Mid-session corruption**: Only detected on `open()`. If a disk block goes bad during a running session, individual SQLite operations will throw but no automatic recovery triggers. This is out of scope — mid-session recovery would require connection pooling or shadow-DB switching, far beyond this PR's scope.
+- **Probe-read coverage**: The secondary probe reads 100 rows from each non-FTS table. Very large tables with corruption only in pages beyond the first 100 rows could theoretically pass the probe. In practice, 100 rows spans multiple 4KB pages, making this unlikely. Full table scans at startup would have unacceptable latency on large DBs.
+- **JSONL truncation**: If the JSONL was itself truncated (disk-full event during a write), the rebuild will be partial — some entities may be missing. The `importFromJsonl()` method handles malformed lines gracefully (skips them), so the rebuild is best-effort. The quarantined DB is preserved for forensic comparison.
+- **Writes not flushed to JSONL**: All mutation paths in SemanticMemory go through `remember()` / `addEdge()` which write to JSONL first (append), then to DB. The JSONL is the source of truth. There is no path where the DB is written first.
+
+## 3. Level-of-abstraction fit
+
+**Is this at the right layer?**
+
+Yes. SemanticMemory owns its DB lifecycle — `open()` is the correct place for integrity checks, matching TopicMemory's pattern. The quarantine logic is a private method, not exposed to callers. The size-gate config is on the existing `SemanticMemoryConfig` interface, which is the established place for tuning knobs.
+
+The alternative (adding a "health check" service layer above SemanticMemory) would scatter recovery logic across modules and require SemanticMemory to expose its DB state — worse encapsulation.
+
+## 4. Blocking authority
+
+- [x] No — this is a startup-time recovery mechanism. It does not gate any runtime operation. The only "decision" is quarantine-vs-keep, which is always quarantine (corruption is binary).
+
+## 5. Interactions
+
+- **Shadowing:** No existing corruption detection to shadow — SemanticMemory had none before this PR.
+- **Double-fire:** `_needsRebuild` is reset after the rebuild block. A second `open()` on the recovered DB is a no-op (tested).
+- **Races:** `open()` is async but the integrity check is synchronous (better-sqlite3 is sync). No concurrent access during startup.
+- **Downstream consumers:** Callers of `SemanticMemory.open()` (currently only `src/commands/server.ts`) see no behavioral change on healthy DBs. On corrupt DBs, `open()` succeeds instead of potentially throwing — strictly better.
+
+## 6. External surfaces
+
+- **Agents:** After corruption recovery, the knowledge graph may be rebuilt from JSONL (common case) or start empty (large JSONL). Agents notice "fewer memories" but server stays up — preferable to a crash loop.
+- **File system:** New files created during recovery: `.corrupt.<ts>` (quarantined DB), `.corrupt-recovery.<ts>.marker.json` (recovery marker). These accumulate over time — an operator might want periodic cleanup, but each occurrence is exceptional (disk errors).
+- **Persistent state:** The JSONL append log is never modified — only read during rebuild. The SQLite DB is replaced (quarantined + fresh). No other persistent state is touched.
+
+## 7. Rollback cost
+
+Pure code change. Revert removes the integrity check — corrupt DBs would again cause `open()` to either throw or silently serve bad data. No migration, no data repair needed on rollback.
+
+---
+
+## 8. Destructive-tool containment compliance
+
+`quarantineCorruptDb()` uses `fs.unlinkSync` to remove the corrupt DB and its WAL/SHM sidecars. Per the Comprehensive Destructive-Tool Containment spec (PRs #98/#99), all destructive filesystem calls must go through `SafeFsExecutor`. Updated:
+
+- `fs.unlinkSync(this.config.dbPath)` → `SafeFsExecutor.safeUnlinkSync(this.config.dbPath, { operation: 'SemanticMemory.quarantineCorruptDb' })`
+- `fs.unlinkSync(this.config.dbPath + ext)` → `SafeFsExecutor.safeUnlinkSync(this.config.dbPath + ext, { operation: 'SemanticMemory.quarantineCorruptDb:sidecar' })`
+
+The test file uses `fs.rmSync` in `afterEach` cleanup only (temp directory in `os.tmpdir()`). Annotated with `// safe-git-allow:` escape comment per the lint spec.
+
+---
+
+## Evidence pointers
+
+- Typecheck: `tsc --noEmit` — 0 errors.
+- Lint: `node scripts/lint-no-direct-destructive.js` — 0 violations.
+- Tests: 12 contract tests covering all recovery paths including partial corruption (valid SQLite header + 4KB corrupted data page in 5000-row DB), size-gate behavior, and skipped-rebuild marker.
+- TopicMemory parity: pattern mirrors `TopicMemory.open()` which has been production-stable since v0.27.x.

package/upgrades/side-effects/url-pathname-path-encoding-fix.md

@@ -0,0 +1,45 @@
+# Side-Effects Review — Eliminate URL.pathname path encoding across the codebase
+
+**Version / slug:** `url-pathname-path-encoding-fix`
+**Date:** 2026-04-28
+**Author:** gfrankgva (contributor)
+
+## Summary of the change
+
+Systematic replacement of `new URL(import.meta.url).pathname` with `__dirname` (or `fileURLToPath()`) across 13 source files. The former preserves `%20`-encoded spaces in filesystem paths, causing `fs.readFileSync`, `path.resolve`, and similar operations to fail when the project directory contains spaces.
+
+**Files changed (source):**
+- `src/commands/init.ts` (4 occurrences)
+- `src/commands/playbook.ts` (1)
+- `src/commands/server.ts` (4)
+- `src/commands/setup.ts` (3)
+- `src/core/Config.ts` (1)
+- `src/core/PostUpdateMigrator.ts` (2)
+- `src/core/SessionManager.ts` (1)
+- `src/core/UpdateChecker.ts` (1)
+- `src/core/UpgradeGuideProcessor.ts` (1)
+- `src/threadline/ThreadlineBootstrap.ts` (1)
+- `src/lifeline/ServerSupervisor.ts` (1)
+
+**Files changed (tests):** 5 test files with unquoted `execSync` paths or test expectation updates.
+
+**Files changed (generated):** `src/data/builtin-manifest.json` — content hashes updated to reflect changed source files.
+
+**Files changed (infrastructure):** `scripts/pre-push-gate.js` — added regression guard (check 5) that prevents re-introduction of the `URL.pathname` antipattern.
+
+## Decision-point inventory
+
+- All `new URL(import.meta.url).pathname` usages — **fix** (replace with `__dirname` or `fileURLToPath`).
+- No behavioral changes — every replacement produces the same decoded path, just without the `%20` encoding bug.
+
+---
+
+## 1–7. Analysis
+
+This is a pure bug fix with no behavioral, architectural, or security implications. Every replacement produces the identical filesystem path on systems without spaces, and the correct path on systems with spaces. No new code paths, no new dependencies, no new failure modes. Fully reversible by reverting the commit.
+
+## Evidence pointers
+
+- Typecheck: `tsc --noEmit` — 0 errors.
+- Full test suite: 740 files passed, 0 failed, 17171 individual tests passed.
+- Zero instances of `new URL(import.meta.url).pathname` remain in `src/`.