instar 1.2.60 → 1.2.62

package/upgrades/1.2.62.md

@@ -0,0 +1,75 @@
+# Upgrade Guide — topic-intent auto-capture loop (rung 0)
+
+<!-- bump: minor -->
+<!-- minor = new features, new APIs, new capabilities (backwards-compatible) -->
+
+## What Changed
+
+**The topic-intent "cabinet" now fills itself from live conversation.**
+
+The Topic-Intent Layer shipped a while back — a per-topic store of the facts
+and decisions a conversation establishes, a session-start briefing that reads
+from it, and an ArcCheck that guards against acting on shaky ground. But its
+defining capability was never switched on: nothing ever read a real
+conversation turn and filed anything. The store stayed empty, so the briefing
+rendered blank and ArcCheck had nothing to gate against. This is why the
+original methodology-drift incident found "topic-intent had no record for the
+topic" — the drift-catching machine was shipped but asleep.
+
+This change wires the capture "clerk" that reads each substantive turn and
+gives it a cheap, fast-tier "anything worth filing here?" read, with **broader
+context** — the topic's already-established refs plus a rolling summary — so it
+judges significance against what the conversation is actually about, not one
+line in isolation.
+
+Built in:
+
+- **A deterministic, fail-open pre-filter** so trivial turns (empty, bare
+  "ok"/"thanks", agent status/heartbeat lines) never reach the model. Registered
+  as a state-detector with a canary (`docs/specs/06-state-detector-registry.md`)
+  that guards against sentinel-format drift silently dropping real captures.
+- **Cost controls**: every extraction is admitted through the shared LlmQueue on
+  the background lane (yields to interactive work, shares the daily cap), runs on
+  the subscription transport (never the raw paid API), is bounded by a per-topic
+  rate ceiling, and backs off under quota pressure. Fire-and-forget, so capture
+  latency can never slow a message reaching you.
+- **Prompt-injection hardening**: prior notes and the rolling summary are fed
+  back to the model only inside delimited untrusted-data blocks, truncated to
+  hard caps.
+- **Concurrency-safe writes**: the store's append path holds a per-topic lock so
+  two sessions capturing the same topic can't drop each other's events.
+- **Whole-loop observability** (`GET /topic-intent/:id/capture-metrics`): the
+  funnel of captured → surfaced → used → corrected. Paired with the
+  human-as-detector heat map (what we MISSED), this is the read for tuning
+  effectiveness over time.
+
+ON by default (ratified), with a kill-switch: `topicIntent.capture.enabled:
+false`. Existing agents get the default on update.
+
+Spec: `docs/specs/topic-intent-capture-loop.md` (converged iter 3, approved).
+ELI16: `docs/specs/topic-intent-capture-loop.eli16.md`.
+Side-effects review: `upgrades/side-effects/topic-intent-capture-loop.md`.
+
+## What to Tell Your User
+
+- **Conversation memory**: "I now quietly remember the facts and decisions from
+  our conversations, so when we pick a topic back up I already know where we
+  left off — instead of you having to remind me."
+
+## Summary of New Capabilities
+
+| Capability | How to Use |
+|-----------|-----------|
+| Per-turn topic-intent capture | Automatic (ON by default; disable via `topicIntent.capture.enabled: false`) |
+| Capture-loop funnel metrics | `GET /topic-intent/:topicId/capture-metrics` (operator-only) |
+
+## Evidence
+
+Not a bug fix in the runtime sense — this wires a capability that shipped inert.
+Verified end-to-end: 30 new tests across all three tiers (20 unit, 6
+integration, 4 e2e) plus the 13 pre-existing capture tests — 138 topic-intent
+tests green. The e2e **wiring-integrity** test fires a real inbound turn through
+the live `onMessageLogged` callback and asserts a ref lands in the store and the
+briefing goes from empty to non-empty (the anti-"shipped-but-asleep" guard), and
+that the extraction went through the queue's background lane on the injected
+provider, never a raw API client. `tsc` + lint clean.

package/upgrades/side-effects/topic-intent-capture-loop.md

@@ -0,0 +1,100 @@
+# Side-effects review — topic-intent auto-capture loop (rung 0)
+
+**Scope**: Wire the topic-intent capture loop so the per-topic store actually
+fills from live conversation (closing the "shipped but asleep" gap — the store,
+read routes, and session-start briefing all shipped, but nothing ever invoked
+`ingest()` on a real turn). Adds the adapter-agnostic capture "clerk", broader
+context (rolling summary + established refs) feeding the extractor, cost
+controls, prompt-injection-hardened extraction, the live wiring on the inbound
+message path, and whole-loop observability. Spec:
+`docs/specs/topic-intent-capture-loop.md` (converged iter 3, approved by justin).
+
+**Files touched**:
+- `src/core/TopicIntent.ts` — add `CaptureCounters` to `TelemetryCounters`
+  (defaulted on read for back-compat) + `defaultCaptureCounters()` +
+  `bumpCaptureCounters()` (atomic under the existing per-topic lock). Switch the
+  two `withTopicLock` lock-dir removals to `SafeFsExecutor.safeRmdirSync`.
+- `src/core/TopicIntentExtractor.ts` — `createLlmExtractFn` gains an optional
+  `onDegrade(reason, topicId)` observability hook; still returns `[]` on every
+  degrade path (degrade-safety unchanged).
+- `src/core/TopicIntentCapture.ts` — NEW. The capture step: `isSubstantiveTurn`
+  pre-filter (deterministic, fail-open) + canary; `createQueuedIntelligence`
+  (queue-backed, subscription-transport); `captureTurn` + `createCaptureLoop`
+  (rate-state-owning closure).
+- `src/server/topicIntentRoutes.ts` — NEW `GET /topic-intent/:id/capture-metrics`
+  (the whole-loop funnel); `briefing_served` metering on the briefing route;
+  `arccheck_fired`/`arccheck_signalled` metering on the arccheck route.
+- `src/server/CapabilityIndex.ts` — add `topic-intent` to `INTERNAL_PREFIXES`
+  (operator-only; not a discoverable agent endpoint).
+- `src/config/ConfigDefaults.ts` — `topicIntent.capture.enabled: true` in
+  `SHARED_DEFAULTS` (auto-applies on init AND migration → migration parity).
+- `src/core/types.ts` — add optional `topicIntent` to `InstarConfig`.
+- `src/commands/server.ts` — construct the queue-backed extractor + capture loop
+  and chain it onto `telegram.onMessageLogged` (preserving prior callbacks),
+  gated on `sharedIntelligence && config.topicIntent.capture.enabled`.
+- Tests: `tests/unit/TopicIntentCapture.test.ts`,
+  `tests/integration/topic-intent-capture-routes.test.ts`,
+  `tests/e2e/topic-intent-capture-lifecycle.test.ts`.
+- `docs/specs/06-state-detector-registry.md` — NEW registry; pre-filter entry.
+
+**Under-block**: The pre-filter is fail-open — when unsure it passes the turn to
+the LLM, so it cannot silently swallow a substantive turn on an ambiguous input.
+Its only confident skips are empty/whitespace, whole-message bare acks, and
+agent sentinel/heartbeat lines (agent turns only). Risk of under-block (a real
+turn skipped) is bounded to sentinel-format drift, which the canary guards.
+
+**Over-block**: The only "block"-shaped behavior is the pre-filter skip and the
+QuotaTracker shed. Over-skipping costs a missed cheap extraction, never a
+delivery failure or a user-visible block. The canary asserts known substantive
+turns (including ack-prefixed ones) are NOT skipped.
+
+**Level-of-abstraction fit**: The capture step is adapter-agnostic — it takes a
+generic `CaptureTurnEntry`, not a Telegram type; Telegram is merely the first
+wiring (other adapters tracked as `cwa-multi-adapter-capture`). The store stays
+the single authority for persistence/projection; the extractor owns extraction;
+the capture helper only orchestrates. Transport is delegated to the injected
+`sharedIntelligence` provider (subscription/REPL-pool) through the shared
+`LlmQueue` — capture never reaches for a raw API client.
+
+**Signal vs authority**: Capture only RECORDS (append-only evidence); it has no
+blocking authority. ArcCheck SIGNALS; neither blocks a send. The pre-filter is a
+brittle low-context detector emitting a skip signal, never a gate. This matches
+`[[feedback_signal_vs_authority]]`.
+
+**Interactions**:
+- `telegram.onMessageLogged` is a single-assignment property already chained by
+  PresenceProxy, human-as-detector, and the keep-watching detector. The capture
+  wiring preserves the prior callback (`const before = ...; cb = (e) => { before?.(e); capture(e); }`),
+  verified by the e2e chain test (prior callback still fires).
+- Capture is fire-and-forget (`void captureLoop(...)`) so extraction latency
+  can never reach the delivery path (acceptance #4).
+- Extraction is admitted on the LlmQueue **background** lane, so it yields to
+  interactive (PresenceProxy/PromiseBeacon) work and shares the daily cap. On
+  cap breach the queue throws → `createLlmExtractFn` catches → degrades to a
+  `degraded_cap_or_error` tick.
+- `bumpCaptureCounters`, `bumpTurn`, and `appendEvidence` each take the per-topic
+  lock separately (multiple short acquisitions per turn). Correct under
+  concurrency (the concurrency test still passes); accepted minor lock churn for
+  v1 since capture runs off the delivery path.
+- The briefing route now has a metering side-effect on a GET (writes
+  `briefing_served`). Intentional per spec §10; best-effort and never blocks the
+  fetch.
+
+**External surfaces**:
+- New endpoint: `GET /topic-intent/:topicId/capture-metrics` (operator-only).
+- New config field: `topicIntent.capture.enabled` (default true; kill-switch).
+- New additive `TopicIntentFile` fields (`telemetry.capture.*`), defaulted on
+  read — old files load unchanged.
+- New exported symbols in `TopicIntentCapture.ts`; no breaking change to
+  existing exports.
+
+**Cost (the one genuinely new ongoing cost)**: This is the product's first
+always-on per-turn LLM path. Bounded by: the deterministic pre-filter (most
+turns never reach the model), a per-topic rate ceiling (30/60s), the LlmQueue
+daily cap (best-effort, per-process), and QuotaTracker load-shedding. ON by
+default is ratified.
+
+**Rollback cost**: Low. Config kill-switch `topicIntent.capture.enabled: false`
+makes capture inert immediately (store + routes remain, as today). Full revert:
+drop the server.ts wiring block + the new file; the store/routes/briefing return
+to the inert pre-capture state. Additive store fields are harmless if left.

package/upgrades/side-effects/wall-is-a-hypothesis-standard.md

@@ -0,0 +1,49 @@
+# Side-Effects Review — A Wall Is a Hypothesis (B16_UNVERIFIED_WALL)
+
+**Slug:** `wall-is-a-hypothesis-standard`
+**Date:** 2026-05-24
+**Author:** echo
+**Second-pass reviewer:** internal conformance pass
+
+## Summary of the change
+
+Adds the constitution standard "A Wall Is a Hypothesis" to `docs/STANDARDS-REGISTRY.md` and its structural enforcement: a new always-evaluated rule **B16_UNVERIFIED_WALL** in `MessagingToneGate` (the existing outbound-message authority that hosts B15). B16 blocks an outbound message that declares a path impossible/blocked/infeasible because an interface/API/mechanism is missing, when the message shows no evidence the agent inventoried its own capabilities first. Also registers the standard in `docs/INSTAR-DESIGN-PRINCIPLES-AND-LESSONS.md` (the catalog the `/spec-converge` reviewer loads).
+
+## Decision-point inventory
+
+- `VALID_RULES` set — **add** `'B16_UNVERIFIED_WALL'`. Without this the gate's drift-detection fails-open on a legitimate B16 citation.
+- `buildPrompt()` rule section — **add** the B16 definition after B15 (always-evaluated, no precondition).
+- Response-format enumeration + two stale doc comments — **modify** to include B16 (the comments already lagged at B14).
+- No route changes: `checkOutboundMessage` → 422 is rule-agnostic; B16 rides the existing outbound paths.
+
+## 1. Over-block
+
+The principal over-block risk: blocking ordinary "I can't do X" messages. Mitigated in the rule text — severity explicitly favors false-negatives; genuinely-external limits ("can't read your email until you connect it"), walls reported after a visible inventory, real either/or questions, real runtime errors, and messages discussing the rule all pass. The rule targets only the precise pattern: an internal feasibility verdict resting on a missing interface with no inventory shown.
+
+## 2. Under-block (a real wall-surrender slipping through)
+
+Possible if the LLM judge misses a borderline case — acceptable by design (favor false-negatives), matching the gate's stated philosophy (high signal, not adversarial correctness). The standard + the `/spec-converge` registration provide the softer review-time catch as backup.
+
+## 3. Level-of-abstraction fit
+
+Correct: the guard lives inside the single outbound authority (where B15 lives), not as a new detector with independent block power. Signal-vs-authority compliant.
+
+## 4. Blocking authority
+
+No new brittle authority. B16 is one more rule the existing authority may cite; the 422 plumbing is unchanged. Fail-open behavior (LLM error/timeout/invalid-rule) is inherited unchanged.
+
+## 5. Interactions
+
+B16 is always evaluated alongside B15 and the signal/health rules in one LLM call — no extra calls, marginally longer prompt. No interaction with the health-alert (B12-B14) or style (B11) rules, which remain gated by their preconditions. The drift-detection branch is unaffected (an invented rule id still fails open — covered by a regression test).
+
+## 6. External surfaces
+
+None. No new endpoints, credentials, or network calls. The standard's enforcement claim was verified against code before authoring (the registry is not parsed at runtime; the conformance gate and Usher are unbuilt North Star designs) — so the "Applied through" line states only what exists.
+
+## 7. Rollback cost
+
+Low. Reverting removes the rule from the set + prompt and the doc entries; no state, no migration, no schema. An older server simply lacks the rule.
+
+## 8. Test evidence
+
+Unit (`messaging-tone-gate-b16.test.ts`, 9 tests) + integration (`telegram-reply-b16-wall.test.ts`, 2 tests) green; tsc clean. Both sides of the decision boundary covered with realistic inputs; the /goal-style wall blocks through the real route (422, message suppressed) and the happy path still delivers (200).