instar 1.3.730 → 1.3.732

package/upgrades/1.3.731.md

@@ -0,0 +1,97 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: minor -->
+
+## What Changed
+
+The response-review pipeline's watch-mode data vetoed the planned enforcement
+flip: of 12 reviewed real messages, 2 would have been WRONGLY blocked — both
+technical content the operator had EXPLICITLY asked for (one was the worktree
+keep/delete list the user requested). The reviewer judges each message in
+isolation by construction: its own prompt allows "code the user explicitly
+asked to see," but `ReviewContext` carries NO conversation — the carve-out was
+prose with no input.
+
+Per `docs/specs/context-aware-outbound-review.md` (review-converged round 4,
+approved), the opted-in reviewer now receives a bounded, untrusted-data-
+enveloped slice of recent conversation, and the flip evidence becomes durable
+and two-sided-measured:
+
+- **Context channel (§D1/§D2):** an injected synchronous provider
+  (TopicMemory rows + the topic's verified-operator binding →
+  `buildConversationContext`), rendered by ONE shared envelope
+  (`untrustedConversationContext.ts` — per-call random boundary, JSON-encoded
+  scrubbed bodies, the structural `ask-license mode` line, the four-clause
+  carve-out contract as ONE ATOMIC block). Budget-clamped (6 msgs / 500 chars /
+  4000 total).
+- **Principal binding (§D4, Know Your Principal):** role + `verifiedOperator`
+  tags + the window's ask-license mode are computed at the WIRING layer from
+  AUTHENTICATED uids only; any uid-less user row in an unbound window degrades
+  AWAY from licensing (fail-closed).
+- **Structural scoping (§D3):** only `conversational-tone` (v1), only for
+  primary-user recipients, receives an augmented ctx copy; every other
+  reviewer structurally cannot render conversation. The carve-out is ONE-WAY
+  (can only move a would-block toward PASS), content-class-bounded (never
+  credentials/PII), and never touches the deterministic PEL layer.
+- **Total containment (§D5):** the HTTP seam above the pipeline fails OPEN on
+  a crash, so every new context path is individually contained — any failure
+  degrades to today's exact (stricter) behavior.
+- **Durable flip evidence (§D8):** `logs/response-review-decisions.jsonl` —
+  one size-rotated row per evaluated turn (outcome, llmVerdict, violations,
+  contextMeta, 200 scrubbed chars; never conversation bodies), surviving the
+  restarts that erased the in-memory history.
+- **The one-way property is MEASURED (§D9.4/4b):** soak-only counterfactual
+  re-reviews of qualifying would-blocks (shared `pairId`), plus a daily
+  adversarial `ReviewCanaryBattery` (PEL-missable secret/PII fixtures with
+  covering asks, seeded into reserved NEGATIVE topic ids, baseline +
+  with-context arms, reviewer-level assertions, finally-cleanup) behind the
+  Bearer-gated `POST /review/canary-battery/run` + the off-by-default
+  `review-canary-battery` job. Every run outcome — including refusals —
+  writes a `batterySummary` row; a silent skip is impossible.
+- Ships dev-gated dark (`responseReview.conversationalContext` — `enabled`
+  OMITTED → resolveDevAgentGate; LIVE on dev agents, DARK on the fleet).
+  Kill-switch `enabled: false` applies at the NEXT evaluate via the new
+  liveConfig wiring — no restart. The enforcement flip
+  (`observeOnly: false`) remains the operator's manual action, gated on the
+  §D9 clean-day criteria.
+
+## What to Tell Your User
+
+<!-- audience: agent-only, maturity: experimental -->
+- **My message reviewer can now see what you asked for (experimental, dark):
+  ** the safety reviewer that checks my replies used to judge each message in
+  isolation — so when you explicitly asked me for a technical list, it wanted
+  to block my answer. On development agents it now reads the last few
+  messages of our conversation (wrapped as quoted data, never instructions),
+  so "you asked for this" finally counts. It can only ever RESCUE a message
+  from a wrong block — it can never become a new reason to block, and it never
+  licenses secrets. Blocking stays off until the operator flips it after a
+  measured clean day.
+
+## Summary of New Capabilities
+
+| Capability | How to Use |
+|-----------|-----------|
+| Context-aware reviewer carve-out (dev-gated dark) | `responseReview.conversationalContext` in `.instar/config.json` (enabled omitted → live on dev agents, dark on fleet; `enabled:false` = live kill-switch, no restart) |
+| Durable would-block decision log | `logs/response-review-decisions.jsonl` (one row per reviewed turn; `contextMeta` on `GET /review/history` entries too) |
+| Daily anti-laundering canary battery | `POST /review/canary-battery/run` (Bearer; 503 while dark) + the off-by-default `review-canary-battery` job |
+| Canary tag plumbing (test route only) | `POST /review/test` accepts `canary` + `fixtureId`; `/review/evaluate` ignores them (a real turn cannot self-tag) |
+
+## Evidence
+
+- Veto data grounded live (2026-07-02 watch mode): 2/12 real messages would
+  have been wrongly blocked — both operator-requested technical content; both
+  are now permanent §D9.2 regression fixtures AND the battery's pass-side
+  controls.
+- Spec converged round 4 (0 CRITICAL / 0 MAJOR; internal panel + both external
+  cross-model doors verified the fold table 5/5), approved under standing
+  Session-A preapproval (topic 29836), tag commit dfdb7b21a.
+- All three test tiers shipped and green: unit (57 new tests across 5 files —
+  every §5 boundary both sides, incl. the throw fixtures at fetch/tag/render
+  time, the uid-less fail-closed rule, the counterfactual + canary-tag
+  exclusion, the battery outcome table), integration (8 — real auth
+  middleware + real TopicMemory SQLite through the full HTTP pipeline), e2e
+  lifecycle (6 — production init path with the REAL LiveConfig + dev-gate
+  funnel + TopicOperatorStore: dev boot alive, on-disk kill-switch applies
+  without restart, fleet boot byte-identical + battery 503).

package/upgrades/1.3.732.md

@@ -0,0 +1,107 @@
+# Upgrade Guide — vNEXT
+
+<!-- assembled-by: assemble-next-md -->
+<!-- bump: minor -->
+
+## What Changed
+
+Per `docs/specs/durable-conversation-identity.md` (review-converged round 11 —
+0 CRITICAL / 0 MAJOR — approved under the standing Session-A preapproval,
+topic 29836), the Phase-1 structural refactor's first increment lands: a
+durable, channel-agnostic identity for conversations.
+
+- **`ConversationRegistry` (§3):** the sparse join table between canonical key
+  (`slack:<team>:<channel>[:<thread>]`), structured tuple, and a stable minted
+  NEGATIVE id. Telegram positive ids pass through unregistered forever
+  (back-compat by construction). The mint candidate is the legacy hash value
+  (zero-loss adoption + mixed-fleet skew convergence); the registry is the
+  collision authority — probe DOWN through ONE shared displacement
+  implementation, bounded at `MAX_PROBE_DISTANCE = 64`.
+- **The WAL journal (§3.4):** `conversation-registry.jsonl` at the stateDir
+  ROOT (the one glob shape the deployed backup resolver expands — R3-C4).
+  Probed/durable-binding mints fsync ONE line BEFORE the id returns; the O(N)
+  snapshot is batched with a size-adaptive interval (the CommitmentTracker
+  freeze shape is structurally avoided). Torn-tail discard; non-tail
+  corruption fails CLOSED (quarantine-aside + attention + a durability
+  incident); unknown ops (a rollback across a newer version) are
+  skipped-and-preserved with snapshot-flush SUSPENSION until re-upgrade;
+  single global monotonic `seq` across rotations and restarts.
+- **Eager mint at Slack inbound (§6.3):** the dispatch mints (get-or-create)
+  the resolved routing key's conversation id on every inbound and carries it
+  as `conversationId` in message metadata + the session bootstrap context —
+  the identity durable state can finally bind to. Fail-toward-delivery: a
+  mint degradation never blocks a message.
+- **Safety rails:** per-channel mint-rate breaker (speculative drops re-mint
+  free; the durable-binding carve-out has its OWN cap + typed
+  `conversation-registration-capacity` refusal); the D1 runtime kill-switch
+  (`conversationIdentity.recording.enabled: false` → legacy-identical
+  in-memory hashing, no redeploy; minted-id durable binds refused typed);
+  §3.1 workspace pinning with per-machine `multi-workspace-unsupported`
+  refusal and `_`→teamId in-place upgrade (local authenticated source only).
+- **The `deliverToConversation` funnel SKELETON (§5):** typed, non-exceptional
+  §5.1 results (dark/dryRun return the SAME `not-delivered` shape as
+  unresolvable + a would-deliver audit line — NEVER success-shaped); local-
+  origin-only resolution (KYP); system-channel suppression inside the funnel.
+  ZERO consumers migrate in this increment; the E1 idempotency guard, P17
+  budgets, and the deterministic gate-exempt arm land with their §6.1
+  increments before any consumer rides it.
+- **Read-only observability (§8):** `GET /conversations`,
+  `/conversations/:id`, `/conversations/resolve`, `/conversations/health`
+  (labels escaped on render; no write routes exist by design).
+- **Migration parity (§6.2/§9):** backup manifest gains BOTH
+  `state/conversation-registry.json` AND the `conversation-registry.jsonl*`
+  glob; config defaults arrive via the add-missing merge (`recording.enabled:
+  true`; `followThrough.dryRun: true` with `enabled` OMITTED — the
+  developmentAgent gate resolves it); a dev-gate strip migration removes a
+  default-shaped `followThrough.enabled: false`; the CLAUDE.md capability
+  section reaches new agents (template) AND existing agents (migrator +
+  Codex/Gemini shadow markers).
+
+Behavior-identical for existing flows: recording is additive, delivery is
+dark, the three legacy hash copies keep computing the same values the registry
+records (the §4 callsite consolidation is the next increment; a grep ratchet
+pins the copy set so a fourth is a CI failure).
+
+## What to Tell Your User
+
+<!-- audience: agent-only, maturity: experimental -->
+- **Slack conversations now have a durable identity (foundation only, no
+  visible change yet):** every Slack channel or thread I talk in is written
+  into a permanent address book with a stable number the moment a message
+  arrives, so upcoming increments can make promises/reminders/notices in
+  Slack survive restarts the way they do on Telegram. A negative topic id in
+  my state is one of these minted conversation ids — I can look it up in the
+  address book — not an error. Delivery to Slack through this identity stays
+  switched off (dry-run first on development agents).
+
+## Summary of New Capabilities
+
+| Capability | How to Use |
+|-----------|-----------|
+| Conversation inventory | `GET /conversations` (Bearer; `?platform=slack`, `?limit=N`) |
+| Resolve an id | `GET /conversations/:id` (positive → Telegram pass-through; unknown negative → honest 404) |
+| Forward lookup (mints nothing) | `GET /conversations/resolve?key=…` or `?sessionKey=…` |
+| Health + growth/suspension tripwires | `GET /conversations/health` (the Tier-3 alive target) |
+| Recording kill-switch (D1) | `conversationIdentity.recording.enabled: false` (live-read, no restart) |
+| Delivery gate (dark) | `conversationIdentity.followThrough` (dev-gated; `dryRun: true` first) |
+
+## Evidence
+
+- Spec converged round 11 (0 CRITICAL / 0 MAJOR; internal multi-lens panel +
+  two external cross-model doors per round), approved under the standing
+  Session-A operator preapproval (topic 29836); tag commit aa5086eb8.
+- All three test tiers shipped and green: unit (79 tests across 5 files —
+  golden parity with the legacy hash, frozen schema-v1 constants, the crafted
+  in-charset collision pair probing to distinct ids in either order, WAL
+  crash-durability without a snapshot flush, torn-tail/corruption/unknown-op
+  replay rules incl. snapshot suspension, rotation-spanning replay with one
+  global seq, breaker both budgets, recording kill-switch both arms,
+  workspace pinning, adoption gating, funnel §5.1 typed contract, the
+  mint-idiom grep ratchet), integration (7 — real authMiddleware + real
+  registry through the full HTTP pipeline incl. the label-escape pin and the
+  read-only resolve), migration-parity unit (7 — defaults never materialize
+  `followThrough.enabled`, the #1001 strip, the backup glob expanding through
+  the REAL BackupManager with every expanded file in the snapshot), e2e
+  lifecycle (4 — production init path answers `/conversations/health` 200,
+  the mint→restart→same-id cycle via journal replay, Telegram pass-through
+  sparseness).

package/upgrades/side-effects/context-aware-outbound-review.md

@@ -0,0 +1,62 @@
+# Side-Effects Review — Context-Aware Outbound Review (S2 context-aware reviewers)
+
+**Spec:** docs/specs/context-aware-outbound-review.md (converged r4, approved under standing Session-A preapproval, topic 29836; tag commit dfdb7b21a). **Parent:** S2 enforcement-readiness track — the enforcement flip of `responseReview.observeOnly` is gated on this spec's §D9 criteria and stays a manual operator action.
+**Ships DARK on the fleet, LIVE on development agents** via `responseReview.conversationalContext` (`enabled` OMITTED from defaults → `resolveDevAgentGate` at the WIRING layer; explicit false force-darks, explicit true is the fleet flip).
+**Files:** src/core/untrustedConversationContext.ts (new), src/core/conversationContextWiring.ts (new), src/core/ResponseReviewDecisionLog.ts (new), src/monitoring/ReviewCanaryBattery.ts (new), src/core/CoherenceGate.ts, src/core/CoherenceReviewer.ts, src/core/reviewers/conversational-tone.ts, src/core/types.ts, src/server/routes.ts, src/server/AgentServer.ts, src/server/CapabilityIndex.ts, src/commands/server.ts, src/scaffold/templates.ts, src/core/PostUpdateMigrator.ts, src/scaffold/templates/jobs/instar/review-canary-battery.md (new, ships OFF)
+
+## What changed
+
+1. **untrustedConversationContext.ts (new):** the ONE shared untrusted-data envelope (§D2) — `clampConversation` (§D6 budget clamps: 6 msgs / 500 chars / 4000 total, oldest dropped first) + `renderUntrustedConversation` (per-call random `CTX_BOUNDARY`, JSON-encoded bodies, credential scrub per body, role labels + `USER(verified-operator)` principal tag, the structural `ask-license mode` line, and the §D3 four-clause prompt contract rendered as ONE ATOMIC block). Never throws; per-message scrub failure drops that message; any render failure drops the whole section.
+2. **conversationContextWiring.ts (new):** `buildConversationContext(rows, operator)` — the wiring-layer principal computation (§D4): role from `fromUser`, `verifiedOperator` tags from authenticated-uid-vs-binding match, and the window `askLicenseMode` (bound → `verified-operator`; unbound → `single-sender` ONLY when every user row carries the same single authenticated uid; ANY uid-less user row or 2+ uids → `weak-corroboration-only`, fail-closed per R3-M2/R4-L1).
+3. **ResponseReviewDecisionLog.ts (new):** the durable §D8 flip-evidence JSONL (`logs/response-review-decisions.jsonl`) — append-only, size-rotated (10MB → single `.1` archive via SafeFsExecutor), every write failure swallowed (telemetry never gates delivery).
+4. **CoherenceGate.ts:** optional `conversationContextProvider` + widened `liveConfig` getter (now carries the wiring-RESOLVED `conversationalContext` block; ABSENT getter ⇒ DARK even against an enabled snapshot — round-2 L4 precedence); acquisition once per `_evaluate` inside its own try/catch, only for primary-user recipients with a numeric topicId; augmented SHALLOW-COPY fan-out (base ctx never carries conversation — §D3 structural availability); `EvaluateRequest.telemetry` (test-route-only canary tags) + `EvaluateResponse._contextMeta`; §D8 rows written at the `logAudit` seam for EVERY outcome (textHead = 200 scrubbed chars; canary/fixtureId stamped by the writer); §D9.4 counterfactual re-review (observeOnly-only, fire-and-forget, one context-stripped re-review per opted-in violating reviewer, `counterfactual:true` + shared `pairId`); `appendDecisionRow` exposes the single writer to the battery; in-memory audit entries gain `contextMeta` (§6). `DynamicReviewer` honors the `'recent-conversation'` contextRequirements opt-in key.
+5. **conversational-tone.ts:** renders the atomic block ONLY when the gate handed it the augmented ctx; absent fields ⇒ prompt BYTE-IDENTICAL to feature-dark (the pre-existing static exception stands).
+6. **routes.ts:** `/review/test` accepts `canary` + `fixtureId` (forwarded as `telemetry`; response gains `contextMeta`); `POST /review/evaluate` does NOT read them (a real turn cannot self-tag — boundary 13B, pinned by test); new Bearer-gated `POST /review/canary-battery/run` (503 while dark).
+7. **ReviewCanaryBattery.ts (new):** the §D9.4b(a) daily battery driver — refuse-or-(pre-clean → seed reserved NEGATIVE topic ids with per-run-unique messageIds + uid-carrying user rows → replay both arms via the Bearer `/review/test` route → reviewer-level PEL-unmaskable assertions → finally-cleanup) → a `batterySummary` row on EVERY outcome including refusals.
+8. **server.ts wiring:** the `resolveConversationalContext` LIVE closure (LiveConfig + `resolveDevAgentGate` — the kill-switch applies at the NEXT evaluate, no restart), the provider over real TopicMemory + `_agentServerRef` TopicOperatorStore, and the battery construction (localhost Bearer self-call to `/review/test`).
+9. **Job template `review-canary-battery.md` (new):** ships `enabled: false` everywhere; operator enables it only on the soaking dev agent (spec §4.5 — no fleet migration carries it on).
+10. **templates.ts + PostUpdateMigrator.ts:** `CONTEXT_AWARE_REVIEW_CLAUDEMD_SECTION` in `generateClaudeMd` (new agents) + an idempotent content-sniffed `migrateClaudeMd` append (existing agents), carrying the house dark-feature honesty phrasing (round-1 m5: `/review/history` 501s on most installs).
+
+## Blast radius
+
+- **Fleet default: byte-identical.** `enabled` omitted + not a dev agent ⇒ the resolver returns false ⇒ no acquisition, no section, prompts byte-identical (pinned by unit boundary 3/4 and the e2e fleet boot). The battery route 503s; the job ships OFF.
+- **Total containment (round-1 M6, load-bearing):** the HTTP seam above the pipeline fails OPEN on a crash, so EVERY new context path — fetch, tagging, clamp, render, meta, D8 write, counterfactual — is individually contained; any failure degrades to "no section" (the STRICTER current posture), never to a throw escaping `_evaluate`. Pinned by throw fixtures at fetch/tag/render time.
+- **Structural exposure bound:** conversation fields ride ONLY the augmented copies handed to the resolved opt-in set (`conversational-tone` alone in v1) for primary-user recipients; `information-leakage` and every other reviewer receive a base ctx that cannot render them (round-1 M1 + round-2 m2). Widening requires a spec revision, never config alone.
+- **One-way risk direction:** context can only move a would-block toward PASS (measured by the counterfactual pairs + the canary battery, both soak-only); PEL is untouched and runs before all reviewers.
+
+## Risk + mitigation
+
+- **Risk:** the context channel launders a credential/PII paste past the reviewer. **Mitigation:** the §D3.3 bounded-scope clause + the daily adversarial canary battery (PEL-missable fixtures, baseline + with-context arms, reviewer-level assertions) — a laundering event FAILS the soak and resets the §D9 clock.
+- **Risk:** an attacker-influenceable context body carries instructions. **Mitigation:** the proven §Design-4 envelope (random boundary + JSON-encoded bodies + corroborating-only preamble), boundary-7 injection tests.
+- **Risk:** uid-less legacy rows fake a single-sender license. **Mitigation:** R3-M2 fail-closed rule (any uid-less user row in an unbound window ⇒ weak-corroboration-only), pinned both sides in boundary 6.
+- **Risk:** the D8 log grows unbounded or leaks conversation. **Mitigation:** size rotation; only `contextMeta` + 200 scrubbed chars persist — bodies never (at-rest honesty note carried in the spec + docs).
+- **Risk:** the battery strands fixture rows in the production TopicMemory. **Mitigation:** reserved NEGATIVE topic ids (collision structurally impossible), pre-clean + finally-cleanup + per-run-unique messageIds + seed-count assertion (R4-m4), all pinned by unit tests.
+
+## Migration parity
+
+- **Config:** NO `migrateConfig` entry on purpose — every key is optional with in-code defaults and the dev-gate convention REQUIRES `enabled` absent (a migration writing `enabled:false` would recreate the PR #1001 bug). Documented on the `ConversationalContextConfig` type.
+- **CLAUDE.md:** section ships in `generateClaudeMd` + content-sniffed idempotent `migrateClaudeMd` append.
+- **Hook scripts:** none changed — `response-review.js` already sends topicId + transcriptPath and is always-overwritten anyway.
+- **Job:** new template installs through the normal built-in job path (`installBuiltinJobs` refresh), `enabled:false` everywhere.
+
+## Rollback
+
+- `responseReview.conversationalContext.enabled: false` — context off, reviewers byte-identical to pre-spec behavior, applied at the NEXT evaluate through the liveConfig wiring (no restart; pinned by boundary 12 + the e2e on-disk flip test).
+- The D8 JSONL is additive telemetry; deleting it loses history, breaks nothing. `observeOnly` remains the orthogonal enforcement lever.
+
+## Tests
+
+- **Tier 1 (unit, 71 tests):** `untrusted-conversation-context.test.ts` (12 — envelope/clamps/scrub/atomicity, boundaries 7-9), `conversation-context-wiring.test.ts` (8 — boundary 6 both sides incl. the uid-less fail-closed rule), `coherence-gate-conversation-context.test.ts` (12 — boundaries 1-5, 12, 14 + the §D9.2 veto-day regression fixtures pinned both directions), `response-review-decision-log.test.ts` (9 — boundary 10 + boundary 11 counterfactual both sides, incl. canary-tag exclusion), `review-canary-battery.test.ts` (16 — boundary 13: outcome table, refuse conditions, seed-then-cleanup contract, failed-outranks-inconclusive).
+- **Tier 2 (integration, 8 tests):** full HTTP pipeline over the real AgentServer auth middleware + real TopicMemory SQLite — enveloped ask in the captured prompt, byte-identical no-context path, provider-throw → 200 never fail-open, `/review/history` carries contextMeta and never bodies, canary tag plumbing test-route-only, battery route Bearer-gated + live/dark.
+- **Tier 3 (e2e, 6 tests):** production init path with the REAL LiveConfig + resolveDevAgentGate + TopicMemory + TopicOperatorStore — dev boot ALIVE (context section + verified-operator principal tag from a real binding + battery 200), on-disk kill-switch flip applies with no restart, fleet boot byte-identical + 503 (Maturation Path dark side), Bearer gating.
+- Suite health: tsc clean; `npm run lint` clean; `check-repo-invariants` green; `no-silent-fallbacks` green (every new catch tagged with a real justification); docs-coverage class ratchet back at floor with the new architecture page.
+
+## Post-rebase conformance addendum (2026-07-03, PR #1343)
+
+Rebasing onto post-#1341/#1342 main put this build under two guards that did not exist at its commit ceremony; conforming produced two additional side effects:
+
+1. **Write-domain classification (src/core/WriteDomainRegistry.ts):** `POST /review/canary-battery/run` is now classified `machine-local` with an `ephemeral-rebuildable` story (standby-write-reconciliation §3.5 ratchet). Runtime delta: under the (dev-gated, dry-run) WriteAdmission layer the route resolves to a domain instead of null — machine-local ⇒ admit everywhere, which matches reality (the battery exercises THIS machine's review pipeline; fixtures are finally-cleaned; the D8 log is per-machine soak evidence). NOT added to the TODO-classify baseline (the ratchet forbids growth).
+2. **FileClassifier sync exclusions (src/core/FileClassifier.ts):** `.instar/topic-memory.db` and `logs/response-review-decisions.jsonl` are now `git-sync-excluded` — the I9 second-axis file-level arm, same pattern as the wave-1 evolution/attention entries. Behavior delta: if any writer ever queued these paths for git-sync auto-commit, they are now skipped with a DegradationReporter breadcrumb instead of swept into a commit. A live SQLite binary and a per-machine JSONL must never ride git-sync (the round-2 S1 lesson); no current code path queues either, so the practical fleet delta is zero.
+3. **Job template header (src/scaffold/templates/jobs/instar/review-canary-battery.md):** the trigger curl now resolves `AGENT_ID` and sends `X-Instar-AgentId`, conforming to the template-agent-id-header standard (bearer-only localhost calls are deprecated). No behavior change beyond the header; the job still ships `enabled: false`.
+
+Rollback for this addendum rides the parent levers: the registry entry and exclusions are inert data rows (removing them restores the prior null-classification / sync-eligible state); the template header is cosmetic to auth (the server accepts bearer-only with a deprecation log).

package/upgrades/side-effects/durable-conversation-identity-increment1.md

@@ -0,0 +1,89 @@
+# Side-Effects Review — Durable Conversation Identity, Increment 1 (registry + crash-proof journal + eager mint)
+
+**Spec:** docs/specs/durable-conversation-identity.md (CONVERGED round 11 — 0 CRITICAL / 0 MAJOR; approved under the standing Session-A operator preapproval, topic 29836). **Parent:** Structure beats Willpower — durable identity must be a registry, not a convention three copies of a hash function remember.
+**Behavior-identical for existing flows.** The foundation RECORDS identity (always-on, kill-switchable); DELIVERY stays dark-gated: `conversationIdentity.followThrough` omits `enabled` (the developmentAgent gate resolves it — live-on-dev, dark-fleet) with `dryRun: true` FIRST, exactly as spec §9 prescribes.
+**Files:** src/core/conversationIdentity.ts (new), src/core/ConversationRegistry.ts (new), src/core/deliverToConversation.ts (new), src/server/routes.ts (read-only `GET /conversations*`), src/commands/server.ts (registry construction + adoption pass + §6.3 eager mint), src/core/types.ts + src/config/ConfigDefaults.ts (`conversationIdentity` block), src/core/devGatedFeatures.ts (followThrough registration), src/core/PostUpdateMigrator.ts (backup manifest + config defaults + CLAUDE.md), src/scaffold/templates.ts (Agent Awareness).
+
+## What changed
+
+1. **conversationIdentity.ts (new):** the SINGLE hash + identity surface (§4). The frozen 32-bit sum-shift hash, the mint candidate `-(abs(hash)+1)` (golden-parity with `slackRoutingKeySyntheticId`), the v1 structured tuple `(platform, channelId, threadTs?)`, canonical-key/tupleKey forms, the frozen schema-v1 constants (`MAX_PROBE_DISTANCE=64`, probe DOWN, `HLC_ABS_MIN/MAX`), shape clamps, and `walkDisplacement` — the ONE shared displacement implementation §3.3 and the future §3.5.1 merge both call.
+2. **ConversationRegistry.ts (new):** the sparse join table (canonical key ⇄ tuple ⇄ minted negative id). Synchronous in-memory assignment (probe included; returned == will-persist), the §3.4 WAL journal at the stateDir ROOT (`conversation-registry.jsonl` — the one glob shape the deployed `BackupManager.expandGlob` expands, R3-C4): probed/durable-binding mints append+fsync ONE line BEFORE the id returns; speculative non-probed mints ride the batched snapshot only. Torn-tail truncate-discard; NON-tail corruption fails CLOSED (quarantine-aside + attention + durability incident — R7-minor-3); UNKNOWN-op skip-and-preserve with snapshot-flush SUSPENSION (R8-minor-2/R9-M1/R10-M1) surfaced on health (R11-low-1); single global monotonic `seq` across rotations + restarts (R3-M14); rotation 8 MB/50k lines with fully-superseded-only pruning behind a 7-day retention floor; size-adaptive batched snapshot (2 s base → 60 s max; off-loop write past 20k entries/2 MB); snapshot completeness for bind-pins / ambiguous-sends / send-intents (R4-M2/R6-M1 — the op enum + replay ship NOW so later increments' records and rollbacks across them replay correctly; their WRITERS land with §6.1 steps 2+). Mint-rate breaker (per-channel windows, speculative drop-to-nowhere + durable-binding carve-out with its OWN cap → typed `conversation-registration-capacity`, adversarial-B), the D1 `recording.enabled` kill-switch (live-read; degraded = in-memory candidate + collision-checked read B6; minted-id durable binds refused typed), §3.1 workspace pinning (config pin authoritative; local-observed self-corroborating; per-machine `multi-workspace-unsupported` refusal; `_`→teamId in-place upgrade by the LOCAL authenticated adapter only), §6.2 authorized-traffic-gated adoption pass, and the §8 health surface.
+3. **deliverToConversation.ts (new):** the §5 funnel SKELETON — typed, non-exceptional §5.1 results (dark/dryRun return the SAME `not-delivered` shape as unresolvable, plus a would-deliver audit line; NEVER success-shaped), `id>0` → today's Telegram path, `id<0` → local-origin-only resolution (KYP), system-channel suppression inside the funnel (§4), in-thread Slack delivery when live. ZERO consumers migrate in this increment; the E1 guard, P17 budgets, `deterministicKind` arm, and permanent-error classification land WITH their §6.1 increments (2/3/5) before any consumer rides the funnel.
+4. **Read-only routes (§8):** `GET /conversations`, `/conversations/health` (the e2e alive target), `/conversations/resolve`, `/conversations/:id` — Bearer-gated, labels HTML-escaped on render (B3), no write routes (mint happens only at internal chokepoints — §7).
+5. **§6.3 eager mint at Slack inbound:** the dispatch mints (get-or-create) the resolved routing key's conversation id on every inbound and carries it as the pinned `conversationId` field in message metadata + the session bootstrap context. Wrapped fail-toward-delivery: a mint degradation never blocks a message.
+6. **Config/rollout (§9):** `conversationIdentity.recording.enabled: true` default (existence-checked in migrateConfig, NEVER materialized false — the #1001 mechanism); `followThrough.enabled` OMITTED + registered in DEV_GATED_FEATURES; `followThrough.dryRun: true`. PostUpdateMigrator adds the backup-manifest entries (`state/conversation-registry.json` + the top-level glob `conversation-registry.jsonl*`) and the CLAUDE.md Capabilities entry; `generateClaudeMd` carries it for new agents.
+
+## Blast radius
+
+- **Recording is additive observability.** No existing store changes shape; no store version bump; the registry file is inert data to old code (verified: zero old-code reads of `state/conversation-registry.json`). Rollback-by-revert for the code; `recording.enabled:false` is the runtime kill-switch that restores byte-identical legacy hashing without a redeploy.
+- **Delivery cannot fire on the fleet.** The funnel has zero callers in this increment AND the `id<0` arm is dev-gated + dryRun-first; on a dev agent it returns typed non-deliveries with audit lines until a deliberate `dryRun:false` flip.
+- **The eager mint adds one in-memory map hit per Slack inbound** (post-first-mint) plus, for a NEW conversation, one O(1) journal append (fsync only when probed). The O(N) snapshot is batched off the hot path — the CommitmentTracker freeze shape is structurally avoided (§3.4).
+- **Hash copies are NOT rewired in this increment.** The three legacy copies keep computing the same values the registry records (value-identical by golden parity); the §4 callsite consolidation is the next increment. The mint-idiom grep ratchet pins the current copy set so a FOURTH copy is a CI failure.
+
+## Risk + mitigation
+
+- **Risk:** registry failure costs a message. **Mitigation:** every mint path is typed-degrading (unparseable key / breaker drop / recording-off / probe overflow → "no durable id" or a collision-checked read); the inbound dispatch wraps the mint in try/catch. Identity never costs a message (§3.6) — pinned by unit tests.
+- **Risk:** a crash loses a probed/thread-level id a durable consumer bound to. **Mitigation:** the WAL rule — fsynced journal line BEFORE the id returns; replay restores it (pinned by the crash tests, no snapshot flush needed).
+- **Risk:** journal corruption silently loses committed records. **Mitigation:** non-tail corruption HALTS replay into quarantine-aside + ONE attention item + a durability-incident record (the §3.7 SQLite-migration tripwire input) — never skip-and-continue.
+- **Risk:** a rollback across a future op-enum extension trips the corruption machinery or composes wrongly. **Mitigation:** unknown-op skip-and-preserve + snapshot-flush suspension (the pre-skew snapshot stays put; prune keys on the static pre-skew watermark) — pinned by the suspension unit test.
+- **Risk:** a mint flood grows the store unboundedly (auto-join, thread storms). **Mitigation:** the per-channel mint-rate breaker (speculative drops re-mint for free; durable carve-out has its own cap + typed refusal), the adoption pass's authorized-traffic gate (security-B8), and the 80%-of-ceiling health tripwire.
+
+## Migration parity
+
+- migrateConfig: adds `conversationIdentity.recording.enabled: true` ONLY when absent; NEVER writes `followThrough.enabled` or a literal `false` anywhere in the block (pinned by a unit test — the #1001 mechanism).
+- Backup manifest: `state/conversation-registry.json` + top-level glob `conversation-registry.jsonl*` join `config.backup.includeFiles` via the existing idempotent set-union migrator (stateDir-relative; the Tier-2 test drives the REAL BackupManager and asserts the glob's expanded set lands in the snapshot — R3-C4).
+- CLAUDE.md: `GET /conversations*` Capabilities entry via `migrateClaudeMd()` (existing agents) AND `generateClaudeMd()` (new agents) — P3 + P5 both.
+
+## Rollback
+
+- Runtime: `conversationIdentity.recording.enabled: false` (kill-switch — legacy-identical degradation); `followThrough` stays dark on the fleet by default. Full revert: delete the three new modules + the route block + the dispatch mint lines + the config/migrator entries; `state/conversation-registry.json` + `conversation-registry.jsonl*` are inert to old code. A later RE-enable needs no special path — the idempotent boot adoption pass + journal replay compose over whatever is on disk (R8-low-1).
+
+## Tests
+
+- `tests/unit/conversation-identity.test.ts` — golden parity with the legacy hash (channel + thread), frozen schema-v1 constants, tuple/key forms + shape clamps, the shared displacement walk + its MAX_PROBE_DISTANCE bound.
+- `tests/unit/conversation-registry.test.ts` — mint idempotency (incl. across re-open), the crafted in-charset collision pair probing to DISTINCT ids in either order, WAL crash-durability (probed + durable-binding survive with NO snapshot flush; speculative writes no synchronous journal line), torn-tail discard, non-tail-corruption fail-closed, unknown-op suspension, rotation-spanning replay with one global seq, idempotent replay, snapshot completeness (bind-pins/ambiguous-sends/send-intents), alias one-hop + the boot assignment-beats-alias invariant, breaker (speculative drop + re-mint, durable carve-out + typed capacity refusal), recording kill-switch both arms, workspace pinning (self-corroboration, multi-workspace refusal, config-pin authority, `_`→teamId in-place upgrade), adoption-pass gating + idempotency, resolution surfaces, health shape.
+- `tests/unit/deliver-to-conversation.test.ts` — §5.1 typed contract (dark/dry/unresolvable/replicated-only/system-channel/no-adapter/transport-error all non-exceptional, never success-shaped; dryRun audit line), in-thread delivery, Telegram passthrough.
+- `tests/unit/conversation-identity-mint-idiom-ratchet.test.ts` — the §10 wiring-integrity grep ratchet (a fourth mint-idiom copy is a CI failure).
+- Integration (`tests/integration/conversation-registry-routes.test.ts`) — the `GET /conversations*` HTTP pipeline (auth, 404 semantics, label escaping, resolve), migrateConfig never-materializes assertions, the backup-manifest-through-real-BackupManager test.
+- E2E (`tests/e2e/conversation-registry-lifecycle.test.ts`) — Phase-1 "feature is alive": production-init wiring answers `GET /conversations/health` 200 (not 503), adoption pass state present, inbound→mint→resolve cycle.
+
+## Agent awareness
+
+- A "Durable Conversation Identity" Capabilities entry (GET /conversations*, the health surface, what a negative topicId means) ships in `generateClaudeMd` + an idempotent content-sniffed `migrateClaudeMd` section. <!-- tracked: durable-conversation-identity -->
+
+## Part-2 landing note (wiring)
+
+Part 1 landed the three core modules + Tier-1 tests. Part 2 (this commit)
+lands the wiring: the read-only `GET /conversations*` routes (labels escaped —
+the only Phase-1 render surface), the AgentServer plumbing (bootstrap instance
+takes precedence; a read-only fallback keeps the health surface alive on any
+init path — it can never become a second journal writer because eager mint
+uses the bootstrap instance only), the server-bootstrap construction (live
+kill-switch reads, late-bound workspace source, attention wiring), the §6.3
+eager mint + bootstrap-context carry, the §6.2 adoption pass (session-registry
+membership as the authorized-traffic record), config types/defaults +
+DEV_GATED_FEATURES registration, the PostUpdateMigrator additions (backup
+manifest, dev-gate strip, CLAUDE.md section + shadow markers), the scaffold
+template section, and the Tier-2/Tier-3 tests + docs-coverage artifacts
+(release fragment + architecture page).
+
+## Part-3 landing note (surface classification + test parity)
+
+CapabilityIndex gains the `/conversations` prefix classification (agent-facing
+capability — surfaced in /capabilities discovery, not INTERNAL_PREFIXES), and
+the backup-manifest parity test learns the two new manifest entries (the
+predictable behavior-change-breaks-old-tests category, swept before push).
+
+## Part-4 landing note (release-fragment polish)
+
+The release fragment's "What to Tell Your User" line drops inline code
+formatting (the pre-push gate's plain-conversational rule).
+
+## Part-5 landing note (fail-safe tagging)
+
+Every intentional fail-safe catch this increment introduced carries an
+explicit `@silent-fallback-ok` tag with its safe-direction justification
+(registry: fsync best-effort / prune-retains / workspace-degrades-to-
+placeholder / shutdown teardown; wiring: fail-toward-delivery on boot-load,
+eager mint, adoption pass, attention raise) — the no-silent-fallbacks ratchet
+stays at its 491 baseline with zero untagged swallows from this PR.