@treeseed/dsh-chapters 0.1.0

package/docs/development.md

@@ -0,0 +1,437 @@
+# Development and Test Loop
+
+The problem is real: **this repository is edited from inside a running DeepSeek Harness, and a plugin
+only takes effect after that host restarts.** Restarting it would kill the session doing the work. So
+the loop has to be built so that it never touches the host it runs in. Everything below was verified on
+this machine, not assumed.
+
+## Environment Facts (checked, not inferred)
+
+| Thing | Value |
+|---|---|
+| Host serving this session | global npm install, `/usr/lib/node_modules/@deepseek-ai/dsh/lib/bin.js`, **0.1.5-rc.1**, listening on `127.0.0.1:3080` |
+| Profile root | `~/.dsh` (`DSH_HOME` default) — `profiles/`, `sessions/`, `storages/`, `.credentials.yaml` |
+| Sandbox effect | `~/.dsh` is **read-only** from the agent sandbox: `dsh web --help` died with `EROFS: read-only file system, open '/home/adrian/.dsh/profiles/web/cordis.yml'` |
+| Dev fork | `~/Projects/deepseek-harness` — git `fb2c4b9e69` (2026-09-10), **0.1.5-rc.2**, pnpm workspace, `node_modules` installed |
+| `/app/checkout` | **does not exist on this machine.** It appeared in my *session context* as a presumed harness checkout, not in this repository's docs — I checked, and no file here references it. The real fork is the row above |
+
+Two consequences worth stating plainly:
+
+1. **The sandbox is an ally, not an obstacle.** Because `~/.dsh` is read-only to an agent shell, the
+   accidental self-destruct path — installing into the live profile and restarting the host running the
+   work — is *structurally unavailable*. The dev loop is the only loop available.
+2. **The fork is newer than the host** (rc.2 vs rc.1), and `examples/deepseek-harness` is a third
+   snapshot. Line numbers in `docs/contract.md` come from the `examples/` copy. Treat any of the three as
+   truth only for the question you asked it; resolve real behaviour against the version you are actually
+   booting.
+
+## The Isolation Mechanism: `DSH_HOME`
+
+`dsh --help` documents profiles as "the profile under `$DSH_HOME/profiles` to boot", and `DSH_HOME`
+relocates the entire root — profiles, sessions, storages, credentials. Verified:
+
+```bash
+mkdir -p .dshdev
+DSH_HOME=$PWD/.dshdev dsh web --port 0 --no-open
+# → dsh web: http://127.0.0.1:32869/?token=…
+```
+
+A **fresh, fully isolated instance** booted: it created its own `profiles/web` under `.dshdev`, chose a
+free port, minted its own token, and left `~/.dsh` and the `:3080` host untouched. `--port 0` means the OS
+picks, so nothing conflicts. This is the whole answer: we get a disposable harness.
+
+`.dshdev/` is gitignored. Never point `DSH_HOME` at `~/.dsh`, and never run `dsh plugin --profile web add`
+without it — that is the command that mutates the profile this session is living in.
+
+### The dev profile for the real target: `scripts/bootstrap-dev-profile.sh`
+
+Everything so far measured on cloud models; the plugin's target is the user's **Local
+qwen3.8-flash-next** at the **64K window the user actually runs** (32K was tried and
+rejected as too small for real work — the 27-tool header alone is ~13–15K; the user
+confirmed the model is configured for 64K). `dev/settings.yaml` +
+`dev/profile-cordis.patch.yml` mirror the live `~/.dsh` Local-provider block exactly —
+same model, window (64K, just under the server's real `n_ctx: 65536`), and timeouts — and
+make **Chapters the default preset** and **Local the default model**, so nothing there is
+"enabled by remembering". The r27 gold run proves the engine's pressure math at this exact
+regime (threshold 57,600; a 93K-token session compacts before its first request).
+Bootstrap is write-if-missing for your tuned settings; `--force` re-templates. Build + link +
+settings + credential-refs copy in one shot:
+
+```bash
+scripts/bootstrap-dev-profile.sh            # -> .dshdev-local (browser-safe: no probe)
+scripts/bootstrap-dev-profile.sh --with-probe   # scripted rounds only (probe self-exits)
+```
+
+### Three test layers (and why the third exists)
+
+| Layer | Command | Runs | Covers |
+|---|---|---|---|
+| unit | `npm test` (`tests/unit`) | node --test, no build, no network | pure core: registry, render, archive, notice, engine-core, tools, commands |
+| integration | (today: probe rounds — see spikes/probe/README pattern) | boots real hosts in scratch homes | host wiring, realm mounts, provider economics |
+| e2e (tape — the default loop) | `npm run test:e2e:replay` | same boots with `E2E_MODEL=replay`; model turns answered from `tests/fixtures/model-tape/<project>` via the record/replay proxy (`tests/e2e/model-proxy.ts`) — seconds, deterministic |
+| e2e (distillation — occasional) | `npm run test:e2e:record` | `E2E_MODEL=record`: forwards every exchange to the real Local model and appends the tape. Run once per scenario; wipe `var/model-tape` first for a clean corpus; tapes are isolated per project (suite/heavy/arrival); re-record when scenarios, prompts, or thresholds change. The GPU model is the DISTILLER, never the test oracle (user directive 2026-09-19) |
+| e2e | `npm run test:e2e` (`tests/e2e`, THREE projects: suite/heavy/arrival) | Playwright + workspace-cached chromium vs a throwaway `var/e2e-home-<port>` rebuilt per boot (port-keyed so no boot can stomp another's state) (isolated from live agents — see FINDINGS 2026-09-19) | the browser: identity, fork button + switch, knowledge loop end-to-end, the oversized turn (mid-turn compactions), the subagent fan-out, arrival-time artifacting (low-floor project) |
+
+The e2e layer exists because client-plane facts were being GUESSED (services, methods, tooltips,
+switching) and every guess cost the user a browser round-trip. Playwright answers those in seconds
+and its suite is the regression net for all future UI surface. Each project boots through
+`boot.ts` (probe bundle removed first — it would otherwise kill the test server mid-suite; per-
+project env pins: suite threshold 0.75, heavy 0.5 + arrival floor off, arrival floor 500 at the
+production 0.9; homes are port-keyed). `discovery.spec.ts` keeps a health assertion on client-entry
+activation, `fork-button.spec.ts` proves the click end-to-end on durable + visible planes, and the
+heavy specs pin their claims to DURABLE facts (own-log turn/end, registry chapters, coverage) —
+never to model verbosity. ACCEPTANCE DEFAULTS TO THE TAPE (`test:e2e:replay`, model turns served
+from `tests/fixtures/model-tape/<project>` in seconds — explore: 14.7 s replayed vs 1.7 min live); the live
+chain remains the distiller (`test:e2e:record`) and the ultimate witness. The one honest
+limit of AGENTS-BLIND masking (measured 2026-09-23): it keys on the `instructions from:` /
+`<system-reminder>` delimiters, so it strips injected instruction blocks — but when a
+near-threshold session compacts, the harness's own checkpoint summary re-emits the doc as
+free prose at msg[1], which no delimiter mask reaches. Effect: `heavy` and `fanout` tapes
+carry the live AGENTS.md text, so an AGENTS.md edit means re-recording THOSE two projects
+(suite/enrich/rules carry too little to compact and stay doc-independent). Rule: finalize
+doc edits, then distill — never the reverse.
+
+Re-record discipline (learned the hard way 2026-09-22/23):
+- **A tape records the CONTEXT it was distilled in.** A spec replayed in a different
+  predecessor set (or a different boot port before HOSTPORT-BLIND landed) sees different
+  injected presence and misses. Re-record the whole project, not a single spec, when specs
+  share a boot — and never run two GPU-consuming suites concurrently (RR2=137 + timeout
+  cascades: a fanout that passes alone failed at 1.1 h while a heal chain shared the server).
+- **Ports are parameterized end to end**: the model proxy derives `E2E_PROXY_PORT = boot port
+  + 1000` (a hard-coded 41799 turned one killed run's orphan into EADDRINUSE for every later
+  boot), and `boot.ts` fails FAST with a named-port message when the boot port is already
+  owned (an orphan's silence cost 20+ minutes of opaque hangs). Kill leftovers:
+  `pkill -f "dsh web --port 417"`.
+- **The tape's matching is normalized on BOTH sides** — legacy stored prefixes are re-run
+  through `substituteAll` at load, so a new volatile rule (HOSTPORT-BLIND, `e2e-home-\d+`)
+  repairs older tapes without a re-record. Prefer that route over distilling again.
+- **When a browser scenario fails, suspect its own tooling before the product.** The
+  parked-for-eras `rules` e2e was a spec reading zstd-compressed directories as flat files
+  (fixed 2026-09-23 with the session.ts `zstd -dc` route + dynamic rule-id capture); the
+  enrich e2e exposed two REAL product bugs (the frontmatter-anchor convention mismatch and
+  the missing host-plane route) that every deterministic layer had missed because each side
+  only ever tested against its own convention. The e2e is the integration witness — keep it.
+
+### The coverage ledgers (added 2026-09-23 by the coverage audit; gated same day)
+
+- `npm run coverage:ci` — **the gate CI enforces.** `c8 --check-coverage
+  --statements 85 --branches 85 --functions 85 --lines 85` over the whole
+  deterministic suite (unit + integration in one process graph; live TreeDX
+  rows self-skip when the container is down, so it passes on a runner with no
+  TreeDX and no LLM). Below any threshold it exits non-zero; the report lands
+  in `var/coverage/` and the summary is posted to the CI job page. Current
+  ledger: **97.98% stmts / 85.61% branch / 93.62% funcs** across 450 tests.
+  The branch number is the binding one (defensive `??`/ternary arms dominate
+  the miss count); it is deliberately kept above 85 by the `*-branches.test.ts`
+  sweep files rather than by lowering the bar.
+- `npm run coverage` — the same deterministic suite under node's native
+  `--experimental-test-coverage` (a quick second opinion; no gate).
+- `npm run coverage:e2e` — replays the seven browser segments with
+  `E2E_COVERAGE=1`, which makes every boot write NODE_V8_COVERAGE profiles
+  under `var/e2e-cov/<port>` (flushed on the boot's clean exit), then runs
+  `scripts/e2e-coverage.mjs` → **c8 with real source maps** (tsconfig
+  `sourceMap: true`; lib/*.js maps back to src/*.ts) for istanbul-grade
+  per-module Stmts/Branch/Funcs. Read it as "covered by SOME boot of the
+  suite" (a union), and mind the script header's c8 gotcha: include/exclude
+  filters apply BEFORE remapping, so `--include=src/**` would silently
+  zero every row. First full ledger: 80% stmts / 62% branches overall;
+  tools.ts 72/49, engine.ts 81/59, index.ts 86/55 — plus the honest plane
+  split (src/treedx reads low HERE because the browser suite rides git;
+  the deterministic + live ledgers own that transport).
+  Both ledgers together are the coverage claim; neither alone is honest.
+
+### Hosted CI (`.github/workflows/`, wired + dress-rehearsed 2026-09-23)
+
+- **`ci.yml`** — two jobs. `test`: `npm ci` (all deps public on
+  registry.npmjs.org — **no install secret**), typecheck, build,
+  `coverage:ci` (the 85/85/85/85 gate). `e2e-replay`: installs the pinned
+  host (`npm i -g @deepseek-ai/dsh@0.1.5-rc.1` — the SAME version as the
+  gate line in `scripts/ci-replay.sh`, which fails loudly on mismatch) plus
+  chromium's apt libs, then `scripts/ci-replay.sh` — seven browser segments
+  served entirely from the committed tapes. **No LLM, no key, no GPU:** in
+  replay mode `localModelUp()` short-circuits to true (specs RUN, they do not
+  skip) and a tape miss is a loud proxy 503, so a scenario that legitimately
+  changed fails the job instead of silently passing.
+- **`release.yml`** — bare-semver tag (`0.1.0`, matching the Production
+     environment's `*.*.*` tag protection) or manual dispatch → the full
+     gated deterministic suite as publish blocker → `npm publish
+     --provenance`. Two token facts learned the hard way (measured on the
+     first ship): the token must be written into the file npm actually reads
+     (setup-node exports `NPM_CONFIG_USERCONFIG` — `~/.npmrc` is ignored),
+     and the FIRST publish of a new package name needs a granular token with
+     'Read and publish' over ALL packages (a package-scoped token 404s — it
+     cannot be scoped to a package that does not exist yet; narrow it to
+     `dsh-chapters` after the first ship).
+  --provenance`. Auth path B is OIDC trusted publishing (no long-lived
+  secret); path A uses `NPM_TOKEN` if configured.
+- The dress rehearsal on a pristine machine caught three real blockers, all
+  fixed and pinned: (1) `bootstrap-dev-profile.sh` refused the comments-only
+  `cordis.patch.yml` scaffold that `dsh plugin add` lays down on a fresh home
+  — the guard now only protects patches with real entries; (2) a credential-
+  less home boots but every session dies composing the local provider —
+  tape-mode boots now inject a dummy `LOCAL_API_KEY` (the proxy ignores
+  auth; a real env key always wins); (3) **a fresh home has no registered
+  workspace**, so the UI's New-session draft never commits — every browser
+  spec dies at `newSessionWithTurn` (this had been masked for weeks by
+  accumulated state in the dev home — the exact class of drift CI exists to
+  catch). The bootstrap now seeds `storages/workspace.json` (exact host
+  schema: ISO dates, `updatedAt`, registered path = the checkout) when absent.
+
+### Safe default guard
+
+Prefix every harness command with `DSH_HOME=$PWD/.dshdev` — or do not type it at all:
+**`scripts/dsh-scratch.sh` is that wrapper** (`--home .dshdev2` for a parallel profile; it refuses any
+resolved home under the real `~/.dsh` and execs `dsh` with `DSH_HOME` exported).
+
+This stopped being advice the hard way: during Stage 0 a bare `dsh plugin --profile web add ...` (no
+`DSH_HOME`) targeted the LIVE profile — the sandbox's read-only `~/.dsh` turned it into a harmless
+`pnpm failed` and the live `package.json` mtime proved nothing was touched, but "the sandbox protects us"
+is not a guard rail. Any future command form that can reach `~/.dsh` must route through the wrapper.
+
+## Test Layers
+
+Adopted from how the reference plugins actually iterate (they mostly *avoid* restarting the host):
+`dsh-session-fork` has ~8,000 lines across 28 `bun test` files and **every test file header says "no
+cordis"** — logic takes an injected `Ports` interface and only `makePorts(ctx)` touches the kernel. Copy
+that shape, and fix its one gap (§L1).
+
+### L0 — Pure logic, no cordis, no harness
+
+Ranges validation, the chapter renderer, TOC assembly, budget math, slug/number allocation. Fast, CI-able,
+no DSH at all. Structure for it from day one:
+
+```ts
+// src/continue.ts — no cordis import anywhere in this module
+export interface ContinuePorts {
+  readSession(id: string): Promise<SourceLog | null>
+  createContinuation(opts: { sessionId: string; seed: readonly SessionEvent[]; cwd?: string }): Promise<void>
+  reserveNumbers(key: string, count: number): Promise<number[]>
+  putRegistry(state: SessionState): Promise<void>
+}
+export async function runContinue(ports: ContinuePorts, args: ContinueArgs): Promise<ContinueResult>
+```
+
+`src/index.ts` is then the **only** cordis-facing file — a thin adapter, deliberately untested.
+
+### L1 — A *stateful* seed fake (where the reference plugin fell short)
+
+`dsh-session-fork`'s `createChildFromSeed` fake records `cut` as a number and validates nothing: it proves
+*which* port calls happened in *what* order, never what the kernel does with a seed. Our seed is one
+synthetic event, so a ~30-line fake can close that hole cheaply — **materialize** the seed and assert:
+
+- seqs contiguous from `0`, our notice at `seq = 0`
+- `surfaceOp: 'append'`
+- `inheritedEventCount` equals `seed.length`
+- the message source we emit is inside the kernel's accepted vocabulary
+
+That last bullet is `dsh-session-fork`'s most expensive lesson: a retired message source **bricked 66
+stored logs** which then refused to load, and they now pin the rule in a `format-watch` test because the
+validator isn't published. Our TOC is a synthetic `user/message` — pin the vocabulary test *first*, not
+after a chain of corrupted session files. Their `replaySurface()` idea (re-fold the durable log, assert it
+equals the live surface) is the right invariant to steal wholesale.
+
+### L2 — Boot-time probe in an isolated profile → **this is how Phase 0 runs**
+
+No LLM, no tokens, no HTTP driving, no restart of anything:
+
+1. Write the probe as a minimal plugin whose `apply()` runs the experiment during boot and writes JSON
+   facts to a file in the workspace.
+2. Install into the **scratch** profile. `dsh plugin add` runs pnpm and may reach for the network; if it
+   does, bypass it by symlinking the package into `.dshdev/profiles/web/node_modules/` and adding its id
+   via a `--patch` overlay (`dsh --patch <file>` applies an extra patch layer at boot).
+3. `DSH_HOME=$PWD/.dshdev dsh web --port 0 --no-open`, let it boot, read the JSON.
+
+Phase 0 questions it answers: is an unseeded/one-event-seeded `agents.create` durable, listed, and
+resumable; does `Session.create` accept the seed; does workspace attach work; does the child's header match.
+
+### L3 — Headless one-shot with a real model (only when a measurement needs it)
+
+`dsh --profile headless "…"` boots a profile, answers one task, prints, and exits — scriptable. The
+headless bundle emits JSON with `usage.cacheReadTokens`, which is the only way to get the cache numbers
+G1–G3 in `docs/verify.md` without a browser.
+
+Costs real model tokens and needs credentials: `$DSH_HOME/.credentials.yaml` is *inside* the relocated
+root, so a scratch home starts unauthenticated. **Copying the user's credentials into a scratch root is
+their call — ask before doing it, and ask before spending tokens on a measured run.** Prefer L0/L1 for
+correctness and reserve L3 for the few numeric assertions.
+
+### L4 — Upstream drift pins
+
+The habit that lets you iterate against a moving host without booting it: text-pin what you depend on in
+the *installed* bundles and fail loudly on drift.
+
+- `parity`-style: run the real published package against any vendored copy of its logic.
+- `upstream-watch`-style: assert literal call sites still look the way our assumption says (e.g. that
+  `session.fork` still seeds with `events.slice(0, cut)` — the fact invariant 2 rests on).
+- `vendor`-style: pin `file:line` + SHA markers in a header comment and assert they still exist.
+
+On failure, the procedure is *re-read upstream and re-anchor*, never "relax the assertion".
+
+## Speed Notes — What Made This Fast
+
+Learned while running eleven probe boots. Each is a concrete loop improvement, not a preference.
+
+**One env var is the whole isolation story.** `DSH_HOME=$PWD/.dshdev` cost nothing to adopt and removed
+the restart problem entirely. Every harness command in this repo should carry it — wrap them in
+`scripts/` or a `Makefile` so the bare form never appears in a doc or a shell history.
+
+**Failures before the provider are free.** Round 10's every turn errored on `{{model}}` assembly and
+spent **zero** tokens. A probe that fails early costs a boot, not a bill — which means it is cheap to
+probe aggressively and wrong. Prefer a probe that might error over reading five more source files.
+
+**Plain `.js` probes beat a build step.** The spike needed no TypeScript, no `tsc`, no `lib/` output:
+write `lib/roundN.js`, point `package.json` `main` at it, boot, read JSON. Ten seconds of edit-build-test
+with no compile errors to fight. Keep the *real* plugin in TypeScript; keep the *spike* in JS.
+
+**Reading the host's own session logs is the fastest source of truth.** `~/.dsh/sessions/<projectKey>/<id>/session.v3.jsonl.zstd`
+shows exactly what a real `user/message` looks like, which `source.kind` values are in play, and how large a
+request header actually is — all without a model call. Two gotchas that each cost a cycle:
+
+- `zstd -dc <path>` **fails** on these paths: they begin `--home-…` and are parsed as flags. Use
+  `zstd -dc -- <path>`.
+- Paths are bucketed by `projectKey(cwd)` (`session-persistence-jsonl/src/format.ts:224`), so a project
+  directory is encoded with `-` separators, e.g. `--home-adrian-Projects-dsh-chapter-fork--`.
+
+**`Function.prototype.toString` is useless on the bundled host** — service methods report
+`function () { [native code] }`. Do not plan to discover signatures by introspection; grep the
+`examples/deepseek-harness` checkout instead. Three signatures cost one wasted boot each when guessed:
+`listSessions(signal?)` is **positional**, `agents.resume({ resumeSessionId })` is an **options object**,
+and `create` needs `agentOptions` plus a preset.
+
+**TypeScript tests run with zero install and zero build — use this before reaching for vitest.**
+`node --test tests/unit/*.test.ts` executes `.ts` directly on Node 24 via type stripping, so the pure core
+(`src/render.ts`, `src/types.ts`) is covered by 15 tests in ~135 ms with **no `node_modules` at all**. That
+keeps the L0/L1 layer free: no dependency resolution, no bundler, no config, and it works inside the agent
+sandbox where an install may not. `npm test` is wired to it.
+
+Reserve the vitest/bun decision for when kernel-facing mocks need more than Node's runner gives, and note
+that `npm run typecheck` **does** need `typescript` + `@types/node` installed — a separate step from
+testing, deliberately not taken yet.
+
+**Keep probe turns tiny.** `Reply with exactly: ALPHA. No explanation.` finishes in one step and a few
+tokens, so a measurement that needs five real turns still costs pocket change.
+
+**Instrument the metric, not just the code.** The cache numbers only became readable once the formula was
+pinned: `totalPrompt = inputTokens + cacheReadTokens`, `hit = cacheReadTokens / totalPrompt`. Dividing by
+the uncached delta instead produces values like 1721% and quietly wrong conclusions — which is exactly what
+happened in these notes before it was caught.
+
+## Implementation Status (kept current)
+
+| Layer | State |
+|---|---|
+| `src/types.ts`, `src/render.ts` — pure chapter renderer, ranges, fence safety, artifact deferral | **done**, 15 tests |
+| `src/archive.ts` — reserve→write→verify ordering, dedup, idempotent retry, coverage, tamper detection | **done**, 11 tests |
+| `src/notice.ts` + `test/notice.test.ts` — TOC-notice builder, kernel-rejection reproduction, vocabulary pins + drift guard (Phase 0b check 5) | **done**, 10 tests |
+| `src/registry.ts` + `src/store.ts` — pure numbering/ancestry/plan state; zod domain spec, storage-domain + node-fs adapters | **done**, 13 + 7 tests; durability witness passed |
+| `src/engine-core.ts` + `src/engine.ts` — ChaptersCompactionEngine | **done & boot-proven** (r18/r19: realm subpath row mounts; real cascade finalizes; catch-path fix — FINDINGS § Phase 1) |
+| `presets/chapters/` + copy-on-boot install | **done & boot-proven** (r21; `!!js` has no `require`, so copying is the delivery mechanism — r20) |
+| `npm test` (unit + integration) | **188 passing, <2 s** — unit imports no harness services; integration runs the sync loop against a fake git driver AND a real git smart-HTTP server (git-http-backend on loopback; the git binary is test-only scaffolding, the product stays pure-JS) |
+| 20-check verify.md pass | **r26/26b**: 18 mechanical on `.dshdev2` (refusals with numbers, tamper marks, idempotent retry, cross-boot registry + resume); human rows per verify.md evidence map |
+| dsh-session-fork coexistence | **moot at this version** (r26a pair-boot): the example fails to boot the installed host ALONE (unguarded `webServer` read in the installed dsh-client-connection) — assumed-neutral until a version-matched pair-boot |
+| `scripts/dsh-scratch.sh` | **done** — refuses any DSH_HOME under the live `~/.dsh`; the near-miss is in FINDINGS |
+| `chapters_segment` / `chapters_continue` / `chapters_fork` tools (`src/tools.ts` → `continue-core.ts`) | **done & boot-proven** (r22 9/9 full loop incl. child read-back; r24 fork siblings reserve nothing; r25 durable title; r26/26b checklist pass) |
+| E3: header cache across a compaction | **measured, per-machine (r23 cloud, r28 local)**: cloud held 7,424 cached across the replacement (refill 13,658 → ~7,000); local llama.cpp reuses NOTHING across a head replacement (0 cached) but steady-state turns cost only the new content. Never quote one machine's number for the other |
+| Fork button | **shipped** (client bundle + slots; `remote.commands` trigger — `connection.rpc.handle` from a plugin is broken on this host, r26a; the button rides the command path instead) |
+| Long-context mechanics (architecture amendments) | **built + verified**: arrival-time artifacting (`src/arrival.ts`, `chapters_artifact`; e2e artifact-arrival green — 373KB book never prefilled, cache intact) and plot carriage (persona PLOT: + elicited fallback); e2e = 7 specs / 2 projects on a throwaway per-boot home (188 unit/integration) |
+| Knowledge layer P1 (record §13) | **complete**: project identity, redaction, live turn signatures, topic-sequential composition (fork proven live, r36; engine wired, r29), sharded index + incremental manifest, `chapters_search`, `/chapters-link`+`/chapters-status`+notice v2 (Project line, `(N msgs)`), the sync loop with §5.3 local-only degradation + diverged-rebuild, both schedulers and all §5.1 triggers wired. 9/9 live (r35j) + the two-machine exit criterion PROVEN as a browser journey (e2e suite green 2026-09-18) (composer→real remote→machine B searches) |
+| Topic mapping (composer) | **live-verified at the fork (r36)**: same-topic turns merge into one chapter on real signatures; topic changes split; pressure fragments stay legacy (measured, documented §4.2/4.3 + FINDINGS r28-36) — engine composition (r29) waits for deep selections |
+| `npm run typecheck` / `npm run build` | **working** — typescript 5.9 + @types/node + host packages as devDeps; `.ts`-import convention kept via `rewriteRelativeImportExtensions` |
+
+Two bugs the pure tests caught that would have been expensive later, which is the argument for this order:
+
+1. `coverage()` started its cursor at the first chapter, so a chapter set that skipped the **opening** of a
+   conversation reported no gap — silent history loss, exactly the failure the plugin exists to prevent.
+2. Substring assertions against `/chapters/` matched the store root `.dsh-chapters/` first. Worth knowing
+   as a general hazard: **the default store root name contains the word `chapters`**, so any path matching
+   in real code must anchor on the separator.
+
+## Unavoidably Manual
+
+Say it rather than pretend the suite covers it:
+
+| Needs a human | Why |
+|---|---|
+| Sidebar listing and visual resume of a continuation | no client-side harness reaches the real module loader contract |
+| Cache/prefill economics across process generations | inherently multi-process; a restart recomposes every header |
+| TOC readability and whether the model actually reloads chapters | judgement about prose and behaviour |
+| The Phase 0 verdict itself, read from probe output | L2 produces the facts; a human confirms the interpretation |
+
+## The TreeDX dev loop (knowledge transport provider #2)
+
+The TreeDX transport (`src/treedx/`, contract in `docs/provider.md`) is developed against an
+**in-process stub service** — `tests/integration/treedx-stub-server.ts` — which is part of the
+normal `npm test` loop (real HTTP on loopback, no docker, ~400 ms). The stub's semantics are
+the documented ones (bearer 401, one writable lease per branch, fail-closed moved-base commit,
+UTF-8 1 MiB File cap); when a real service disagrees, the live test below catches it and the
+stub changes to match — never the other way.
+
+The real service, for the exit-criterion run:
+
+```bash
+# the HOST user drives docker (the agent sandbox has the docker group stripped; its
+# HTTP view of the booted service is fine — that is where live tests execute from)
+scripts/treedx-local.sh up                     # builds examples/treedx (dev target); FIRST BOOT compiles the Rust NIF — minutes
+scripts/treedx-local.sh token                  # dev-token (30d), writes var/treedx-dev.env (gitignored, 0600)
+scripts/treedx-local.sh smoke                  # the six-step transport walk via curl
+node --test tests/integration/treedx-live.test.ts   # the §13 exit criterion over the real API
+scripts/treedx-local.sh down                   # stop; the data volume persists (down -v wipes)
+```
+
+`treedx-live.test.ts` skips ONLY when unconfigured (no env, no `var/treedx-dev.env`):
+configured-but-unreachable FAILS loudly — the skip flag is decided at load time, and a
+test that silently declines to witness after being told to run is broken tooling (a real
+bug class caught 2026-09-21, by the human, by running it). Proven live 2026-09-21:
+6/6 live tests green, full suite 286/286 with zero skips.
+`dev/treedx.compose.yaml` exists because the cloned repo's own `compose.yaml` is bound to the
+Treeseed platform network + connected auth and will not boot standalone (measured; recorded in
+`spikes/treedx/FINDINGS.md` §Environment). The cloned checkout under `examples/treedx/` is
+gitignored reference material — the live loop works from any path via `TREEDX_LOCAL_*` overrides.
+
+## Command Reference
+
+```bash
+# disposable instance — prefer the wrapper (DSH_HOME is its problem, not yours):
+#   NOTE: .dshdev mounts dsh-chapters-probe, whose rounds process.exit when done — fine for
+#   scripted one-shot boots, WRONG for an interactive test (the server dies seconds after the
+#   URL prints). Manual testing belongs to .dshdev2 (plugin only; probe removed 2026-09-17).
+scripts/dsh-scratch.sh web --port 0 --no-open                    # .dshdev (probe home)
+scripts/dsh-scratch.sh --home .dshdev2 web --port 0 --no-open    # clean home for the real plugin
+
+# manual form, if the wrapper is unavailable:
+mkdir -p .dshdev
+DSH_HOME=$PWD/.dshdev dsh web --port 0 --no-open
+
+# inspect the composed bundle tree of a profile without booting it
+DSH_HOME=$PWD/.dshdev dsh --profile web --dump-config | head
+
+# add our plugin to the SCRATCH profile only
+scripts/dsh-scratch.sh --home .dshdev2 plugin --profile web add "link:$(pwd)"
+
+# one-shot headless run (needs credentials in the scratch root; costs tokens)
+DSH_HOME=$PWD/.dshdev dsh --profile headless "list your tools"
+
+# unit tests
+bun test        # or vitest — decide in Phase 0, see docs/architecture.md § Open Questions
+```
+
+**Never** run `dsh plugin --profile web add` or `dsh web` without `DSH_HOME=` set to the scratch root:
+that is the profile and port this session is running on.
+
+## Correction to Earlier Claims in This Repo
+
+Two things I asserted before checking, now that they are checkable:
+
+1. **"`dsh web` runs in a tmux pane on the dev fork at `/app/checkout`"** — wrong for this machine. There
+   is no `tmux` binary and no `/app/checkout`; the harness serving this session is the global npm install
+   on `:3080`. The dev fork is `~/Projects/deepseek-harness`.
+2. **An isolated instance's port.** I first reported `34159` from a garbled read; the actual boot printed
+   `32869`. The curl to `34159` failing was the tell — it was my number, not the server's.
+
+Both are worth recording rather than quietly fixing, since the whole point of this file is that its
+commands were *verified*.
+
+## Suggested Additions to the Other Docs
+
+`docs/contract.md`'s file:line citations are from the `examples/` snapshot (rc.1-era), not the rc.2 fork —
+re-anchor them against whichever version a test actually boots.

package/docs/host-compaction-seam.md

@@ -0,0 +1,117 @@
+# The Host Compaction Seam — and Why It Shrinks the MVP
+
+Found in Phase 0b by reading `examples/deepseek-harness/packages/compaction/`, before writing any plugin
+code. The plan in `docs/architecture.md § Phases` assumed we would build pressure detection, surface
+replacement, and a TOC-delivery mechanism. **The harness already has all three.** Cited so this can be
+re-verified on upgrade.
+
+## What already exists
+
+| Capability | Where | Consequence for us |
+|---|---|---|
+| Automatic pressure compaction with a configurable ratio | `compaction-basic/src/index.ts:126` `if (this.config.auto) this._registerAutomaticCompaction()`; `thresholdRatio` default **0.8**, `retainRatio` **0.16** | Do not build `pressure.ts`. **Delete it from the plan.** We set `thresholdRatio: 0.9` in config. |
+| **Provider-confirmed context-overflow recovery** | `index.ts:220-221` (`context overflow recovery`), `maxOverflowRetries`, returns `{ kind: 'retry' }` | The user's "compact when it runs out in the current conversation" is already implemented, including retry. |
+| Between-step trigger, replayed at step boundaries | `_registerAutomaticCompaction`, `compactIfNeeded` "dynamically dispatched so subclass overrides are honored" | Correct trigger timing is inherited, and safe: never mid-step. |
+| Per-model policies | `modelPolicies: z.array(modelPolicy)`, `resolveTargetPolicy(config, target)` | 32K models get their own ratios without code. |
+| **One sanctioned extension hook** | `index.ts:215-217`: "`summarize()` is the **sole subclass customization hook**; the replay and durable mutation strategy stays fixed" | Our entire integration surface: subclass `BasicCompactionEngine` (exported, `export default BasicCompactionEngine`, `:434`) and override `summarize()`. |
+| Cache-aware auxiliary call shape | `:224-227`: the summary call's "prefix reuses the conversation's own system prompt, tools, and messages so the provider's KV cache is not invalidated" | The pattern to imitate if we ever need a model call — already solved for us. |
+| Transactional result contract | `ManualCompactionError` codes `busy / cancelled / changed / summary / commit / persistence` (`summary` = shrink floor refused; observed live in r12); `compaction/summary` event records summary inputs, route, model, usage, `shadowedSeqs`, `shadowedRange` | Real failure semantics to render, not to invent. `changed` = history moved under us; `commit` = partial application is a documented state. |
+| **Tool-result pruning** | `compaction-tool-result-pruner` — "replay-safe, model-free", `PRUNE_MARKER`, shadow-price event | Overlaps our deferral idea. Ours differs by writing a **retrievable file and a path reference**; its pruning replaces content with a marker. Verify they compose before assuming. |
+| Image offload | `compaction-image-offload` (`agents`, `sessions`) | Precedent for exactly the "attachments leave the body" pattern. |
+
+## The correction this forces on our founding premise
+
+`compaction/types.ts:3` describes `compaction/*` events as **"log-only, no surface events"**, and a
+separate replacement `user/message` carries the summary while `result.shadowedSeqs` names the hidden
+surface nodes.
+
+So DSH's compaction **shadows surface nodes; it does not delete the durable log.** Every claim in
+`README.md` and `AGENTS.md` that in-place compaction destroys the raw text is therefore **wrong about the
+record** — the text stays in the log. What is actually lost is *model accessibility*: the content is
+hidden from the projection and there is no addressable handle the agent can use to get it back.
+
+That is a smaller and more honest claim, and it redefines our value:
+
+> dsh-chapters does not prevent data loss the harness would otherwise cause. It makes compacted content
+> **addressable and re-loadable by the model** — topic-named chapter files with stable paths, an index the
+> model can consult, and a decision about which oversized tool results are worth carrying — plus branching.
+
+Everything in both docs that says "the raw conversation is gone" must be rewritten to say "the model can no
+longer see it." *(Done — README §the two costs and AGENTS.md invariant 1 now phrase it as loss of
+accessibility, with the check-2 qualification from Phase 0b.)*
+
+## Revised MVP (much smaller than the approved plan)
+
+1. **`ChaptersCompactionEngine extends BasicCompactionEngine`**, overriding only `summarize()`. Instead of
+   returning an LLM summary it returns **deterministic TOC blocks** over chapters written to disk — **zero
+   summarization tokens spent**, which also means no `summarizationProvider`/`summarizationModel` config,
+   no extra call, no summary-accuracy risk.
+2. **Archive module**: `render` (ranges → Markdown + frontmatter), artifact deferral
+   (content-addressed, dedup), `budget` check. Mount (corrected by Phase 0b — see "closed" section):
+   a **shipped user-root preset** whose compaction group names `dsh-chapters` instead of
+   `compaction-basic` (the web profile already ships the host rows `disabled: true`; the live engine
+   sits in the agent-preset realm), plus `agents.create({ setup: ctx => agentPresets.mount(ctx, id) })`
+   for continuations. The kernel refuses double-provision of `compaction` on one plane at boot.
+3. **Registry** (storage domain `dsh_chapters`): chapter records, ranges, hashes, per-session index.
+4. **`chapters_fork`** tool for branching/related conversations (Phase 0 already proved the mechanics), plus
+   `chapters_segment` if model-assisted ranges prove better than deterministic chunking.
+5. **Config**: `thresholdRatio: 0.9`, `retainRatio` tuned so the retained tail plus TOC fits a 32K window
+   after the measured ~12K header.
+
+Dropped from the plan: `pressure.ts`, `compact.ts`, custom surface mutation, custom trigger wiring,
+`chapters_compact` as our own command (`/compact` already exists via `command-compact`), and the summarizer.
+
+## Phase 0b closed — all five items measured (rounds 12-17)
+
+Full evidence in [spikes/probe/FINDINGS.md § Phase 0b](../spikes/probe/FINDINGS.md). One-liners:
+
+1. **Engine replacement: PROVEN.** Subclass + host-mounted service + `/compact` resolution + the
+   automatic `agent/pre-step` path all dispatch into our override, manual and automatic transactions
+   commit real durable records, zero provider calls (r12/r13). Double-provision on one plane fails boot
+   loudly (`service "compaction" has been registered at <BasicCompactionEngine>`, r14). The live mount
+   point is the **preset realm** — the web profile disables host rows and the `standard` preset mounts
+   the engine inside an `isolate: { compaction: true }` cordis group; a user-root preset naming our
+   package works end-to-end (r17, 6/6). New founding fact: **`agents.create` composes no preset** —
+   `meta.agentPreset` is a label; without `setup: agentCtx => agentPresets.mount(agentCtx, id)` a child
+   is a tool-less agent (974-token bare head, no `read`), with it the head is the real ~13.3K header
+   (27 tools; r16). This lands on invariant 2.
+2. **Shadowed content reachability:** no default model path exists (the web profile mounts no session
+   query tool; the sqlite index ships `openAt: never`), **but** an opt-in package
+   (`dsh-tool-session-query`, five tools incl. `session_event_read` = full unabridged event JSON)
+   already ships in rc.2 — mounting it costs five schemas + fixed guidance per request. Differentiation
+   survives as *readable topic-named files + zero-token index + forking*; the README wording was
+   tightened in the same change.
+3. **Prefix-cache sharing — CLOSED by r23:** cross-session sharing is ~one leading block (1,024 of
+   13.7K on a cold start with siblings warm; r10/11 stand). In-place compaction across a head-position
+   replacement: `cacheReadTokens` HELD (7,424 on both post-compaction turns) while uncached refill
+   halved (13,658 → ~7,000, stable) — the header survives replacement, "refill halves" is the honest
+   phrase, "fully cached" is not. So the cache story favors the engine as designed: the engine is the
+   relief mechanism; `chapters_continue` is for branching and topic resets (it pays one full cold
+   ~13.3-15K head).
+4. **Pruner × deferral: compose**, given one rule — render chapter bodies from the **durable log**
+   following each replacement's `sourceEventSeqs` (the pruner only shadows surface nodes; originals are
+   untouched), never from `summarize()`'s input, which already shows `PRUNE_MARKER` truncations.
+5. **Vocabulary: pinned** — `src/notice.ts` + `test/notice.test.ts` (kernel-measured rejections
+   reproduced; published `MessageSourceMap`/`ContextForm` constants; installed-`.d.ts` drift guard).
+
+### Also true, learned late (fix on contact)
+
+- `summarize()` receives no seqs; the manual path bypasses `compactRegion` dispatch; correlation is
+  post-commit via `CompactionResult.shadowedSeqs` / the `compaction/summary` event.
+- `compactIfNeeded` no-ops until the session has a durable routed request — the first step can never
+  compact (index.ts:260).
+- `retainRatio ∈ (0, 1]`; zero-tail tests use `retainTokens: 0`.
+- Subclassing inherits `static inject` — extend it or `ctx.agents` throws `cannot get property "agents"
+  without inject`.
+- Host consumers do `instanceof` against THEIR copy of `ManualCompactionError`; our engine's throws are
+  a different class when mounted from a different tree — pair `command-compact` with us in one tree or
+  ship our own command (r12 section J observed the raw-throw degradation).
+
+## Verification delta
+
+Landed as `docs/verify.md § Engine-Path Checks` (E1-E6): after automatic compaction, assert (a) the
+header prefix stays cached — **E3 still OPEN, never measured on a composed session**; (b) the
+durable log length **never decreased** (shadowing, not deletion — r12 proved the host honors this);
+(c) a compacted session can still answer a question whose answer lives only in a chapter it chooses to
+`read`; (d) the summary call cost is **zero** tokens, since our `summarize()` is deterministic —
+proved with the r12 stub (no `usage` field), must re-prove with the real TOC.