@treeseed/dsh-chapters 0.1.24 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (103) hide show
  1. package/README.md +2 -1
  2. package/cordis.patch.yml +49 -1
  3. package/docs/architecture.md +1 -496
  4. package/docs/backlog.md +234 -4
  5. package/docs/contract.md +127 -1
  6. package/docs/development.md +248 -12
  7. package/docs/host-compaction-seam.md +108 -0
  8. package/docs/knowledge-repo.md +532 -16
  9. package/docs/project-state.md +323 -0
  10. package/docs/verify.md +217 -0
  11. package/lib/archive.d.ts +49 -1
  12. package/lib/archive.js +56 -2
  13. package/lib/archive.js.map +1 -1
  14. package/lib/arrival.js +24 -10
  15. package/lib/arrival.js.map +1 -1
  16. package/lib/attach.d.ts +46 -0
  17. package/lib/attach.js +120 -0
  18. package/lib/attach.js.map +1 -0
  19. package/lib/auto-save.d.ts +74 -0
  20. package/lib/auto-save.js +162 -26
  21. package/lib/auto-save.js.map +1 -1
  22. package/lib/client.js +413 -11
  23. package/lib/client.js.map +1 -1
  24. package/lib/commands.d.ts +8 -1
  25. package/lib/commands.js +135 -7
  26. package/lib/commands.js.map +1 -1
  27. package/lib/compose.js +25 -6
  28. package/lib/compose.js.map +1 -1
  29. package/lib/continue-core.d.ts +24 -1
  30. package/lib/continue-core.js +41 -9
  31. package/lib/continue-core.js.map +1 -1
  32. package/lib/diag.d.ts +47 -0
  33. package/lib/diag.js +141 -0
  34. package/lib/diag.js.map +1 -0
  35. package/lib/engine-core.d.ts +51 -1
  36. package/lib/engine-core.js +63 -7
  37. package/lib/engine-core.js.map +1 -1
  38. package/lib/engine.d.ts +93 -0
  39. package/lib/engine.js +477 -36
  40. package/lib/engine.js.map +1 -1
  41. package/lib/enrich-queue.js +28 -2
  42. package/lib/enrich-queue.js.map +1 -1
  43. package/lib/enrich-wire.d.ts +92 -2
  44. package/lib/enrich-wire.js +135 -17
  45. package/lib/enrich-wire.js.map +1 -1
  46. package/lib/enrich.d.ts +27 -2
  47. package/lib/enrich.js +114 -18
  48. package/lib/enrich.js.map +1 -1
  49. package/lib/index.d.ts +51 -0
  50. package/lib/index.js +180 -2
  51. package/lib/index.js.map +1 -1
  52. package/lib/indexing.d.ts +7 -0
  53. package/lib/indexing.js.map +1 -1
  54. package/lib/injections.d.ts +1 -0
  55. package/lib/injections.js +38 -0
  56. package/lib/injections.js.map +1 -1
  57. package/lib/last-resort.d.ts +24 -0
  58. package/lib/last-resort.js +27 -0
  59. package/lib/last-resort.js.map +1 -0
  60. package/lib/prompt.d.ts +130 -0
  61. package/lib/prompt.js +255 -0
  62. package/lib/prompt.js.map +1 -0
  63. package/lib/registry.d.ts +39 -1
  64. package/lib/registry.js +76 -9
  65. package/lib/registry.js.map +1 -1
  66. package/lib/render.d.ts +31 -3
  67. package/lib/render.js +75 -3
  68. package/lib/render.js.map +1 -1
  69. package/lib/repair-run.d.ts +49 -0
  70. package/lib/repair-run.js +423 -0
  71. package/lib/repair-run.js.map +1 -0
  72. package/lib/repair.d.ts +200 -0
  73. package/lib/repair.js +354 -0
  74. package/lib/repair.js.map +1 -0
  75. package/lib/rules-commands.d.ts +22 -0
  76. package/lib/rules-commands.js +177 -20
  77. package/lib/rules-commands.js.map +1 -1
  78. package/lib/rules.d.ts +184 -7
  79. package/lib/rules.js +312 -27
  80. package/lib/rules.js.map +1 -1
  81. package/lib/save-core.d.ts +31 -1
  82. package/lib/save-core.js +153 -17
  83. package/lib/save-core.js.map +1 -1
  84. package/lib/session-rules.d.ts +222 -0
  85. package/lib/session-rules.js +454 -0
  86. package/lib/session-rules.js.map +1 -0
  87. package/lib/store.d.ts +22 -1
  88. package/lib/store.js +94 -17
  89. package/lib/store.js.map +1 -1
  90. package/lib/sync.js +13 -1
  91. package/lib/sync.js.map +1 -1
  92. package/lib/tools.d.ts +14 -0
  93. package/lib/tools.js +311 -32
  94. package/lib/tools.js.map +1 -1
  95. package/lib/types.d.ts +12 -1
  96. package/lib/types.js.map +1 -1
  97. package/lib/version.d.ts +9 -0
  98. package/lib/version.js +10 -0
  99. package/lib/version.js.map +1 -0
  100. package/lib/warnings.d.ts +10 -0
  101. package/lib/warnings.js +18 -0
  102. package/lib/warnings.js.map +1 -0
  103. package/package.json +9 -3
package/README.md CHANGED
@@ -17,7 +17,7 @@ dsh plugin --profile web update @treeseed/dsh-chapters # keep current — rest
17
17
  ```
18
18
 
19
19
  `--profile <name>` is required on every `dsh plugin` command (`web` = the browser harness profile).
20
- After restart you have the fork button, six agent tools, four `/chapters-*` commands — and the
20
+ After restart you have the fork button, seven agent tools, nine `/chapters-*` commands — and the
21
21
  optional `chapters` preset that routes `/compact` through the zero-token engine.
22
22
 
23
23
  ## Find the right page for your question
@@ -35,6 +35,7 @@ optional `chapters` preset that routes `/compact` through the zero-token engine.
35
35
  | read the design rules the system won't break | [§6 The four invariants](https://treeseed-ai.github.io/dsh-chapters/concepts/invariants/) |
36
36
  | the theory behind retrieval-over-compression (RLM) | [§8 Recursive context theory](https://treeseed-ai.github.io/dsh-chapters/concepts/recursive-context/) |
37
37
  | propose/approve rules for agents | [§13 Rules & governance](https://treeseed-ai.github.io/dsh-chapters/knowledge/rules/) |
38
+ | let a second model propose your next prompt | [§28 Prompt recommendations](https://treeseed-ai.github.io/dsh-chapters/concepts/prompt-recommendations/) |
38
39
  | debug something (“seems broken, is it?”) | [§4 FAQ & troubleshooting](https://treeseed-ai.github.io/dsh-chapters/start/faq/) |
39
40
  | prompt-cache / llama.cpp behavior | [§21 Cache behavior on llama.cpp](https://treeseed-ai.github.io/dsh-chapters/evidence/llamacpp-cache/) |
40
41
  | see measured results (production run + benchmarks) | [§19 Benchmarks](https://treeseed-ai.github.io/dsh-chapters/evidence/benchmarks/) · [§20 Field study](https://treeseed-ai.github.io/dsh-chapters/evidence/field-study/) |
package/cordis.patch.yml CHANGED
@@ -56,6 +56,31 @@
56
56
  # # ---- chapters & composition ----------------------------------------
57
57
  # chapterTokenTarget: 8000 # soft per-chapter size target
58
58
  # chapterLimit: 8000 # hard split threshold
59
+ # warningsDisplayCap: 6 # how many warning TEXTS a door prints before '…and M more' (#61).
60
+ # # The field failure was delivery: 86 warnings — naming a hollow
61
+ # # chapter and an unarchived 0-4082 hole — reached a human as
62
+ # # '(86 warning(s))'. A count is not a message.
63
+ # segmentCandidateCapTokens: 4000 # ceiling on chapters_segment's oversized-result candidate list (#61):
64
+ # # it emitted 49,932 chars while reporting 'Omitted 275266 bytes',
65
+ # # ~81K est tokens into a 32K window. Ranked by size DESC, so the cap
66
+ # # removes the small stuff, and the response names what was withheld.
67
+ # diagnosticsLog: '' # where door/child failures are appended (#61). Empty ⇒ the FIRST door
68
+ # # that touches a project resolves <store-root>/diagnostics.log;
69
+ # # DSH_CHAPTERS_ENGINE_ERRORS overrides it; the literal 'off' is the
70
+ # # only silence. Before this, unset meant NO TRACE ON DISK — the three
71
+ # # SAIP compaction finalization failures left none.
72
+ # childAutoArchive: true # the seventh archive moment (#61): a ch-* continuation child is not a
73
+ # # subagent, so door 5 never sees it. `ch-dc3be3ab` reached 2,363
74
+ # # events and 3 completed turns with chapters: [] — nothing failed,
75
+ # # NO MOMENT WAS WATCHING. Set false to close this door.
76
+ # childAutoArchiveMinTurns: 4 # after the first archive, how many unarchived completed turns re-arm it
77
+ # # (the first archive always fires: never-archived is the defect).
78
+ # chapterOversizeFactor: 2 # hard oversize multiple of chapterTokenTarget (#59): over the
79
+ # # TARGET warns ("split at the next boundary, never clip"), over
80
+ # # target × factor every door REFUSES with its numbers before
81
+ # # reserving one. The field shipped an 89,982-token chapter and a
82
+ # # 237,870-token orphan against an 8K target; a chapter nobody can
83
+ # # re-read is not an archive. 0/unset behaviour = warn only.
59
84
  # mergeThreshold: 0.3 # τ for merging turns into one chapter at
60
85
  # # archive time. Composition is archive-time
61
86
  # # only: tune for future archives.
@@ -104,6 +129,16 @@
104
129
  # # THIS machine (approval = per-machine
105
130
  # # curation fact, record §15)
106
131
  #
132
+ # # ---- prompt recommendations (#71, site §28) ----------------------------
133
+ # promptModel: '' # the model that DEFINES the next work.
134
+ # # '' ⇒ the feature does not exist (zero calls,
135
+ # # zero surface); setting one IS the enable.
136
+ # # Runtime: /chapters-prompt model <p/m|clear>
137
+ # promptContextChars: 4000 # hard budget on the assembled input slice
138
+ # # (chapter summaries + screened live tail)
139
+ # promptMaxChars: 240 # cap on the recommendation; over-cap ⇒ retry
140
+ # # once ⇒ refuse with numbers, never truncate
141
+ #
107
142
  # # ---- transport (2026-09-21 provider-seam amendment) -------------------
108
143
  # knowledgeProvider: 'auto' # 'auto' | 'git' | 'treedx'; 'auto' dispatches
109
144
  # # on the remote scheme (treedx+ → TreeDX),
@@ -115,6 +150,18 @@
115
150
  # treedxLeaseRetryDelayMs: 1000 # delay between lease retries
116
151
  #
117
152
  # Realm-row-only keys (in the preset's engine row, not this patch):
153
+ # harnessId, sessionRulesEnabled, rulesMarkerPrefix, coreRulesBudgetTokens,
154
+ # enrichmentMaxTokens (4000 — one auxiliary completion budget; sized to cover
155
+ # reasoning tokens, since the measured route reports reasoning inside completion usage
156
+ # and does not enforce max_tokens)
157
+ # rulesEnrichment (true — the ladder mines standing orders),
158
+ # rulesMinedPerChapter (3), rulesAutoActivate (true — mined inferences govern on
159
+ # arrival; false holds them for approve), rulesModel ('' — or a dedicated
160
+ # 'provider/model' for mining, one extra bounded call per chapter),
161
+ # projectRulesDisplayCap, sessionRulesDisplayCap, rulesRelevance (#50 §7/§7.5 —
162
+ # the realm plane owns what an in-place checkpoint renders; the host plugin config
163
+ # owns what a continuation notice renders; same names, same defaults, set both to
164
+ # diverge from the default)
118
165
  # thresholdRatio, retainRatio, retainTokens, maxTokens, compactionRetries,
119
166
  # maxOverflowRetries, auto, summarizationProvider/summarizationModel (legacy
120
167
  # override), enrichmentTrigger (afterPush|idle|both|manual),
@@ -169,7 +216,8 @@
169
216
  name: '@deepseek-ai/dsh-persona'
170
217
  config:
171
218
  prefix: You are a coding agent powered by the {{model}} model.
172
- suffix: "When a task spans multiple turns, end each reply with one line beginning 'PLOT:' (max 60 words — objective, current hypothesis, immediate next step). A checkpoint carries this plot forward; if it looks stale, revise it on your next plot line."
219
+ suffix: "When a task spans multiple turns, end each reply with one line beginning 'PLOT:' (max 60 words — objective, current hypothesis, immediate next step). A checkpoint carries this plot forward; if it looks stale, revise it on your next plot line.\n\n## Chapters memory and rules\nThis session keeps an archive of its own history, and a rule tier on top of it.\nBefore starting a task, run `chapters_rules` with the task description: it returns the standing orders in force here — project rules your team approved, plus session rules captured from this conversation, including ones nobody spelled out (each says which). Name the rules you actually followed, or say none; a rule cited wrongly twice is one the human revokes with a single command.\nWhen you need earlier work, a past decision, or bug history, run `chapters_search` FIRST — before re-reading files or asking the human. Read a cited chapter for the exact bytes; `chapters_artifact` holds the oversized tool outputs.\nDelegated subagents are covered by the same rules and the same archive: a child inherits the orders of the session that delegated it, so query them rather than assuming a fresh start."
220
+
173
221
  - id: agent-instructions
174
222
  name: '@deepseek-ai/dsh-agent-instructions'
175
223
  config:
@@ -1,496 +1 @@
1
- # Architecture and Design Decisions
2
-
3
- Decisions with their reasoning, so they can be revisited knowingly rather than rediscovered.
4
- API mechanics and file:line citations live in `docs/contract.md`; this file is the *why*.
5
-
6
- ## Naming
7
-
8
- **Decided: `dsh-chapters`** — package `dsh-chapters`, tools `chapters_segment` and `chapters_continue`,
9
- storage domain `dsh_chapters`, store root `.dsh-chapters/`. Recorded with its reasoning because the
10
- reasoning constrains future renames, not because the question is open.
11
-
12
- Why the two most available-sounding words were rejected: DSH's "fork" means *copy the parent's history
13
- into the child*, which is the one thing this plugin must not do. And "compact"/"summarize" implies the
14
- lossy in-place editing this replaces.
15
-
16
- | Candidate | Tools | Why taken or left |
17
- |---|---|---|
18
- | **`dsh-chapters`** ✓ | `chapters_segment`, `chapters_continue` | Names the durable artifact, makes no mechanism claim, greppable, and `continue` is exactly what happens: the conversation continues in a new session. Short enough for a plugin list. |
19
- | `dsh-chapter-continue` | `segment_chapters`, `continue_with_chapters` | Verb-forward and self-describing to a model scanning a tool list; clunkier as a package. |
20
- | `dsh-tome` | `tome_index`, `tome_continue` | Memorable and literally apt — a tome is a volume of chapters with a table of contents, which *is* the mechanic. But opaque to anyone who hasn't read the README. |
21
-
22
- Also rejected: `dsh-archive` (sounds read-only, and the plugin's job is to keep working).
23
-
24
- **One leftover: the repository directory.** The checkout is still `~/Projects/dsh-chapter-fork`. The
25
- folder name is cosmetic — installs use `link:$(pwd)`, and the bundle id comes from `package.json` and
26
- `cordis.patch.yml`, both of which say `dsh-chapters`. Rename it out-of-band with `mv` rather than from
27
- inside a session whose working directory *is* that folder, which would strand the tree. Keeping the
28
- identifiers in sync is what actually matters, and they are in sync.
29
-
30
- ## File Format: Markdown with Frontmatter, Not XML
31
-
32
- Asked and answered: chapters stay **Markdown**, with structure in YAML frontmatter.
33
-
34
- The reasoning against XML, strongest first:
35
-
36
- 1. **The consumer is a model reading text, not a parser.** Every markup token is a context token, in a
37
- plugin whose entire purpose is bounding context. `<turn role="assistant" seq="184">` costs several
38
- times what `**Assistant:**` does, repeated across a whole chapter.
39
- 2. **Structure already has two better homes.** The session log *is* the lossless machine-readable
40
- record, keyed by seq, and the registry stores ranges and hashes. Duplicating turn structure into the
41
- readable artifact creates a third source of truth that can silently drift from the other two.
42
- 3. **`.md` gets rendered; `.xml` gets downloaded.** The harness's own markdown component renders `.md`
43
- inline in the web UI, and the `read` tool, editors, and git diffs are all Markdown-shaped. XML files
44
- become write-only artifacts nobody looks at.
45
- 4. **Markdown is the established convention for exactly this shape.** DSH's skill loader parses
46
- YAML-frontmatter-plus-Markdown-body files — the same idiom, already understood by the tooling.
47
-
48
- What we *do* take from the XML instinct, because it was a real concern:
49
-
50
- - **Frontmatter carries the parseable structure** — `chapter`, `session`, `root`, `seqRange`, `sha256`,
51
- `tokens`, `artifacts`. Queryable with a YAML parser, invisible to the reading experience.
52
- Chapters authored by a **subagent** carry two more lines — `subagent: true` and
53
- `parentSession: <id>` — copied from the child's durable session header, which the host itself
54
- stamps (`childSessionMeta`: `parentSession`, `origin: subagent`, `delegationDepth`, `agentPreset`;
55
- measured, `spikes/subagent-end/FINDINGS.md`). Emitted ONLY when the header has `parentSession`,
56
- so existing chapter bytes never change — which is also why no tape needs re-recording.
57
- - **There is one index, not three.** The TOC lists the chapters for reading; the storage domain already
58
- holds the machine-readable enumeration. A separate `MANIFEST.md` was proposed and **rejected as
59
- redundant** — a regenerated duplicate that can drift, whose growth becomes a context cost when read
60
- whole, and the sole exception to append-only. Dropping it removes all three problems at once.
61
- - **Fence collision is a renderer bug, not a format problem.** Conversation text containing ```` ``` ````
62
- must be escaped by choosing a longer fence per block. This is the actual failure mode an XML format
63
- would have prevented, and it is solved in ~10 lines of the renderer. Get it wrong and chapters
64
- silently truncate at an embedded fence.
65
-
66
- **The one case that would reopen this:** if a future feature needs to *rewrite* chapter structure
67
- programmatically — re-segmenting an existing chapter by turn boundaries without a session log — then a
68
- losslessly round-trippable format earns its tokens. Rendering is currently one-way from log to file, so
69
- it doesn't. Revisit if chapter *editing* ever enters scope.
70
-
71
- ## Chapter Rendering and the Artifact Store
72
-
73
- The model supplies ranges; the plugin renders. Within a range:
74
-
75
- | Event content | Handling |
76
- |---|---|
77
- | user / assistant text | inline, verbatim, attributed |
78
- | tool **invocation** (name + arguments) | inline as a bounded head (400 chars) — the "what was attempted" half of the record; above `callArgArtifactTokens` the full arguments are **artifact-backed** like any deferred blob (0 = off by default) |
79
- | session **plan** (`exit_plan_mode` call + its verdict result) | full text always artifact-backed + a labeled `## Plans in this chapter` block after the transcript (docs/plans-carriage.md) |
80
- | tool **result** judged **not worth keeping** | deferred to `artifacts/`, replaced by a reference line |
81
- | tool **result** judged worth keeping, ≤ floor | inline, verbatim, no model input needed |
82
- | tool **result** judged worth keeping, > floor | inline by model override **and** still written to `artifacts/` |
83
- | images / attachments | deferred to `artifacts/` by path (Markdown cannot carry them) |
84
- | surface ops, projections, lifecycle events | omitted from the body; their seq ranges are recorded in frontmatter |
85
-
86
- ### Who decides: the model, not a threshold
87
-
88
- The keep/defer call is **semantic**, so it belongs to the agent that lived the conversation. "This diff
89
- matters; this npm install log does not" is not inferable from size — a 4-line compiler error is load-bearing
90
- and a 400-line `read` of a config file may be pure noise. A size threshold gets this backwards exactly often
91
- enough to matter. *(Amended 2026-09-19 below: one band — results too large to ever be reasoning context —
92
- is artifacted at arrival without judgment; this section's rule governs everything that can genuinely fit.)*
93
-
94
- Division of labour:
95
-
96
- - **Plugin enumerates.** It has the log, so `chapters_segment` returns, per chapter, the candidate tool
97
- results with `seq`, tool name, and size. The model never guesses at byte counts.
98
- - **Default is defer** for anything above `toolResultDeferFloorTokens`; trivial results stay inline without
99
- being itemized, so the narrative reads naturally and the exception list stays short.
100
- - **Model submits exceptions only**: `toolResultOverrides: [{seq, inline}]`. Usually empty or one entry, so
101
- the judgment costs almost no output tokens — which is why this is affordable where "the model writes the
102
- chapter bodies" was not.
103
- - **Invocations stay inline as a bounded head.** Plugin rule, not a choice; it is the "what was
104
- attempted" half of the record and the thing a successor most needs. The head is 400 characters, so
105
- a large argument (`write` carrying a whole file, a plan carrying a whole design) is *also* written to
106
- `artifacts/` whenever `callArgArtifactTokens` is above zero — the inline text is then a head with a
107
- resolution, never the only copy. Session plans get this treatment unconditionally and by name: the
108
- plan's own bytes are always an artifact, cited with its outline and verdict
109
- (docs/plans-carriage.md § 2/§ 3).
110
- - **The artifact is written either way.** Inlining is a convenience copy *alongside* its reference. So a
111
- wrong call is a context-economy miss, never data loss — that asymmetry is what makes delegating this
112
- judgment to the model safe.
113
-
114
- The floor is a formatting device, not the policy, and its value is a Phase 1 calibration guess. Oversized
115
- output is caught by the continuation budget and the chapter split signal, both of which **report or refuse
116
- rather than clip**.
117
-
118
- Reference line format — stable, greppable, and `read`-able:
119
-
120
- ```
121
- ↳ tool: bash `npm run migrate -- --dry-run`
122
- result: 41.2 KB, sha256 8f3a1c… → .dsh-chapters/<root>/artifacts/8f/8f3a1c9d….txt
123
- ```
124
-
125
- Artifacts are **content-addressed** at `artifacts/<first-2-hex>/<sha256>.txt`, which means identical
126
- results across an entire subtree — the same 200-line file read ten times — are stored once. That
127
- deduplication is a large part of why this design stays cheap over long chains.
128
-
129
- ### This changes the losslessness claim, and the claim must change with it
130
-
131
- Chapters are no longer self-contained. The honest formulation is:
132
-
133
- > **Every byte of the conversation remains retrievable** — inline in its chapter, or by an exact path
134
- > from it. Nothing is paraphrased, summarized, or dropped.
135
-
136
- Do **not** write "the chapter contains the full history" in a TOC preamble or a README. The distinction
137
- matters operationally too: following a reference costs the model a turn and real tokens, so the default
138
- rendering must keep the 99% case (who said what, and what was attempted) inline. That is precisely the
139
- trade the user's judgment called for: long dumps are the least valuable thing to re-read, and the most
140
- expensive to carry.
141
-
142
- If a chapter still exceeds `chapterTokenTarget` after deferral, that is the *split* signal — not a
143
- license to truncate. Truncation is refused by invariant 4 and by "never truncate to fit."
144
-
145
- ### Amendment (user-directed, 2026-09-19, APPROVED, **landed**): arrival-time artifacting for the impossible band
146
-
147
- > Shipped with phases 2-4 of the artifacting plan: `src/arrival.ts` writes the stub, `chapters_artifact`
148
- > is in `src/tools.ts`. Verified by unit rows plus browser journey **E7** in
149
- > [verify.md](verify.md) — one `read` of a 373 KB book yields exactly one arrival stub at the tail, zero
150
- > compaction events at the production 0.9 threshold, the blob fingerprint absent from every `request/*`
151
- > event, and `cacheReadTokens` monotonic across the turn (the prefix was never rewritten).
152
-
153
- The section above lets the model decide what to defer **at archive time** — a judgment about material
154
- that fit in the window to begin with. This amendment carves off the band where there is no judgment to
155
- make: a tool result above a hard threshold (config key `toolResultArtifactTokens`, default 8000,
156
- floored at 256) **never enters the surface at all**. A 300-page book does not fit a 32K desk; loading it
157
- to "see what happens" is not deference to the model's semantics, it is an emergency waiting to happen.
158
-
159
- Mechanics:
160
-
161
- - The result is written to the artifact store (content-addressed, same path scheme as archival
162
- deferral — dedup applies) and the surface receives a **reference stub** instead: tool, size, sha,
163
- path, plus the query instruction below. A few hundred tokens, appended at the tail — prefix cache
164
- untouched. (This is distinct from the pruner's mid-history rewrite, whose re-prefill cliff on
165
- llama.cpp we measured r28; arrival-time interception never rewrites the past.)
166
- - A new read-only tool, **`chapters_artifact`**, queries artifacts *inside* their file: `search`
167
- (terms/regex → matching blocks with line numbers, bounded by a maxTokens pack), `read`
168
- (offset/limit — the model can still pull an exact section when its judgment says so), and `toc`
169
- (for structured documents: markdown headings parsed to a line-numbered contents list — a 300-page
170
- paper arrives with its own table of contents in hand).
171
- - Consequence on the failure inventory: the **retention paradox** (a newest node too large to file,
172
- too new to keep ⇒ bounded refusal) becomes structurally unreachable for tool results — the monster
173
- is never on the desk to be caught in retention. This is the "handle all abortions" directive
174
- realized at the gate rather than at the rescue.
175
- - External validation: this is the Recursive Language Model result (Zhang, Kraska & Khattab,
176
- arXiv:2512.24601) — arbitrarily long material as *external environment* the agent inspects
177
- programmatically. Their mechanism is a Python REPL over the prompt; ours is a file + query tool
178
- + parsed TOC, which delivers the same decomposition without code execution or a sandbox. Their
179
- reported compaction-baseline gap (median 26%) is the quality argument; the SRLM follow-up
180
- (arXiv:2603.15653) supplies the policy guard: recursion that splits what already fits **hurts** —
181
- hence the hard band boundary, not a general split-everything stance, and bounded pulls remain the
182
- model's choice.
183
-
184
- The archive-time model-judgment machinery above is unchanged for results below the floor.
185
-
186
- **Verification (2026-09-19):** `tests/e2e/artifact-arrival.spec.ts` (green 9.3m and
187
- 4.1m runs) — a 373,861-byte book fetched in one read produced exactly the arrival
188
- prune+stub pair, ZERO blob occurrences in any `request/*` event (never prefilled),
189
- 5–15 `chapters_artifact` calls recovering facts from inside it, and a cache table
190
- (FINDINGS) showing `cacheReadTokens` monotone across the stub boundary — the tail
191
- rewrite costs the prefix nothing, unlike the pruner's mid-history cliff. Row config
192
- `toolResultArtifactTokens` is pinned per e2e project (500 for arrival, default
193
- 8000 elsewhere).
194
-
195
- ### Amendment (user-directed, 2026-09-19, APPROVED, **landed**): the plot note across in-place compaction
196
-
197
- > Shipped: the forward path is the preset persona suffix (the agent is told to end substantive turns
198
- > with a `PLOT:` line) and the engine copies the latest such note verbatim into the replacement above
199
- > the index cards; the elicited fallback is one bounded call gated by `elicitedPlot` (default `true`) —
200
- > tail-heavy excerpt capped at ~8K chars in, `maxTokens: 220`, note clamped to 900 chars, any failure
201
- > degrading to `null` (no note, never a crash). Both halves pinned in unit; the forward path proven live
202
- > in verify **E8** (a suite-B run carried `PLOT:` blocks through seven compactions). The 60-word figure
203
- > below supersedes the ~120 originally proposed.
204
-
205
- Heavy mid-turn tidying measurably erodes *plan* continuity (FINDINGS, oversized-turn runs: model
206
- finished the reading but abandoned the wrap-up). The TOC preserves what was **done**; this adds what
207
- is **going on**, authored while the model still holds the thread:
208
-
209
- - **Forward-maintained (preferred):** preset discipline — at each turn's end the agent leaves a
210
- compact *plot note* (objective, current hypothesis, next step; ~40–80 words, no tools). Compaction
211
- copies the latest note verbatim into the replacement, above the index cards. Cost: a little output
212
- per turn; zero extra prefill at compaction, which is when cost bites.
213
- - **Elicited (fallback, bounded):** if no plot note exists at compaction time, one auxiliary
214
- completion over the **TOC + retained tail only** (tail-heavy excerpt, ~8K chars in, ≤60 words out,
215
- `maxTokens: 220`; key `elicitedPlot`, default `true`) asks the model to state the plan. This is
216
- deliberately not a re-read of the archived span — it costs a fraction of what a naive summary call
217
- would, and only when the free path failed.
218
- - Invariant check: the model authors *state*, never archive text (invariant 4 intact); the note rides
219
- the replacement event append-only; every byte of the conversation still lives in chapters.
220
-
221
- ## The archive moments
222
-
223
- Chapters are written by exactly five doors, and knowledge that trips none of them is knowledge
224
- that never enters the index. Four are session-facing, all idempotent over a per-session watermark
225
- (`fromSeq = max(chapter.endSeq) + 1`) and anchored on the last **completed** turn (`turn/end`):
226
-
227
- 1. `chapters_continue` (tool) and `/chapters-continue` (deterministic door) — archive + child session.
228
- 2. `chapters_fork` (tool) and `/chapters-fork` (door) — archive + branch session (no new bytes).
229
- 3. `/chapters-save` — archive only, parent untouched.
230
- 4. Compaction finalize — the engine's `summarize()` commits a plan pre-durable, and the post-commit
231
- finalizer writes the chapter (issue #47 note: the fanout field study shows this is the ONLY one
232
- of these that subagents historically reached — 20 child compactions, or nothing).
233
- 5. **Subagent settlement (issue #47, this round)** — a host-plane listener on the published
234
- `subagent/end` lifecycle event runs the SAME save pipeline (one implementation, shared with
235
- door 3 by extraction — never a second renderer) for children whose header carries
236
- `origin: subagent` and `agentPreset === chapters`. The emitter fires `end` synchronously before
237
- the holder's dispose continuation (measured, spike), so the fast path is a synchronous
238
- `agents.get(info.id).session.snapshotEvents()`; the fallback arm is `sessionQuery.listEvents`
239
- over the durable log (ambient-readable — measured); when neither reads, the child is skipped
240
- with a loud diagnostic, never guessed. Save-only, so the "no continuation-creating actions
241
- mid-step" rule is not engaged: no session is created, no surface is compacted. Enrichment is
242
- **queued** per settlement — a 10-child fanout must not `runNow` ten annotator passes. A
243
- continuable child settles repeatedly; the watermark makes every repeat after the first a
244
- save of only-new-turns. Config: `subagentAutoSave` (default on).
245
-
246
- ## Budgets and the Reserve Margin
247
-
248
- Two distinct ceilings. Conflating them caused the original design's silent-failure mode.
249
-
250
- **Continuation budget** — the *newly added* context of a child must be small and predictable, as a share
251
- of the space left after the fixed header:
252
-
253
- ```
254
- tocTokens + handoffNoteTokens ≤ continuationBudgetRatio × (window − systemPromptTokens)
255
- ```
256
-
257
- **Chapter target** — per-file size, governing rendering and splitting. Not a hard error.
258
-
259
- Deriving the budget from the **remaining** window rather than the whole window is not a nitpick. On a 32K
260
- model with a ~9K system prompt and tool schemas, a 25%-of-window rule gives an 8K ceiling that the TOC and
261
- note may blow through before any work happens, and the plugin would refuse every continuation on exactly
262
- the small-context hardware it targets. The header is a fixed cost to subtract, not a competitor for the
263
- index. Measure it once per generation and cache it.
264
-
265
- The budget is enforced as a **preflight**: measure, then refuse with concrete numbers
266
- (`TOC 4,120 + note 2,800 = 6,920 > budget 5,750 (0.25 × (32,000 − 9,000))`) *before* creating any
267
- session. Never truncate to fit — a clipped handoff note is the lossy behaviour this plugin exists to
268
- avoid, and a silent clip is exactly the bug that takes a month to notice.
269
-
270
- There is deliberately **no cap on the handoff note.** The budget check subsumes it: a model that dumps the
271
- whole conversation into the note fails preflight and is told by how much. The note should be state of play
272
- — decisions held, the next step, anything in flight — and the tool description should say so rather than a
273
- numeric limit doing the talking.
274
-
275
- This margin is also what automatic triggering needs, which is why the phases were reordered: pressure
276
- measurement, the preflight, and a *suggestion* at `turn/end` all land in Phase 2; only autonomous
277
- action stays late.
278
-
279
- ### The native branch button: cosmetic hide (user-approved 2026-09-17)
280
-
281
- The message-row branch glyph is host chrome, not a slot entry. The chosen lever is
282
- **display:none on its accessible label, scoped to action-row buttons** (en+zh spellings);
283
- the sidebar fork entry stays. If a harness update rewords the label, the native button
284
- reappears beside ours — accepted deliberately: a visible, temporary duplicate beats
285
- behavioral monkey-patching of the client's fork call (the dsh-session-fork route, rejected
286
- for its version-drift surface). Do not "upgrade" the hide into an intercept without asking.
287
-
288
- ## Ancestry: A DAG We Flatten For The Model
289
-
290
- History becomes a tree the moment someone continues an older ancestor twice. The worry that agents
291
- won't traverse a DAG is correct, and the answer is that **they never should have to.**
292
-
293
- Three rules:
294
-
295
- 1. **The plugin walks; the model reads a list.** At continuation time the plugin walks the ancestor path
296
- in the registry and emits a flat, chronologically ordered, numbered TOC. Reading order *is* causal
297
- order. Depth 10 or mid-branch, the model sees a numbered index and a `read` path — no graph, no
298
- inference, no traversal.
299
- 2. **Back-links are printed, not implied.** The TOC footer names the root session and the parent session,
300
- so the trace is visible without being reconstructed.
301
- 3. **The TOC is the complete index for the current line of work.** Every chapter on the ancestor path is
302
- listed with its summary, so ordinary follow-up needs no traversal at all.
303
-
304
- Cross-**branch** recall — "what did the other attempt try?" — is the only case the linear TOC cannot serve,
305
- and it is rare. It is answered by the registry (which holds the whole tree) behind a future read-only
306
- command, never by a per-fork index file. Carrying it in every session's prefix to serve an occasional
307
- question would tax all work for one.
308
-
309
- Two structural consequences:
310
-
311
- - **Chapters are keyed by creating session**, not by a linear chain id. Siblings branching from one
312
- ancestor get different, correct TOCs and cannot collide on numbering.
313
- - **Append-only has no exceptions.** Nothing derived is written back into the store; the registry is the
314
- only mutable state, and it is a domain table designed for it.
315
-
316
- ## Storage
317
-
318
- Two stores because they have opposite requirements.
319
-
320
- **Chapters and artifacts: workspace files.** The `read` tool must reach them, and it runs under the
321
- session's file sandbox (`workspace-write` permits only the workspace tree). The original
322
- `~/.dsh/chapters` design fails asymmetrically: writes to `$HOME` may succeed while `read` cannot reach
323
- the path, so every TOC bullet is silently dead and the plugin looks like it worked. `ctx.fs` is also
324
- **optional**, so its absence must be a clean refusal.
325
-
326
- Layout:
327
-
328
- ```
329
- <store-root>/ # default .dsh-chapters/
330
- <rootSession>/
331
- chapters/001-project-setup.md # append-only, never renumbered
332
- artifacts/8f/8f3a1c9d….txt # content-addressed, deduplicated
333
- ```
334
-
335
- Add `<store-root>/` to `.gitignore` on first write and say so — polluting a user's tree silently is its
336
- own small bug.
337
-
338
- **Registry: a DSH storage domain.** Ancestry, numbering, hashes, ranges, bullet lists. Sanctioned
339
- durability, survives restart, and gives the append-only counter a real allocation home instead of a
340
- directory scan racing a restart.
341
-
342
- Trade-off to keep honest: a workspace-relative root makes a chain **cwd-scoped**. Crossing workspaces
343
- means an absolute configured root, at which point `read` reachability must be re-verified — do not assume
344
- an absolute path inside one workspace stays reachable in another.
345
-
346
- ## Atomicity
347
-
348
- Writing files, allocating numbers, creating a session, and committing registry state is not
349
- transactional. Order them so the worst failure is an orphan, never a lie:
350
-
351
- 1. **Reserve** numbering, chapter ids, and slug assignments in the registry *before* writing files.
352
- 2. Render and write chapter files and artifacts; verify each path resolves.
353
- 3. Create the child session with the TOC notice.
354
- 5. **Only then** commit the registry entry linking parent → child — this is the commit point that makes
355
- the continuation visible.
356
-
357
- Retry must be **idempotent**: key the attempt on `sourceSession + archiveCeiling`, so a second call reuses
358
- reserved numbers and existing files rather than duplicating chapters. Failure before step 5 leaves
359
- unreferenced files, which is recoverable garbage; failure after step 4 without step 5 leaves a child
360
- citing chapters that don't exist, which is not. Hence the ordering.
361
-
362
- Concurrency is real, not theoretical: parallel branch work is a first-class pattern in this ecosystem, so
363
- two agents can continue from one ancestor at the same time. A directory-scan counter races; registry
364
- allocation does not.
365
-
366
- ## Integrity
367
-
368
- Chapter files are plain Markdown in a writable tree whose content is replayed into a child as an
369
- authoritative `user/message` — the model's own claimed memory. Anything that can write those files,
370
- including a tool executed inside the workspace, injects text into every session downstream. That is a
371
- durable prompt-injection surface, and worse than the ordinary kind because it arrives wearing the
372
- guise of the agent's own past.
373
-
374
- Mitigation, in the order that pays:
375
-
376
- 1. **Hash every chapter body and artifact at write time** (`sha256` in frontmatter *and* registry).
377
- 2. **Verify before citing** at continuation time; a mismatch gets an explicit `⚠ modified since
378
- archived` marker on its TOC bullet rather than silent trust.
379
- 3. **Record the authoring session** per chapter, so tampering can be correlated with a turn.
380
-
381
- Not solved: a determined in-workspace writer can update the registry hash too. The honest claim is
382
- *accidental drift and single-shot injection are detected*, not *the archive is tamper-proof*. Say it
383
- that way.
384
-
385
- ## Bounds — What Still Grows
386
-
387
- The head is bounded; the index and the disk are not, and neither doc should imply otherwise.
388
-
389
- - **The flattened TOC must stay an ancestor-path list, not a whole-tree dump.** Listing every chapter of
390
- every branch would make "flatten the DAG" a new way to exhaust context — the § Ancestry rule (walk the
391
- ancestor path only; siblings stay behind a future read-only command) is what bounds the index; folding
392
- below handles depth itself.
393
- *(This bullet reconstructs a line corrupted in the file's first commit — the fragment `DAG" becomes a
394
- new way to exhaust context` survived without its opening. The intent was restated, not recalled; see
395
- spikes/probe/FINDINGS.md § Doc corrections. Delete this parenthetical once re-reviewed.)*
396
- - **TOC length grows with depth.** At ~30 tokens per bullet and 3 chapters per continuation, a depth-10
397
- chain costs ~900 tokens — fine. Depth 100 costs ~9K, which is not. **Bullet folding** belongs in
398
- Phase 4: when ancestor bullets exceed `tocBulletBudgetTokens`, fold the oldest group into one summary
399
- bullet citing a fold chapter. Folding happens only at continuation boundaries.
400
- - **Reload is a feedback loop.** A model that `read`s three 8K chapters immediately re-triggers pressure
401
- and may continue again. State the remaining reload headroom in the TOC preamble so the model spends it
402
- deliberately, and have the pressure detector count reloaded content like anything else.
403
- - **Retention.** Append-only with no GC is unbounded disk growth across abandoned chains. A retention
404
- policy, or at minimum a documented manual prune command, is required before anyone runs this for
405
- months. Content-addressed artifacts make pruning safe to reason about — refcount them.
406
-
407
- ## Open Questions
408
-
409
- Carried into Phase 0 deliberately, not by omission:
410
-
411
- 1. **Is an unseeded `agents.create` durable, sidebar-listed, and resumable?** The premise of everything.
412
- If not, the design needs a different carrier and should not be funded as written.
413
- 2. **Does creating a session schedule a turn?** If so, the child burns its first turn acknowledging its
414
- own TOC. Fix by instructing the notice to stay silent until addressed, or by creating without
415
- running.
416
- 3. **Coexistence with `dsh-session-fork`.** It already maintains session lineage and a branch registry,
417
- and patches client `sessions.fork` globally. Depend on it as a peer, vendor against the same pinned
418
- version, or declare mutual exclusion — all defensible; picking silently is not.
419
- 4. **How do tool results that reference deleted files render?** A `read` of a file later removed is
420
- still verbatim text, which is correct — but a `bash` output naming a path that no longer exists is
421
- misleading without context. Probably a renderer annotation, not a format change.
422
- 5. **Test strategy.** The verification checklist is manual, but "the child is smaller than the parent"
423
- and "ranges validate" are exactly what an automated regression suite should own. Both reference
424
- plugins have `tests/`; we should decide the harness (vitest, like `dsh-compact`) before Phase 1.
425
-
426
- ## References
427
-
428
- `examples/` is gitignored; read the **one** file that answers the question. Do not read all four.
429
-
430
- | Concern | File |
431
- |---|---|
432
- | Workspace attach, preset compose, resume kernel | `dsh-session-fork/src/vendor/fork.ts` — header lists upstream sources + pinned commit |
433
- | Why `agents.create` is the only durable route; notice event | `dsh-session-fork/src/branch.ts:8-14,154` |
434
- | Storage-domain registry | `dsh-session-fork/src/store.ts:36-43` |
435
- | `defineTool`, `jsonOutput`, `exec` caller | `dsh-session-fork/src/tools.ts:110-121,200` |
436
- | `inject`, registration | `dsh-session-fork/src/index.ts:161,786` |
437
- | Real fork handler — confirm what we avoid | `deepseek-harness/packages/api/session-controller/src/commands.ts:185-282` (`seed` at `:260`) |
438
- | `CreateAgentOptions`, optional `seed` | `deepseek-harness/packages/core/agent/src/index.ts:95,388` |
439
- | `SessionHeader` | `deepseek-harness/packages/core/session/src/types.ts:93-122` |
440
- | `defineTool` signature | `deepseek-harness/packages/core/tools/src/schema.ts:545` |
441
- | `ctx.llm.stream()` prefix reuse | `deepseek-harness/packages/compaction/compaction-basic/src/index.ts:225,290` |
442
- | `ctx.fs` is optional | `deepseek-harness/packages/context/agent-instructions/src/index.ts:6` |
443
- | `Schema<Config>` idiom | `dsh-compact/src/index.ts:15,26-37` |
444
- | Packaging (`dsh.bundle.patch`) | `dsh-session-fork/package.json`, `cordis.patch.yml` |
445
-
446
- **Do not port from:** `dsh-session-fork/src/vendor/compact.ts`, `squash*.ts`, `merge-region.ts` (span
447
- rewriting — forbidden); `dsh-session-fork/src/prompt.ts` (system-prompt contribution — invariant 3);
448
- `dsh-compact`'s mutation strategy (in-place by design). `dsh-plugin-dev-kb` is a minimal plugin shell
449
- only; skip it otherwise.
450
-
451
- ## Phases
452
-
453
- > **Superseded in part.** `docs/host-compaction-seam.md` found that the harness already ships automatic
454
- > pressure compaction (`thresholdRatio`, default 0.8), provider-confirmed overflow recovery, per-model
455
- > policies, and a single sanctioned `summarize()` subclass hook on `BasicCompactionEngine`. Phases 1-3 below
456
- > predate that discovery: `pressure.ts`, `compact.ts`, custom surface mutation, and our own compact command
457
- > are all **dropped**. The MVP becomes a subclassed engine that returns a deterministic TOC over archived
458
- > chapters, plus the archive, registry, and fork tool. **Phase 0b (rounds 12-17) verified that revised MVP
459
- > end-to-end** and settled the mount: our engine rides a preset realm (`isolate: { compaction: true }`
460
- > group, our row replacing `compaction-basic`), continuations compose via
461
- > `setup: agentCtx => agentPresets.mount(agentCtx, presetId)`, and `chapters_continue` remains the
462
- > *branching* mechanism — automatic relief is the engine's. Read
463
- > [docs/host-compaction-seam.md](host-compaction-seam.md) and FINDINGS § Phase 0b before implementing.
464
-
465
- Reordered so the reserve-margin and pressure machinery — which automatic triggering needs — is built
466
- early rather than last.
467
-
468
- **Phase 0 — Prove the premise.** Skeleton, one trivial tool with an `output` schema, then: unseeded
469
- create → durable? listed? resumable? header-matching? Single-event seed at `seq = 0` accepted by
470
- continuity validation? Child first-request tokens materially below the parent's? Settle naming, the test
471
- harness, and the `dsh-session-fork` coexistence question here.
472
- *Gate: if the child is not durable and listed, stop and redesign.*
473
-
474
- **Phase 1 — Render, cite, and bound.** `chapters_continue` with validated ranges; bodies rendered from
475
- the log; artifact deferral and content-addressed dedup; registry with ancestry, numbering, hashes; flat
476
- ancestor-path TOC with handoff note and back-links; the continuation **preflight** with
477
- concrete refusal numbers; atomic ordering and idempotent retry. Manual trigger, agent supplies ranges.
478
-
479
- **Phase 2 — Segmentation as a function, plus pressure detection.** `chapters_segment` returning validated
480
- ranges, implemented as a callable usable **with or without an agent turn**; `turn/end` pressure check
481
- that **surfaces a suggestion** (no autonomous action). Both the UI button and Phase 3 depend on this
482
- seam, which is why it is here rather than near the end.
483
-
484
- **Phase 3 — Automatic continuation.** Fires on the Phase 2 detector at `turn/end` only, through the same
485
- Phase 1 preflight. Never mid-step, never `agent/pre-step`.
486
-
487
- **Phase 4 — Bound the index.** Chapter splitting at continuation boundaries; bullet folding past
488
- `tocBulletBudgetTokens`; reload budget in the TOC preamble; artifact refcounting and a retention/prune
489
- command.
490
-
491
- **Phase 5 — UI trigger.** Title-bar button sharing Phase 2's segmentation. Deliberately last: it needs a
492
- client bundle, `tsdown.client.config.ts`, and the `slots`/`locale`/`connection` client injects — the
493
- heaviest packaging in the project, for ergonomics rather than correctness.
494
-
495
- Permanently out of scope: in-place span compaction, prefix rewriting, seeding from parent events,
496
- system-prompt contributions, and truncating anything to fit.
1
+ docs/architecture.md