@try-works/dsh-memory 0.1.19 → 0.1.21

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (55) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +585 -585
  3. package/STATUS.md +294 -287
  4. package/cordis.patch.yml +69 -69
  5. package/package.json +1 -1
  6. package/scripts/check-package.mjs +76 -76
  7. package/scripts/inspect-log.ts +40 -0
  8. package/scripts/repair-ignorable.ts +130 -0
  9. package/scripts/scan-sessions.ts +43 -0
  10. package/scripts/verify-log.ts +26 -0
  11. package/src/SYSTEM.template.md +311 -311
  12. package/src/capture.js +242 -242
  13. package/src/client/index.js +886 -886
  14. package/src/config.js +57 -57
  15. package/src/consolidate-run.js +132 -132
  16. package/src/consolidate.js +330 -330
  17. package/src/context.js +88 -88
  18. package/src/decisions.js +106 -106
  19. package/src/defaults.js +93 -93
  20. package/src/digest.js +255 -255
  21. package/src/entry.js +179 -179
  22. package/src/eval.js +72 -72
  23. package/src/files.js +60 -60
  24. package/src/fixture.js +56 -56
  25. package/src/forget.js +71 -71
  26. package/src/freshness.js +65 -65
  27. package/src/fts.js +166 -166
  28. package/src/git.js +53 -53
  29. package/src/history.js +226 -226
  30. package/src/host.js +884 -866
  31. package/src/index.js +115 -94
  32. package/src/inject.js +108 -108
  33. package/src/layers.js +178 -178
  34. package/src/ledger.js +19 -19
  35. package/src/map.js +184 -184
  36. package/src/models.js +136 -136
  37. package/src/queue.js +602 -602
  38. package/src/rank.js +31 -31
  39. package/src/recover.js +91 -91
  40. package/src/refresh.js +80 -80
  41. package/src/remember.js +58 -58
  42. package/src/rerank.js +90 -90
  43. package/src/search.js +75 -75
  44. package/src/select.js +57 -57
  45. package/src/session.js +60 -60
  46. package/src/state.js +240 -240
  47. package/src/store.js +128 -128
  48. package/src/summarize.js +195 -195
  49. package/src/system-doc.js +171 -171
  50. package/src/tasks.js +617 -617
  51. package/src/telemetry.js +122 -122
  52. package/src/tools.js +828 -818
  53. package/src/working.js +288 -288
  54. package/src/workspace-key.js +68 -68
  55. package/src/workspace.js +87 -87
package/README.md CHANGED
@@ -1,585 +1,585 @@
1
- # dsh-memory
2
-
3
- Git-backed, workspace-scoped memory for the DeepSeek Harness.
4
-
5
- > **Where things stand:** [`STATUS.md`](STATUS.md) is the tracker and planner — what is verified
6
- > working, what is written but not yet loaded, what is unverified, and what is left. Updated
7
- > against measurements rather than intentions, so if you want to know whether a capability is
8
- > real, read that rather than this.
9
-
10
- ## Features
11
-
12
- - **Global layer** (`global/`) + **per-workspace layers** (`workspaces/<key>/`).
13
- - Workspaces auto-associate with local folders via DSH `ctx.workspaceRegistry`.
14
- - The whole memory folder is a git repo: durable markdown is committed, derived state is gitignored.
15
- GitHub sync (pull + push) runs hourly (+ boot + explicit).
16
- - **Web-UI settings** (storage root, GitHub origin, token, cadences) via `ctx.settings`.
17
- - **Deep DSH hooks**: `session/event` capture, `agent/pre-step` injection + cadence, in-process
18
- `ctx.llm` consolidation, `ctx.subprocess` git, `ctx.sessionQuery` capture, `ctx.goals`.
19
-
20
- ## Self-describing: `SYSTEM.md`
21
-
22
- The store explains itself. On every boot the plugin renders `SYSTEM.md` at the store root from
23
- `src/SYSTEM.template.md`, filling in the live configuration and a table of what the store
24
- currently holds. It is committed, so the same description travels with the memory to every
25
- machine — and regenerated every boot, so it can never describe a version of the plugin that no
26
- longer exists.
27
-
28
- It documents the *intent*, not just the mechanics: why the files are the truth and everything
29
- else is derived, why a model decides what to store but never how it is written, why each layer
30
- exists and what belongs in it, what the reply protocol is and what the earlier JSON contract
31
- got wrong, and how the whole thing is meant to be used. An agent that reads it can place a fact
32
- in the right layer and know what it is safe to trust.
33
-
34
- The document version is stamped into the file and exposed as `systemDocStale(root)`, so a stale
35
- copy is detectable rather than silently wrong.
36
-
37
- ## Store layout
38
-
39
- <memoryRoot>/
40
- ├── .gitignore # ignores index/ state/ digests/ logs/
41
- ├── memory.json # schemaVersion + git.remote/branch + workspace registry
42
- ├── global/ # GLOBAL layer (identity, prefs, standing rules)
43
- │ ├── INDEX.md # hand-written topic map over the whole scope
44
- │ ├── semantic.md # INDEX of semantic topics
45
- │ ├── semantic/ # one file per topic
46
- │ │ └── <topic>.md
47
- │ ├── episodic.md + episodic/<topic>.md
48
- │ ├── procedural.md + procedural/<topic>.md
49
- │ ├── learnings.md + learnings/<topic>.md
50
- │ ├── working.md # current-state head (single file, never indexed)
51
- │ └── junk.md # archaeology (single file, never indexed)
52
- └── workspaces/<key>/ # WORKSPACE layer (one per associated local folder/repo)
53
- ├── meta.json
54
- ├── codebase.md + codebase/<topic>.md # repository map (WORKSPACE ONLY)
55
- └── (same shape as global/)
56
-
57
- ### Layered layers
58
-
59
- `semantic`, `episodic`, `procedural`, `learnings` and `codebase` are **multi-layered**: the
60
- layer file is an index, and the entries live in one file per topic under a sibling directory.
61
- A single append-only file per layer grows without bound and becomes hard to read, diff and
62
- grep; one file per topic keeps every file small and searchable.
63
-
64
- ```markdown
65
- # global/semantic.md — the index
66
-
67
- Topic index. Entries live in one file per topic under semantic/.
68
-
69
- - [Store root](semantic/store-root.md) — 1 entry, updated 2026-09-30
70
- - [Plugin config rows](semantic/plugin-config-rows.md) — 2 entries, updated 2026-10-01
71
- ```
72
-
73
- Properties this buys:
74
-
75
- - The index is **regenerated** from the topic files after every write, so it can never drift.
76
- - A write reindexes **only the files it touched** (`updateIndex`). A full rebuild re-reads and re-tokenizes the whole store and costs seconds once the store holds a few hundred entries; `buildIndex` remains the repair path.
77
- - A topic file is ordinary Format-2 markdown, so `parseEntries`, the ledger, search and the
78
- context card work on it unchanged — the layer is resolved from the *path*, not the basename.
79
- - Id allocation spans the whole layer (`NNNN` is unique per scope+layer, not per file).
80
- - Adding a topic file by hand is fine: the next rebuild picks it up.
81
- - Existing stores migrate on load: entries still in a single layer file are moved into topic
82
- files and the index is rebuilt. The migration is idempotent.
83
-
84
- ## Install
85
-
86
- The package is a self-contained DSH plugin: plain ESM, no build step, and a bundle patch that
87
- mounts it by name. Install it from a path or a registry:
88
-
89
- # from a local checkout
90
- dsh plugin --profile web add D:/DEV/dsh-memory
91
-
92
- # or from a registry, once published
93
- dsh plugin --profile web add dsh-memory
94
-
95
- Adding it composes `cordis.patch.yml` into the profile bundles, which mounts one entry:
96
-
97
- - insert:
98
- - id: dsh-memory
99
- name: dsh-memory
100
-
101
- The entry carries **no config** on purpose. The store root, memory model, git origin and branch
102
- belong to the user, so they are written by **Settings -> Memory** as a separate top-level
103
- `config:` row for the same id in your profile patch. Mounting by hand works the same way:
104
-
105
- - insert:
106
- - id: dsh-memory
107
- name: D:/DEV/dsh-memory/src/index.js
108
- - id: dsh-memory # written by the settings page
109
- config:
110
- memoryRoot: E:\dsh-memory
111
-
112
- The entry id is the settings namespace, so either `memory` or `dsh-memory` works as a row id;
113
- the client half resolves whichever the host actually serves.
114
-
115
- ### Package invariants
116
-
117
- `npm run check` (or `node scripts/check-package.mjs`) verifies what an install-by-name needs:
118
- the entry exports `name`/`inject`/`apply`/`Config` and **no** default export, the bundle patch
119
- mounts the package by name, the client half and `SYSTEM.template.md` ship, and every imported
120
- peer is declared. `test/package.test.js` asserts the same invariants so packaging cannot drift.
121
-
122
- ## Configure (web UI → Settings → **Memory**)
123
-
124
- The plugin ships a browser client half (`src/client/index.js`, declared via `dsh.client`
125
- in package.json), so Settings shows a **Memory** page with editable fields. Saving writes to
126
- this profile's layer and applies live.
127
-
128
- The client half is plain JavaScript that requests only `react` from the browser module table.
129
- It deliberately imports no harness client package: those change without notice, plain JS gets no
130
- type check against them, and a throwing component blanks the slot entry. Its styles are local,
131
- class-prefixed, and built only from theme tokens that exist on the shipped settings pages
132
- (`--dsw-alias-*`, `--dsw-radius-*`, `--ds-font-family-code`). A token that does not exist is an
133
- invalid declaration, the browser drops it, and the page degrades into unstyled prose — which is
134
- exactly what happened before the current styling was added.
135
-
136
- ### Settings -> **Memory**
137
-
138
- | Field | Purpose |
139
- | --- | --- |
140
- | memoryRoot | local folder/drive for the store (default `~/.dsh-memory`) |
141
- | summaryModel | model for memory writes (`provider/model`); empty = the session's own model |
142
- | gitRemote | GitHub origin URL (empty = local-only) |
143
- | gitBranch | default `main` |
144
-
145
- ### Config file only (advanced tuning)
146
-
147
- Set these in the profile patch's `config:` block. They have sane defaults and are
148
- deliberately kept out of the settings page.
149
-
150
- | Field | Purpose |
151
- | --- | --- |
152
- | gitToken | optional PAT (secret; empty = OS git credential helper) |
153
- | gitName / gitEmail | optional commit identity |
154
- | autoCaptureIntervalMs | per-session capture throttle (default 300000) |
155
- | commitDebounceMs | local commit debounce (default 10000) |
156
- | autoSyncIntervalMs | GitHub push+pull cadence (default 3600000 = 1h) |
157
- | autoConsolidateIntervalMs / autoConsolidateMinPending / autoConsolidateBatchSize | consolidation cadence |
158
- | autoRefreshIntervalMs | freshness scan cadence (default 86400000 = 1d) |
159
- | injectionMaxChars / injectionRefreshMs | injection budget |
160
- | digestMaxUserTurns / digestMaxAssistantGists / digestMaxMessageChars | what the summarizer may see per pass |
161
- | digestMaxBytes | hard ceiling on summarizer input per pass |
162
- | summaryMaxBullets | durable bullets kept from one summary |
163
- | consolidateMaxPromptBytes / consolidateMaxDigestBytes | consolidation prompt + per-digest ceilings |
164
-
165
- ## Agent tools
166
-
167
- - `memory_search(query, scope?, workspace?, limit?)` — ranked facts with file:line citations.
168
- - `memory_remember(text, layer, scope?, workspace?, topic?, source?, importance?, until?)` — deterministic write.
169
- - `memory_workspaces()` / `memory_associate(key, title?, path?)` — workspace association.
170
- - `memory_sync()` — commit local changes and sync (pull + push) with the GitHub origin.
171
- - `memory_forget(id, undo?)` — forget a memory entry (supersede + suppress re-adding), or release it.
172
- - `memory_consolidate(limit?)` — run the LLM consolidation pass now (digests → entries) instead of waiting for the cadence.
173
- - `memory_codebase(topic, text, importance?)` — record what a module or subsystem of the current repository does (workspace codebase map).
174
- - `memory_decision(decision, rationale, context?, alternatives?, consequences?, scope?, workspace?, importance?)` — record a settled choice with why it won. Available in both scopes.
175
- - `memory_state(section, content)` — record or refresh one section of the **repository** state overview.
176
- - `memory_working(section?, content?, activity?, clearActivity?)` — record what **you** are working on: a current section, or activity notes.
177
- - `memory_status()` — where the store lives, how many digests await consolidation, whether the last sync conflicted.
178
- - `memory_refresh(limit?)` — run the freshness pass now instead of waiting for the daily cadence.
179
- - `memory_status()` — now also reports model calls, event counts and saved calls (thin/trivial).
180
- - `memory_history(limit?)` — ingest repository commit history into the workspace history layer.
181
-
182
- `ctx.memory` service: `search`, `remember`, `workspaces`, `associate`, `resolveKey`, `status`, `workspaceOf`, `pending`, `consolidate`.
183
-
184
- ## Capture pipeline
185
-
186
- Capture turns sessions into memory without ever storing raw transcripts:
187
-
188
- 1. **Relevance filter** (`digest.js`) — deterministic and bounded. Tool calls reduce to a
189
- *count* (tool mechanics are not memory); user turns and assistant *gists* are clipped;
190
- bare acknowledgements are dropped; the window is trimmed oldest-first to `digestMaxBytes`.
191
- 2. **Summarization** (`summarize.js`) — the session's own model turns that material into
192
- 5-12 durable bullets under explicit instructions: bullets, `**label:**`, date-led when
193
- time-bound, keep decisions/constraints/preferences/environment facts, drop tool chatter,
194
- code and pleasantries. A model that finds nothing durable replies `NO_DURABLE_CONTENT`
195
- and nothing is stored.
196
- 3. **Incremental storage** (`capture.js`) — a `lastSeq` watermark in `state/sessions.json`
197
- means each pass summarizes only events that arrived since the previous one, so capture is
198
- O(new events) rather than O(session). Bullets are appended as a dated section.
199
-
200
- A digest file therefore holds a bounded metadata header plus dated bullet sections — never
201
- message bodies. Without a model, capture degrades to deterministic clipped notes instead of
202
- failing; a `null` `consolidatedAt` marks the digest as work for the consolidation pass.
203
-
204
- ## Working memory and sync safety
205
-
206
- - `working.md` is **per repo/workspace**, never global: it is the current-state head of one
207
- repository, so mixing repositories would be meaningless. The global layer has no working
208
- file, and a template-only `global/working.md` left by an older version is retired on boot.
209
- - A failed `git pull`/`push` is never swallowed. The failure is logged, recorded in
210
- `state/sync.json`, surfaced by `status()`, reported by the `memory_sync` tool, and leaves a
211
- `state/CONFLICT.md` marker that the next clean sync clears.
212
- - Consolidation attributes every entry to the digest it came from. A digest the model never
213
- attributes stays pending and is retried; after `maxAttempts` it is marked `needsReview`
214
- instead of being silently dropped.
215
- - A model reply is parsed with escalating recovery: exact JSON, then repaired JSON (trailing
216
- commas, placeholder values), then whatever complete objects the text contains — so a
217
- truncated or sloppy reply degrades instead of failing the pass. If nothing is recoverable,
218
- the raw prompt and reply are written to `logs/consolidate-failed-<stamp>.txt` and the tool
219
- reports `parseFailed` rather than discarding the evidence.
220
- - Replies are assembled from **text blocks only**. A reasoning model streams its thinking as
221
- `reasoning` blocks; folding those into the reply prepends prose to the answer, and any
222
- "parse the JSON" step then matches the wrong braces. This was the cause of a real failure:
223
- the reasoning preamble plus the echoed prompt example were handed to the parser.
224
- - The reply protocol is **line-oriented, not JSON**: one
225
- `- [<digest> | global|<workspace:key> | <layer> | <importance>] <topic> :: <text>` line per
226
- fact, plus a `DONE <digest>` line per digest processed. Nested JSON proved fragile in
227
- practice — one invalid placeholder, one truncated brace or one reasoning preamble lost the
228
- entire pass — while a line protocol parses per line, so one bad line cannot kill the batch
229
- and prose is simply ignored. A JSON object is still accepted if a model volunteers one, with
230
- escalating recovery (exact, repaired, then salvaged objects).
231
- - Nothing recoverable is discarded: the raw prompt and reply are written to
232
- `logs/consolidate-failed-<stamp>.txt` and the tool reports `parseFailed`.
233
- - Each digest heading carries the workspace key resolved from that session's own working
234
- directory, and the model is told to use it. It was previously left to derive a key, which it
235
- did from the filesystem path — and a path key created a workspace directory that the write
236
- then could not use. `remember()` now sanitizes the key once, up front, so a key and the
237
- directory it targets can never disagree.
238
- - The last consolidation exchange is always kept at `logs/consolidate-last.txt`, and the tool
239
- reports `writeFailed` with the reasons, so a pass that writes nothing is visible.
240
- - One malformed entry cannot fail a whole pass: it is recorded, and a workspace request with
241
- no usable key falls back to the global layer instead of throwing.
242
-
243
- ## Codebase map
244
-
245
- `codebase` is the repository map: one entry per module, file or subsystem, written **as the
246
- agent discovers it**. It is deliberately **workspace-only** — a global codebase map would mix
247
- unrelated repositories together, exactly like working memory.
248
-
249
- Two things feed it:
250
-
251
- - `memory_codebase(topic, text, importance?)` — the agent records what a module does. The
252
- topic is the path or subsystem name, so re-discovery updates the same topic file. It
253
- resolves the workspace from the session, so no key is needed.
254
- - **Traversal evidence.** Capture records the distinct file paths a session's tool *calls*
255
- point at (arguments only — results are full of paths mentioned in file contents, which say
256
- nothing about intent), bounded and deduped. Those paths ride the digest, so summarization
257
- and consolidation can turn exploration into map entries without the agent writing anything.
258
-
259
- The injected context lists the topics already mapped for this workspace, so the agent can see
260
- what it can rely on and what is still unexplored.
261
-
262
- ## Commit history
263
-
264
- `history` records **how the repository changed**, complementing the codebase map (which says
265
- what each module *is*). Also workspace-only and multi-layered, but its topics are **calendar
266
- days** — one file per day, so a long-lived repo yields many small readable files:
267
-
268
- ```markdown
269
- # workspaces/role-model/history.md
270
- - [2026-09-27](history/2026-09-27.md) — 11 entries
271
- - [2026-09-28](history/2026-09-28.md) — 16 entries
272
- - [2026-10-01](history/2026-10-01.md) — 2 entries
273
- ```
274
-
275
- Each entry is an ordinary Format-2 entry (subject + short hash as the heading, the full commit
276
- body, the changed files, and `**Source:** git:<hash>`), so the ledger, ranking and search work
277
- on it unchanged.
278
-
279
- Ingest is incremental in two layers:
280
-
281
- 1. The log range starts at the last recorded commit (`<head>..HEAD`), so a normal pass reads
282
- only new commits. When the range is unreachable (a rebase or force-push rewrote history) the
283
- pass falls back to a bounded walk and reports `fellBack`.
284
- 2. A hash watermark in `state/history.json` skips what is already recorded — an optimisation,
285
- not the authority.
286
- 3. **The layer itself is the authority.** Every entry carries a `Commit: <full hash>` line and
287
- ingest reads them back, so a lost watermark, a fresh clone or a rewound head cannot produce a
288
- duplicate. Ingest is idempotent even if `state/` is deleted entirely.
289
-
290
- Runners report status inconsistently — `git.js` documents `code`, while the host runner and
291
- `ctx.subprocess` return `exitCode` — so the check reads either. Reading only one made every
292
- successful ranged query look like a failure and silently fell back to a full re-read.
293
-
294
- Recording happens on a cadence (`autoHistoryIntervalMs`, default hourly, `autoHistoryLimit`
295
- commits per pass) and on demand via `memory_history`. A repository that is missing, not a repo,
296
- or has no new commits is skipped silently — it never interrupts a turn.
297
-
298
- ## Decisions
299
-
300
- A `decision` is neither a fact nor a lesson: it is a **settled choice and why it won** — a
301
- dependency, a convention, an architecture call. It exists so a later session does not
302
- re-litigate a closed question or silently reverse it.
303
-
304
- Unlike `codebase` and `history`, it is available in **both scopes**: a convention can be a
305
- global preference, while a dependency choice belongs to the repository that made it.
306
-
307
- The shape is the point. The decision statement is the entry heading, so the layer index reads
308
- as a list of decisions, and the reasoning lives in fixed sections:
309
-
310
- ```markdown
311
- ## Use plain ESM with no build step for dsh-memory
312
-
313
- **Context:**
314
- This plugin is plain JavaScript loaded directly by path...
315
-
316
- **Rationale:**
317
- Plain ESM keeps the shipped source equal to the code under test...
318
-
319
- **Alternatives considered:**
320
- - Compile TypeScript or bundle the host half
321
- - Wrap the entry in a default export
322
-
323
- **Consequences:**
324
- The plugin entry must keep named exports and must not add a default export...
325
- ```
326
-
327
- Fixed sections make decisions comparable and greppable, and `parseDecisionBody` reads them back
328
- so a stored decision can be inspected without re-deriving it from prose. The write path renders
329
- this shape — a model deciding what is durable never has to produce it.
330
-
331
- Decisions in force are surfaced in the injected context next to learnings, because they are the
332
- constraints a new session has to respect.
333
-
334
- ## Freshness metadata
335
-
336
- Every entry carries its own decay metadata, so staleness is determinable **from the file alone** —
337
- no external clock, no global assumption, and it survives a clone because it travels with the text.
338
-
339
- | Field | Meaning |
340
- | --- | --- |
341
- | `**Volatility:**` | How fast this decays: `stable` (years — a language, a license, a convention), `normal` (months — how a subsystem works), `volatile` (days — a port, a version in flight). |
342
- | `**Verified:**` | When the fact was last *confirmed true*. This is the age basis. |
343
- | `**Valid-until:**` | A hard expiry for facts that genuinely end. |
344
-
345
- Freshness is therefore a property of the entry, not a constant applied to a write date. A plugin
346
- using plain ESM stays true for years; the port it listens on can change tomorrow — and only the
347
- writer knows which is which:
348
-
349
- ```text
350
- same age (40 days), different verdicts
351
- dsh-memory-sem-0029 Volatility: volatile -> stale (window 3d)
352
- dsh-memory-sem-0028 Volatility: stable -> fresh (window 365d)
353
- ```
354
-
355
- `**Verified:**` matters because writing and confirming are different acts: an entry dated 400 days
356
- ago but re-confirmed yesterday is fresh, and status quo previously could not express that. Where
357
- volatility is absent (entries written before the field existed) time-bounded language still hints
358
- at it, then the normal window applies. `memory_remember` accepts `volatility`, and the
359
- consolidation protocol carries it as an optional fifth field.
360
-
361
- ## Workspace state
362
-
363
- `state.md` is a short, section-based overview of **what the repository is right now** — layout,
364
- what is running, what is pending, health. It is the one layer whose entire value is being current,
365
- so it is treated differently from every other:
366
-
367
- - **Replaced per section, never appended.** State describes now, so a newer statement about a
368
- section supersedes the old one instead of piling up beside it. Other sections keep their own
369
- timestamps — editing one area must not make an unrelated section look freshly verified.
370
- - **Every section carries `**Verified:**`**, and freshness is graded rather than binary:
371
- `fresh` / `aging` / `stale` / `unverified`.
372
- - **Staleness is always shown, never hidden.** A stale section appears in the injected context
373
- with its age (`[STALE 5d]`), so it reads as a hypothesis to check rather than a fact to trust.
374
- An unverified section is marked `[UNVERIFIED]`.
375
- - **Workspace-only** and a single file: a repository state file in the global layer would
376
- describe no repository, and an overview that needs an index is not an overview.
377
-
378
- ## What capture keeps (and what it refuses)
379
-
380
- A session is projected to candidate material before any model sees it. Two rules were added
381
- after observing real research sessions silently produce nothing:
382
-
383
- - **Injected context is not user intent.** The harness prepends runtime snapshots, skill
384
- catalogs, policy notices and the memory plugin's own context block as *user* turns. They are
385
- dropped, because in one measured session they were **82% of the material**. Capturing the
386
- memory block specifically is a feedback loop: the system treating its own output as new input.
387
- - **A research answer is its findings, not its status line.** Analysis sessions open with a
388
- one-line announcement ("Analysis complete.") and put everything in the body. Capture now skips
389
- a bare status opener and carries the substance, and takes the assistant `text` block rather
390
- than its private `reasoning`.
391
-
392
- Both were invisible in unit tests and obvious the moment one real session was traced.
393
-
394
- ## Memory belongs to the main agent
395
-
396
- A **subagent does not write memory**. It reads freely - it can search and consult the store -
397
- but recording is the main agent's job.
398
-
399
- The reason is ownership, not permission. A subagent is a delegate working inside a task the
400
- parent already owns, and the parent's transcript carries the delegation and its result. Letting
401
- the child write as well produces two partial accounts of one task, and the child's - written
402
- without the parent's context - competes with the parent's in search.
403
-
404
- Enforced in three places:
405
-
406
- - **Capture** skips a subagent session entirely, so it never becomes a digest.
407
- - **Every writing tool** refuses with an explanation naming the tool; reads are unrestricted.
408
- - **The injected context** tells a subagent to report its findings to the caller instead of
409
- instructing it to write.
410
-
411
- A session counts as a subagent when `delegationDepth > 0` **or** `origin === "subagent"` - both
412
- are checked because they are set independently, and an absent depth means top-level.
413
-
414
- ## Which model does memory work
415
-
416
- Memory work needs a model that returns **text**. Reasoning-heavy models sometimes stream only
417
- `reasoning` blocks and no answer at all - measured here as 7 of 11 consolidation passes writing
418
- nothing, on the same prompts that another model turned into 35 entries.
419
-
420
- So the model is not configured as a constant. Each call builds a **ladder from the live
421
- registry**:
422
-
423
- 1. the configured `summaryModel`, when set (a preference, not a requirement)
424
- 2. the session's own route
425
- 3. every other model this deployment can route to right now
426
-
427
- The first model that returns text wins. A model that returns nothing, or throws, is skipped and
428
- the **same prompt** goes to the next rung. The winner is remembered, so the normal case still
429
- costs one call; a model that returned nothing is benched for 15 minutes and sinks to the end of
430
- the ladder - never dropped, so a deployment where everything is benched still tries.
431
-
432
- The ladder is capped at 4 models, and if every one returns nothing the call raises an error
433
- rather than reporting success, so the session is retried instead of silently settled.
434
-
435
- ## Failures retry; conclusions do not
436
-
437
- A summary that *fails* (empty reply, unparseable prose, or a thrown call) is not the same as a
438
- model concluding there is nothing durable. Failure keeps its material and is retried up to
439
- `summaryMaxAttempts` (default 2) before being laid to rest as failed. A deliberate
440
- `NO_DURABLE_CONTENT` settles immediately. Treating a failure as a conclusion is how whole
441
- sessions were marked handled and lost for good.
442
-
443
- ## Memory may correctly write nothing
444
-
445
- A worker is **not** required to add a memory every capture interval or every turn. In fact,
446
- writing something merely to fill a gap is a failure: it pollutes search, creates false context,
447
- and makes later decisions harder to trust.
448
-
449
- This is enforced at multiple levels:
450
-
451
- - The injected footer says explicitly that writing NOTHING is a correct and expected outcome.
452
- - The capture relevance filter drops tool mechanics, acknowledgements and low-signal material.
453
- - `summaryMinSignalChars` (default 400) skips the model on a thin new delta entirely.
454
- - The summarizer may return `NO_DURABLE_CONTENT` without creating pending work.
455
- - Consolidation may return only `DONE <digest>` lines: a handled digest with zero facts is
456
- successful, writes zero entries, and is not retried. It never invents a fact for output.
457
- - `memory_remember` itself says not to call it to fill a gap.
458
-
459
- The desired ratio is not "one turn in, one memory out". It is **high signal, low noise**, with
460
- a cheap no-op path for the common case where nothing durable changed.
461
-
462
- ## Working memory belongs to the agent
463
-
464
- Two files describe "now", and they are **not** the same thing:
465
-
466
- | File | Owner | Answers |
467
- | --- | --- | --- |
468
- | `state.md` | the repository | *What is this repo right now?* — layout, what is running, pending, health |
469
- | `working.md` | **the agent** | *What am I in the middle of?* — task, recent activity, next steps |
470
-
471
- A repository has one current state, but several agents may work in it with entirely different
472
- intents — and one intent may span several repositories. Merging them would lose both meanings.
473
-
474
- `working.md` states its own ownership in the file, on every write, so an agent reading it
475
- directly (not just through the injected card) knows it is theirs:
476
-
477
- ```text
478
- THIS FILE BELONGS TO THE AGENT. It is not repository state (that is state.md) and not
479
- durable memory (those are the layered entries). It is your own head for this workspace.
480
-
481
- You own it. Read it, edit it, and keep it honest. Write it with memory_working, or edit
482
- this file directly. Nothing else writes here, so a stale or empty file means the intent
483
- was never recorded.
484
- ```
485
-
486
- Two section behaviours, because the information has two shapes:
487
-
488
- - **Task / Next steps / Plan / Blocked by** — *replaced* on write: there is one current task.
489
- - **Recent activity** — a *rolling log*, newest last, trimmed oldest-first by entries and
490
- characters, with identical consecutive notes not repeated.
491
-
492
- It decays in **hours**, not days, so it is graded on its own scale (fresh / aging / stale / not
493
- started), and an unrecorded file is **called out** in the injected card rather than omitted —
494
- silence would read as "no working memory" when it actually means "nothing written yet".
495
-
496
- ## Operational observability
497
-
498
- The derived `logs/` and `state/` surfaces are intentionally observable even though they are
499
- gitignored:
500
-
501
- - `logs/memory.log` is newline-delimited JSON for capture, consolidation, refresh, history,
502
- pruning, sync errors and model calls. It rotates at 2 MB.
503
- - `logs/consolidate-last.txt` preserves the latest prompt/reply exchange for diagnosis, and
504
- `consolidate-failed-*.txt` preserves parse failures rather than discarding the evidence.
505
- - `state/metrics.json` counts cadence events, model calls by purpose, and calls avoided by
506
- cost control (`thin`, `trivial`). `memory_status` exposes the useful subset.
507
- - `state/activity.jsonl` is a bounded human-readable operational feed.
508
-
509
- ## Cost control
510
-
511
- Every stage can decide **not** to spend a model call:
512
-
513
- - A `trivial` session (a ping) never reaches a model at all.
514
- - A content session whose **new** delta is below `summaryMinSignalChars` (default 400) is skipped as
515
- `thin`. The check counts the *filtered* candidates, not raw events, so a burst of tool noise
516
- cannot masquerade as content — a model call on a two-line delta costs more than it captures.
517
- - Consolidation batches and caps its prompt (`consolidateMaxPromptBytes`, `consolidateMaxDigestBytes`).
518
-
519
- Captured digests are **pruned** on the idle cadence down to the newest `digestKeepCount` (default
520
- 50) *consolidated* ones. A pending digest is never removed: it is the only copy of work not yet in
521
- the store. Ordering is by when the session was captured, not file mtime, so digests written in the
522
- same instant cannot be pruned in the wrong order.
523
-
524
- ## Hooks
525
-
526
- | Direction | Hook | Effect |
527
- | --- | --- | --- |
528
- | read | `agent/pre-step` | injects the `## DSH memory context` card at step 1 (re-injected per `injectionRefreshMs`) |
529
- | write | `session/event` turn/end | incremental capture: new events → relevance filter → session-model summary → `digests/` (throttled by `autoCaptureIntervalMs`) |
530
- | write | `agent/pre-step` cadence | LLM consolidation: pending digests → entries |
531
- | write | `agent/pre-step` cadence | freshness scan + LLM refresh (keep/update/supersede) |
532
- | life | `session/event` turn/start | workspace auto-association via `ctx.workspaceRegistry` |
533
- | sync | `agent/pre-step` cadence | debounced git commit + hourly pull/push |
534
- | all | idle ticker (60 s) | runs the cadences even when nobody is chatting; needs no agent when `summaryModel` is set |
535
-
536
- On-demand equivalents: `memory_consolidate` (or `ctx.memory.consolidate(agent)`), `memory_sync` (or `ctx.memory` sync).
537
-
538
- ## How it works
539
-
540
- observe (`session/event`) → capture (`sessionQuery` → deterministic digest) → consolidate (`ctx.llm` → entries)
541
- → index (FTS5) → query (in-process search) → maintain (freshness/forget) → sync (git).
542
-
543
- ## Development
544
-
545
- node --test # 72 tests (unit + behavior fixtures + eval)
546
-
547
- - `src/fixture.js` — versioned behavior-fixture suite (ranking/freshness/forgetting/budget/workspace-key).
548
- - `src/eval.js` — sufficiency eval: fact-recall probes, a judge layer, and deterministic anchor checks.
549
-
550
- Local validation against an installed DSH (needs the profile node_modules on the resolve path):
551
-
552
- node --input-type=module -e "import('file:///D:/DEV/dsh-memory/src/index.js').then(m => console.log(m.name, m.inject))"
553
-
554
- ## Search
555
-
556
- Normal queries use the unicode61 word index. CJK and other no-space scripts route to a
557
- trigram substring index; two-character queries (which are shorter than a trigram) use an
558
- escaped LIKE fallback. Partial English words also retry through substring matching. The
559
- parallel index is regenerated or updated alongside the word index, and scope filtering still
560
- applies to every result.
561
-
562
- ### Optional rerank
563
-
564
- `memory_search` can ask the session model to reorder hits by usefulness. It is **off by
565
- default** (`rerankSearch`), costs one call, and is strictly **advisory**: rerankHits returns
566
- the original deterministic order whenever the model is missing, throws, or replies with
567
- something unusable — and it refuses a partial answer that merely agrees with the existing
568
- order. Search never depends on a model.
569
-
570
- ## Settings safety
571
-
572
- The Memory page exposes only store root, summary model, GitHub origin and branch. The optional
573
- git token is a password field that is **never pre-filled** from the redacted host snapshot:
574
- typing a new token replaces it; leaving the field empty leaves the current token untouched.
575
-
576
- ## Status
577
-
578
- Implemented greenfield, no legacy code. **Seven layers** (`semantic`, `episodic`, `procedural`,
579
- `learning`, plus workspace-only `codebase`, `history` and `state`), all multi-layered except
580
- `state.md` and `working.md`. **Eleven tools**, a `ctx.memory` service for other plugins, deep DSH
581
- hooks (capture, consolidation, refresh, history ingest, git sync, idle ticker) and a self-describing
582
- `SYSTEM.md`. **170+ unit tests**, a behaviour-fixture suite over five axes, and a wiring test that
583
- mounts the real host and executes every tool. See `dsh-memory-v2-plan.md` for the plan and
584
- `SYSTEM.md` in the store for the as-built description.
585
-
1
+ # dsh-memory
2
+
3
+ Git-backed, workspace-scoped memory for the DeepSeek Harness.
4
+
5
+ > **Where things stand:** [`STATUS.md`](STATUS.md) is the tracker and planner — what is verified
6
+ > working, what is written but not yet loaded, what is unverified, and what is left. Updated
7
+ > against measurements rather than intentions, so if you want to know whether a capability is
8
+ > real, read that rather than this.
9
+
10
+ ## Features
11
+
12
+ - **Global layer** (`global/`) + **per-workspace layers** (`workspaces/<key>/`).
13
+ - Workspaces auto-associate with local folders via DSH `ctx.workspaceRegistry`.
14
+ - The whole memory folder is a git repo: durable markdown is committed, derived state is gitignored.
15
+ GitHub sync (pull + push) runs hourly (+ boot + explicit).
16
+ - **Web-UI settings** (storage root, GitHub origin, token, cadences) via `ctx.settings`.
17
+ - **Deep DSH hooks**: `session/event` capture, `agent/pre-step` injection + cadence, in-process
18
+ `ctx.llm` consolidation, `ctx.subprocess` git, `ctx.sessionQuery` capture, `ctx.goals`.
19
+
20
+ ## Self-describing: `SYSTEM.md`
21
+
22
+ The store explains itself. On every boot the plugin renders `SYSTEM.md` at the store root from
23
+ `src/SYSTEM.template.md`, filling in the live configuration and a table of what the store
24
+ currently holds. It is committed, so the same description travels with the memory to every
25
+ machine — and regenerated every boot, so it can never describe a version of the plugin that no
26
+ longer exists.
27
+
28
+ It documents the *intent*, not just the mechanics: why the files are the truth and everything
29
+ else is derived, why a model decides what to store but never how it is written, why each layer
30
+ exists and what belongs in it, what the reply protocol is and what the earlier JSON contract
31
+ got wrong, and how the whole thing is meant to be used. An agent that reads it can place a fact
32
+ in the right layer and know what it is safe to trust.
33
+
34
+ The document version is stamped into the file and exposed as `systemDocStale(root)`, so a stale
35
+ copy is detectable rather than silently wrong.
36
+
37
+ ## Store layout
38
+
39
+ <memoryRoot>/
40
+ ├── .gitignore # ignores index/ state/ digests/ logs/
41
+ ├── memory.json # schemaVersion + git.remote/branch + workspace registry
42
+ ├── global/ # GLOBAL layer (identity, prefs, standing rules)
43
+ │ ├── INDEX.md # hand-written topic map over the whole scope
44
+ │ ├── semantic.md # INDEX of semantic topics
45
+ │ ├── semantic/ # one file per topic
46
+ │ │ └── <topic>.md
47
+ │ ├── episodic.md + episodic/<topic>.md
48
+ │ ├── procedural.md + procedural/<topic>.md
49
+ │ ├── learnings.md + learnings/<topic>.md
50
+ │ ├── working.md # current-state head (single file, never indexed)
51
+ │ └── junk.md # archaeology (single file, never indexed)
52
+ └── workspaces/<key>/ # WORKSPACE layer (one per associated local folder/repo)
53
+ ├── meta.json
54
+ ├── codebase.md + codebase/<topic>.md # repository map (WORKSPACE ONLY)
55
+ └── (same shape as global/)
56
+
57
+ ### Layered layers
58
+
59
+ `semantic`, `episodic`, `procedural`, `learnings` and `codebase` are **multi-layered**: the
60
+ layer file is an index, and the entries live in one file per topic under a sibling directory.
61
+ A single append-only file per layer grows without bound and becomes hard to read, diff and
62
+ grep; one file per topic keeps every file small and searchable.
63
+
64
+ ```markdown
65
+ # global/semantic.md — the index
66
+
67
+ Topic index. Entries live in one file per topic under semantic/.
68
+
69
+ - [Store root](semantic/store-root.md) — 1 entry, updated 2026-09-30
70
+ - [Plugin config rows](semantic/plugin-config-rows.md) — 2 entries, updated 2026-10-01
71
+ ```
72
+
73
+ Properties this buys:
74
+
75
+ - The index is **regenerated** from the topic files after every write, so it can never drift.
76
+ - A write reindexes **only the files it touched** (`updateIndex`). A full rebuild re-reads and re-tokenizes the whole store and costs seconds once the store holds a few hundred entries; `buildIndex` remains the repair path.
77
+ - A topic file is ordinary Format-2 markdown, so `parseEntries`, the ledger, search and the
78
+ context card work on it unchanged — the layer is resolved from the *path*, not the basename.
79
+ - Id allocation spans the whole layer (`NNNN` is unique per scope+layer, not per file).
80
+ - Adding a topic file by hand is fine: the next rebuild picks it up.
81
+ - Existing stores migrate on load: entries still in a single layer file are moved into topic
82
+ files and the index is rebuilt. The migration is idempotent.
83
+
84
+ ## Install
85
+
86
+ The package is a self-contained DSH plugin: plain ESM, no build step, and a bundle patch that
87
+ mounts it by name. Install it from a path or a registry:
88
+
89
+ # from a local checkout
90
+ dsh plugin --profile web add D:/DEV/dsh-memory
91
+
92
+ # or from a registry, once published
93
+ dsh plugin --profile web add dsh-memory
94
+
95
+ Adding it composes `cordis.patch.yml` into the profile bundles, which mounts one entry:
96
+
97
+ - insert:
98
+ - id: dsh-memory
99
+ name: dsh-memory
100
+
101
+ The entry carries **no config** on purpose. The store root, memory model, git origin and branch
102
+ belong to the user, so they are written by **Settings -> Memory** as a separate top-level
103
+ `config:` row for the same id in your profile patch. Mounting by hand works the same way:
104
+
105
+ - insert:
106
+ - id: dsh-memory
107
+ name: D:/DEV/dsh-memory/src/index.js
108
+ - id: dsh-memory # written by the settings page
109
+ config:
110
+ memoryRoot: E:\dsh-memory
111
+
112
+ The entry id is the settings namespace, so either `memory` or `dsh-memory` works as a row id;
113
+ the client half resolves whichever the host actually serves.
114
+
115
+ ### Package invariants
116
+
117
+ `npm run check` (or `node scripts/check-package.mjs`) verifies what an install-by-name needs:
118
+ the entry exports `name`/`inject`/`apply`/`Config` and **no** default export, the bundle patch
119
+ mounts the package by name, the client half and `SYSTEM.template.md` ship, and every imported
120
+ peer is declared. `test/package.test.js` asserts the same invariants so packaging cannot drift.
121
+
122
+ ## Configure (web UI → Settings → **Memory**)
123
+
124
+ The plugin ships a browser client half (`src/client/index.js`, declared via `dsh.client`
125
+ in package.json), so Settings shows a **Memory** page with editable fields. Saving writes to
126
+ this profile's layer and applies live.
127
+
128
+ The client half is plain JavaScript that requests only `react` from the browser module table.
129
+ It deliberately imports no harness client package: those change without notice, plain JS gets no
130
+ type check against them, and a throwing component blanks the slot entry. Its styles are local,
131
+ class-prefixed, and built only from theme tokens that exist on the shipped settings pages
132
+ (`--dsw-alias-*`, `--dsw-radius-*`, `--ds-font-family-code`). A token that does not exist is an
133
+ invalid declaration, the browser drops it, and the page degrades into unstyled prose — which is
134
+ exactly what happened before the current styling was added.
135
+
136
+ ### Settings -> **Memory**
137
+
138
+ | Field | Purpose |
139
+ | --- | --- |
140
+ | memoryRoot | local folder/drive for the store (default `~/.dsh-memory`) |
141
+ | summaryModel | model for memory writes (`provider/model`); empty = the session's own model |
142
+ | gitRemote | GitHub origin URL (empty = local-only) |
143
+ | gitBranch | default `main` |
144
+
145
+ ### Config file only (advanced tuning)
146
+
147
+ Set these in the profile patch's `config:` block. They have sane defaults and are
148
+ deliberately kept out of the settings page.
149
+
150
+ | Field | Purpose |
151
+ | --- | --- |
152
+ | gitToken | optional PAT (secret; empty = OS git credential helper) |
153
+ | gitName / gitEmail | optional commit identity |
154
+ | autoCaptureIntervalMs | per-session capture throttle (default 300000) |
155
+ | commitDebounceMs | local commit debounce (default 10000) |
156
+ | autoSyncIntervalMs | GitHub push+pull cadence (default 3600000 = 1h) |
157
+ | autoConsolidateIntervalMs / autoConsolidateMinPending / autoConsolidateBatchSize | consolidation cadence |
158
+ | autoRefreshIntervalMs | freshness scan cadence (default 86400000 = 1d) |
159
+ | injectionMaxChars / injectionRefreshMs | injection budget |
160
+ | digestMaxUserTurns / digestMaxAssistantGists / digestMaxMessageChars | what the summarizer may see per pass |
161
+ | digestMaxBytes | hard ceiling on summarizer input per pass |
162
+ | summaryMaxBullets | durable bullets kept from one summary |
163
+ | consolidateMaxPromptBytes / consolidateMaxDigestBytes | consolidation prompt + per-digest ceilings |
164
+
165
+ ## Agent tools
166
+
167
+ - `memory_search(query, scope?, workspace?, limit?)` — ranked facts with file:line citations.
168
+ - `memory_remember(text, layer, scope?, workspace?, topic?, source?, importance?, until?)` — deterministic write.
169
+ - `memory_workspaces()` / `memory_associate(key, title?, path?)` — workspace association.
170
+ - `memory_sync()` — commit local changes and sync (pull + push) with the GitHub origin.
171
+ - `memory_forget(id, undo?)` — forget a memory entry (supersede + suppress re-adding), or release it.
172
+ - `memory_consolidate(limit?)` — run the LLM consolidation pass now (digests → entries) instead of waiting for the cadence.
173
+ - `memory_codebase(topic, text, importance?)` — record what a module or subsystem of the current repository does (workspace codebase map).
174
+ - `memory_decision(decision, rationale, context?, alternatives?, consequences?, scope?, workspace?, importance?)` — record a settled choice with why it won. Available in both scopes.
175
+ - `memory_state(section, content)` — record or refresh one section of the **repository** state overview.
176
+ - `memory_working(section?, content?, activity?, clearActivity?)` — record what **you** are working on: a current section, or activity notes.
177
+ - `memory_status()` — where the store lives, how many digests await consolidation, whether the last sync conflicted.
178
+ - `memory_refresh(limit?)` — run the freshness pass now instead of waiting for the daily cadence.
179
+ - `memory_status()` — now also reports model calls, event counts and saved calls (thin/trivial).
180
+ - `memory_history(limit?)` — ingest repository commit history into the workspace history layer.
181
+
182
+ `ctx.memory` service: `search`, `remember`, `workspaces`, `associate`, `resolveKey`, `status`, `workspaceOf`, `pending`, `consolidate`.
183
+
184
+ ## Capture pipeline
185
+
186
+ Capture turns sessions into memory without ever storing raw transcripts:
187
+
188
+ 1. **Relevance filter** (`digest.js`) — deterministic and bounded. Tool calls reduce to a
189
+ *count* (tool mechanics are not memory); user turns and assistant *gists* are clipped;
190
+ bare acknowledgements are dropped; the window is trimmed oldest-first to `digestMaxBytes`.
191
+ 2. **Summarization** (`summarize.js`) — the session's own model turns that material into
192
+ 5-12 durable bullets under explicit instructions: bullets, `**label:**`, date-led when
193
+ time-bound, keep decisions/constraints/preferences/environment facts, drop tool chatter,
194
+ code and pleasantries. A model that finds nothing durable replies `NO_DURABLE_CONTENT`
195
+ and nothing is stored.
196
+ 3. **Incremental storage** (`capture.js`) — a `lastSeq` watermark in `state/sessions.json`
197
+ means each pass summarizes only events that arrived since the previous one, so capture is
198
+ O(new events) rather than O(session). Bullets are appended as a dated section.
199
+
200
+ A digest file therefore holds a bounded metadata header plus dated bullet sections — never
201
+ message bodies. Without a model, capture degrades to deterministic clipped notes instead of
202
+ failing; a `null` `consolidatedAt` marks the digest as work for the consolidation pass.
203
+
204
+ ## Working memory and sync safety
205
+
206
+ - `working.md` is **per repo/workspace**, never global: it is the current-state head of one
207
+ repository, so mixing repositories would be meaningless. The global layer has no working
208
+ file, and a template-only `global/working.md` left by an older version is retired on boot.
209
+ - A failed `git pull`/`push` is never swallowed. The failure is logged, recorded in
210
+ `state/sync.json`, surfaced by `status()`, reported by the `memory_sync` tool, and leaves a
211
+ `state/CONFLICT.md` marker that the next clean sync clears.
212
+ - Consolidation attributes every entry to the digest it came from. A digest the model never
213
+ attributes stays pending and is retried; after `maxAttempts` it is marked `needsReview`
214
+ instead of being silently dropped.
215
+ - A model reply is parsed with escalating recovery: exact JSON, then repaired JSON (trailing
216
+ commas, placeholder values), then whatever complete objects the text contains — so a
217
+ truncated or sloppy reply degrades instead of failing the pass. If nothing is recoverable,
218
+ the raw prompt and reply are written to `logs/consolidate-failed-<stamp>.txt` and the tool
219
+ reports `parseFailed` rather than discarding the evidence.
220
+ - Replies are assembled from **text blocks only**. A reasoning model streams its thinking as
221
+ `reasoning` blocks; folding those into the reply prepends prose to the answer, and any
222
+ "parse the JSON" step then matches the wrong braces. This was the cause of a real failure:
223
+ the reasoning preamble plus the echoed prompt example were handed to the parser.
224
+ - The reply protocol is **line-oriented, not JSON**: one
225
+ `- [<digest> | global|<workspace:key> | <layer> | <importance>] <topic> :: <text>` line per
226
+ fact, plus a `DONE <digest>` line per digest processed. Nested JSON proved fragile in
227
+ practice — one invalid placeholder, one truncated brace or one reasoning preamble lost the
228
+ entire pass — while a line protocol parses per line, so one bad line cannot kill the batch
229
+ and prose is simply ignored. A JSON object is still accepted if a model volunteers one, with
230
+ escalating recovery (exact, repaired, then salvaged objects).
231
+ - Nothing recoverable is discarded: the raw prompt and reply are written to
232
+ `logs/consolidate-failed-<stamp>.txt` and the tool reports `parseFailed`.
233
+ - Each digest heading carries the workspace key resolved from that session's own working
234
+ directory, and the model is told to use it. It was previously left to derive a key, which it
235
+ did from the filesystem path — and a path key created a workspace directory that the write
236
+ then could not use. `remember()` now sanitizes the key once, up front, so a key and the
237
+ directory it targets can never disagree.
238
+ - The last consolidation exchange is always kept at `logs/consolidate-last.txt`, and the tool
239
+ reports `writeFailed` with the reasons, so a pass that writes nothing is visible.
240
+ - One malformed entry cannot fail a whole pass: it is recorded, and a workspace request with
241
+ no usable key falls back to the global layer instead of throwing.
242
+
243
+ ## Codebase map
244
+
245
+ `codebase` is the repository map: one entry per module, file or subsystem, written **as the
246
+ agent discovers it**. It is deliberately **workspace-only** — a global codebase map would mix
247
+ unrelated repositories together, exactly like working memory.
248
+
249
+ Two things feed it:
250
+
251
+ - `memory_codebase(topic, text, importance?)` — the agent records what a module does. The
252
+ topic is the path or subsystem name, so re-discovery updates the same topic file. It
253
+ resolves the workspace from the session, so no key is needed.
254
+ - **Traversal evidence.** Capture records the distinct file paths a session's tool *calls*
255
+ point at (arguments only — results are full of paths mentioned in file contents, which say
256
+ nothing about intent), bounded and deduped. Those paths ride the digest, so summarization
257
+ and consolidation can turn exploration into map entries without the agent writing anything.
258
+
259
+ The injected context lists the topics already mapped for this workspace, so the agent can see
260
+ what it can rely on and what is still unexplored.
261
+
262
+ ## Commit history
263
+
264
+ `history` records **how the repository changed**, complementing the codebase map (which says
265
+ what each module *is*). Also workspace-only and multi-layered, but its topics are **calendar
266
+ days** — one file per day, so a long-lived repo yields many small readable files:
267
+
268
+ ```markdown
269
+ # workspaces/role-model/history.md
270
+ - [2026-09-27](history/2026-09-27.md) — 11 entries
271
+ - [2026-09-28](history/2026-09-28.md) — 16 entries
272
+ - [2026-10-01](history/2026-10-01.md) — 2 entries
273
+ ```
274
+
275
+ Each entry is an ordinary Format-2 entry (subject + short hash as the heading, the full commit
276
+ body, the changed files, and `**Source:** git:<hash>`), so the ledger, ranking and search work
277
+ on it unchanged.
278
+
279
+ Ingest is incremental in two layers:
280
+
281
+ 1. The log range starts at the last recorded commit (`<head>..HEAD`), so a normal pass reads
282
+ only new commits. When the range is unreachable (a rebase or force-push rewrote history) the
283
+ pass falls back to a bounded walk and reports `fellBack`.
284
+ 2. A hash watermark in `state/history.json` skips what is already recorded — an optimisation,
285
+ not the authority.
286
+ 3. **The layer itself is the authority.** Every entry carries a `Commit: <full hash>` line and
287
+ ingest reads them back, so a lost watermark, a fresh clone or a rewound head cannot produce a
288
+ duplicate. Ingest is idempotent even if `state/` is deleted entirely.
289
+
290
+ Runners report status inconsistently — `git.js` documents `code`, while the host runner and
291
+ `ctx.subprocess` return `exitCode` — so the check reads either. Reading only one made every
292
+ successful ranged query look like a failure and silently fell back to a full re-read.
293
+
294
+ Recording happens on a cadence (`autoHistoryIntervalMs`, default hourly, `autoHistoryLimit`
295
+ commits per pass) and on demand via `memory_history`. A repository that is missing, not a repo,
296
+ or has no new commits is skipped silently — it never interrupts a turn.
297
+
298
+ ## Decisions
299
+
300
+ A `decision` is neither a fact nor a lesson: it is a **settled choice and why it won** — a
301
+ dependency, a convention, an architecture call. It exists so a later session does not
302
+ re-litigate a closed question or silently reverse it.
303
+
304
+ Unlike `codebase` and `history`, it is available in **both scopes**: a convention can be a
305
+ global preference, while a dependency choice belongs to the repository that made it.
306
+
307
+ The shape is the point. The decision statement is the entry heading, so the layer index reads
308
+ as a list of decisions, and the reasoning lives in fixed sections:
309
+
310
+ ```markdown
311
+ ## Use plain ESM with no build step for dsh-memory
312
+
313
+ **Context:**
314
+ This plugin is plain JavaScript loaded directly by path...
315
+
316
+ **Rationale:**
317
+ Plain ESM keeps the shipped source equal to the code under test...
318
+
319
+ **Alternatives considered:**
320
+ - Compile TypeScript or bundle the host half
321
+ - Wrap the entry in a default export
322
+
323
+ **Consequences:**
324
+ The plugin entry must keep named exports and must not add a default export...
325
+ ```
326
+
327
+ Fixed sections make decisions comparable and greppable, and `parseDecisionBody` reads them back
328
+ so a stored decision can be inspected without re-deriving it from prose. The write path renders
329
+ this shape — a model deciding what is durable never has to produce it.
330
+
331
+ Decisions in force are surfaced in the injected context next to learnings, because they are the
332
+ constraints a new session has to respect.
333
+
334
+ ## Freshness metadata
335
+
336
+ Every entry carries its own decay metadata, so staleness is determinable **from the file alone** —
337
+ no external clock, no global assumption, and it survives a clone because it travels with the text.
338
+
339
+ | Field | Meaning |
340
+ | --- | --- |
341
+ | `**Volatility:**` | How fast this decays: `stable` (years — a language, a license, a convention), `normal` (months — how a subsystem works), `volatile` (days — a port, a version in flight). |
342
+ | `**Verified:**` | When the fact was last *confirmed true*. This is the age basis. |
343
+ | `**Valid-until:**` | A hard expiry for facts that genuinely end. |
344
+
345
+ Freshness is therefore a property of the entry, not a constant applied to a write date. A plugin
346
+ using plain ESM stays true for years; the port it listens on can change tomorrow — and only the
347
+ writer knows which is which:
348
+
349
+ ```text
350
+ same age (40 days), different verdicts
351
+ dsh-memory-sem-0029 Volatility: volatile -> stale (window 3d)
352
+ dsh-memory-sem-0028 Volatility: stable -> fresh (window 365d)
353
+ ```
354
+
355
+ `**Verified:**` matters because writing and confirming are different acts: an entry dated 400 days
356
+ ago but re-confirmed yesterday is fresh, and status quo previously could not express that. Where
357
+ volatility is absent (entries written before the field existed) time-bounded language still hints
358
+ at it, then the normal window applies. `memory_remember` accepts `volatility`, and the
359
+ consolidation protocol carries it as an optional fifth field.
360
+
361
+ ## Workspace state
362
+
363
+ `state.md` is a short, section-based overview of **what the repository is right now** — layout,
364
+ what is running, what is pending, health. It is the one layer whose entire value is being current,
365
+ so it is treated differently from every other:
366
+
367
+ - **Replaced per section, never appended.** State describes now, so a newer statement about a
368
+ section supersedes the old one instead of piling up beside it. Other sections keep their own
369
+ timestamps — editing one area must not make an unrelated section look freshly verified.
370
+ - **Every section carries `**Verified:**`**, and freshness is graded rather than binary:
371
+ `fresh` / `aging` / `stale` / `unverified`.
372
+ - **Staleness is always shown, never hidden.** A stale section appears in the injected context
373
+ with its age (`[STALE 5d]`), so it reads as a hypothesis to check rather than a fact to trust.
374
+ An unverified section is marked `[UNVERIFIED]`.
375
+ - **Workspace-only** and a single file: a repository state file in the global layer would
376
+ describe no repository, and an overview that needs an index is not an overview.
377
+
378
+ ## What capture keeps (and what it refuses)
379
+
380
+ A session is projected to candidate material before any model sees it. Two rules were added
381
+ after observing real research sessions silently produce nothing:
382
+
383
+ - **Injected context is not user intent.** The harness prepends runtime snapshots, skill
384
+ catalogs, policy notices and the memory plugin's own context block as *user* turns. They are
385
+ dropped, because in one measured session they were **82% of the material**. Capturing the
386
+ memory block specifically is a feedback loop: the system treating its own output as new input.
387
+ - **A research answer is its findings, not its status line.** Analysis sessions open with a
388
+ one-line announcement ("Analysis complete.") and put everything in the body. Capture now skips
389
+ a bare status opener and carries the substance, and takes the assistant `text` block rather
390
+ than its private `reasoning`.
391
+
392
+ Both were invisible in unit tests and obvious the moment one real session was traced.
393
+
394
+ ## Memory belongs to the main agent
395
+
396
+ A **subagent does not write memory**. It reads freely - it can search and consult the store -
397
+ but recording is the main agent's job.
398
+
399
+ The reason is ownership, not permission. A subagent is a delegate working inside a task the
400
+ parent already owns, and the parent's transcript carries the delegation and its result. Letting
401
+ the child write as well produces two partial accounts of one task, and the child's - written
402
+ without the parent's context - competes with the parent's in search.
403
+
404
+ Enforced in three places:
405
+
406
+ - **Capture** skips a subagent session entirely, so it never becomes a digest.
407
+ - **Every writing tool** refuses with an explanation naming the tool; reads are unrestricted.
408
+ - **The injected context** tells a subagent to report its findings to the caller instead of
409
+ instructing it to write.
410
+
411
+ A session counts as a subagent when `delegationDepth > 0` **or** `origin === "subagent"` - both
412
+ are checked because they are set independently, and an absent depth means top-level.
413
+
414
+ ## Which model does memory work
415
+
416
+ Memory work needs a model that returns **text**. Reasoning-heavy models sometimes stream only
417
+ `reasoning` blocks and no answer at all - measured here as 7 of 11 consolidation passes writing
418
+ nothing, on the same prompts that another model turned into 35 entries.
419
+
420
+ So the model is not configured as a constant. Each call builds a **ladder from the live
421
+ registry**:
422
+
423
+ 1. the configured `summaryModel`, when set (a preference, not a requirement)
424
+ 2. the session's own route
425
+ 3. every other model this deployment can route to right now
426
+
427
+ The first model that returns text wins. A model that returns nothing, or throws, is skipped and
428
+ the **same prompt** goes to the next rung. The winner is remembered, so the normal case still
429
+ costs one call; a model that returned nothing is benched for 15 minutes and sinks to the end of
430
+ the ladder - never dropped, so a deployment where everything is benched still tries.
431
+
432
+ The ladder is capped at 4 models, and if every one returns nothing the call raises an error
433
+ rather than reporting success, so the session is retried instead of silently settled.
434
+
435
+ ## Failures retry; conclusions do not
436
+
437
+ A summary that *fails* (empty reply, unparseable prose, or a thrown call) is not the same as a
438
+ model concluding there is nothing durable. Failure keeps its material and is retried up to
439
+ `summaryMaxAttempts` (default 2) before being laid to rest as failed. A deliberate
440
+ `NO_DURABLE_CONTENT` settles immediately. Treating a failure as a conclusion is how whole
441
+ sessions were marked handled and lost for good.
442
+
443
+ ## Memory may correctly write nothing
444
+
445
+ A worker is **not** required to add a memory every capture interval or every turn. In fact,
446
+ writing something merely to fill a gap is a failure: it pollutes search, creates false context,
447
+ and makes later decisions harder to trust.
448
+
449
+ This is enforced at multiple levels:
450
+
451
+ - The injected footer says explicitly that writing NOTHING is a correct and expected outcome.
452
+ - The capture relevance filter drops tool mechanics, acknowledgements and low-signal material.
453
+ - `summaryMinSignalChars` (default 400) skips the model on a thin new delta entirely.
454
+ - The summarizer may return `NO_DURABLE_CONTENT` without creating pending work.
455
+ - Consolidation may return only `DONE <digest>` lines: a handled digest with zero facts is
456
+ successful, writes zero entries, and is not retried. It never invents a fact for output.
457
+ - `memory_remember` itself says not to call it to fill a gap.
458
+
459
+ The desired ratio is not "one turn in, one memory out". It is **high signal, low noise**, with
460
+ a cheap no-op path for the common case where nothing durable changed.
461
+
462
+ ## Working memory belongs to the agent
463
+
464
+ Two files describe "now", and they are **not** the same thing:
465
+
466
+ | File | Owner | Answers |
467
+ | --- | --- | --- |
468
+ | `state.md` | the repository | *What is this repo right now?* — layout, what is running, pending, health |
469
+ | `working.md` | **the agent** | *What am I in the middle of?* — task, recent activity, next steps |
470
+
471
+ A repository has one current state, but several agents may work in it with entirely different
472
+ intents — and one intent may span several repositories. Merging them would lose both meanings.
473
+
474
+ `working.md` states its own ownership in the file, on every write, so an agent reading it
475
+ directly (not just through the injected card) knows it is theirs:
476
+
477
+ ```text
478
+ THIS FILE BELONGS TO THE AGENT. It is not repository state (that is state.md) and not
479
+ durable memory (those are the layered entries). It is your own head for this workspace.
480
+
481
+ You own it. Read it, edit it, and keep it honest. Write it with memory_working, or edit
482
+ this file directly. Nothing else writes here, so a stale or empty file means the intent
483
+ was never recorded.
484
+ ```
485
+
486
+ Two section behaviours, because the information has two shapes:
487
+
488
+ - **Task / Next steps / Plan / Blocked by** — *replaced* on write: there is one current task.
489
+ - **Recent activity** — a *rolling log*, newest last, trimmed oldest-first by entries and
490
+ characters, with identical consecutive notes not repeated.
491
+
492
+ It decays in **hours**, not days, so it is graded on its own scale (fresh / aging / stale / not
493
+ started), and an unrecorded file is **called out** in the injected card rather than omitted —
494
+ silence would read as "no working memory" when it actually means "nothing written yet".
495
+
496
+ ## Operational observability
497
+
498
+ The derived `logs/` and `state/` surfaces are intentionally observable even though they are
499
+ gitignored:
500
+
501
+ - `logs/memory.log` is newline-delimited JSON for capture, consolidation, refresh, history,
502
+ pruning, sync errors and model calls. It rotates at 2 MB.
503
+ - `logs/consolidate-last.txt` preserves the latest prompt/reply exchange for diagnosis, and
504
+ `consolidate-failed-*.txt` preserves parse failures rather than discarding the evidence.
505
+ - `state/metrics.json` counts cadence events, model calls by purpose, and calls avoided by
506
+ cost control (`thin`, `trivial`). `memory_status` exposes the useful subset.
507
+ - `state/activity.jsonl` is a bounded human-readable operational feed.
508
+
509
+ ## Cost control
510
+
511
+ Every stage can decide **not** to spend a model call:
512
+
513
+ - A `trivial` session (a ping) never reaches a model at all.
514
+ - A content session whose **new** delta is below `summaryMinSignalChars` (default 400) is skipped as
515
+ `thin`. The check counts the *filtered* candidates, not raw events, so a burst of tool noise
516
+ cannot masquerade as content — a model call on a two-line delta costs more than it captures.
517
+ - Consolidation batches and caps its prompt (`consolidateMaxPromptBytes`, `consolidateMaxDigestBytes`).
518
+
519
+ Captured digests are **pruned** on the idle cadence down to the newest `digestKeepCount` (default
520
+ 50) *consolidated* ones. A pending digest is never removed: it is the only copy of work not yet in
521
+ the store. Ordering is by when the session was captured, not file mtime, so digests written in the
522
+ same instant cannot be pruned in the wrong order.
523
+
524
+ ## Hooks
525
+
526
+ | Direction | Hook | Effect |
527
+ | --- | --- | --- |
528
+ | read | `agent/pre-step` | injects the `## DSH memory context` card at step 1 (re-injected per `injectionRefreshMs`) |
529
+ | write | `session/event` turn/end | incremental capture: new events → relevance filter → session-model summary → `digests/` (throttled by `autoCaptureIntervalMs`) |
530
+ | write | `agent/pre-step` cadence | LLM consolidation: pending digests → entries |
531
+ | write | `agent/pre-step` cadence | freshness scan + LLM refresh (keep/update/supersede) |
532
+ | life | `session/event` turn/start | workspace auto-association via `ctx.workspaceRegistry` |
533
+ | sync | `agent/pre-step` cadence | debounced git commit + hourly pull/push |
534
+ | all | idle ticker (60 s) | runs the cadences even when nobody is chatting; needs no agent when `summaryModel` is set |
535
+
536
+ On-demand equivalents: `memory_consolidate` (or `ctx.memory.consolidate(agent)`), `memory_sync` (or `ctx.memory` sync).
537
+
538
+ ## How it works
539
+
540
+ observe (`session/event`) → capture (`sessionQuery` → deterministic digest) → consolidate (`ctx.llm` → entries)
541
+ → index (FTS5) → query (in-process search) → maintain (freshness/forget) → sync (git).
542
+
543
+ ## Development
544
+
545
+ node --test # 72 tests (unit + behavior fixtures + eval)
546
+
547
+ - `src/fixture.js` — versioned behavior-fixture suite (ranking/freshness/forgetting/budget/workspace-key).
548
+ - `src/eval.js` — sufficiency eval: fact-recall probes, a judge layer, and deterministic anchor checks.
549
+
550
+ Local validation against an installed DSH (needs the profile node_modules on the resolve path):
551
+
552
+ node --input-type=module -e "import('file:///D:/DEV/dsh-memory/src/index.js').then(m => console.log(m.name, m.inject))"
553
+
554
+ ## Search
555
+
556
+ Normal queries use the unicode61 word index. CJK and other no-space scripts route to a
557
+ trigram substring index; two-character queries (which are shorter than a trigram) use an
558
+ escaped LIKE fallback. Partial English words also retry through substring matching. The
559
+ parallel index is regenerated or updated alongside the word index, and scope filtering still
560
+ applies to every result.
561
+
562
+ ### Optional rerank
563
+
564
+ `memory_search` can ask the session model to reorder hits by usefulness. It is **off by
565
+ default** (`rerankSearch`), costs one call, and is strictly **advisory**: rerankHits returns
566
+ the original deterministic order whenever the model is missing, throws, or replies with
567
+ something unusable — and it refuses a partial answer that merely agrees with the existing
568
+ order. Search never depends on a model.
569
+
570
+ ## Settings safety
571
+
572
+ The Memory page exposes only store root, summary model, GitHub origin and branch. The optional
573
+ git token is a password field that is **never pre-filled** from the redacted host snapshot:
574
+ typing a new token replaces it; leaving the field empty leaves the current token untouched.
575
+
576
+ ## Status
577
+
578
+ Implemented greenfield, no legacy code. **Seven layers** (`semantic`, `episodic`, `procedural`,
579
+ `learning`, plus workspace-only `codebase`, `history` and `state`), all multi-layered except
580
+ `state.md` and `working.md`. **Eleven tools**, a `ctx.memory` service for other plugins, deep DSH
581
+ hooks (capture, consolidation, refresh, history ingest, git sync, idle ticker) and a self-describing
582
+ `SYSTEM.md`. **170+ unit tests**, a behaviour-fixture suite over five axes, and a wiring test that
583
+ mounts the real host and executes every tool. See `dsh-memory-v2-plan.md` for the plan and
584
+ `SYSTEM.md` in the store for the as-built description.
585
+