memhtml 0.2.5 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +114 -245
- package/agent/instructions.md +74 -84
- package/dist/{dist-t84Q_98w.mjs → dist-BCsav-EP.mjs} +341 -6
- package/dist/dist-BCsav-EP.mjs.map +1 -0
- package/dist/{dist-Uj47oBRC.mjs → dist-D73gfqLc.mjs} +4087 -370
- package/dist/dist-D73gfqLc.mjs.map +1 -0
- package/dist/dist-DuzGralO.mjs +3 -0
- package/dist/memhtml-mcp.mjs +45 -7
- package/dist/memhtml-mcp.mjs.map +1 -1
- package/dist/memhtml.mjs +64 -11
- package/dist/memhtml.mjs.map +1 -1
- package/package.json +1 -1
- package/src/client.ts +180 -8
- package/src/contract.ts +232 -5
- package/state-migrations/S0002_entity_corroboration.sql +55 -0
- package/dist/dist-Dj-MYf9q.mjs +0 -3
- package/dist/dist-Uj47oBRC.mjs.map +0 -1
- package/dist/dist-t84Q_98w.mjs.map +0 -1
package/README.md
CHANGED
|
@@ -1,28 +1,19 @@
|
|
|
1
1
|
# memhtml
|
|
2
2
|
|
|
3
|
-
[](https://github.com/memhtml/memhtml/actions/workflows/check.yml)
|
|
4
|
-
[](https://github.com/memhtml/memhtml/actions/workflows/security.yml)
|
|
5
|
-
[](https://scorecard.dev/viewer/?uri=github.com/memhtml/memhtml)
|
|
6
|
-
[](https://opensource.org/licenses/Apache-2.0)
|
|
3
|
+
[](https://github.com/memhtml/memhtml/actions/workflows/check.yml) [](https://github.com/memhtml/memhtml/actions/workflows/security.yml) [](https://scorecard.dev/viewer/?uri=github.com/memhtml/memhtml) [](https://opensource.org/licenses/Apache-2.0)
|
|
7
4
|
|
|
8
|
-
memhtml stores an agent's long-term memory as a git repository of semantic HTML5 files, one fact per
|
|
9
|
-
file. A rebuildable SQLite index sits over that tree, retrieval fuses four ranking arms, and a nightly
|
|
10
|
-
curation pipeline commits its work to a branch a human reviews before it lands.
|
|
5
|
+
memhtml stores an agent's long-term memory as a git repository of semantic HTML5 files, one fact per file. A rebuildable SQLite index sits over that tree, retrieval fuses four ranking arms, and a nightly curation pipeline commits its work to a branch a human reviews before it lands.
|
|
11
6
|
|
|
12
7
|
## Install
|
|
13
8
|
|
|
14
|
-
One package carries the whole system — the CLI, code mode, the sleep cycle, the trace indexer, and the
|
|
15
|
-
MCP server — and installs two binaries, `memhtml` and `memhtml-mcp`. Node 24 or newer.
|
|
9
|
+
One package carries the whole system — the CLI, code mode, the sleep cycle, the trace indexer, and the MCP server — and installs two binaries, `memhtml` and `memhtml-mcp`. Node 24 or newer.
|
|
16
10
|
|
|
17
11
|
```bash
|
|
18
12
|
npm i -g memhtml # or: pnpm add -g memhtml, bun add -g memhtml
|
|
19
13
|
npx memhtml manifest # every command, flag, and error code, without installing anything
|
|
20
14
|
```
|
|
21
15
|
|
|
22
|
-
Point it at a corpus with `MEMHTML_ROOT` (default `~/memhtml`), and at your transcripts with
|
|
23
|
-
`MEMHTML_TRACE_ROOT` (default `~/.claude`). Reading and writing memories needs no credentials;
|
|
24
|
-
embeddings and the sleep cycle's model calls use Bedrock through the default AWS credential chain, and
|
|
25
|
-
`MEMHTML_EMBED=off` / `MEMHTML_LLM=off` turn both off.
|
|
16
|
+
Point it at a corpus with `MEMHTML_ROOT` (default `~/memhtml`), and at your transcripts with `MEMHTML_TRACE_ROOT` (default `~/.claude`). Reading and writing memories needs no credentials; embeddings and the sleep cycle's model calls use Bedrock through the default AWS credential chain, and `MEMHTML_EMBED=off` / `MEMHTML_LLM=off` turn both off.
|
|
26
17
|
|
|
27
18
|
To register the MCP server with a client, the command is `memhtml-mcp` over stdio.
|
|
28
19
|
|
|
@@ -34,24 +25,15 @@ memhtml search "one writer many readers" # FTS + vector + recency + salienc
|
|
|
34
25
|
memhtml serve mcp # the same store over stdio: 14 tools, 2 resources
|
|
35
26
|
```
|
|
36
27
|
|
|
37
|
-
`memhtml manifest` (or a bare `memhtml`) answers with every command, flag, response type, and error code
|
|
38
|
-
the binary accepts, and it answers on a machine with no repo, no database, and no credentials. Every
|
|
39
|
-
command writes exactly one JSON envelope to stdout, logs go to stderr, and the exit code is 0 for
|
|
40
|
-
success, 2 for a usage error, 1 for a runtime failure. `AGENTS.md` is generated from the same table that
|
|
41
|
-
drives parsing, so the doc cannot drift from the binary.
|
|
28
|
+
`memhtml manifest` (or a bare `memhtml`) answers with every command, flag, response type, and error code the binary accepts, and it answers on a machine with no repo, no database, and no credentials. Every command writes exactly one JSON envelope to stdout, logs go to stderr, and the exit code is 0 for success, 2 for a usage error, 1 for a runtime failure. `AGENTS.md` is generated from the same table that drives parsing, so the doc cannot drift from the binary.
|
|
42
29
|
|
|
43
30
|
## The design in three sentences
|
|
44
31
|
|
|
45
|
-
The git tree is the system of record, and `.memhtml/index.db` is a projection of it that can be deleted
|
|
46
|
-
and rebuilt without loss. Anything that must survive `rm index.db` lives in the files, where authored
|
|
47
|
-
links are `<link>` elements and metadata is `<meta>` elements, while re-derivable artifacts such as
|
|
48
|
-
embeddings and mined edges live only in the index. Every removal is a `git mv` into `archive/<YYYY>/`
|
|
49
|
-
that mirrors the original path, so the file stays in the tree and `git log --follow` reads straight
|
|
50
|
-
through a memory's whole life.
|
|
32
|
+
The git tree is the system of record, and `.memhtml/index.db` is a projection of it that can be deleted and rebuilt without loss. Anything that must survive `rm index.db` lives in the files, where authored links are `<link>` elements and metadata is `<meta>` elements, while re-derivable artifacts such as embeddings and mined edges live only in the index. Every removal is a `git mv` into `archive/<YYYY>/` that mirrors the original path, so the file stays in the tree and `git log --follow` reads straight through a memory's whole life.
|
|
51
33
|
|
|
52
|
-
Figure 1 draws that. It is built from monospace box characters, which a screen reader reads as noise, so
|
|
53
|
-
skip to the paragraph below the figure: it says the same thing in words.
|
|
34
|
+
Figure 1 draws that. It is built from monospace box characters, which a screen reader reads as noise, so skip to the paragraph below the figure: it says the same thing in words.
|
|
54
35
|
|
|
36
|
+
<!-- dprint-ignore-start -->
|
|
55
37
|
<!-- figure:system-topology -->
|
|
56
38
|
```text
|
|
57
39
|
+--------+ +---------------+ +----------------+
|
|
@@ -101,41 +83,25 @@ skip to the paragraph below the figure: it says the same thing in words.
|
|
|
101
83
|
+------------+
|
|
102
84
|
```
|
|
103
85
|
<!-- /figure:system-topology -->
|
|
86
|
+
<!-- dprint-ignore-end -->
|
|
104
87
|
|
|
105
|
-
**Figure 1: every write door lands in the git tree, and every read is served from projections of it.**
|
|
106
|
-
Three doors reach in from outside: `memhtml write` and `memhtml apply`, the MCP server's 14 tools, and
|
|
107
|
-
your own file tools. All three commit into one git tree, and they differ only in who owns the commit.
|
|
108
|
-
Eviction moves a file to `archive/YYYY/` and leaves it in the tree. From the tree, a git-driven indexer
|
|
109
|
-
derives `index.db`, which supplies three of the four ranking arms. The fourth, salience, comes from
|
|
110
|
-
`state.db`, the one plane git cannot reproduce. A query enters the ranker at the bottom, RRF fuses the
|
|
111
|
-
four arms, MMR diversifies the result, and ranked hits come out. On the docs site the same drawing puts
|
|
112
|
-
into its borders what this caption has to spell out: a heavy border is a door, a double border is the
|
|
113
|
-
system of record, a dashed border is a projection that can be deleted and rebuilt, and a cylinder is a
|
|
114
|
-
database on disk.
|
|
88
|
+
**Figure 1: every write door lands in the git tree, and every read is served from projections of it.** Three doors reach in from outside: `memhtml write` and `memhtml apply`, the MCP server's 14 tools, and your own file tools. All three commit into one git tree, and they differ only in who owns the commit. Eviction moves a file to `archive/YYYY/` and leaves it in the tree. From the tree, a git-driven indexer derives `index.db`, which supplies three of the four ranking arms. The fourth, salience, comes from `state.db`, the one plane git cannot reproduce. A query enters the ranker at the bottom, RRF fuses the four arms, MMR diversifies the result, and ranked hits come out. On the docs site the same drawing puts into its borders what this caption has to spell out: a heavy border is a door, a double border is the system of record, a dashed border is a projection that can be deleted and rebuilt, and a cylinder is a database on disk.
|
|
115
89
|
|
|
116
90
|
## Why files
|
|
117
91
|
|
|
118
|
-
A memory an agent can be trusted with has to be reviewable, diffable, and recoverable. git supplies all
|
|
119
|
-
three directly, and that is why every write below lands as a commit:
|
|
92
|
+
A memory an agent can be trusted with has to be reviewable, diffable, and recoverable. git supplies all three directly, and that is why every write below lands as a commit:
|
|
120
93
|
|
|
121
|
-
- A correction is a commit. `memhtml correct` writes the new file and archives the old one in one
|
|
122
|
-
|
|
123
|
-
- A
|
|
124
|
-
one commit, and reindex once. The batch is atomic by default, per-op results come back in input
|
|
125
|
-
order, and a duplicate succeeds with `deduped: true` and the existing path.
|
|
126
|
-
- A nightly curation run is a branch. `memhtml sleep run` walks fifteen phases and commits each one's
|
|
127
|
-
work on its own, so a human reads the curation one phase-shaped diff at a time, and
|
|
128
|
-
`memhtml sleep merge` fast-forwards `main` only after a quality gate that can refuse.
|
|
94
|
+
- A correction is a commit. `memhtml correct` writes the new file and archives the old one in one commit, so an interrupted run cannot leave two live memories contradicting each other.
|
|
95
|
+
- A batch is a commit. `memhtml apply` (JSONL ops) and `memory_write_batch` (MCP) stage N files, make one commit, and reindex once. The batch is atomic by default, per-op results come back in input order, and a duplicate succeeds with `deduped: true` and the existing path.
|
|
96
|
+
- A nightly curation run is a branch. `memhtml sleep run` walks sixteen phases and commits each one's work on its own, so a human reads the curation one phase-shaped diff at a time, and `memhtml sleep merge` fast-forwards `main` only after a quality gate that can refuse.
|
|
129
97
|
|
|
130
98
|
## Who does what
|
|
131
99
|
|
|
132
|
-
Three actors share one tree. The agent writes facts, and it resolves only the conflicts it found itself.
|
|
133
|
-
Sleep curates nightly on a branch, and it detects conflicts without resolving them. The human owns the
|
|
134
|
-
gate and every one-way door.
|
|
100
|
+
Three actors share one tree. The agent writes facts, and it resolves only the conflicts it found itself. Sleep curates nightly on a branch, and it detects conflicts without resolving them. The human owns the gate and every one-way door.
|
|
135
101
|
|
|
136
|
-
Figure 2 draws the cycle they form. A screen reader reads its box characters as noise, so the paragraph
|
|
137
|
-
beneath the figure carries the same content in words.
|
|
102
|
+
Figure 2 draws the cycle they form. A screen reader reads its box characters as noise, so the paragraph beneath the figure carries the same content in words.
|
|
138
103
|
|
|
104
|
+
<!-- dprint-ignore-start -->
|
|
139
105
|
<!-- figure:three-actors -->
|
|
140
106
|
```text
|
|
141
107
|
+----------+
|
|
@@ -181,20 +147,13 @@ beneath the figure carries the same content in words.
|
|
|
181
147
|
+------------+
|
|
182
148
|
```
|
|
183
149
|
<!-- /figure:three-actors -->
|
|
150
|
+
<!-- dprint-ignore-end -->
|
|
184
151
|
|
|
185
|
-
**Figure 2: the three actors form a cycle through `main`, and only one of them may settle a
|
|
186
|
-
contradiction.** Reading top to bottom: the agent writes to `main` at any hour, one fact per file.
|
|
187
|
-
Sleep reads `main` nightly and puts its fifteen commits on a `sleep/<date>` branch, leaving `main`
|
|
188
|
-
untouched. Those phases deduplicate, resolve entities, decay confidence, compress, and synthesize arcs,
|
|
189
|
-
and they flag a contradiction without choosing a winner. The human reviews that branch and merges, which
|
|
190
|
-
returns the cycle to `main` and to the agent. The two heavy-bordered boxes are the actors outside the
|
|
191
|
-
system, and `main` and the branch are double-bordered because they are the system of record.
|
|
152
|
+
**Figure 2: the three actors form a cycle through `main`, and only one of them may settle a contradiction.** Reading top to bottom: the agent writes to `main` at any hour, one fact per file. Sleep reads `main` nightly and puts its sixteen commits on a `sleep/<date>` branch, leaving `main` untouched. Those phases deduplicate, resolve entities, decay confidence, compress, and synthesize arcs, and they flag a contradiction without choosing a winner. The human reviews that branch and merges, which returns the cycle to `main` and to the agent. The two heavy-bordered boxes are the actors outside the system, and `main` and the branch are double-bordered because they are the system of record.
|
|
192
153
|
|
|
193
154
|
## The file format
|
|
194
155
|
|
|
195
|
-
One fact per file, in standard HTML5 that a browser displays and a person can read in view-source. The
|
|
196
|
-
element vocabulary is closed, and `memhtml doctor` warns on anything outside it. Each element carries
|
|
197
|
-
meaning the indexer reads, which is the structure Markdown gives you no way to express:
|
|
156
|
+
One fact per file, in standard HTML5 that a browser displays and a person can read in view-source. The element vocabulary is closed, and `memhtml doctor` warns on anything outside it. Each element carries meaning the indexer reads, which is the structure Markdown gives you no way to express:
|
|
198
157
|
|
|
199
158
|
```html
|
|
200
159
|
<article>
|
|
@@ -207,37 +166,23 @@ meaning the indexer reads, which is the structure Markdown gives you no way to e
|
|
|
207
166
|
</article>
|
|
208
167
|
```
|
|
209
168
|
|
|
210
|
-
The single `<mark>` is the claim. It becomes the gist every listing shows, and it is the span a
|
|
211
|
-
correction targets. `<time datetime>` records when the fact happened in the world, so an episodic memory
|
|
212
|
-
ranks by that date instead of by its write time. `<dl>` pairs index as facets and `<cite>` as citations.
|
|
213
|
-
`<details>` folds elaboration behind a summary, and recall always discloses that a fold is there.
|
|
214
|
-
`docs/format.md` is the full vocabulary. `docs/tasks.md` covers the task type (`memhtml-task-status`,
|
|
215
|
-
`memhtml-due`), which rides the same format.
|
|
169
|
+
The single `<mark>` is the claim. It becomes the gist every listing shows, and it is the span a correction targets. `<time datetime>` records when the fact happened in the world, so an episodic memory ranks by that date instead of by its write time. `<dl>` pairs index as facets and `<cite>` as citations. `<details>` folds elaboration behind a summary, and recall always discloses that a fold is there. `docs/format.md` is the full vocabulary. `docs/tasks.md` covers the task type (`memhtml-task-status`, `memhtml-due`), which rides the same format.
|
|
216
170
|
|
|
217
171
|
## Writing
|
|
218
172
|
|
|
219
173
|
Three doors, all supported, all landing in the same tree:
|
|
220
174
|
|
|
221
|
-
1. The CLI. `memhtml write` takes one memory. Give it `--claim` plus `--body` and the template owns the
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
repo: write, read, search, recall, correct, link, archive, batch writes, and trace search. A CLI
|
|
227
|
-
command and a running server share one store, because WAL admits one writer and any number of
|
|
228
|
-
readers, and a contended write retries on `SQLITE_BUSY` (see `RUNBOOK.md`, section 4).
|
|
229
|
-
3. Your file tools. The tree is the system of record, so a hand-written file is as real as one the CLI
|
|
230
|
-
wrote. You take on what the write path would have done: format validity (`memhtml doctor`), path
|
|
231
|
-
choice, dedup, and the commit. Sleep refuses to start on a dirty tree.
|
|
232
|
-
|
|
233
|
-
Dedup is enforced by the schema: a partial unique index over active files makes a duplicate write
|
|
234
|
-
impossible to index, so the write returns the existing path with `deduped: true` and creates nothing.
|
|
175
|
+
1. The CLI. `memhtml write` takes one memory. Give it `--claim` plus `--body` and the template owns the markup; give it `--article-html` and you own the markup, with the format check refusing violations before anything is written. `memhtml apply` takes many: one JSONL op per line, every op validated for shape before any of them executes, then one commit and one index pass.
|
|
176
|
+
2. The MCP server. `memhtml serve mcp` speaks stdio and exposes 14 tools and 2 resources over the same repo: write, read, search, recall, correct, link, archive, batch writes, and trace search. A CLI command and a running server share one store, because WAL admits one writer and any number of readers, and a contended write retries on `SQLITE_BUSY` (see `RUNBOOK.md`, section 4).
|
|
177
|
+
3. Your file tools. The tree is the system of record, so a hand-written file is as real as one the CLI wrote. You take on what the write path would have done: format validity (`memhtml doctor`), path choice, dedup, and the commit. Sleep refuses to start on a dirty tree.
|
|
178
|
+
|
|
179
|
+
Dedup is enforced by the schema: a partial unique index over active files makes a duplicate write impossible to index, so the write returns the existing path with `deduped: true` and creates nothing.
|
|
235
180
|
|
|
236
181
|
A memory's whole life is commits in one tree.
|
|
237
182
|
|
|
238
|
-
Figure 3 draws that life. A screen reader sounds out its box characters, so read the paragraph beneath
|
|
239
|
-
the figure, which states the same four transitions in words.
|
|
183
|
+
Figure 3 draws that life. A screen reader sounds out its box characters, so read the paragraph beneath the figure, which states the same four transitions in words.
|
|
240
184
|
|
|
185
|
+
<!-- dprint-ignore-start -->
|
|
241
186
|
<!-- figure:memory-lifecycle -->
|
|
242
187
|
```text
|
|
243
188
|
+--------+
|
|
@@ -265,77 +210,44 @@ the figure, which states the same four transitions in words.
|
|
|
265
210
|
+-----------+ +---------+ +-----------+
|
|
266
211
|
```
|
|
267
212
|
<!-- /figure:memory-lifecycle -->
|
|
213
|
+
<!-- dprint-ignore-end -->
|
|
268
214
|
|
|
269
|
-
**Figure 3: a memory has one entry and three exits, and every one of them is a commit.** A write enters
|
|
270
|
-
the corpus as a single dedup-checked commit, and the file is then active. It stays active while sleep
|
|
271
|
-
reinforces or decays its confidence in place. Three things can end that state: `memhtml correct` writes a
|
|
272
|
-
replacement and archives the original in one commit, which makes it superseded; retention triage scores
|
|
273
|
-
it into the EVICT band and archives it, which makes it archived; or compress folds it into a synthesized
|
|
274
|
-
canonical memory and archives it with a `supersedes` link, which makes it compressed. Each of the three
|
|
275
|
-
exits is a `git mv` into `archive/YYYY/` mirroring the original path, so the file survives all of them
|
|
276
|
-
and `git log --follow` reads straight through the whole life.
|
|
215
|
+
**Figure 3: a memory has one entry and three exits, and every one of them is a commit.** A write enters the corpus as a single dedup-checked commit, and the file is then active. It stays active while sleep reinforces or decays its confidence in place. Three things can end that state: `memhtml correct` writes a replacement and archives the original in one commit, which makes it superseded; retention triage scores it into the EVICT band and archives it, which makes it archived; or compress folds it into a synthesized canonical memory and archives it with a `supersedes` link, which makes it compressed. Each of the three exits is a `git mv` into `archive/YYYY/` mirroring the original path, so the file survives all of them and `git log --follow` reads straight through the whole life.
|
|
277
216
|
|
|
278
217
|
## Retrieval
|
|
279
218
|
|
|
280
|
-
Four ranking arms are fused by reciprocal rank fusion (RRF, k=60) inside one SQL statement, and maximal
|
|
281
|
-
marginal relevance (MMR) diversifies the result afterwards in TypeScript:
|
|
219
|
+
Four ranking arms are fused by reciprocal rank fusion (RRF, k=60) inside one SQL statement, and maximal marginal relevance (MMR) diversifies the result afterwards in TypeScript:
|
|
282
220
|
|
|
283
|
-
| arm
|
|
284
|
-
|
|
285
|
-
| fts
|
|
286
|
-
| vector
|
|
287
|
-
| recency
|
|
288
|
-
| salience | 0.4
|
|
221
|
+
| arm | weight | what it ranks |
|
|
222
|
+
| -------- | ------ | ------------------------------------------------------------------------------------------------------------------- |
|
|
223
|
+
| fts | 1.0 | one denormalized title+gist+body column |
|
|
224
|
+
| vector | 1.0 | exact brute force over Cohere Embed v4 (1024-dim), grouped by path so a long memory does not outrank a relevant one |
|
|
225
|
+
| recency | 0.5 | `coalesce(event_at, updated_at)`, so an episodic memory sorts by when the fact happened |
|
|
226
|
+
| salience | 0.4 | the durable access plane, attached in the same statement, with tasks and `resources/people/` excluded |
|
|
289
227
|
|
|
290
|
-
A Bedrock outage narrows retrieval instead of stopping it. Arms that need a query vector are dropped
|
|
291
|
-
before the statement is assembled, the response carries `degraded: true`, and the remaining arms answer.
|
|
228
|
+
A Bedrock outage narrows retrieval instead of stopping it. Arms that need a query vector are dropped before the statement is assembled, the response carries `degraded: true`, and the remaining arms answer.
|
|
292
229
|
|
|
293
|
-
Salience counts the opens a caller chose. `memhtml read` and `memory_read` of a named path bump the access
|
|
294
|
-
plane; a path that `memhtml search` or `memhtml recall` merely returned does not, and neither does a sleep
|
|
295
|
-
phase. Bumping on a hit would make today's top five rank higher tomorrow for having been listed, while
|
|
296
|
-
the memory that should displace them never gets a first bump. `memhtml reinforce` is the explicit outcome
|
|
297
|
-
channel, and it moves the same exponentially weighted moving average. The arm also stays out of the way
|
|
298
|
-
of a `task` row and a `resources/people/` reference record: both are reached by predicate and by key, and
|
|
299
|
-
salience there would reward a stale task and decay a person's identity.
|
|
230
|
+
Salience counts the opens a caller chose. `memhtml read` and `memory_read` of a named path bump the access plane; a path that `memhtml search` or `memhtml recall` merely returned does not, and neither does a sleep phase. Bumping on a hit would make today's top five rank higher tomorrow for having been listed, while the memory that should displace them never gets a first bump. `memhtml reinforce` is the explicit outcome channel, and it moves the same exponentially weighted moving average. The arm also stays out of the way of a `task` row and a `resources/people/` reference record: both are reached by predicate and by key, and salience there would reward a stale task and decay a person's identity.
|
|
300
231
|
|
|
301
|
-
`memhtml recall` adds a disclosure fold on top. Arcs get their own character envelope, so a summary does
|
|
302
|
-
not compete with the memories it summarizes. Each fold quotes at most 2 memories per entity name, and
|
|
303
|
-
everything past the budget collapses to one index line plus a path to drill into.
|
|
232
|
+
`memhtml recall` adds a disclosure fold on top. Arcs get their own character envelope, so a summary does not compete with the memories it summarizes. Each fold quotes at most 2 memories per entity name, and everything past the budget collapses to one index line plus a path to drill into.
|
|
304
233
|
|
|
305
234
|
## The discrimination gate
|
|
306
235
|
|
|
307
|
-
`memhtml eval discriminate` reports the number that says whether retrieval can tell two similar facts
|
|
308
|
-
apart.
|
|
309
|
-
Embeddings are weakest on the tokens that carry a fact's polarity: "drain the VIP before reverting" and
|
|
310
|
-
"do not drain the VIP before reverting" sit above 0.99 cosine similarity while asserting opposite things.
|
|
311
|
-
So the gate derives every control from the probe's own target by flipping a negation, a number, or a
|
|
312
|
-
qualifier, which makes each control a high-cosine wrong answer by construction. Every target has to
|
|
313
|
-
strictly outrank all of its own controls, mean reciprocal rank has to clear 0.85, and one inversion fails
|
|
314
|
-
the run.
|
|
236
|
+
`memhtml eval discriminate` reports the number that says whether retrieval can tell two similar facts apart. Embeddings are weakest on the tokens that carry a fact's polarity: "drain the VIP before reverting" and "do not drain the VIP before reverting" sit above 0.99 cosine similarity while asserting opposite things. So the gate derives every control from the probe's own target by flipping a negation, a number, or a qualifier, which makes each control a high-cosine wrong answer by construction. Every target has to strictly outrank all of its own controls, mean reciprocal rank has to clear 0.85, and one inversion fails the run.
|
|
315
237
|
|
|
316
|
-
Two places run it. `pnpm check` runs it, and CI runs `pnpm check`. `memhtml sleep merge` runs it a second
|
|
317
|
-
time, so a sleep run that degrades retrieval cannot land. Fake-embedder mode is deterministic and needs
|
|
318
|
-
no credentials. `live` mode is an operator diagnostic, and it reports `skipped: true` when it cannot
|
|
319
|
-
reach the model, so a skipped gate reads as skipped rather than as green.
|
|
238
|
+
Two places run it. `pnpm check` runs it, and CI runs `pnpm check`. `memhtml sleep merge` runs it a second time, so a sleep run that degrades retrieval cannot land. Fake-embedder mode is deterministic and needs no credentials. `live` mode is an operator diagnostic, and it reports `skipped: true` when it cannot reach the model, so a skipped gate reads as skipped rather than as green.
|
|
320
239
|
|
|
321
240
|
## Sleep
|
|
322
241
|
|
|
323
|
-
`memhtml sleep run` executes
|
|
324
|
-
|
|
325
|
-
and the
|
|
326
|
-
`memhtml sleep resume` re-runs only what is missing. Two phases commit nothing by design. `preflight`
|
|
327
|
-
refreshes the index, and `relationship-mining` writes derived edges to the index alone, because thousands
|
|
328
|
-
of re-derivable edges would bury every real diff. `trace-consolidation` hands unread session transcripts
|
|
329
|
-
to an agent and lands each distilled memory as its own commit, one per memory, so a reviewer reads one
|
|
330
|
-
claim at a time. A failed phase leaves the phases before it committed.
|
|
242
|
+
`memhtml sleep run` executes sixteen curation phases on a `sleep/<date>` branch: dedup-merge, entity resolution, edge typing, confidence decay, arc synthesis, retention triage, compress, integrity, and the rest. Each committing phase makes its own isolated commit with a machine-readable trailer, so `memhtml sleep resume` re-runs only what is missing. Two phases commit nothing by design. `preflight` refreshes the index, and `relationship-mining` writes derived edges to the index alone, because thousands of re-derivable edges would bury every real diff. `trace-consolidation` hands unread session transcripts to an agent and lands each distilled memory as its own commit, one per memory, so a reviewer reads one claim at a time. A failed phase leaves the phases before it committed.
|
|
243
|
+
|
|
244
|
+
A run also opens TASKS, for work the corpus records and nobody opened. `task-detection` reads the recent memories in batches and asks which of them carry a commitment nobody closed, quoting the sentence it found; three other phases do the same for the decisions they decline to make — an alias pair too close to ignore and too far to merge, a near-duplicate pair the divergence veto refused, a contradiction seen only once. Every detected task is authored `agent:sleep`, cites its evidence verbatim, is capped at ten a night across all four detectors, and closes itself when its finding stops appearing. A detection is a proposal for a human, never a fact the corpus asserts.
|
|
331
245
|
|
|
332
|
-
`memhtml sleep review` classifies every touched file. `memhtml sleep merge` re-runs the discrimination gate
|
|
333
|
-
and refuses to move `main` on a regression. Detecting a conflict is nightly and automatic; resolving one
|
|
334
|
-
stays with the writer or a human, because choosing a winner is a one-way door.
|
|
246
|
+
`memhtml sleep review` classifies every touched file. `memhtml sleep merge` re-runs the discrimination gate and refuses to move `main` on a regression. Detecting a conflict is nightly and automatic; resolving one stays with the writer or a human, because choosing a winner is a one-way door.
|
|
335
247
|
|
|
336
|
-
Figure 4 draws the branch and the gate. A screen reader sounds out its box characters, so read the
|
|
337
|
-
paragraph beneath the figure, which carries the same content in words.
|
|
248
|
+
Figure 4 draws the branch and the gate. A screen reader sounds out its box characters, so read the paragraph beneath the figure, which carries the same content in words.
|
|
338
249
|
|
|
250
|
+
<!-- dprint-ignore-start -->
|
|
339
251
|
<!-- figure:sleep-branch -->
|
|
340
252
|
```text
|
|
341
253
|
+-------+
|
|
@@ -352,10 +264,10 @@ paragraph beneath the figure, which carries the same content in words.
|
|
|
352
264
|
+-------------+
|
|
353
265
|
|
|
|
354
266
|
v
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
267
|
+
+---------------+
|
|
268
|
+
|sixteen phases |
|
|
269
|
+
| |
|
|
270
|
+
+---------------+
|
|
359
271
|
|
|
|
360
272
|
review
|
|
361
273
|
|
|
|
@@ -376,45 +288,26 @@ paragraph beneath the figure, which carries the same content in words.
|
|
|
376
288
|
+-----------+ +-------------+
|
|
377
289
|
```
|
|
378
290
|
<!-- /figure:sleep-branch -->
|
|
291
|
+
<!-- dprint-ignore-end -->
|
|
379
292
|
|
|
380
|
-
**Figure 4: `main` moves only after a gate that can refuse says so.** A run branches `main` into
|
|
381
|
-
`sleep/<date>` before any phase executes and walks its fifteen phases there, thirteen of them committing,
|
|
382
|
-
each on its own, with `preflight` and `relationship-mining` committing nothing by design. Then it submits
|
|
383
|
-
the branch for review. That review re-runs the discrimination gate and has two outcomes, both drawn: it
|
|
384
|
-
passes and `main` moves, or it refuses and `main` stays exactly where it was. Those are the only two
|
|
385
|
-
outcomes, and neither needs a rollback, because nothing on `main` ever moved. The abort is
|
|
386
|
-
`git branch -D`.
|
|
293
|
+
**Figure 4: `main` moves only after a gate that can refuse says so.** A run branches `main` into `sleep/<date>` before any phase executes and walks its sixteen phases there, fourteen of them committing, each on its own, with `preflight` and `relationship-mining` committing nothing by design. Then it submits the branch for review. That review re-runs the discrimination gate and has two outcomes, both drawn: it passes and `main` moves, or it refuses and `main` stays exactly where it was. Those are the only two outcomes, and neither needs a rollback, because nothing on `main` ever moved. The abort is `git branch -D`.
|
|
387
294
|
|
|
388
295
|
## Code-mode
|
|
389
296
|
|
|
390
|
-
The closed vocabulary makes the corpus a queryable API with no new surface: `article mark` is always the
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
written from that spelling reports zero claims while looking correct. An agent that writes parser code
|
|
394
|
-
against `$MEMHTML_ROOT` composes multi-hop traversals in one execution and answers corpus-shaped
|
|
395
|
-
questions no tool enumerates: live contradiction pairs, an orphan census, a walk up a supersedence chain.
|
|
396
|
-
The contract is read-only, and writes stay behind the three doors above.
|
|
397
|
-
|
|
398
|
-
`memhtml exec` ships this as a command. It runs your script in a QuickJS sandbox with the corpus mounted
|
|
399
|
-
read-only at `/mnt/memhtml` and a helper preloaded at `/workspace/lib/corpus.mjs`, and it answers one
|
|
400
|
-
`exec.report` envelope. Measured on the 305-file fixture corpus: 305/305 claims parsed and 410/410 edges
|
|
401
|
-
resolved in one execution. The script reads a pinned commit, so an answer is reproducible and an
|
|
402
|
-
uncommitted edit stays invisible to it. The sandbox reaches the structural and lexical planes and holds
|
|
403
|
-
no index handle, so a script that wants ranked retrieval shells out to `memhtml search` and consumes its
|
|
404
|
-
envelope. `docs/code-mode.md` is the cookbook, with a measured helper and five recipes.
|
|
297
|
+
The closed vocabulary makes the corpus a queryable API with no new surface: `article mark` is always the claim, `link[rel^="memhtml-"]` is always an authored edge, and `dl` pairs are always facets. Use the descendant selector. The markup is `<article><p><mark>`, so `article > mark` matches nothing and a helper written from that spelling reports zero claims while looking correct. An agent that writes parser code against `$MEMHTML_ROOT` composes multi-hop traversals in one execution and answers corpus-shaped questions no tool enumerates: live contradiction pairs, an orphan census, a walk up a supersedence chain. The contract is read-only, and writes stay behind the three doors above.
|
|
298
|
+
|
|
299
|
+
`memhtml exec` ships this as a command. It runs your script in a QuickJS sandbox with the corpus mounted read-only at `/mnt/memhtml` and a helper preloaded at `/workspace/lib/corpus.mjs`, and it answers one `exec.report` envelope. Measured on the 305-file fixture corpus: 305/305 claims parsed and 410/410 edges resolved in one execution. The script reads a pinned commit, so an answer is reproducible and an uncommitted edit stays invisible to it. The sandbox reaches the structural and lexical planes and holds no index handle, so a script that wants ranked retrieval shells out to `memhtml search` and consumes its envelope. `docs/code-mode.md` is the cookbook, with a measured helper and five recipes.
|
|
405
300
|
|
|
406
301
|
## Measured
|
|
407
302
|
|
|
408
|
-
| benchmark
|
|
409
|
-
|
|
410
|
-
| MemoryAgentBench FactConsolidation single-hop (26KB to 1.1MB stores) | 92% to 97% | ~60% at 26KB only
|
|
411
|
-
| MemoryAgentBench FactConsolidation multi-hop
|
|
412
|
-
| BEAM Contradiction Resolution (100K split, 40 probes)
|
|
413
|
-
| LongMemEval-S (full 500, judged 2026-08-07)
|
|
303
|
+
| benchmark | memhtml | published reference |
|
|
304
|
+
| -------------------------------------------------------------------- | ---------- | ----------------------------------- |
|
|
305
|
+
| MemoryAgentBench FactConsolidation single-hop (26KB to 1.1MB stores) | 92% to 97% | ~60% at 26KB only |
|
|
306
|
+
| MemoryAgentBench FactConsolidation multi-hop | 37% to 49% | ≤7% all methods |
|
|
307
|
+
| BEAM Contradiction Resolution (100K split, 40 probes) | 43.8% mean | 0% to 5% all systems |
|
|
308
|
+
| LongMemEval-S (full 500, judged 2026-08-07) | 67.0% | ~55% to 65% typical agent baselines |
|
|
414
309
|
|
|
415
|
-
Read the cross-judge numbers as reference points rather than as a ranking: the judges here are verbatim
|
|
416
|
-
prompt ports running haiku-4.5, where the papers used gpt-4o and gpt-4.1-mini. `ROADMAP.md` carries these
|
|
417
|
-
numbers and the horizons they rank.
|
|
310
|
+
Read the cross-judge numbers as reference points rather than as a ranking: the judges here are verbatim prompt ports running haiku-4.5, where the papers used gpt-4o and gpt-4.1-mini. `ROADMAP.md` carries these numbers and the horizons they rank.
|
|
418
311
|
|
|
419
312
|
## Layout
|
|
420
313
|
|
|
@@ -437,38 +330,29 @@ $MEMHTML_ROOT/ # its own git repo, one global memory stor
|
|
|
437
330
|
|
|
438
331
|
## Packages
|
|
439
332
|
|
|
440
|
-
The layering is strict and TypeScript project references enforce it. `@memhtml/contracts` and
|
|
441
|
-
|
|
442
|
-
names
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
|
452
|
-
|
|
453
|
-
| `@memhtml/
|
|
454
|
-
| `@memhtml/
|
|
455
|
-
| `@memhtml/
|
|
456
|
-
| `@memhtml/
|
|
457
|
-
| `@memhtml/
|
|
458
|
-
| `@memhtml/
|
|
459
|
-
| `@memhtml/sleep` | The fifteen curation phases, each an isolated commit. |
|
|
460
|
-
| `@memhtml/llm` | Bedrock: Cohere embeddings and forced-tool structured output. |
|
|
461
|
-
| `@memhtml/eval` | The fixture corpus generator and the refusable discrimination gate. |
|
|
462
|
-
| `@memhtml/cli` | The `memhtml` binary, the envelope contract, and the one composition root. |
|
|
463
|
-
| `@memhtml/mcp` | The `memhtml-mcp` stdio server: 14 tools, 2 resources. |
|
|
464
|
-
| `@memhtml/consolidator` | The sandboxed eve agent that distills candidate memories from raw transcripts. |
|
|
465
|
-
| `@memhtml/docs` | The documentation site. |
|
|
333
|
+
The layering is strict and TypeScript project references enforce it. `@memhtml/contracts` and `@memhtml/domain` import `effect` and nothing else, and a test reads `domain`'s own `dist` to confirm it names no database driver, no SDK, and no `node:fs`.
|
|
334
|
+
|
|
335
|
+
None of them is published. Every workspace package is `private`, and `mise run package:assemble` bundles the libraries and the binary-bearing apps into the single `memhtml` package that carries the two binaries — the docs site and the integration-test harness stay outside the bundle (`tsdown.config.ts` names the exact set). The table below is a map of the source, not a list of things to install. `RELEASING.md` covers how the artifact is built and what must stay outside the bundle.
|
|
336
|
+
|
|
337
|
+
| Package | What it owns |
|
|
338
|
+
| ----------------------- | ------------------------------------------------------------------------------- |
|
|
339
|
+
| `@memhtml/contracts` | Schemas, the closed vocabularies, errors, path algebra. Zero I/O. |
|
|
340
|
+
| `@memhtml/domain` | Pure math: retention, decay, RRF, MMR, PageRank, the anti-merge guards. |
|
|
341
|
+
| `@memhtml/html` | The memory file format: parse, serialize, hash, surgical head editors. |
|
|
342
|
+
| `@memhtml/store` | The git-backed file store. One commit per operation, typed conflicts. |
|
|
343
|
+
| `@memhtml/index` | SQLite schema, the git-driven indexer, four-arm RRF retrieval, the state plane. |
|
|
344
|
+
| `@memhtml/traces` | Streaming JSONL parser over `~/.claude`, with a size+mtime+offset watermark. |
|
|
345
|
+
| `@memhtml/sleep` | The sixteen curation phases, each an isolated commit. |
|
|
346
|
+
| `@memhtml/llm` | Bedrock: Cohere embeddings and forced-tool structured output. |
|
|
347
|
+
| `@memhtml/eval` | The fixture corpus generator and the refusable discrimination gate. |
|
|
348
|
+
| `@memhtml/cli` | The `memhtml` binary, the envelope contract, and the one composition root. |
|
|
349
|
+
| `@memhtml/mcp` | The `memhtml-mcp` stdio server: 14 tools, 2 resources. |
|
|
350
|
+
| `@memhtml/consolidator` | The sandboxed eve agent that distills candidate memories from raw transcripts. |
|
|
351
|
+
| `@memhtml/docs` | The documentation site. |
|
|
466
352
|
|
|
467
353
|
## Development
|
|
468
354
|
|
|
469
|
-
[`mise`](https://mise.jdx.dev) is the command surface. It installs the toolchain from `mise.toml`, which
|
|
470
|
-
declares node, pnpm, lefthook, and the scanners, each pinned by checksum and provenance in the committed
|
|
471
|
-
`mise.lock`, so a clone resolves the same binaries CI does:
|
|
355
|
+
[`mise`](https://mise.jdx.dev) is the command surface. It installs the toolchain from `mise.toml`, which declares node, pnpm, lefthook, and the scanners, each pinned by checksum and provenance in the committed `mise.lock`, so a clone resolves the same binaries CI does:
|
|
472
356
|
|
|
473
357
|
```bash
|
|
474
358
|
mise install # node 24, pnpm 11.21.0, lefthook, scanners, from mise.lock
|
|
@@ -476,60 +360,45 @@ mise run install # dependencies from the lockfile + the git hooks
|
|
|
476
360
|
mise run check # the definition of done: lint, typecheck, tests, integration, eval, a11y, budget
|
|
477
361
|
```
|
|
478
362
|
|
|
479
|
-
CI runs that same `mise run check`, so the gate cannot drift from the one you run locally. Every task
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
|
|
490
|
-
| Command | Delegates to | What it runs |
|
|
491
|
-
|---|---|---|
|
|
492
|
-
| `mise run build` | `pnpm build` | `tsc -b` across the project graph |
|
|
493
|
-
| `mise run lint` | `pnpm lint` | biome |
|
|
494
|
-
| `mise run typecheck` | `pnpm typecheck` | strict `tsc --noEmit`, tests included |
|
|
495
|
-
| `mise run test` | `pnpm test` | every package's unit and property suites |
|
|
363
|
+
CI runs that same `mise run check`, so the gate cannot drift from the one you run locally. Every task delegates to the pnpm script underneath it, and turbo owns the task graph and the cache. No mise task declares `sources` or `outputs`, because mise decides freshness by mtime and turbo by content hash, so a mise-level skip would preempt turbo's per-package hashing.
|
|
364
|
+
|
|
365
|
+
`check` includes the discrimination gate in fake mode, so a change that degrades retrieval fails the build. Tests run against a real temp-dir git repo and a real SQLite database with the shipped migrations. Fakes are limited to the two edges that reach the network, the embedder and the model, because a stateless fake verifies the shape of a call and misses the state semantics behind it, which is where the defects in this system have actually lived.
|
|
366
|
+
|
|
367
|
+
| Command | Delegates to | What it runs |
|
|
368
|
+
| --------------------------- | ----------------------- | ---------------------------------------------------------------- |
|
|
369
|
+
| `mise run build` | `pnpm build` | `tsc -b` across the project graph |
|
|
370
|
+
| `mise run lint` | `pnpm lint` | biome |
|
|
371
|
+
| `mise run typecheck` | `pnpm typecheck` | strict `tsc --noEmit`, tests included |
|
|
372
|
+
| `mise run test` | `pnpm test` | every package's unit and property suites |
|
|
496
373
|
| `mise run test:integration` | `pnpm test:integration` | the cross-package contracts over a real repo and a real database |
|
|
497
|
-
| `mise run test:eval`
|
|
498
|
-
| `mise run test:a11y`
|
|
499
|
-
| `mise run test:budget`
|
|
500
|
-
| `mise run gen:fixture`
|
|
501
|
-
| `mise run agents-doc`
|
|
502
|
-
| `mise run security`
|
|
503
|
-
| `mise run tools:bump`
|
|
374
|
+
| `mise run test:eval` | `pnpm test:eval` | the discrimination gate (fake mode) |
|
|
375
|
+
| `mise run test:a11y` | `pnpm test:a11y` | WCAG 2.2 AA over the built docs site, in a real browser |
|
|
376
|
+
| `mise run test:budget` | `pnpm test:budget` | Lighthouse category floors and the byte budget for that site |
|
|
377
|
+
| `mise run gen:fixture` | `pnpm gen:fixture` | write a browsable fixture corpus (pure function of a seed) |
|
|
378
|
+
| `mise run agents-doc` | none | regenerate `AGENTS.md` from the built CLI's own table |
|
|
379
|
+
| `mise run security` | none | osv-scanner + semgrep + betterleaks, SARIF into `.sarif/` |
|
|
380
|
+
| `mise run tools:bump` | none | re-resolve every `latest` tool in `mise.lock` |
|
|
504
381
|
|
|
505
|
-
To narrow a run to one package, use `mise run test-pkg <package> [vitest args]`. The package name takes
|
|
506
|
-
either spelling, and everything after it goes to vitest:
|
|
382
|
+
To narrow a run to one package, use `mise run test-pkg <package> [vitest args]`. The package name takes either spelling, and everything after it goes to vitest:
|
|
507
383
|
|
|
508
384
|
```bash
|
|
509
385
|
mise run test-pkg domain rrf -t "strictly" # one test
|
|
510
386
|
mise run test-pkg index retrieval # one file
|
|
511
387
|
```
|
|
512
388
|
|
|
513
|
-
That path goes straight to the package's vitest, so it skips turbo and builds nothing first. Every
|
|
514
|
-
`@memhtml/*` package's exports resolve only to `./dist`, so run `mise run build` after editing another
|
|
515
|
-
package's `src/`.
|
|
389
|
+
That path goes straight to the package's vitest, so it skips turbo and builds nothing first. Every `@memhtml/*` package's exports resolve only to `./dist`, so run `mise run build` after editing another package's `src/`.
|
|
516
390
|
|
|
517
|
-
`mise.toml`'s `[tools] pnpm` and `package.json`'s `packageManager` both declare the pnpm that runs, and
|
|
518
|
-
neither can be derived from the other. `mise run tools:verify` fails when they disagree, and
|
|
519
|
-
`mise run install` depends on it, so the check runs on every install. Declaration order matters too:
|
|
520
|
-
`pnpm` sits above `node` because node's own bin holds a `pnpm` symlink into corepack wherever
|
|
521
|
-
`corepack enable` has run, and with node first every pnpm call would resolve to corepack rather than to
|
|
522
|
-
the pinned binary.
|
|
391
|
+
`mise.toml`'s `[tools] pnpm` and `package.json`'s `packageManager` both declare the pnpm that runs, and neither can be derived from the other. `mise run tools:verify` fails when they disagree, and `mise run install` depends on it, so the check runs on every install. Declaration order matters too: `pnpm` sits above `node` because node's own bin holds a `pnpm` symlink into corepack wherever `corepack enable` has run, and with node first every pnpm call would resolve to corepack rather than to the pinned binary.
|
|
523
392
|
|
|
524
393
|
## Docs
|
|
525
394
|
|
|
526
|
-
| File
|
|
527
|
-
|
|
528
|
-
| `AGENTS.md`
|
|
529
|
-
| `RUNBOOK.md`
|
|
530
|
-
| `ROADMAP.md`
|
|
531
|
-
| `docs/design.md`
|
|
532
|
-
| `docs/format.md`
|
|
533
|
-
| `docs/tasks.md`
|
|
534
|
-
| `docs/code-mode.md` | Navigating the corpus with code
|
|
535
|
-
| `docs/backlog.md`
|
|
395
|
+
| File | What it holds |
|
|
396
|
+
| ------------------- | --------------------------------------------------------------- |
|
|
397
|
+
| `AGENTS.md` | The full command surface, generated from the binary's own table |
|
|
398
|
+
| `RUNBOOK.md` | Operating the store day to day |
|
|
399
|
+
| `ROADMAP.md` | The system-level view: measured standing, ranked horizons |
|
|
400
|
+
| `docs/design.md` | Every architectural decision with its evidence |
|
|
401
|
+
| `docs/format.md` | The file format and the closed vocabulary |
|
|
402
|
+
| `docs/tasks.md` | The task memory type |
|
|
403
|
+
| `docs/code-mode.md` | Navigating the corpus with code |
|
|
404
|
+
| `docs/backlog.md` | The fine-grained ledger |
|