plugmem 0.4.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +527 -196
- package/index.d.ts +231 -4
- package/index.js +3 -1
- package/package.json +8 -8
package/README.md
CHANGED
|
@@ -1,20 +1,26 @@
|
|
|
1
|
-
# plugmem
|
|
1
|
+
# plugmem
|
|
2
2
|
|
|
3
3
|
> ⚠️ Experimental. plugmem is mostly an AI-built experiment — written with
|
|
4
4
|
> the help of a small local model (Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) and various
|
|
5
5
|
> Claude models, in roughly equal measure. Expect non-professional design
|
|
6
6
|
> choices, rough edges, broken behavior, or mistakes. Use it at your own risk.
|
|
7
7
|
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
(real mmap, file locking, cross-process MVCC — the whole engine, unchanged) and
|
|
12
|
-
exposes it to **JavaScript / TypeScript** as a `Plugmem` class. It is published
|
|
13
|
-
to npm as **`plugmem`**.
|
|
8
|
+
An embeddable bitemporal memory database for local-first applications and
|
|
9
|
+
agents, embedded in your Node process. It stores short facts and answers a
|
|
10
|
+
query with ranked facts and edges plus an optional bounded rendered block.
|
|
14
11
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
12
|
+
File-backed on disk, no server, no daemon. It links into your process the way
|
|
13
|
+
SQLite does: the engine is [`plugmem-host`](https://docs.rs/plugmem-host/latest)
|
|
14
|
+
compiled to a native addon through [napi-rs](https://napi.rs), so there is no
|
|
15
|
+
WebAssembly copy of the file in RAM and no 4 GiB ceiling. Runs on Node, Deno and
|
|
16
|
+
Bun.
|
|
17
|
+
|
|
18
|
+
**Contents:** [Install](#install) · [Quick start](#quick-start) ·
|
|
19
|
+
[What it stores](#what-it-stores) · [Two clocks](#two-clocks) ·
|
|
20
|
+
[How recall works](#how-recall-works) · [API](#api) · [Errors](#errors) ·
|
|
21
|
+
[Configuration](#configuration-and-embeddings) ·
|
|
22
|
+
[Async](#async-and-the-event-loop) · [Many memories](#many-memories-in-one-directory) ·
|
|
23
|
+
[What it is not for](#what-it-is-not-for)
|
|
18
24
|
|
|
19
25
|
## Install
|
|
20
26
|
|
|
@@ -22,115 +28,324 @@ Rust library. It loads in **Node, and any N-API host (Deno, Bun)**.
|
|
|
22
28
|
$ npm install plugmem
|
|
23
29
|
```
|
|
24
30
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
31
|
+
That pulls a meta package which, through `optionalDependencies`, installs only
|
|
32
|
+
the prebuilt binary for your platform: one of `plugmem-{linux-x64-gnu,
|
|
33
|
+
linux-arm64-gnu, darwin-x64, darwin-arm64, win32-x64-msvc, win32-arm64-msvc}`.
|
|
34
|
+
No toolchain, no build step.
|
|
35
|
+
|
|
36
|
+
## Quick start
|
|
37
|
+
|
|
38
|
+
```typescript
|
|
39
|
+
import { Plugmem } from "plugmem";
|
|
40
|
+
|
|
41
|
+
const db = await Plugmem.open("agent.plugmem");
|
|
42
|
+
|
|
43
|
+
await db.remember({ text: "the user prefers tokio", entity: "user", tags: ["pref"] });
|
|
44
|
+
await db.remember({ text: "the release ships on friday", entity: "release" });
|
|
45
|
+
|
|
46
|
+
const res = await db.recall({ query: "which runtime?", k: 5 });
|
|
47
|
+
console.log(res.rendered); // paste this into the prompt
|
|
48
|
+
// - [f0] user: the user prefers tokio (2026-08; active)
|
|
49
|
+
|
|
50
|
+
db.close();
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
`Plugmem.open` is a static method, not a constructor, because opening replays a
|
|
54
|
+
journal and maps a snapshot — work proportional to the file — and a JavaScript
|
|
55
|
+
constructor has no way to hand that to a worker thread. Everything is typed:
|
|
56
|
+
`index.d.ts` is generated from the Rust, so a TypeScript host gets real
|
|
57
|
+
autocomplete on arguments and results.
|
|
58
|
+
|
|
59
|
+
## What it stores
|
|
60
|
+
|
|
61
|
+
One **fact** is one statement. It carries:
|
|
62
|
+
|
|
63
|
+
| Field | Meaning |
|
|
64
|
+
|---|---|
|
|
65
|
+
| `text` | the statement itself, and what lexical search indexes |
|
|
66
|
+
| `entity` | the subject, by name — created on first mention, shared across facts |
|
|
67
|
+
| `tags` | filters, not ranking: a query asking for a tag requires it |
|
|
68
|
+
| `metadata` | an opaque `Record<string,string>`. The engine stores and returns it and never looks inside — use it for a URI to the real payload elsewhere, a mime type, an external key |
|
|
69
|
+
| `vector` | an optional embedding; supply your own, or let a configured embedder produce it |
|
|
70
|
+
| `validFrom` | when the statement became true |
|
|
71
|
+
|
|
72
|
+
Entities are joined by **typed edges**: `link({ src: "ann", rel: "hires", dst: "bob" })`.
|
|
73
|
+
An edge can name the fact it follows from (`provenance`), so a later reader can
|
|
74
|
+
answer "why does the memory think this" instead of trusting a bare relationship.
|
|
75
|
+
|
|
76
|
+
Facts are never rewritten in place. `revise` closes the old one and chains a
|
|
77
|
+
successor; `forget` tombstones a fact and the next `maintain` erases it from
|
|
78
|
+
disk. `unlink` closes an edge the same way `revise` closes a fact.
|
|
29
79
|
|
|
30
|
-
##
|
|
80
|
+
## Two clocks
|
|
31
81
|
|
|
32
|
-
|
|
82
|
+
This is the part worth reading, because it is the one thing that behaves
|
|
83
|
+
differently from every other store.
|
|
33
84
|
|
|
34
|
-
|
|
35
|
-
|---|---|---|
|
|
36
|
-
| **writing JavaScript / TypeScript for Node** | **`plugmem-napi`** (this, npm `plugmem`) | The engine *in your Node process*, native speed, typed for TS. |
|
|
37
|
-
| **writing Rust** | [`plugmem-host`](https://docs.rs/plugmem-host/latest) | The engine in your process, like linking SQLite. |
|
|
38
|
-
| **an agent, or another language** (Python, Go…) | [`plugmem-mcp`](https://docs.rs/plugmem-mcp/latest) | A long-lived stdio JSON-RPC sidecar; language-independent. |
|
|
39
|
-
| a person at a **terminal / script** | [`plugmem-cli`](https://docs.rs/plugmem-cli/latest) | The human door. |
|
|
85
|
+
Every fact carries **two** timestamps, not one:
|
|
40
86
|
|
|
41
|
-
|
|
42
|
-
the
|
|
87
|
+
- **`validFrom` / `validTo`** — when the statement was true.
|
|
88
|
+
- **`recordedAt`** — when the memory learned it. Set by the engine, never by you.
|
|
43
89
|
|
|
44
|
-
|
|
90
|
+
They are different questions, and one timestamp cannot hold both.
|
|
45
91
|
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
`Plugmem.open`, not `new`: opening replays a journal and maps a snapshot, and a
|
|
49
|
-
JavaScript constructor has no way to hand that to a worker thread.
|
|
92
|
+
Unlike the Rust library, this binding reads the system clock on every call — you
|
|
93
|
+
never pass `now` — so `recordedAt` is always the moment of the write:
|
|
50
94
|
|
|
51
95
|
```typescript
|
|
52
|
-
|
|
96
|
+
await db.remember({ text: "lives in Moscow", entity: "kim" });
|
|
97
|
+
const between = Date.now();
|
|
98
|
+
await db.revise(0, { text: "lives in Berlin", entity: "kim" });
|
|
53
99
|
|
|
54
|
-
|
|
100
|
+
(await db.recall({ entities: ["kim"] })).rendered;
|
|
101
|
+
// - [f1] kim: lives in Berlin (2026-08; active)
|
|
55
102
|
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
tags: ["pref"],
|
|
60
|
-
links: [{ rel: "works_at", entity: "acme" }],
|
|
61
|
-
metadata: { source: "chat", uri: "s3://bucket/note.txt" }, // opaque; a pointer
|
|
62
|
-
});
|
|
63
|
-
out.id; // number
|
|
64
|
-
out.similar; // Similar[] — the engine surfaces conflicts, you decide
|
|
103
|
+
(await db.recall({ entities: ["kim"], asOf: between })).rendered;
|
|
104
|
+
// - [f0] kim: lives in Moscow (2026-08 → 2026-08; closed)
|
|
105
|
+
```
|
|
65
106
|
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
res.facts; // RecalledFact[] — { id, score, entity, recordedAt, … }
|
|
107
|
+
`revise` closed the first fact's interval rather than deleting it, which is why
|
|
108
|
+
the second query has something to answer with.
|
|
69
109
|
|
|
70
|
-
|
|
71
|
-
|
|
110
|
+
`asOf` moves **both** clocks: a fact answers only if it was valid at that
|
|
111
|
+
instant *and* had already been recorded by then. The second half is the one
|
|
112
|
+
people trip over — an `asOf` earlier than a fact's `recordedAt` sees nothing,
|
|
113
|
+
because the memory genuinely knew nothing then. Answering with today's knowledge
|
|
114
|
+
would be the wrong answer to "what did I hold".
|
|
72
115
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
116
|
+
`validFrom` is the other half: a statement that became true before you heard of
|
|
117
|
+
it. Recording today that someone moved a week ago closes the previous interval a
|
|
118
|
+
week ago rather than now, so a query as of three days back finds **neither** —
|
|
119
|
+
the old fact had stopped being true, and the new one was not yet known. That is
|
|
120
|
+
not a hole in the model; it is the honest answer for that instant, and it is
|
|
121
|
+
what a single timestamp cannot express.
|
|
76
122
|
|
|
77
|
-
|
|
78
|
-
|
|
123
|
+
Two more queries over the same axes:
|
|
124
|
+
|
|
125
|
+
```typescript
|
|
126
|
+
await db.recall({ range: [from, to] }); // what did I record in this window
|
|
127
|
+
await db.recall({ query: "kim", closed: true }); // include closed revisions
|
|
79
128
|
```
|
|
80
129
|
|
|
81
|
-
|
|
130
|
+
Use `revise` when something changed and `forget` only when a fact was simply
|
|
131
|
+
wrong: `forget` destroys the "what was true then" answer, `revise` keeps it.
|
|
132
|
+
|
|
133
|
+
Edges are temporal too, so `asOf` walks the graph as it stood then — through
|
|
134
|
+
relationships that have since been unlinked.
|
|
135
|
+
|
|
136
|
+
## How recall works
|
|
137
|
+
|
|
138
|
+
Not a vector lookup. Four sources run and are fused by
|
|
139
|
+
[reciprocal-rank fusion](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf)
|
|
140
|
+
with a recency boost; tags filter and are not a source:
|
|
82
141
|
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
142
|
+
| Source | What it finds |
|
|
143
|
+
|---|---|
|
|
144
|
+
| **Lexical** — [BM25](https://en.wikipedia.org/wiki/Okapi_BM25) over a Unicode ([UAX #29](https://unicode.org/reports/tr29/)) tokenizer | exact terms, keyword overlap |
|
|
145
|
+
| **Semantic** — int8-quantized cosine, a flat scan below a threshold and an [HNSW](https://arxiv.org/abs/1603.09320) graph above | meaning, nearest neighbours |
|
|
146
|
+
| **Graph** — typed edges walked from the query's anchor entities | relational knowledge |
|
|
147
|
+
| **Temporal** — range scans over the `recordedAt` index, plus the validity test | "what was true then", time windows |
|
|
148
|
+
|
|
149
|
+
The sources compose. A query with no `query` string still answers from tags,
|
|
150
|
+
entities and time. **Without an embedder the system is complete** — the other
|
|
151
|
+
three sources need no model, no network and no API key.
|
|
152
|
+
|
|
153
|
+
The result carries both a `rendered` block, selected greedily under a token
|
|
154
|
+
budget and ready to paste, and the structured `facts`/`edges` behind it:
|
|
89
155
|
|
|
90
156
|
```typescript
|
|
91
|
-
const
|
|
157
|
+
const res = await db.recall({
|
|
158
|
+
query: "release plans",
|
|
159
|
+
entities: ["ann"], // graph anchors
|
|
160
|
+
tags: ["work"], // filter: a fact must carry all of these
|
|
161
|
+
k: 10, // cap the number of facts
|
|
162
|
+
tokenBudget: 400, // cap the size of the block — your context budget
|
|
163
|
+
graphDepth: 3, // how far to walk from the anchors (default 2)
|
|
164
|
+
});
|
|
165
|
+
|
|
166
|
+
res.rendered; // string, prompt-ready
|
|
167
|
+
res.facts; // { id, score, entity, recordedAt, validFrom, validTo, sources }[]
|
|
168
|
+
res.edges; // { src, rel, dst, provenance }[] — what the graph walked
|
|
169
|
+
res.truncated; // true if selection stopped at k or the budget with more left
|
|
92
170
|
```
|
|
93
171
|
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
path = "/path/to/memory.plugmem" # optional example
|
|
172
|
+
`remember` returns the new fact's id **plus any live facts it may duplicate or
|
|
173
|
+
contradict**. The engine never merges or deletes on its own — it surfaces the
|
|
174
|
+
tension and your code decides:
|
|
98
175
|
|
|
99
|
-
|
|
100
|
-
|
|
176
|
+
```typescript
|
|
177
|
+
const out = await db.remember({ text: "the user prefers async-std", entity: "user" });
|
|
178
|
+
for (const s of out.similar) {
|
|
179
|
+
// s.id, s.score, s.reason: "LexicalOverlap" | "VectorCosine"
|
|
180
|
+
// → revise it, forget it, or keep both. Your call.
|
|
181
|
+
}
|
|
182
|
+
```
|
|
101
183
|
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
184
|
+
## API
|
|
185
|
+
|
|
186
|
+
Every method wraps the identically-named verb of the Rust `Database`; this layer
|
|
187
|
+
only moves arguments and results across the boundary.
|
|
188
|
+
|
|
189
|
+
**Writing** — all return promises:
|
|
190
|
+
|
|
191
|
+
| Method | Does |
|
|
192
|
+
|---|---|
|
|
193
|
+
| `remember(args)` | store a fact; resolves with its id and similar facts |
|
|
194
|
+
| `rememberMany(args[])` | store a batch: one embedding round-trip, one journal sync |
|
|
195
|
+
| `revise(id, args)` | close a fact and record its successor |
|
|
196
|
+
| `forget(id)` | tombstone a fact; resolves with whether it was live |
|
|
197
|
+
| `link(args)` | upsert a typed edge, optionally with `provenance` |
|
|
198
|
+
| `unlink(args)` | close the current edge; resolves with whether one was open |
|
|
199
|
+
|
|
200
|
+
**Reading** — synchronous ones touch mapped memory and return in microseconds:
|
|
201
|
+
|
|
202
|
+
| Method | Does |
|
|
203
|
+
|---|---|
|
|
204
|
+
| `recall(args?)` | ranked, fused, token-budgeted result (async) |
|
|
205
|
+
| `get(id)` | one fact's full card, or `null` (sync) |
|
|
206
|
+
| `tagsOf(id)` | that fact's tags (sync) |
|
|
207
|
+
| `stats()` | engine size counters (sync) |
|
|
208
|
+
| `path()` | the file this handle resolved to (sync) |
|
|
209
|
+
| `export()` | every open fact as one array (async, unbounded — see below) |
|
|
210
|
+
| `exportPage(cursor?)` | the same data in bounded pages of 128 (async) |
|
|
211
|
+
| `exportEdges(onBatch)` | every current edge, streamed in batches (async) |
|
|
212
|
+
| `configWarnings()` | anything in `config.toml` nothing claimed (sync) |
|
|
213
|
+
|
|
214
|
+
**Upkeep** — all async, all on a worker thread:
|
|
215
|
+
|
|
216
|
+
| Method | Does |
|
|
217
|
+
|---|---|
|
|
218
|
+
| `maintain(mode?)` | `"auto"` (default), `"compact"`, `"reindex-text"`, `"optimize-vectors"`, `"full"`. No mode ever drops a revision or an edge version |
|
|
219
|
+
| `checkpoint()` | flush the journal into a fresh snapshot |
|
|
220
|
+
| `verify()` | full content-integrity sweep; rejects on the first inconsistency |
|
|
221
|
+
| `scrub(options?)` | start a resumable byte-level check of the snapshot |
|
|
222
|
+
| `recover(src, dst, options?)` | module function: salvage a damaged file into a clean copy |
|
|
223
|
+
|
|
224
|
+
**Read-only handles** (`{ readOnly: true }`) observe another process's writer
|
|
225
|
+
over a published snapshot. The read verbs answer, the write verbs throw, and two
|
|
226
|
+
more appear: `generation()` (the pinned snapshot number) and `refresh()` (adopt
|
|
227
|
+
the writer's latest checkpoint, returning whether a newer one existed).
|
|
228
|
+
|
|
229
|
+
`close()` releases the file and its lock; every verb afterwards throws, and
|
|
230
|
+
calling it twice is a no-op.
|
|
231
|
+
|
|
232
|
+
### Bringing your own embedding
|
|
233
|
+
|
|
234
|
+
`remember`, `revise`, `rememberMany` and `recall` all take an optional `vector`
|
|
235
|
+
whose length must equal the configured `dim`. Given one, it **replaces** the
|
|
236
|
+
embedder for that call — nothing is sent to the provider:
|
|
237
|
+
|
|
238
|
+
```typescript
|
|
239
|
+
const own = await myEmbedder(text);
|
|
240
|
+
await db.remember({ text, vector: own });
|
|
241
|
+
const res = await db.recall({ query: text, vector: own });
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
Use it for vectors you already have, for a model that is not an OpenAI-shaped
|
|
245
|
+
HTTP endpoint, or for a deterministic test with no network.
|
|
246
|
+
|
|
247
|
+
### Backing up: facts are only half of it
|
|
248
|
+
|
|
249
|
+
`export`/`exportPage` dump facts. An **edge** is a statement between two
|
|
250
|
+
entities — `kim -works_on-> plugmem` — and belongs to no single fact, so a dump
|
|
251
|
+
of facts alone loses the graph. `exportEdges` is the other half.
|
|
252
|
+
|
|
253
|
+
It streams: the walk runs on a worker and hands your callback one batch at a
|
|
254
|
+
time, so memory stays flat whether the graph has ten edges or ten million. When
|
|
255
|
+
a callback is slower than the walk, the *worker* waits — never the event loop.
|
|
256
|
+
|
|
257
|
+
```javascript
|
|
258
|
+
const edges = [];
|
|
259
|
+
const count = await db.exportEdges((batch) => edges.push(...batch));
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
`count` is `2` here, and `edges` is complete the moment the promise resolves —
|
|
263
|
+
no extra tick needed:
|
|
264
|
+
|
|
265
|
+
```json
|
|
266
|
+
[
|
|
267
|
+
{ "src": "kim", "rel": "works_on", "dst": "plugmem", "provenance": 0 },
|
|
268
|
+
{ "src": "kim", "rel": "reports_to", "dst": "ann" }
|
|
269
|
+
]
|
|
106
270
|
```
|
|
107
271
|
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
272
|
+
`provenance` is the fact the edge follows from, when it was recorded with one.
|
|
273
|
+
It is **absent** rather than zero when there is none, so it can never be
|
|
274
|
+
mistaken for fact `0` — as the second edge above shows.
|
|
275
|
+
|
|
276
|
+
### Checking a file has not rotted
|
|
277
|
+
|
|
278
|
+
`verify()` and `scrub()` ask different questions, and neither replaces the other:
|
|
279
|
+
|
|
280
|
+
- **`verify()`** — does the *content* agree with itself? Text is valid UTF-8,
|
|
281
|
+
each vector belongs to its fact, both directions of every edge match.
|
|
282
|
+
- **`scrub()`** — are the *bytes* the ones that were written? It recomputes each
|
|
283
|
+
section's checksum and the whole-file hash. This is what catches a flipped bit
|
|
284
|
+
that the structure happily accepts.
|
|
285
|
+
|
|
286
|
+
A scrub is paced by you rather than run in one go, so it stays affordable on a
|
|
287
|
+
live database — the model ZFS uses. Each step checks up to a budget's worth of
|
|
288
|
+
bytes and returns:
|
|
289
|
+
|
|
290
|
+
```javascript
|
|
291
|
+
const scrub = await db.scrub(); // default budget: 1 MiB per step
|
|
292
|
+
let step;
|
|
293
|
+
while ((step = await scrub.next()) !== null) {
|
|
294
|
+
// step.doneBytes of step.totalBytes — progress through the snapshot file
|
|
295
|
+
}
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
`next()` returns a promise because a step reads from disk, not because hashing
|
|
299
|
+
is slow: over a memory-mapped file the bytes are paged in as they are read, so a
|
|
300
|
+
step is I/O of whatever length your storage takes. On the JS thread that would
|
|
301
|
+
freeze the process.
|
|
302
|
+
|
|
303
|
+
Two things to know. **Holding the object holds a lock** on the snapshot
|
|
304
|
+
generation it is scanning, so run it to completion or `close()` it. And it is
|
|
305
|
+
one-shot: after it returns `null`, or throws, `active()` is `false` and you ask
|
|
306
|
+
the database for another.
|
|
307
|
+
|
|
308
|
+
```javascript
|
|
309
|
+
const partial = await db.scrub({ budget: 16 * 1024 });
|
|
310
|
+
await partial.next(); // { doneBytes: 16384, totalBytes: <the file's size> }
|
|
311
|
+
partial.close(); // released; further next() calls return null
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
Damage rejects with `PLUGMEM_ENGINE` naming what failed its checksum.
|
|
315
|
+
|
|
316
|
+
### Repairing a damaged file
|
|
317
|
+
|
|
318
|
+
`recover` is a module function, not a method: it works on **paths**, and takes
|
|
319
|
+
the source's exclusive lock, so close your handle first.
|
|
320
|
+
|
|
321
|
+
```javascript
|
|
322
|
+
import { recover } from "plugmem";
|
|
323
|
+
|
|
324
|
+
const report = await recover("memory.plugmem", "repaired.plugmem");
|
|
325
|
+
// { kept: 1, droppedText: 0, droppedVector: 0, droppedMetadata: 0 }
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
**The source is never written.** It stays exactly as it was, as evidence; this
|
|
329
|
+
produces a repaired copy beside it, and swapping them is your decision. `dst`
|
|
330
|
+
must therefore be a different path — passing the same one throws.
|
|
331
|
+
|
|
332
|
+
The three `dropped` counts are the damage: each is a fact the source could not
|
|
333
|
+
produce intact. All zero means the image was content-clean and this was a
|
|
334
|
+
compaction. Memory stays proportional to the record count rather than the file,
|
|
335
|
+
so a database far larger than RAM can be recovered.
|
|
336
|
+
|
|
337
|
+
It handles **content** damage — the kind `verify()` reports. A snapshot whose
|
|
338
|
+
container will not parse at all is not salvageable here; that is what a backup
|
|
339
|
+
is for.
|
|
340
|
+
|
|
341
|
+
### The one thing the Rust library has and this does not
|
|
342
|
+
|
|
343
|
+
`import`. The JSONL dump format is defined by
|
|
344
|
+
[`plugmem-cli`](https://docs.rs/plugmem-cli/latest), not by the engine — there is
|
|
345
|
+
no `import` verb to mirror. A Node program holding records already has
|
|
346
|
+
`rememberMany` and `link`, which is what an importer is made of.
|
|
347
|
+
|
|
348
|
+
## Errors
|
|
134
349
|
|
|
135
350
|
Every failure plugmem itself decides carries a stable `code`, so a program
|
|
136
351
|
branches on it instead of on wording:
|
|
@@ -149,153 +364,269 @@ try {
|
|
|
149
364
|
`PLUGMEM_INVALID_NAME` from an argument that was refused; `PLUGMEM_CLOSED`,
|
|
150
365
|
`PLUGMEM_READ_ONLY`, `PLUGMEM_WRITER_ONLY` and `PLUGMEM_BUSY` from calling a
|
|
151
366
|
verb the handle cannot serve; `PLUGMEM_ENGINE` from the engine itself, carrying
|
|
152
|
-
|
|
367
|
+
its own message.
|
|
153
368
|
|
|
154
369
|
The code is there whether the verb threw or the promise rejected — the two are
|
|
155
370
|
the same contract, so nothing has to be handled twice.
|
|
156
371
|
|
|
157
|
-
An argument that
|
|
158
|
-
be exactly `[from, to]`, and `range`, `asOf` and `validFrom` must each be a
|
|
159
|
-
finite, non-negative instant. Silently ignoring one
|
|
160
|
-
without it
|
|
372
|
+
An argument that would shape an answer is refused rather than dropped: `range`
|
|
373
|
+
must be exactly `[from, to]`, and `range`, `asOf` and `validFrom` must each be a
|
|
374
|
+
finite, non-negative instant. Silently ignoring one produces an answer computed
|
|
375
|
+
without it, indistinguishable from a correct one.
|
|
161
376
|
|
|
162
|
-
|
|
377
|
+
## Configuration and embeddings
|
|
163
378
|
|
|
164
|
-
|
|
165
|
-
|
|
379
|
+
Settings resolve from an explicit `config` path, then `$PLUGMEM_CONFIG`, then
|
|
380
|
+
the platform config directory, then defaults. The database path resolves from an
|
|
381
|
+
explicit argument, then `$PLUGMEM_DB`, then `[database].path`, then the platform
|
|
382
|
+
data directory.
|
|
166
383
|
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
384
|
+
```typescript
|
|
385
|
+
const db = await Plugmem.open(undefined, { config: "./plugmem.toml" });
|
|
386
|
+
```
|
|
387
|
+
|
|
388
|
+
```toml
|
|
389
|
+
# plugmem.toml
|
|
390
|
+
[database]
|
|
391
|
+
path = "/path/to/memory.plugmem"
|
|
173
392
|
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
when it already owns the input records.
|
|
393
|
+
[engine]
|
|
394
|
+
dim = 768 # embedding size (0 = vectors off)
|
|
177
395
|
|
|
178
|
-
|
|
396
|
+
[recall] # optional — every key has a tuned default
|
|
397
|
+
w_vec = 2.0 # trust meaning over keywords in this memory
|
|
398
|
+
half_life_days = 30 # and treat anything older than a month as stale
|
|
179
399
|
|
|
180
|
-
|
|
400
|
+
[embedder] # optional — omit for lexical/tag/graph/time only
|
|
401
|
+
kind = "ollama" # or openai / lmstudio / vllm / llamacpp
|
|
402
|
+
url = "http://localhost:11434/v1/embeddings"
|
|
403
|
+
model = "nomic-embed-text"
|
|
181
404
|
|
|
182
|
-
|
|
183
|
-
|
|
405
|
+
[maintenance]
|
|
406
|
+
fsync = "each_op" # or "on_snapshot": faster, loses the journal tail on an OS crash
|
|
407
|
+
```
|
|
184
408
|
|
|
185
|
-
|
|
186
|
-
|
|
409
|
+
`[engine]` is what a database is *built* with; changing one of those on an
|
|
410
|
+
existing file is refused. `[recall]` and `[index]` are the opposite — reopening
|
|
411
|
+
with different weights is how you change the ranking, so tune them freely. All
|
|
412
|
+
of them are in the [full settings reference](https://github.com/m62624/plugmem/blob/main/crates/plugmem-host/SETTINGS.md).
|
|
187
413
|
|
|
188
|
-
|
|
414
|
+
### When a key is misspelled
|
|
189
415
|
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
416
|
+
Unknown keys and sections do not stop anything, but they are not swallowed
|
|
417
|
+
either — a misspelled `w_vec` changes no behaviour, and silence would leave you
|
|
418
|
+
believing you had tuned something. **Read them once after opening**, because a
|
|
419
|
+
native addon has nowhere sensible to print:
|
|
193
420
|
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
421
|
+
```javascript
|
|
422
|
+
const db = await Plugmem.open("agent.plugmem", { config: "./plugmem.toml" });
|
|
423
|
+
for (const warning of db.configWarnings()) console.warn(warning);
|
|
424
|
+
// unknown setting [recall].w_vector — did you mean `w_vec`?
|
|
198
425
|
```
|
|
199
426
|
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
427
|
+
With an `[embedder]`, a text-only `remember`/`recall` embeds automatically, and
|
|
428
|
+
the provider's HTTP call happens outside the engine lock. The `dim` open option
|
|
429
|
+
sets the embedding size when there is no config; if the config built an
|
|
430
|
+
embedder, its dimension governs and `dim` must agree.
|
|
431
|
+
|
|
432
|
+
A read-only handle cannot embed inside the engine — writing into a zero-copy
|
|
433
|
+
mapping is exactly what read-only exists to avoid — so this binding embeds the
|
|
434
|
+
query itself before the read. A text `recall` reaches the vector source in both
|
|
435
|
+
modes.
|
|
436
|
+
|
|
437
|
+
The [full settings reference](https://github.com/m62624/plugmem/blob/main/crates/plugmem-host/SETTINGS.md)
|
|
438
|
+
lists every field and the OS-specific paths.
|
|
439
|
+
|
|
440
|
+
## Async and the event loop
|
|
205
441
|
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
process that
|
|
209
|
-
|
|
442
|
+
Node runs all JavaScript on one thread, so a native call that waits on an
|
|
443
|
+
embedder's HTTP round trip or an fsync would freeze every timer, socket and
|
|
444
|
+
callback in the process. Anything that can do that runs on a libuv worker and
|
|
445
|
+
returns a promise instead.
|
|
210
446
|
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
from your code, so put the policy there.
|
|
447
|
+
Promises: `Plugmem.open`, `remember`, `rememberMany`, `revise`, `recall`,
|
|
448
|
+
`forget`, `link`, `unlink`, `export`, `exportPage`, `verify`, `maintain`,
|
|
449
|
+
`checkpoint`, and every `Workspace` verb except `closeIdle`, `openCount` and
|
|
450
|
+
`close`.
|
|
216
451
|
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
opened. `closeIdle()` and `openCount()` stay synchronous.
|
|
452
|
+
Synchronous: `path`, `get`, `stats`, `tagsOf`, `generation`, `refresh`, `close`.
|
|
453
|
+
These touch mapped memory and return in microseconds, where a promise would be
|
|
454
|
+
pure ceremony.
|
|
221
455
|
|
|
222
|
-
|
|
223
|
-
|
|
456
|
+
Arguments are still checked on your thread: a refused one **throws** at the call
|
|
457
|
+
site rather than rejecting later, so a mistake in your code and a failure in the
|
|
458
|
+
engine never arrive the same way.
|
|
224
459
|
|
|
225
|
-
|
|
460
|
+
### Two costs a promise does not hide
|
|
226
461
|
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
loop
|
|
230
|
-
|
|
462
|
+
**The worker pool is shared, and it has four threads by default.** libuv runs
|
|
463
|
+
`fs`, `dns.lookup`, `zlib` and `crypto.pbkdf2` on the same pool this addon uses.
|
|
464
|
+
The event loop stays free either way, but four concurrent plugmem tasks fill the
|
|
465
|
+
default pool and everything else queues behind them. Measured on one machine, a
|
|
466
|
+
4 MiB `fs.readFile` in the same process:
|
|
231
467
|
|
|
232
|
-
|
|
233
|
-
|
|
468
|
+
| | `fs.readFile` |
|
|
469
|
+
|---|---|
|
|
470
|
+
| idle pool | 1.7 ms |
|
|
471
|
+
| 4 plugmem tasks in flight | 29 577 ms |
|
|
472
|
+
| the same, `UV_THREADPOOL_SIZE=8` | 1.4 ms |
|
|
473
|
+
|
|
474
|
+
plugmem's tasks are unusually long — `maintain('full')` is minutes on a large
|
|
475
|
+
memory — so raise `UV_THREADPOOL_SIZE` if the process does anything else with
|
|
476
|
+
libuv while maintenance runs.
|
|
477
|
+
|
|
478
|
+
The pool is also the ceiling on **concurrent embedding**. With an `[embedder]`
|
|
479
|
+
configured, each `remember`/`recall` occupies one worker for its HTTP round
|
|
480
|
+
trip, so at the default four, four is as parallel as it gets. Against a mock
|
|
481
|
+
provider with a fixed 100 ms latency, 16 concurrent recalls took 404 ms on the
|
|
482
|
+
default pool and 101 ms at `UV_THREADPOOL_SIZE=16` — the same 16 requests, four
|
|
483
|
+
waves or one. If a process issues many concurrent recalls against a remote
|
|
484
|
+
provider, size the pool for that, not for the CPU.
|
|
485
|
+
|
|
486
|
+
**`export()` builds its whole result on your thread.** The scan is on a worker,
|
|
487
|
+
but every fact becomes a JavaScript object during the promise's resolution, and
|
|
488
|
+
that part is main-thread work by definition. On 100 000 facts it holds the
|
|
489
|
+
thread for about 244 ms of the call's 289 ms. `exportPage()` over the same
|
|
490
|
+
memory holds it for **0 ms**, in 128-fact pages:
|
|
234
491
|
|
|
235
492
|
```ts
|
|
236
493
|
let cursor: number | undefined;
|
|
237
494
|
do {
|
|
238
495
|
const page = await db.exportPage(cursor);
|
|
239
|
-
for (const fact of page.facts)
|
|
240
|
-
await destination.write(fact);
|
|
241
|
-
}
|
|
496
|
+
for (const fact of page.facts) await destination.write(fact);
|
|
242
497
|
cursor = page.nextCursor;
|
|
243
498
|
} while (cursor !== undefined);
|
|
244
499
|
```
|
|
245
500
|
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
501
|
+
Each promise owns exactly one page and resolves only after its native scan
|
|
502
|
+
completed; no database lock is held while your loop body runs. A writer may
|
|
503
|
+
change between pages, so do not mutate it during a snapshot-style dump — a
|
|
504
|
+
read-only handle pages one immutable checkpoint and is stable.
|
|
250
505
|
|
|
251
|
-
|
|
252
|
-
then resolves with outcomes in input order. A maintenance call may also return
|
|
253
|
-
a no-op report when there is nothing to purge, reindex or optimize.
|
|
506
|
+
### Concurrency
|
|
254
507
|
|
|
255
|
-
`
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
508
|
+
A `Plugmem` handle is safe to use from anywhere in your process. Reads run
|
|
509
|
+
concurrently; writes serialize behind the engine's lock for the microseconds
|
|
510
|
+
they take. A second *process* opening the same file for writing is refused with
|
|
511
|
+
`PLUGMEM_LOCKED` rather than corrupting it, while any number of read-only
|
|
512
|
+
handles map the same file at once — a writer and its readers coexist across
|
|
513
|
+
processes, sharing the OS page cache.
|
|
259
514
|
|
|
260
|
-
|
|
515
|
+
## Many memories in one directory
|
|
261
516
|
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
`forget`, `link`, `unlink`, `verify`, `export`, `path`, `generation`,
|
|
265
|
-
`refresh` — is synchronous.
|
|
517
|
+
**Default: one logical memory backed by a local database layout.** `Plugmem.open(path)` and nothing here
|
|
518
|
+
applies.
|
|
266
519
|
|
|
267
|
-
The
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
`remember` and a text `recall` each cost a request to the provider — so they run
|
|
272
|
-
on a libuv worker and hand JavaScript a promise. The rest touch only mapped
|
|
273
|
-
memory and return in microseconds, where a promise would be pure ceremony.
|
|
520
|
+
The problem this solves: a process serving many conversations, tenants or
|
|
521
|
+
projects wants each to have its **own** memory — nothing from one surfacing in
|
|
522
|
+
another — without managing a pile of file paths by hand. Give a name, get a
|
|
523
|
+
memory:
|
|
274
524
|
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
engine never arrive the same way.
|
|
525
|
+
```ts
|
|
526
|
+
import { Workspace, type DbEntry } from "plugmem";
|
|
278
527
|
|
|
279
|
-
|
|
528
|
+
const ws = new Workspace("/srv/memories");
|
|
280
529
|
|
|
281
|
-
|
|
282
|
-
|
|
530
|
+
// The same `Plugmem` class comes back, so a named memory has exactly the verbs
|
|
531
|
+
// a path-opened one has. A first write to an unused name creates it.
|
|
532
|
+
const chat = await ws.open("chat-42");
|
|
533
|
+
await chat.remember({ text: "prefers tokio" });
|
|
283
534
|
|
|
284
|
-
|
|
285
|
-
const
|
|
286
|
-
await
|
|
287
|
-
|
|
535
|
+
// Another name is another memory. They cannot see each other.
|
|
536
|
+
const other = await ws.open("chat-99");
|
|
537
|
+
(await other.recall({ query: "tokio" })).facts.length; // 0
|
|
538
|
+
```
|
|
539
|
+
|
|
540
|
+
Memories are **independent by design**: nothing searches across them and no
|
|
541
|
+
entity links between them. A fact filed under the wrong name is not merely
|
|
542
|
+
misplaced, it is unreachable from the other memory.
|
|
543
|
+
|
|
544
|
+
If you do not know the name, ask what each memory is for. Descriptions are
|
|
545
|
+
searchable, and so are owners, even though an owner is stored as a graph edge
|
|
546
|
+
rather than as text:
|
|
547
|
+
|
|
548
|
+
```ts
|
|
549
|
+
await ws.describe("chat-42", { description: "release planning", owner: "ann" });
|
|
550
|
+
|
|
551
|
+
const hits: DbEntry[] = await ws.find("release planning"); // → [{ db: "chat-42", … }]
|
|
552
|
+
const byOwner: DbEntry[] = await ws.find("ann"); // → the same memory
|
|
288
553
|
```
|
|
289
554
|
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
555
|
+
A name is `[a-z0-9][a-z0-9_-]*` and **cannot express a path**, so it resolves to
|
|
556
|
+
exactly one named database inside the directory — traversal is not filtered out, it is
|
|
557
|
+
unconstructible. `ws.open(name, false)` refuses a name that does not exist yet,
|
|
558
|
+
which is what a read should do so a typo is diagnosed rather than answered with
|
|
559
|
+
an empty result.
|
|
560
|
+
|
|
561
|
+
**Who may reach which memory is not this package's job.** The name comes from
|
|
562
|
+
your code, so the policy belongs there.
|
|
563
|
+
|
|
564
|
+
### Running it
|
|
565
|
+
|
|
566
|
+
| Method | Does |
|
|
567
|
+
|---|---|
|
|
568
|
+
| `open(name, create?)` | open (default: create if missing) and hand back a `Plugmem` |
|
|
569
|
+
| `list()` | every memory in the directory, from the filesystem — including undescribed ones |
|
|
570
|
+
| `entries()` | every described memory, from the registry |
|
|
571
|
+
| `find(query, k?)` | memories whose description or owner best matches |
|
|
572
|
+
| `describe(name, args)` | record what a memory is for; revises rather than duplicating |
|
|
573
|
+
| `archive(name)` | label it archived, keeping its description. Nothing is moved or deleted |
|
|
574
|
+
| `reindex()` | rebuild the registry from the memories' own descriptions |
|
|
575
|
+
| `verify()` | report disagreements between registry and directory; repairs nothing |
|
|
576
|
+
| `closeIdle()` | close memories unused past the idle timeout (sync) |
|
|
577
|
+
| `openCount()` | how many are open right now (sync) |
|
|
578
|
+
| `close()` | close every pooled memory and the registry |
|
|
579
|
+
|
|
580
|
+
`closeIdle()` matters more than it looks. An open memory holds its file's
|
|
581
|
+
exclusive lock, so a long-running process that never lets go makes its memories
|
|
582
|
+
unreachable from anything else on the machine. Call it on a timer — that is what
|
|
583
|
+
the idle timeout is for, liveness rather than memory. The pool bounds how many
|
|
584
|
+
stay open at once (`maxOpen`, default 16, least-recently-used closed to make
|
|
585
|
+
room):
|
|
586
|
+
|
|
587
|
+
```ts
|
|
588
|
+
const ws = new Workspace("/srv/memories", { maxOpen: 16, idleTimeoutMs: 60_000 });
|
|
589
|
+
setInterval(() => ws.closeIdle(), 30_000);
|
|
590
|
+
```
|
|
591
|
+
|
|
592
|
+
A `Plugmem` handed out by `open()` is **not** closed by `ws.close()`: it is its
|
|
593
|
+
own handle holding its own lock until you close it or it is garbage collected.
|
|
594
|
+
|
|
595
|
+
`verify()` reports and never repairs, because a workspace is a directory a
|
|
596
|
+
person can edit, and guessing at their intent is how a consistency check loses
|
|
597
|
+
data.
|
|
598
|
+
|
|
599
|
+
## What it is not for
|
|
600
|
+
|
|
601
|
+
plugmem is for local-first application and agent memory: one process, one local database,
|
|
602
|
+
no service to operate. Its design centre is around 100 000 active facts on one machine, and
|
|
603
|
+
the benchmarks track 1M-operation profiles to show how the same engine behaves
|
|
604
|
+
under heavier local load.
|
|
605
|
+
|
|
606
|
+
It is **not** a vector database and not built for multi-million vector
|
|
607
|
+
workloads, cluster sharding, multi-tenant serving or managed nearest-neighbour
|
|
608
|
+
search. For those, use a dedicated system — [Qdrant](https://qdrant.tech),
|
|
609
|
+
[Milvus](https://milvus.io), [Weaviate](https://weaviate.io),
|
|
610
|
+
[Pinecone](https://www.pinecone.io) or
|
|
611
|
+
[pgvector](https://github.com/pgvector/pgvector).
|
|
612
|
+
|
|
613
|
+
## Other ways in
|
|
614
|
+
|
|
615
|
+
The same engine ships four ways. This package is the Node one.
|
|
616
|
+
|
|
617
|
+
| You are | Use |
|
|
618
|
+
|---|---|
|
|
619
|
+
| writing JavaScript / TypeScript for Node | **this package** |
|
|
620
|
+
| writing Rust | [`plugmem-host`](https://docs.rs/plugmem-host/latest) — the engine in your process |
|
|
621
|
+
| an agent, or another language | [`plugmem-mcp`](https://docs.rs/plugmem-mcp/latest) — a stdio JSON-RPC sidecar |
|
|
622
|
+
| a person at a terminal | [`plugmem-cli`](https://docs.rs/plugmem-cli/latest) |
|
|
294
623
|
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
624
|
+
**Working with an LLM agent?** There is a companion
|
|
625
|
+
[skill](https://github.com/m62624/plugmem/blob/main/skill/SKILL.md) describing
|
|
626
|
+
the remember/recall loop, the contradiction workflow and the verbs. This package
|
|
627
|
+
ships it: `skill()` returns the text and `skillVersion()` the version it was
|
|
628
|
+
written against.
|
|
298
629
|
|
|
299
630
|
## License
|
|
300
631
|
|
|
301
|
-
MIT.
|
|
632
|
+
MIT. Source: <https://github.com/m62624/plugmem>
|
package/index.d.ts
CHANGED
|
@@ -86,6 +86,28 @@ export interface RecallArgs {
|
|
|
86
86
|
k?: number
|
|
87
87
|
/** Include closed revisions (default false). */
|
|
88
88
|
closed?: boolean
|
|
89
|
+
/**
|
|
90
|
+
* Token budget of the `rendered` block (default 512). That block is what
|
|
91
|
+
* goes into a prompt, so this is the knob deciding how much of the
|
|
92
|
+
* context window a recall may spend.
|
|
93
|
+
*/
|
|
94
|
+
tokenBudget?: number
|
|
95
|
+
/**
|
|
96
|
+
* HNSW beam width for the vector source (default: the configured
|
|
97
|
+
* `hnsw_ef_search`). Higher is more accurate and slower; ignored while
|
|
98
|
+
* the engine is still in the flat regime, below `flat_to_hnsw`.
|
|
99
|
+
*/
|
|
100
|
+
ef?: number
|
|
101
|
+
/**
|
|
102
|
+
* How many edges the graph source may follow from an anchor entity
|
|
103
|
+
* (default: the configured `graph_depth`). `0` asks for the anchors' own
|
|
104
|
+
* facts and no neighbours.
|
|
105
|
+
*
|
|
106
|
+
* Per call for the same reason `k` and `tokenBudget` are: how wide a net
|
|
107
|
+
* to cast belongs to the question. "What is known around this person"
|
|
108
|
+
* wants more hops than "what is this person's stated preference".
|
|
109
|
+
*/
|
|
110
|
+
graphDepth?: number
|
|
89
111
|
/**
|
|
90
112
|
* A precomputed embedding. Its length must equal the configured `dim`.
|
|
91
113
|
*
|
|
@@ -104,6 +126,59 @@ export interface LinkArgs {
|
|
|
104
126
|
rel: string
|
|
105
127
|
/** Destination entity name. */
|
|
106
128
|
dst: string
|
|
129
|
+
/**
|
|
130
|
+
* The fact this edge follows from, recorded on the edge and returned by
|
|
131
|
+
* graph recall — the answer to "why is this edge here". Ignored by
|
|
132
|
+
* `unlink`, which closes an edge rather than opening one.
|
|
133
|
+
*/
|
|
134
|
+
provenance?: number
|
|
135
|
+
}
|
|
136
|
+
/** Options for [`recover`]. */
|
|
137
|
+
export interface RecoverOptions {
|
|
138
|
+
/**
|
|
139
|
+
* Embedding dimension, as `OpenOptions.dim`. It must match what the source
|
|
140
|
+
* database stores, so a memory with vectors needs this (or a `config`
|
|
141
|
+
* naming the embedder) or the salvage cannot open it.
|
|
142
|
+
*/
|
|
143
|
+
dim?: number
|
|
144
|
+
/** Path to a `config.toml`; the standard discovery applies when omitted. */
|
|
145
|
+
config?: string
|
|
146
|
+
}
|
|
147
|
+
/**
|
|
148
|
+
* Salvages a content-corrupt memory: reads `src`, drops the facts that fail
|
|
149
|
+
* the content checks, and writes a clean, compacted image to `dst`.
|
|
150
|
+
*
|
|
151
|
+
* **`src` is never written.** The damaged file stays exactly as it was, as
|
|
152
|
+
* evidence — this produces a repaired copy beside it, and swapping them is
|
|
153
|
+
* your decision, not this function's. `dst` must therefore be a different
|
|
154
|
+
* path.
|
|
155
|
+
*
|
|
156
|
+
* **It handles *content* damage**, the kind `verify` reports: text that is not
|
|
157
|
+
* valid UTF-8, a vector slot that disagrees with its fact, metadata that will
|
|
158
|
+
* not decode. *Structural* damage — a snapshot whose container will not parse
|
|
159
|
+
* at all — is not salvageable here and rejects; that is what a backup is for.
|
|
160
|
+
*
|
|
161
|
+
* Memory stays proportional to the record count rather than the image: the
|
|
162
|
+
* two large pools are streamed through temp files, so a database far bigger
|
|
163
|
+
* than RAM can be recovered as long as its graph fits.
|
|
164
|
+
*
|
|
165
|
+
* **Async**: it opens, sweeps and rewrites a whole database.
|
|
166
|
+
*
|
|
167
|
+
* @throws `PLUGMEM_LOCKED` if either path is open elsewhere; `PLUGMEM_ENGINE`
|
|
168
|
+
* if `dst` resolves to the same file as `src`, or the source will not parse;
|
|
169
|
+
* `PLUGMEM_OPEN` for an IO failure on either path.
|
|
170
|
+
*/
|
|
171
|
+
export declare function recover(src: string, dst: string, options?: RecoverOptions | undefined | null): Promise<RecoverReport>
|
|
172
|
+
/** Options for `Plugmem#scrub`. */
|
|
173
|
+
export interface ScrubOptions {
|
|
174
|
+
/**
|
|
175
|
+
* Bytes to hash per step, at most. Default: the engine's own (1 MiB).
|
|
176
|
+
*
|
|
177
|
+
* This is the knob trading responsiveness for throughput: smaller steps
|
|
178
|
+
* return to JavaScript more often, larger ones spend less time crossing
|
|
179
|
+
* back and forth. It changes no answer, only the grain.
|
|
180
|
+
*/
|
|
181
|
+
budget?: number
|
|
107
182
|
}
|
|
108
183
|
/** One similar / potentially-conflicting live fact surfaced by `remember`. */
|
|
109
184
|
export interface Similar {
|
|
@@ -226,6 +301,12 @@ export interface FactSnapshot {
|
|
|
226
301
|
}
|
|
227
302
|
/** One exported fact — the id-free, import-ready shape. */
|
|
228
303
|
export interface ExportedFact {
|
|
304
|
+
/**
|
|
305
|
+
* The fact's id in the database it came from. Informational: an import
|
|
306
|
+
* assigns fresh ids. Present because edges name their provenance fact by
|
|
307
|
+
* id, so a dump carrying edges needs something for them to point at.
|
|
308
|
+
*/
|
|
309
|
+
id: number
|
|
229
310
|
/** The fact text. */
|
|
230
311
|
text: string
|
|
231
312
|
/** Subject entity name, if any. */
|
|
@@ -239,6 +320,52 @@ export interface ExportedFact {
|
|
|
239
320
|
/** Validity start (unix ms; preserved on import). */
|
|
240
321
|
validFrom: number
|
|
241
322
|
}
|
|
323
|
+
/**
|
|
324
|
+
* One exported edge — the shape `exportEdges` streams, and the same fields the
|
|
325
|
+
* CLI's JSONL dump writes for an edge.
|
|
326
|
+
*
|
|
327
|
+
* Edges are not part of a fact's dump: a fact names its tags and metadata, but
|
|
328
|
+
* an edge is a statement *between* two entities and outlives any single fact.
|
|
329
|
+
* That is why a complete backup is the two streams together.
|
|
330
|
+
*/
|
|
331
|
+
export interface ExportedEdge {
|
|
332
|
+
/** Source entity name. */
|
|
333
|
+
src: string
|
|
334
|
+
/** The relation, verbatim. */
|
|
335
|
+
rel: string
|
|
336
|
+
/** Destination entity name. */
|
|
337
|
+
dst: string
|
|
338
|
+
/**
|
|
339
|
+
* The fact this edge follows from, if it was recorded with one. Absent
|
|
340
|
+
* rather than a sentinel, so "no provenance" cannot be mistaken for fact 0.
|
|
341
|
+
*/
|
|
342
|
+
provenance?: number
|
|
343
|
+
}
|
|
344
|
+
/**
|
|
345
|
+
* What a `recover` salvaged, and what it had to leave behind.
|
|
346
|
+
*
|
|
347
|
+
* The three `dropped` counts are the damage: each is a fact the source could
|
|
348
|
+
* not produce intact, so a non-zero total means the recovered copy is smaller
|
|
349
|
+
* than the original claimed to be. Zeroes across the board mean the image was
|
|
350
|
+
* content-clean and this was a compaction.
|
|
351
|
+
*/
|
|
352
|
+
export interface RecoverReport {
|
|
353
|
+
/** Facts written to the destination — the survivors. */
|
|
354
|
+
kept: number
|
|
355
|
+
/** Dropped: the stored text was not valid UTF-8. */
|
|
356
|
+
droppedText: number
|
|
357
|
+
/** Dropped: the vector slot was out of range or disagreed with the fact. */
|
|
358
|
+
droppedVector: number
|
|
359
|
+
/** Dropped: the metadata blob did not decode to a well-formed map. */
|
|
360
|
+
droppedMetadata: number
|
|
361
|
+
}
|
|
362
|
+
/** How far a `Scrub` has got. */
|
|
363
|
+
export interface ScrubProgress {
|
|
364
|
+
/** Bytes checksummed so far; equals `totalBytes` on the last step. */
|
|
365
|
+
doneBytes: number
|
|
366
|
+
/** The generation file's length — the total to checksum. */
|
|
367
|
+
totalBytes: number
|
|
368
|
+
}
|
|
242
369
|
/** One bounded page returned by `exportPage`. */
|
|
243
370
|
export interface ExportPage {
|
|
244
371
|
/** Open facts in fact-id order; never longer than the native page bound. */
|
|
@@ -474,6 +601,18 @@ export declare class Plugmem {
|
|
|
474
601
|
* snapshot (`PLUGMEM_NEEDS_CHECKPOINT`), or on an IO error.
|
|
475
602
|
*/
|
|
476
603
|
static open(path?: string | undefined | null, options?: OpenOptions | undefined | null): Promise<Plugmem>
|
|
604
|
+
/**
|
|
605
|
+
* What `config.toml` said that nothing claimed — a misspelled key, a
|
|
606
|
+
* misspelled section — one human-readable line each, empty when the file
|
|
607
|
+
* was clean.
|
|
608
|
+
*
|
|
609
|
+
* It is a value rather than a printed warning because a native addon has
|
|
610
|
+
* nowhere sensible to print: stderr belongs to the host application, which
|
|
611
|
+
* may be a server that logs as JSON. **Read it once after opening and log
|
|
612
|
+
* it your own way** — ignoring it puts you back where a typo silently
|
|
613
|
+
* changes nothing.
|
|
614
|
+
*/
|
|
615
|
+
configWarnings(): Array<string>
|
|
477
616
|
/**
|
|
478
617
|
* The file this memory is open on.
|
|
479
618
|
*
|
|
@@ -549,10 +688,14 @@ export declare class Plugmem {
|
|
|
549
688
|
/**
|
|
550
689
|
* Every currently-open fact, as one array (id-free, import-ready).
|
|
551
690
|
*
|
|
552
|
-
* **Async, but still unbounded
|
|
553
|
-
* whole memory is materialized into a single array before it
|
|
554
|
-
*
|
|
555
|
-
*
|
|
691
|
+
* **Async, but still unbounded, in two ways.** The scan runs on a worker,
|
|
692
|
+
* yet the whole memory is materialized into a single array before it
|
|
693
|
+
* resolves, so peak memory is the whole export. And `resolve` runs on the
|
|
694
|
+
* JS thread by definition, so building one JavaScript object per fact
|
|
695
|
+
* stalls it: measured at ~244 ms of a 289 ms call over 100 000 facts,
|
|
696
|
+
* against 0 ms for the same data through `exportPage`. A promise hides
|
|
697
|
+
* neither cost. Prefer `exportPage` for anything but a small memory or a
|
|
698
|
+
* script.
|
|
556
699
|
*/
|
|
557
700
|
export(): Promise<ExportedFact[]>
|
|
558
701
|
/**
|
|
@@ -566,6 +709,50 @@ export declare class Plugmem {
|
|
|
566
709
|
* mutate it during a snapshot-style backup; a read-only handle is stable.
|
|
567
710
|
*/
|
|
568
711
|
exportPage(cursor?: number | undefined | null): Promise<ExportPage>
|
|
712
|
+
/**
|
|
713
|
+
* Streams every current edge to `onBatch`, in batches, and resolves with
|
|
714
|
+
* the number streamed.
|
|
715
|
+
*
|
|
716
|
+
* **The other half of a backup.** `export`/`exportPage` dump facts; an edge
|
|
717
|
+
* is a statement between two entities and belongs to no single fact, so a
|
|
718
|
+
* dump of facts alone silently loses the graph.
|
|
719
|
+
*
|
|
720
|
+
* **Async and bounded.** The walk runs on a libuv worker; batches reach
|
|
721
|
+
* `onBatch` on the JS thread through a threadsafe function with a queue of
|
|
722
|
+
* four. When the callback is slower than the walk, the *worker* waits — the
|
|
723
|
+
* event loop never does, and peak memory is four batches whatever the
|
|
724
|
+
* database's size.
|
|
725
|
+
*
|
|
726
|
+
* **When the promise resolves, every batch has been delivered.** Worth
|
|
727
|
+
* saying because a threadsafe-function call only queues work for the JS
|
|
728
|
+
* thread, so the natural implementation resolves early and hands you a
|
|
729
|
+
* half-filled accumulator on some runs and a full one on others. Each
|
|
730
|
+
* batch is acknowledged from the JS thread and the worker waits for the
|
|
731
|
+
* receipts, so reading your results straight after the `await` is correct.
|
|
732
|
+
*
|
|
733
|
+
* `onBatch` throwing is not caught here: it surfaces as an uncaught error,
|
|
734
|
+
* as it would from any callback Node invokes on your behalf.
|
|
735
|
+
*/
|
|
736
|
+
exportEdges(onBatch: (edges: ExportedEdge[]) => void): Promise<number>
|
|
737
|
+
/**
|
|
738
|
+
* Starts a resumable byte-level check of the current published snapshot,
|
|
739
|
+
* and resolves with the [`Scrub`](crate::scrub::Scrub) that paces it.
|
|
740
|
+
*
|
|
741
|
+
* The counterpart to `verify`, not a replacement: `verify` asks whether
|
|
742
|
+
* the *content* agrees with itself, this asks whether the *bytes* are the
|
|
743
|
+
* ones that were written. Only the second catches a flipped bit that the
|
|
744
|
+
* structure accepts.
|
|
745
|
+
*
|
|
746
|
+
* Available on a writer as well as a read-only handle: a scrub reads the
|
|
747
|
+
* published generation from disk and has no opinion about which handle
|
|
748
|
+
* asked. It does need something published — checkpoint once first.
|
|
749
|
+
*
|
|
750
|
+
* **The returned object holds a shared lock** on the generation it scans
|
|
751
|
+
* for as long as it lives, so run it to completion or `close()` it.
|
|
752
|
+
*
|
|
753
|
+
* @throws `PLUGMEM_NEEDS_CHECKPOINT` when nothing has been published yet.
|
|
754
|
+
*/
|
|
755
|
+
scrub(options?: ScrubOptions | undefined | null): Promise<Scrub>
|
|
569
756
|
/** One fact's tags, or an empty array for an unknown or tombstoned id. */
|
|
570
757
|
tagsOf(id: number): Array<string>
|
|
571
758
|
/**
|
|
@@ -610,6 +797,46 @@ export declare class Plugmem {
|
|
|
610
797
|
*/
|
|
611
798
|
close(): void
|
|
612
799
|
}
|
|
800
|
+
/**
|
|
801
|
+
* A resumable byte-level check of one snapshot generation.
|
|
802
|
+
*
|
|
803
|
+
* **Holding this object holds a lock.** It pins the generation it is scanning
|
|
804
|
+
* with a shared lock for its whole life, so the writer's garbage collection
|
|
805
|
+
* cannot reclaim that generation under it. Finish the scan, or `close()` it —
|
|
806
|
+
* do not park one indefinitely.
|
|
807
|
+
*
|
|
808
|
+
* One-shot: once it has returned `null`, or thrown, it is done. Ask the
|
|
809
|
+
* database for another to scan again.
|
|
810
|
+
*/
|
|
811
|
+
export declare class Scrub {
|
|
812
|
+
/**
|
|
813
|
+
* Hashes the next slice and resolves with the progress so far, or `null`
|
|
814
|
+
* when the scan is complete.
|
|
815
|
+
*
|
|
816
|
+
* **Async**: see the module note — a step is disk I/O, not arithmetic.
|
|
817
|
+
*
|
|
818
|
+
* Concurrent calls are serialized rather than refused: the cursor has one
|
|
819
|
+
* position, so two overlapping steps would be two different slices of the
|
|
820
|
+
* same scan, which is what a caller pacing it from two places asked for.
|
|
821
|
+
*
|
|
822
|
+
* @throws on the first mismatch, naming what failed its checksum. The
|
|
823
|
+
* scrub is finished at that point; a further call resolves `null`.
|
|
824
|
+
*/
|
|
825
|
+
next(): Promise<ScrubProgress | null>
|
|
826
|
+
/**
|
|
827
|
+
* Releases the pinned generation now, abandoning an unfinished scan.
|
|
828
|
+
*
|
|
829
|
+
* Idempotent, and the same bargain as `Plugmem#close`: waiting for the
|
|
830
|
+
* garbage collector to do it means the writer cannot reclaim that
|
|
831
|
+
* generation until it happens.
|
|
832
|
+
*/
|
|
833
|
+
close(): void
|
|
834
|
+
/**
|
|
835
|
+
* Whether this scrub still has work — `false` once it finished, failed or
|
|
836
|
+
* was closed. Cheap: it inspects the handle, not the file.
|
|
837
|
+
*/
|
|
838
|
+
active(): boolean
|
|
839
|
+
}
|
|
613
840
|
/**
|
|
614
841
|
* A directory of named memories — the napi mirror of
|
|
615
842
|
* [`plugmem_host::Workspace`].
|
package/index.js
CHANGED
|
@@ -310,9 +310,11 @@ if (!nativeBinding) {
|
|
|
310
310
|
throw new Error(`Failed to load native binding`)
|
|
311
311
|
}
|
|
312
312
|
|
|
313
|
-
const { Plugmem, MaintainMode, Workspace, version, about, settingsHelp, skill, skillFull, skillVersion } = nativeBinding
|
|
313
|
+
const { Plugmem, recover, Scrub, MaintainMode, Workspace, version, about, settingsHelp, skill, skillFull, skillVersion } = nativeBinding
|
|
314
314
|
|
|
315
315
|
module.exports.Plugmem = Plugmem
|
|
316
|
+
module.exports.recover = recover
|
|
317
|
+
module.exports.Scrub = Scrub
|
|
316
318
|
module.exports.MaintainMode = MaintainMode
|
|
317
319
|
module.exports.Workspace = Workspace
|
|
318
320
|
module.exports.version = version
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "plugmem",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "Native Node.js addon for plugmem: an embedded
|
|
3
|
+
"version": "0.6.0",
|
|
4
|
+
"description": "Native Node.js addon for plugmem: an embedded bitemporal memory and retrieval engine for local-first applications and agents (remember / recall / revise / forget over one local database).",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
7
7
|
"url": "git+https://github.com/m62624/plugmem.git"
|
|
@@ -41,11 +41,11 @@
|
|
|
41
41
|
"typecheck": "tsc -p tsconfig.json"
|
|
42
42
|
},
|
|
43
43
|
"optionalDependencies": {
|
|
44
|
-
"plugmem-linux-x64-gnu": "0.
|
|
45
|
-
"plugmem-linux-arm64-gnu": "0.
|
|
46
|
-
"plugmem-darwin-x64": "0.
|
|
47
|
-
"plugmem-darwin-arm64": "0.
|
|
48
|
-
"plugmem-win32-x64-msvc": "0.
|
|
49
|
-
"plugmem-win32-arm64-msvc": "0.
|
|
44
|
+
"plugmem-linux-x64-gnu": "0.6.0",
|
|
45
|
+
"plugmem-linux-arm64-gnu": "0.6.0",
|
|
46
|
+
"plugmem-darwin-x64": "0.6.0",
|
|
47
|
+
"plugmem-darwin-arm64": "0.6.0",
|
|
48
|
+
"plugmem-win32-x64-msvc": "0.6.0",
|
|
49
|
+
"plugmem-win32-arm64-msvc": "0.6.0"
|
|
50
50
|
}
|
|
51
51
|
}
|