plugmem 0.4.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +408 -198
  2. package/index.d.ts +32 -4
  3. package/package.json +7 -7
package/README.md CHANGED
@@ -1,20 +1,25 @@
1
- # plugmem-napi
1
+ # plugmem
2
2
 
3
3
  > ⚠️ Experimental. plugmem is mostly an AI-built experiment — written with
4
4
  > the help of a small local model (Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) and various
5
5
  > Claude models, in roughly equal measure. Expect non-professional design
6
6
  > choices, rough edges, broken behavior, or mistakes. Use it at your own risk.
7
7
 
8
- `plugmem-napi` is the **native Node.js addon** for the plugmem
9
- [temporal-memory engine](https://docs.rs/plugmem-core/latest) — it embeds
10
- [`plugmem-host`](https://docs.rs/plugmem-host/latest) **in the Node process**
11
- (real mmap, file locking, cross-process MVCC — the whole engine, unchanged) and
12
- exposes it to **JavaScript / TypeScript** as a `Plugmem` class. It is published
13
- to npm as **`plugmem`**.
8
+ A memory database for LLM agents, embedded in your Node process. It stores short
9
+ facts and answers a query with a ranked block of text ready to put in a prompt.
14
10
 
15
- Because it is native (not WebAssembly), there is no whole-file-in-RAM copy and no
16
- 4 GiB ceiling: the OS pages the snapshot in and out exactly as it does for the
17
- Rust library. It loads in **Node, and any N-API host (Deno, Bun)**.
11
+ One file on disk, no server, no daemon. It links into your process the way
12
+ SQLite does: the engine is [`plugmem-host`](https://docs.rs/plugmem-host/latest)
13
+ compiled to a native addon through [napi-rs](https://napi.rs), so there is no
14
+ WebAssembly copy of the file in RAM and no 4 GiB ceiling. Runs on Node, Deno and
15
+ Bun.
16
+
17
+ **Contents:** [Install](#install) · [Quick start](#quick-start) ·
18
+ [What it stores](#what-it-stores) · [Two clocks](#two-clocks) ·
19
+ [How recall works](#how-recall-works) · [API](#api) · [Errors](#errors) ·
20
+ [Configuration](#configuration-and-embeddings) ·
21
+ [Async](#async-and-the-event-loop) · [Many memories](#many-memories-in-one-directory) ·
22
+ [What it is not for](#what-it-is-not-for)
18
23
 
19
24
  ## Install
20
25
 
@@ -22,115 +27,226 @@ Rust library. It loads in **Node, and any N-API host (Deno, Bun)**.
22
27
  $ npm install plugmem
23
28
  ```
24
29
 
25
- `npm i plugmem` pulls the meta package, which through `optionalDependencies`
26
- installs only the prebuilt binary for your platform — one of
27
- `plugmem-{linux-x64-gnu, linux-arm64-gnu, darwin-x64, darwin-arm64,
28
- win32-x64-msvc, win32-arm64-msvc}`. No toolchain, no build step.
30
+ That pulls a meta package which, through `optionalDependencies`, installs only
31
+ the prebuilt binary for your platform: one of `plugmem-{linux-x64-gnu,
32
+ linux-arm64-gnu, darwin-x64, darwin-arm64, win32-x64-msvc, win32-arm64-msvc}`.
33
+ No toolchain, no build step.
34
+
35
+ ## Quick start
36
+
37
+ ```typescript
38
+ import { Plugmem } from "plugmem";
39
+
40
+ const db = await Plugmem.open("agent.plugmem");
41
+
42
+ await db.remember({ text: "the user prefers tokio", entity: "user", tags: ["pref"] });
43
+ await db.remember({ text: "the release ships on friday", entity: "release" });
44
+
45
+ const res = await db.recall({ query: "which runtime?", k: 5 });
46
+ console.log(res.rendered); // paste this into the prompt
47
+ // - [f0] user: the user prefers tokio (2026-08; active)
48
+
49
+ db.close();
50
+ ```
51
+
52
+ `Plugmem.open` is a static method, not a constructor, because opening replays a
53
+ journal and maps a snapshot — work proportional to the file — and a JavaScript
54
+ constructor has no way to hand that to a worker thread. Everything is typed:
55
+ `index.d.ts` is generated from the Rust, so a TypeScript host gets real
56
+ autocomplete on arguments and results.
57
+
58
+ ## What it stores
59
+
60
+ One **fact** is one statement. It carries:
61
+
62
+ | Field | Meaning |
63
+ |---|---|
64
+ | `text` | the statement itself, and what lexical search indexes |
65
+ | `entity` | the subject, by name — created on first mention, shared across facts |
66
+ | `tags` | filters, not ranking: a query asking for a tag requires it |
67
+ | `metadata` | an opaque `Record<string,string>`. The engine stores and returns it and never looks inside — use it for a URI to the real payload elsewhere, a mime type, an external key |
68
+ | `vector` | an optional embedding; supply your own, or let a configured embedder produce it |
69
+ | `validFrom` | when the statement became true |
70
+
71
+ Entities are joined by **typed edges**: `link({ src: "ann", rel: "hires", dst: "bob" })`.
72
+ An edge can name the fact it follows from (`provenance`), so a later reader can
73
+ answer "why does the memory think this" instead of trusting a bare relationship.
29
74
 
30
- ## Which door is this?
75
+ Facts are never rewritten in place. `revise` closes the old one and chains a
76
+ successor; `forget` tombstones a fact and the next `maintain` erases it from
77
+ disk. `unlink` closes an edge the same way `revise` closes a fact.
31
78
 
32
- plugmem is **embedded-first, like SQLite**. Pick the door for your language:
79
+ ## Two clocks
33
80
 
34
- | You are… | Use | Why |
35
- |---|---|---|
36
- | **writing JavaScript / TypeScript for Node** | **`plugmem-napi`** (this, npm `plugmem`) | The engine *in your Node process*, native speed, typed for TS. |
37
- | **writing Rust** | [`plugmem-host`](https://docs.rs/plugmem-host/latest) | The engine in your process, like linking SQLite. |
38
- | **an agent, or another language** (Python, Go…) | [`plugmem-mcp`](https://docs.rs/plugmem-mcp/latest) | A long-lived stdio JSON-RPC sidecar; language-independent. |
39
- | a person at a **terminal / script** | [`plugmem-cli`](https://docs.rs/plugmem-cli/latest) | The human door. |
81
+ This is the part worth reading, because it is the one thing that behaves
82
+ differently from every other store.
40
83
 
41
- So: **Node/TS → napi; Rust → host; an agent or other language → MCP; a human →
42
- the CLI.**
84
+ Every fact carries **two** timestamps, not one:
43
85
 
44
- ## Usage (TypeScript)
86
+ - **`validFrom` / `validTo`** — when the statement was true.
87
+ - **`recordedAt`** — when the memory learned it. Set by the engine, never by you.
45
88
 
46
- Every argument and result is typed — napi generates `index.d.ts`, so a TS host
47
- gets full autocomplete and checking. A memory is opened with the static
48
- `Plugmem.open`, not `new`: opening replays a journal and maps a snapshot, and a
49
- JavaScript constructor has no way to hand that to a worker thread.
89
+ They are different questions, and one timestamp cannot hold both.
90
+
91
+ Unlike the Rust library, this binding reads the system clock on every call — you
92
+ never pass `now` — so `recordedAt` is always the moment of the write:
50
93
 
51
94
  ```typescript
52
- import { Plugmem } from "plugmem";
95
+ await db.remember({ text: "lives in Moscow", entity: "kim" });
96
+ const between = Date.now();
97
+ await db.revise(0, { text: "lives in Berlin", entity: "kim" });
53
98
 
54
- const db = await Plugmem.open("agent.plugmem"); // or { readOnly: true }
99
+ (await db.recall({ entities: ["kim"] })).rendered;
100
+ // - [f1] kim: lives in Berlin (2026-08; active)
55
101
 
56
- const out = await db.remember({
57
- text: "prefers tokio",
58
- entity: "user",
59
- tags: ["pref"],
60
- links: [{ rel: "works_at", entity: "acme" }],
61
- metadata: { source: "chat", uri: "s3://bucket/note.txt" }, // opaque; a pointer
62
- });
63
- out.id; // number
64
- out.similar; // Similar[] — the engine surfaces conflicts, you decide
102
+ (await db.recall({ entities: ["kim"], asOf: between })).rendered;
103
+ // - [f0] kim: lives in Moscow (2026-08 → 2026-08; closed)
104
+ ```
65
105
 
66
- const res = await db.recall({ query: "runtime?", k: 5 });
67
- res.rendered; // the prompt-ready block
68
- res.facts; // RecalledFact[] — { id, score, entity, recordedAt, … }
106
+ `revise` closed the first fact's interval rather than deleting it, which is why
107
+ the second query has something to answer with.
69
108
 
70
- const card = db.get(out.id);
71
- card.metadata; // Record<string,string> — keys sorted, {} when none
109
+ `asOf` moves **both** clocks: a fact answers only if it was valid at that
110
+ instant *and* had already been recorded by then. The second half is the one
111
+ people trip over — an `asOf` earlier than a fact's `recordedAt` sees nothing,
112
+ because the memory genuinely knew nothing then. Answering with today's knowledge
113
+ would be the wrong answer to "what did I hold".
72
114
 
73
- await db.revise(out.id, { text: "prefers async-std" });
74
- await db.link({ src: "user", rel: "works_at", dst: "acme" });
75
- db.unlink({ src: "user", rel: "works_at", dst: "acme" }); // closes the current edge
115
+ `validFrom` is the other half: a statement that became true before you heard of
116
+ it. Recording today that someone moved a week ago closes the previous interval a
117
+ week ago rather than now, so a query as of three days back finds **neither** —
118
+ the old fact had stopped being true, and the new one was not yet known. That is
119
+ not a hole in the model; it is the honest answer for that instant, and it is
120
+ what a single timestamp cannot express.
76
121
 
77
- await db.checkpoint(); // async (see below)
78
- db.close(); // release the file + lock explicitly
122
+ Two more queries over the same axes:
123
+
124
+ ```typescript
125
+ await db.recall({ range: [from, to] }); // what did I record in this window
126
+ await db.recall({ query: "kim", closed: true }); // include closed revisions
79
127
  ```
80
128
 
81
- ## Configuration & embeddings
129
+ Use `revise` when something changed and `forget` only when a fact was simply
130
+ wrong: `forget` destroys the "what was true then" answer, `revise` keeps it.
131
+
132
+ Edges are temporal too, so `asOf` walks the graph as it stood then — through
133
+ relationships that have since been unlinked.
134
+
135
+ ## How recall works
82
136
 
83
- The constructor resolves settings **exactly like the CLI and MCP server**: an
84
- explicit `config` path wins, else `$PLUGMEM_CONFIG`, else the platform config
85
- directory, else all defaults. The database path is resolved as an explicit
86
- constructor path, then `$PLUGMEM_DB`, then `[database].path`, then the platform
87
- data directory. See the [full settings reference](https://github.com/m62624/plugmem/blob/main/crates/plugmem-host/SETTINGS.md)
88
- for all fields and OS-specific paths.
137
+ Not a vector lookup. Four sources run and are fused by
138
+ [reciprocal-rank fusion](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf)
139
+ with a recency boost; tags filter and are not a source:
140
+
141
+ | Source | What it finds |
142
+ |---|---|
143
+ | **Lexical** — [BM25](https://en.wikipedia.org/wiki/Okapi_BM25) over a Unicode ([UAX #29](https://unicode.org/reports/tr29/)) tokenizer | exact terms, keyword overlap |
144
+ | **Semantic** — int8-quantized cosine, a flat scan below a threshold and an [HNSW](https://arxiv.org/abs/1603.09320) graph above | meaning, nearest neighbours |
145
+ | **Graph** — typed edges walked from the query's anchor entities | relational knowledge |
146
+ | **Temporal** — range scans over the `recordedAt` index, plus the validity test | "what was true then", time windows |
147
+
148
+ The sources compose. A query with no `query` string still answers from tags,
149
+ entities and time. **Without an embedder the system is complete** — the other
150
+ three sources need no model, no network and no API key.
151
+
152
+ The result carries both a `rendered` block, selected greedily under a token
153
+ budget and ready to paste, and the structured `facts`/`edges` behind it:
89
154
 
90
155
  ```typescript
91
- const db = await Plugmem.open(undefined, { config: "./plugmem.toml" });
156
+ const res = await db.recall({
157
+ query: "release plans",
158
+ entities: ["ann"], // graph anchors
159
+ tags: ["work"], // filter: a fact must carry all of these
160
+ k: 10, // cap the number of facts
161
+ tokenBudget: 400, // cap the size of the block — your context budget
162
+ });
163
+
164
+ res.rendered; // string, prompt-ready
165
+ res.facts; // { id, score, entity, recordedAt, validFrom, validTo, sources }[]
166
+ res.edges; // { src, rel, dst, provenance }[] — what the graph walked
167
+ res.truncated; // true if selection stopped at k or the budget with more left
92
168
  ```
93
169
 
94
- ```toml
95
- # plugmem.toml
96
- [database]
97
- path = "/path/to/memory.plugmem" # optional example
170
+ `remember` returns the new fact's id **plus any live facts it may duplicate or
171
+ contradict**. The engine never merges or deletes on its own — it surfaces the
172
+ tension and your code decides:
98
173
 
99
- [engine]
100
- dim = 768 # embedding size (0 = vectors off)
174
+ ```typescript
175
+ const out = await db.remember({ text: "the user prefers async-std", entity: "user" });
176
+ for (const s of out.similar) {
177
+ // s.id, s.score, s.reason: "LexicalOverlap" | "VectorCosine"
178
+ // → revise it, forget it, or keep both. Your call.
179
+ }
180
+ ```
101
181
 
102
- [embedder] # optional — omit for lexical/tag/graph/time only
103
- kind = "ollama" # or openai / lmstudio / vllm / llamacpp
104
- url = "http://localhost:11434/v1/embeddings"
105
- model = "nomic-embed-text"
182
+ ## API
183
+
184
+ Every method wraps the identically-named verb of the Rust `Database`; this layer
185
+ only moves arguments and results across the boundary.
186
+
187
+ **Writing** — all return promises:
188
+
189
+ | Method | Does |
190
+ |---|---|
191
+ | `remember(args)` | store a fact; resolves with its id and similar facts |
192
+ | `rememberMany(args[])` | store a batch: one embedding round-trip, one journal sync |
193
+ | `revise(id, args)` | close a fact and record its successor |
194
+ | `forget(id)` | tombstone a fact; resolves with whether it was live |
195
+ | `link(args)` | upsert a typed edge, optionally with `provenance` |
196
+ | `unlink(args)` | close the current edge; resolves with whether one was open |
197
+
198
+ **Reading** — synchronous ones touch mapped memory and return in microseconds:
199
+
200
+ | Method | Does |
201
+ |---|---|
202
+ | `recall(args?)` | ranked, fused, token-budgeted result (async) |
203
+ | `get(id)` | one fact's full card, or `null` (sync) |
204
+ | `tagsOf(id)` | that fact's tags (sync) |
205
+ | `stats()` | engine size counters (sync) |
206
+ | `path()` | the file this handle resolved to (sync) |
207
+ | `export()` | every open fact as one array (async, unbounded — see below) |
208
+ | `exportPage(cursor?)` | the same data in bounded pages of 128 (async) |
209
+
210
+ **Upkeep** — all async, all on a worker thread:
211
+
212
+ | Method | Does |
213
+ |---|---|
214
+ | `maintain(mode?)` | `"auto"` (default), `"compact"`, `"reindex-text"`, `"optimize-vectors"`, `"full"`. No mode ever drops a revision or an edge version |
215
+ | `checkpoint()` | flush the journal into a fresh snapshot |
216
+ | `verify()` | full content-integrity sweep; rejects on the first inconsistency |
217
+
218
+ **Read-only handles** (`{ readOnly: true }`) observe another process's writer
219
+ over a published snapshot. The read verbs answer, the write verbs throw, and two
220
+ more appear: `generation()` (the pinned snapshot number) and `refresh()` (adopt
221
+ the writer's latest checkpoint, returning whether a newer one existed).
222
+
223
+ `close()` releases the file and its lock; every verb afterwards throws, and
224
+ calling it twice is a no-op.
225
+
226
+ ### Bringing your own embedding
227
+
228
+ `remember`, `revise`, `rememberMany` and `recall` all take an optional `vector`
229
+ whose length must equal the configured `dim`. Given one, it **replaces** the
230
+ embedder for that call — nothing is sent to the provider:
231
+
232
+ ```typescript
233
+ const own = await myEmbedder(text);
234
+ await db.remember({ text, vector: own });
235
+ const res = await db.recall({ query: text, vector: own });
106
236
  ```
107
237
 
108
- With an `[embedder]`, a text-only `remember`/`recall` **auto-embeds** — the
109
- provider's HTTP call runs outside the engine lock. Without one, there is no
110
- embedder and vector recall is skipped (lexical, tag, graph and time recall still
111
- answer). The optional `dim` open option sets the embedding size when there is no
112
- config; if the config configured an embedder, its dimension governs and `dim`
113
- must agree. A `{ readOnly: true }` handle cannot auto-embed inside the engine —
114
- embedding into a zero-copy mapping is what read-only exists to avoid — so this
115
- binding embeds the query itself before the read, exactly as the CLI and the MCP
116
- server do. A text `recall` therefore reaches the vector source in both modes.
117
-
118
- ## The verbs
119
-
120
- Every method here is the identically-named `plugmem-host` `Database` verb; the
121
- engine logic is entirely the host's.
122
-
123
- **Writer** (default): `remember`, `rememberMany`, `recall`, `revise(id, args)`,
124
- `forget(id)`, `link`, `unlink`, `get(id)`, `tagsOf(id)`, `stats`, `export`,
125
- `exportPage(cursor?)`, `verify`, and the async maintenance verbs below.
126
- **Read-only** (`{ readOnly: true }`, observing another process's writer):
127
- `recall`, `get`, `tagsOf`, `stats`, `export`, `exportPage(cursor?)`, `verify`,
128
- plus `generation()` (the pinned snapshot generation) and `refresh()` (advance
129
- to the writer's latest checkpoint); the write verbs throw.
130
- **Both**: `path()` — the file the handle resolved to, which is the only way to
131
- learn it when the constructor was given no path.
132
-
133
- ### Errors
238
+ Use it for vectors you already have, for a model that is not an OpenAI-shaped
239
+ HTTP endpoint, or for a deterministic test with no network.
240
+
241
+ ### What the Rust library has and this does not
242
+
243
+ | Not here | Why |
244
+ |---|---|
245
+ | `recover` | salvaging a damaged file is a path-level operation on the host's disk — the CLI's job |
246
+ | `import` | the JSONL file format lives in [`plugmem-cli`](https://docs.rs/plugmem-cli/latest). A Node program holding records already uses `rememberMany` |
247
+ | `scrub` | byte-level container integrity, likewise CLI-side; `verify()` covers content |
248
+
249
+ ## Errors
134
250
 
135
251
  Every failure plugmem itself decides carries a stable `code`, so a program
136
252
  branches on it instead of on wording:
@@ -149,153 +265,247 @@ try {
149
265
  `PLUGMEM_INVALID_NAME` from an argument that was refused; `PLUGMEM_CLOSED`,
150
266
  `PLUGMEM_READ_ONLY`, `PLUGMEM_WRITER_ONLY` and `PLUGMEM_BUSY` from calling a
151
267
  verb the handle cannot serve; `PLUGMEM_ENGINE` from the engine itself, carrying
152
- the host's own message.
268
+ its own message.
153
269
 
154
270
  The code is there whether the verb threw or the promise rejected — the two are
155
271
  the same contract, so nothing has to be handled twice.
156
272
 
157
- An argument that shapes an answer is refused rather than dropped: `range` must
158
- be exactly `[from, to]`, and `range`, `asOf` and `validFrom` must each be a
159
- finite, non-negative instant. Silently ignoring one produced an answer computed
160
- without it — indistinguishable from a correct one.
273
+ An argument that would shape an answer is refused rather than dropped: `range`
274
+ must be exactly `[from, to]`, and `range`, `asOf` and `validFrom` must each be a
275
+ finite, non-negative instant. Silently ignoring one produces an answer computed
276
+ without it, indistinguishable from a correct one.
161
277
 
162
- ### What the host has and this does not
278
+ ## Configuration and embeddings
163
279
 
164
- The list above is the whole supported `Plugmem` surface. The only host
165
- operation intentionally kept out of this boundary is path-level recovery:
166
-
167
- | host verb | boundary note |
168
- |---|---|
169
- | `recover` | salvaging a damaged file is a path-level operation on the disk the process is running on — [`plugmem-cli recover`](https://docs.rs/plugmem-cli/latest)'s job, like `import` and `scrub`. |
170
- | `remember_many` | Exposed as async `rememberMany(items)`. It writes a batch with one embedding round-trip and resolves with outcomes in input order. |
171
- | `export_each` | Exposed as pull-based `exportPage(cursor?)`. Each Promise returns at most 128 facts and releases the native read lock before JS processes them; this gives Node backpressure without a cross-thread callback. |
172
- | `tags_of` | Exposed as synchronous `tagsOf(id)`, returning one fact's tags or an empty array. |
280
+ Settings resolve from an explicit `config` path, then `$PLUGMEM_CONFIG`, then
281
+ the platform config directory, then defaults. The database path resolves from an
282
+ explicit argument, then `$PLUGMEM_DB`, then `[database].path`, then the platform
283
+ data directory.
173
284
 
174
- **No `import` verb** either — bulk-loading a `backup.jsonl` reads a file on disk,
175
- which is the CLI's job. A Node host can use `rememberMany` for bounded batches
176
- when it already owns the input records.
285
+ ```typescript
286
+ const db = await Plugmem.open(undefined, { config: "./plugmem.toml" });
287
+ ```
177
288
 
178
- ## Many memories in one directory (optional)
289
+ ```toml
290
+ # plugmem.toml
291
+ [database]
292
+ path = "/path/to/memory.plugmem"
179
293
 
180
- **Default: one memory, one file.** `Plugmem.open(path)` and nothing here applies.
294
+ [engine]
295
+ dim = 768 # embedding size (0 = vectors off)
181
296
 
182
- A process that serves many independent memories — one per conversation, per
183
- tenant, per project — can point at a directory and address them by name:
297
+ [embedder] # optional — omit for lexical/tag/graph/time only
298
+ kind = "ollama" # or openai / lmstudio / vllm / llamacpp
299
+ url = "http://localhost:11434/v1/embeddings"
300
+ model = "nomic-embed-text"
184
301
 
185
- ```ts
186
- import { Workspace, type DbEntry } from "plugmem";
302
+ [maintenance]
303
+ fsync = "each_op" # or "on_snapshot": faster, loses the journal tail on an OS crash
304
+ ```
187
305
 
188
- const ws = new Workspace("/srv/memories", { maxOpen: 16, idleTimeoutMs: 60_000 });
306
+ With an `[embedder]`, a text-only `remember`/`recall` embeds automatically, and
307
+ the provider's HTTP call happens outside the engine lock. The `dim` open option
308
+ sets the embedding size when there is no config; if the config built an
309
+ embedder, its dimension governs and `dim` must agree.
189
310
 
190
- // `open` hands back the same `Plugmem` class, so a named memory has exactly the
191
- // verbs a path-opened one has. A first write to an unused name creates it.
192
- await (await ws.open("chat-42")).remember({ text: "prefers tokio" });
311
+ A read-only handle cannot embed inside the engine — writing into a zero-copy
312
+ mapping is exactly what read-only exists to avoid — so this binding embeds the
313
+ query itself before the read. A text `recall` reaches the vector source in both
314
+ modes.
193
315
 
194
- // Do not know the name? Ask what each memory is for. Owners are searchable too,
195
- // even though an owner is a graph edge rather than text.
196
- await ws.describe("chat-42", { description: "release planning", owner: "ann" });
197
- const hits: DbEntry[] = await ws.find("release planning"); // → [{ db: "chat-42", … }]
198
- ```
316
+ The [full settings reference](https://github.com/m62624/plugmem/blob/main/crates/plugmem-host/SETTINGS.md)
317
+ lists every field and the OS-specific paths.
199
318
 
200
- A name is `[a-z0-9][a-z0-9_-]*` and **cannot represent a path**, so it resolves
201
- to exactly one file inside the directory — traversal is not filtered out, it is
202
- unconstructible. `open(name, false)` refuses a name that does not exist yet,
203
- which is what a read should do so a typo is diagnosed rather than answered with
204
- an empty result.
319
+ ## Async and the event loop
205
320
 
206
- The pool bounds how many stay open; `closeIdle()` releases the rest. Call it on
207
- a timer: an open memory holds the file's **exclusive lock**, so a long-running
208
- process that never let go would make its memories unreachable from anything else
209
- on the machine. That is what the idle timeout is for — liveness, not memory.
321
+ Node runs all JavaScript on one thread, so a native call that waits on an
322
+ embedder's HTTP round trip or an fsync would freeze every timer, socket and
323
+ callback in the process. Anything that can do that runs on a libuv worker and
324
+ returns a promise instead.
210
325
 
211
- Two things to know before building on it. Memories are **independent**: nothing
212
- searches across them and no entity links between them, so a fact filed in the
213
- wrong one is unreachable from the other rather than merely misplaced. And **who
214
- may reach which memory is not this package's responsibility** — the name comes
215
- from your code, so put the policy there.
326
+ Promises: `Plugmem.open`, `remember`, `rememberMany`, `revise`, `recall`,
327
+ `forget`, `link`, `unlink`, `export`, `exportPage`, `verify`, `maintain`,
328
+ `checkpoint`, and every `Workspace` verb except `closeIdle`, `openCount` and
329
+ `close`.
216
330
 
217
- Every `Workspace` verb that touches a database returns a promise — `open`,
218
- `list`, `entries`, `find`, `describe`, `archive`, `reindex`, `verify` — because
219
- the registry is itself a plugmem memory and a named memory is a real file being
220
- opened. `closeIdle()` and `openCount()` stay synchronous.
331
+ Synchronous: `path`, `get`, `stats`, `tagsOf`, `generation`, `refresh`, `close`.
332
+ These touch mapped memory and return in microseconds, where a promise would be
333
+ pure ceremony.
221
334
 
222
- `reindex()` and `verify()` return promises: they open and read every memory in
223
- the directory, which is not work for the main thread.
335
+ Arguments are still checked on your thread: a refused one **throws** at the call
336
+ site rather than rejecting later, so a mistake in your code and a failure in the
337
+ engine never arrive the same way.
224
338
 
225
- ## Async and concurrency
339
+ ### Two costs a promise does not hide
226
340
 
227
- Operations with unbounded storage or batch work use napi-rs `AsyncTask`: they
228
- return a **`Promise`** and run on Node's **libuv** worker pool, keeping the event
229
- loop available for application code. This includes `rememberMany`,
230
- `exportPage`, `maintain`, `checkpoint`, `reindex` and `verify`.
341
+ **The worker pool is shared, and it has four threads by default.** libuv runs
342
+ `fs`, `dns.lookup`, `zlib` and `crypto.pbkdf2` on the same pool this addon uses.
343
+ The event loop stays free either way, but four concurrent plugmem tasks fill the
344
+ default pool and everything else queues behind them. Measured on one machine, a
345
+ 4 MiB `fs.readFile` in the same process:
231
346
 
232
- For a bounded export, call `exportPage()` once, process its `facts`, then pass
233
- `nextCursor` to the next call until it is absent:
347
+ | | `fs.readFile` |
348
+ |---|---|
349
+ | idle pool | 1.7 ms |
350
+ | 4 plugmem tasks in flight | 29 577 ms |
351
+ | the same, `UV_THREADPOOL_SIZE=8` | 1.4 ms |
352
+
353
+ plugmem's tasks are unusually long — `maintain('full')` is minutes on a large
354
+ memory — so raise `UV_THREADPOOL_SIZE` if the process does anything else with
355
+ libuv while maintenance runs.
356
+
357
+ The pool is also the ceiling on **concurrent embedding**. With an `[embedder]`
358
+ configured, each `remember`/`recall` occupies one worker for its HTTP round
359
+ trip, so at the default four, four is as parallel as it gets. Against a mock
360
+ provider with a fixed 100 ms latency, 16 concurrent recalls took 404 ms on the
361
+ default pool and 101 ms at `UV_THREADPOOL_SIZE=16` — the same 16 requests, four
362
+ waves or one. If a process issues many concurrent recalls against a remote
363
+ provider, size the pool for that, not for the CPU.
364
+
365
+ **`export()` builds its whole result on your thread.** The scan is on a worker,
366
+ but every fact becomes a JavaScript object during the promise's resolution, and
367
+ that part is main-thread work by definition. On 100 000 facts it holds the
368
+ thread for about 244 ms of the call's 289 ms. `exportPage()` over the same
369
+ memory holds it for **0 ms**, in 128-fact pages:
234
370
 
235
371
  ```ts
236
372
  let cursor: number | undefined;
237
373
  do {
238
374
  const page = await db.exportPage(cursor);
239
- for (const fact of page.facts) {
240
- await destination.write(fact);
241
- }
375
+ for (const fact of page.facts) await destination.write(fact);
242
376
  cursor = page.nextCursor;
243
377
  } while (cursor !== undefined);
244
378
  ```
245
379
 
246
- The Promise is the completion boundary for that page: no callback remains
247
- queued, and no database lock is held while the loop body runs. Each page has at
248
- most 128 facts. A read-only handle pages one immutable checkpoint; when paging a
249
- writer for a snapshot-style backup, do not mutate it between calls.
380
+ Each promise owns exactly one page and resolves only after its native scan
381
+ completed; no database lock is held while your loop body runs. A writer may
382
+ change between pages, so do not mutate it during a snapshot-style dump — a
383
+ read-only handle pages one immutable checkpoint and is stable.
250
384
 
251
- `rememberMany(items)` performs one batch embedding pass and one journal sync,
252
- then resolves with outcomes in input order. A maintenance call may also return
253
- a no-op report when there is nothing to purge, reindex or optimize.
385
+ ### Concurrency
254
386
 
255
- `maintain(mode?)` takes `"auto"` (the default), `"compact"`, `"reindex-text"`,
256
- `"optimize-vectors"` or `"full"`. No mode ever drops a fact revision or an edge
257
- version; the heavier ones buy bytes and index freshness. `"full"` is the only
258
- one that repacks the edge arenas, which a relink-heavy workload fragments.
387
+ A `Plugmem` handle is safe to use from anywhere in your process. Reads run
388
+ concurrently; writes serialize behind the engine's lock for the microseconds
389
+ they take. A second *process* opening the same file for writing is refused with
390
+ `PLUGMEM_LOCKED` rather than corrupting it, while any number of read-only
391
+ handles map the same file at once — a writer and its readers coexist across
392
+ processes, sharing the OS page cache.
259
393
 
260
- ### What is async, and why
394
+ ## Many memories in one directory
261
395
 
262
- `remember`, `revise`, `recall`, `rememberMany`, `exportPage`, `maintain` and
263
- `checkpoint` return promises. Everything else — `get`, `stats`, `tagsOf`,
264
- `forget`, `link`, `unlink`, `verify`, `export`, `path`, `generation`,
265
- `refresh` — is synchronous.
396
+ **Default: one memory, one file.** `Plugmem.open(path)` and nothing here
397
+ applies.
266
398
 
267
- The line is drawn at blocking work. Node runs all JavaScript on **one** thread,
268
- so a native call that waits on an embedder's HTTP round trip or on an fsync
269
- freezes every timer, socket and callback in the process for as long as it takes.
270
- The verbs above can do exactly that — with an `[embedder]` configured,
271
- `remember` and a text `recall` each cost a request to the provider — so they run
272
- on a libuv worker and hand JavaScript a promise. The rest touch only mapped
273
- memory and return in microseconds, where a promise would be pure ceremony.
399
+ The problem this solves: a process serving many conversations, tenants or
400
+ projects wants each to have its **own** memory — nothing from one surfacing in
401
+ another — without managing a pile of file paths by hand. Give a name, get a
402
+ memory:
274
403
 
275
- Arguments are still checked on your thread: a refused one **throws** at the call
276
- site rather than rejecting later, so a mistake in your code and a failure in the
277
- engine never arrive the same way.
404
+ ```ts
405
+ import { Workspace, type DbEntry } from "plugmem";
278
406
 
279
- ### Bringing your own embedding
407
+ const ws = new Workspace("/srv/memories");
280
408
 
281
- `remember`, `revise`, `rememberMany` and `recall` all take an optional
282
- `vector` — a precomputed embedding whose length must equal the configured `dim`:
409
+ // The same `Plugmem` class comes back, so a named memory has exactly the verbs
410
+ // a path-opened one has. A first write to an unused name creates it.
411
+ const chat = await ws.open("chat-42");
412
+ await chat.remember({ text: "prefers tokio" });
283
413
 
284
- ```typescript
285
- const own = await myEmbedder(text); // your model, your pipeline
286
- await db.remember({ text, vector: own }); // nothing is sent to `[embedder]`
287
- const res = await db.recall({ query: text, vector: own });
414
+ // Another name is another memory. They cannot see each other.
415
+ const other = await ws.open("chat-99");
416
+ (await other.recall({ query: "tokio" })).facts.length; // 0
417
+ ```
418
+
419
+ Memories are **independent by design**: nothing searches across them and no
420
+ entity links between them. A fact filed under the wrong name is not merely
421
+ misplaced, it is unreachable from the other memory.
422
+
423
+ If you do not know the name, ask what each memory is for. Descriptions are
424
+ searchable, and so are owners, even though an owner is stored as a graph edge
425
+ rather than as text:
426
+
427
+ ```ts
428
+ await ws.describe("chat-42", { description: "release planning", owner: "ann" });
429
+
430
+ const hits: DbEntry[] = await ws.find("release planning"); // → [{ db: "chat-42", … }]
431
+ const byOwner: DbEntry[] = await ws.find("ann"); // → the same memory
288
432
  ```
289
433
 
290
- Given one, it **replaces** the embedder for that call — the engine embeds only
291
- when the field is absent. Use it for vectors you already have, for a model that
292
- is not an OpenAI-shaped HTTP endpoint, or for a deterministic test with no
293
- network. The CLI (`--vector`) and the MCP tools (`vector`) take the same thing.
434
+ A name is `[a-z0-9][a-z0-9_-]*` and **cannot express a path**, so it resolves to
435
+ exactly one file inside the directory — traversal is not filtered out, it is
436
+ unconstructible. `ws.open(name, false)` refuses a name that does not exist yet,
437
+ which is what a read should do so a typo is diagnosed rather than answered with
438
+ an empty result.
439
+
440
+ **Who may reach which memory is not this package's job.** The name comes from
441
+ your code, so the policy belongs there.
442
+
443
+ ### Running it
444
+
445
+ | Method | Does |
446
+ |---|---|
447
+ | `open(name, create?)` | open (default: create if missing) and hand back a `Plugmem` |
448
+ | `list()` | every memory in the directory, from the filesystem — including undescribed ones |
449
+ | `entries()` | every described memory, from the registry |
450
+ | `find(query, k?)` | memories whose description or owner best matches |
451
+ | `describe(name, args)` | record what a memory is for; revises rather than duplicating |
452
+ | `archive(name)` | label it archived, keeping its description. Nothing is moved or deleted |
453
+ | `reindex()` | rebuild the registry from the memories' own descriptions |
454
+ | `verify()` | report disagreements between registry and directory; repairs nothing |
455
+ | `closeIdle()` | close memories unused past the idle timeout (sync) |
456
+ | `openCount()` | how many are open right now (sync) |
457
+ | `close()` | close every pooled memory and the registry |
458
+
459
+ `closeIdle()` matters more than it looks. An open memory holds its file's
460
+ exclusive lock, so a long-running process that never lets go makes its memories
461
+ unreachable from anything else on the machine. Call it on a timer — that is what
462
+ the idle timeout is for, liveness rather than memory. The pool bounds how many
463
+ stay open at once (`maxOpen`, default 16, least-recently-used closed to make
464
+ room):
465
+
466
+ ```ts
467
+ const ws = new Workspace("/srv/memories", { maxOpen: 16, idleTimeoutMs: 60_000 });
468
+ setInterval(() => ws.closeIdle(), 30_000);
469
+ ```
470
+
471
+ A `Plugmem` handed out by `open()` is **not** closed by `ws.close()`: it is its
472
+ own handle holding its own lock until you close it or it is garbage collected.
473
+
474
+ `verify()` reports and never repairs, because a workspace is a directory a
475
+ person can edit, and guessing at their intent is how a consistency check loses
476
+ data.
477
+
478
+ ## What it is not for
479
+
480
+ plugmem is for a local agent's memory: one process, one file, no service to
481
+ operate. Its design centre is around 100 000 active facts on one machine, and
482
+ the benchmarks track 1M-operation profiles to show how the same engine behaves
483
+ under heavier local load.
484
+
485
+ It is **not** a vector database and not built for multi-million vector
486
+ workloads, cluster sharding, multi-tenant serving or managed nearest-neighbour
487
+ search. For those, use a dedicated system — [Qdrant](https://qdrant.tech),
488
+ [Milvus](https://milvus.io), [Weaviate](https://weaviate.io),
489
+ [Pinecone](https://www.pinecone.io) or
490
+ [pgvector](https://github.com/pgvector/pgvector).
491
+
492
+ ## Other ways in
493
+
494
+ The same engine ships four ways. This package is the Node one.
495
+
496
+ | You are | Use |
497
+ |---|---|
498
+ | writing JavaScript / TypeScript for Node | **this package** |
499
+ | writing Rust | [`plugmem-host`](https://docs.rs/plugmem-host/latest) — the engine in your process |
500
+ | an agent, or another language | [`plugmem-mcp`](https://docs.rs/plugmem-mcp/latest) — a stdio JSON-RPC sidecar |
501
+ | a person at a terminal | [`plugmem-cli`](https://docs.rs/plugmem-cli/latest) |
294
502
 
295
- `close()` releases the file and its lock; every verb afterwards throws, and it is
296
- idempotent (the handle is also released on garbage collection, but `close()`
297
- makes the moment explicit — e.g. before reopening the same file read-only).
503
+ **Working with an LLM agent?** There is a companion
504
+ [skill](https://github.com/m62624/plugmem/blob/main/skill/SKILL.md) describing
505
+ the remember/recall loop, the contradiction workflow and the verbs. This package
506
+ ships it: `skill()` returns the text and `skillVersion()` the version it was
507
+ written against.
298
508
 
299
509
  ## License
300
510
 
301
- MIT.
511
+ MIT. Source: <https://github.com/m62624/plugmem>
package/index.d.ts CHANGED
@@ -86,6 +86,18 @@ export interface RecallArgs {
86
86
  k?: number
87
87
  /** Include closed revisions (default false). */
88
88
  closed?: boolean
89
+ /**
90
+ * Token budget of the `rendered` block (default 512). That block is what
91
+ * goes into a prompt, so this is the knob deciding how much of the
92
+ * context window a recall may spend.
93
+ */
94
+ tokenBudget?: number
95
+ /**
96
+ * HNSW beam width for the vector source (default: the configured
97
+ * `hnsw_ef_search`). Higher is more accurate and slower; ignored while
98
+ * the engine is still in the flat regime, below `flat_to_hnsw`.
99
+ */
100
+ ef?: number
89
101
  /**
90
102
  * A precomputed embedding. Its length must equal the configured `dim`.
91
103
  *
@@ -104,6 +116,12 @@ export interface LinkArgs {
104
116
  rel: string
105
117
  /** Destination entity name. */
106
118
  dst: string
119
+ /**
120
+ * The fact this edge follows from, recorded on the edge and returned by
121
+ * graph recall — the answer to "why is this edge here". Ignored by
122
+ * `unlink`, which closes an edge rather than opening one.
123
+ */
124
+ provenance?: number
107
125
  }
108
126
  /** One similar / potentially-conflicting live fact surfaced by `remember`. */
109
127
  export interface Similar {
@@ -226,6 +244,12 @@ export interface FactSnapshot {
226
244
  }
227
245
  /** One exported fact — the id-free, import-ready shape. */
228
246
  export interface ExportedFact {
247
+ /**
248
+ * The fact's id in the database it came from. Informational: an import
249
+ * assigns fresh ids. Present because edges name their provenance fact by
250
+ * id, so a dump carrying edges needs something for them to point at.
251
+ */
252
+ id: number
229
253
  /** The fact text. */
230
254
  text: string
231
255
  /** Subject entity name, if any. */
@@ -549,10 +573,14 @@ export declare class Plugmem {
549
573
  /**
550
574
  * Every currently-open fact, as one array (id-free, import-ready).
551
575
  *
552
- * **Async, but still unbounded**: the scan is off the JS thread, yet the
553
- * whole memory is materialized into a single array before it resolves, so
554
- * the peak memory is the whole export. `exportPage` is the same data in
555
- * bounded pages — prefer it for anything but a small memory or a script.
576
+ * **Async, but still unbounded, in two ways.** The scan runs on a worker,
577
+ * yet the whole memory is materialized into a single array before it
578
+ * resolves, so peak memory is the whole export. And `resolve` runs on the
579
+ * JS thread by definition, so building one JavaScript object per fact
580
+ * stalls it: measured at ~244 ms of a 289 ms call over 100 000 facts,
581
+ * against 0 ms for the same data through `exportPage`. A promise hides
582
+ * neither cost. Prefer `exportPage` for anything but a small memory or a
583
+ * script.
556
584
  */
557
585
  export(): Promise<ExportedFact[]>
558
586
  /**
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "plugmem",
3
- "version": "0.4.0",
3
+ "version": "0.5.0",
4
4
  "description": "Native Node.js addon for plugmem: an embedded long-term memory engine for LLM agents (remember / recall / revise / forget over one local file).",
5
5
  "repository": {
6
6
  "type": "git",
@@ -41,11 +41,11 @@
41
41
  "typecheck": "tsc -p tsconfig.json"
42
42
  },
43
43
  "optionalDependencies": {
44
- "plugmem-linux-x64-gnu": "0.4.0",
45
- "plugmem-linux-arm64-gnu": "0.4.0",
46
- "plugmem-darwin-x64": "0.4.0",
47
- "plugmem-darwin-arm64": "0.4.0",
48
- "plugmem-win32-x64-msvc": "0.4.0",
49
- "plugmem-win32-arm64-msvc": "0.4.0"
44
+ "plugmem-linux-x64-gnu": "0.5.0",
45
+ "plugmem-linux-arm64-gnu": "0.5.0",
46
+ "plugmem-darwin-x64": "0.5.0",
47
+ "plugmem-darwin-arm64": "0.5.0",
48
+ "plugmem-win32-x64-msvc": "0.5.0",
49
+ "plugmem-win32-arm64-msvc": "0.5.0"
50
50
  }
51
51
  }