plugmem 0.5.0 → 0.6.0

package/README.md

@@ -5,10 +5,11 @@
 > Claude models, in roughly equal measure. Expect non-professional design
 > choices, rough edges, broken behavior, or mistakes. Use it at your own risk.
 
-A memory database for LLM agents, embedded in your Node process. It stores short
-facts and answers a query with a ranked block of text ready to put in a prompt.
+An embeddable bitemporal memory database for local-first applications and
+agents, embedded in your Node process. It stores short facts and answers a
+query with ranked facts and edges plus an optional bounded rendered block.
 
-One file on disk, no server, no daemon. It links into your process the way
+File-backed on disk, no server, no daemon. It links into your process the way
 SQLite does: the engine is [`plugmem-host`](https://docs.rs/plugmem-host/latest)
 compiled to a native addon through [napi-rs](https://napi.rs), so there is no
 WebAssembly copy of the file in RAM and no 4 GiB ceiling. Runs on Node, Deno and
@@ -159,6 +160,7 @@ const res = await db.recall({
   tags: ["work"],          // filter: a fact must carry all of these
   k: 10,                   // cap the number of facts
   tokenBudget: 400,        // cap the size of the block — your context budget
+  graphDepth: 3,           // how far to walk from the anchors (default 2)
 });
 
 res.rendered;   // string, prompt-ready
@@ -206,6 +208,8 @@ only moves arguments and results across the boundary.
 | `path()` | the file this handle resolved to (sync) |
 | `export()` | every open fact as one array (async, unbounded — see below) |
 | `exportPage(cursor?)` | the same data in bounded pages of 128 (async) |
+| `exportEdges(onBatch)` | every current edge, streamed in batches (async) |
+| `configWarnings()` | anything in `config.toml` nothing claimed (sync) |
 
 **Upkeep** — all async, all on a worker thread:
 
@@ -214,6 +218,8 @@ only moves arguments and results across the boundary.
 | `maintain(mode?)` | `"auto"` (default), `"compact"`, `"reindex-text"`, `"optimize-vectors"`, `"full"`. No mode ever drops a revision or an edge version |
 | `checkpoint()` | flush the journal into a fresh snapshot |
 | `verify()` | full content-integrity sweep; rejects on the first inconsistency |
+| `scrub(options?)` | start a resumable byte-level check of the snapshot |
+| `recover(src, dst, options?)` | module function: salvage a damaged file into a clean copy |
 
 **Read-only handles** (`{ readOnly: true }`) observe another process's writer
 over a published snapshot. The read verbs answer, the write verbs throw, and two
@@ -238,13 +244,106 @@ const res = await db.recall({ query: text, vector: own });
 Use it for vectors you already have, for a model that is not an OpenAI-shaped
 HTTP endpoint, or for a deterministic test with no network.
 
-### What the Rust library has and this does not
+### Backing up: facts are only half of it
 
-| Not here | Why |
-|---|---|
-| `recover` | salvaging a damaged file is a path-level operation on the host's disk — the CLI's job |
-| `import` | the JSONL file format lives in [`plugmem-cli`](https://docs.rs/plugmem-cli/latest). A Node program holding records already uses `rememberMany` |
-| `scrub` | byte-level container integrity, likewise CLI-side; `verify()` covers content |
+`export`/`exportPage` dump facts. An **edge** is a statement between two
+entities — `kim -works_on-> plugmem` — and belongs to no single fact, so a dump
+of facts alone loses the graph. `exportEdges` is the other half.
+
+It streams: the walk runs on a worker and hands your callback one batch at a
+time, so memory stays flat whether the graph has ten edges or ten million. When
+a callback is slower than the walk, the *worker* waits — never the event loop.
+
+```javascript
+const edges = [];
+const count = await db.exportEdges((batch) => edges.push(...batch));
+```
+
+`count` is `2` here, and `edges` is complete the moment the promise resolves —
+no extra tick needed:
+
+```json
+[
+  { "src": "kim", "rel": "works_on", "dst": "plugmem", "provenance": 0 },
+  { "src": "kim", "rel": "reports_to", "dst": "ann" }
+]
+```
+
+`provenance` is the fact the edge follows from, when it was recorded with one.
+It is **absent** rather than zero when there is none, so it can never be
+mistaken for fact `0` — as the second edge above shows.
+
+### Checking a file has not rotted
+
+`verify()` and `scrub()` ask different questions, and neither replaces the other:
+
+- **`verify()`** — does the *content* agree with itself? Text is valid UTF-8,
+  each vector belongs to its fact, both directions of every edge match.
+- **`scrub()`** — are the *bytes* the ones that were written? It recomputes each
+  section's checksum and the whole-file hash. This is what catches a flipped bit
+  that the structure happily accepts.
+
+A scrub is paced by you rather than run in one go, so it stays affordable on a
+live database — the model ZFS uses. Each step checks up to a budget's worth of
+bytes and returns:
+
+```javascript
+const scrub = await db.scrub();          // default budget: 1 MiB per step
+let step;
+while ((step = await scrub.next()) !== null) {
+  // step.doneBytes of step.totalBytes — progress through the snapshot file
+}
+```
+
+`next()` returns a promise because a step reads from disk, not because hashing
+is slow: over a memory-mapped file the bytes are paged in as they are read, so a
+step is I/O of whatever length your storage takes. On the JS thread that would
+freeze the process.
+
+Two things to know. **Holding the object holds a lock** on the snapshot
+generation it is scanning, so run it to completion or `close()` it. And it is
+one-shot: after it returns `null`, or throws, `active()` is `false` and you ask
+the database for another.
+
+```javascript
+const partial = await db.scrub({ budget: 16 * 1024 });
+await partial.next();     // { doneBytes: 16384, totalBytes: <the file's size> }
+partial.close();          // released; further next() calls return null
+```
+
+Damage rejects with `PLUGMEM_ENGINE` naming what failed its checksum.
+
+### Repairing a damaged file
+
+`recover` is a module function, not a method: it works on **paths**, and takes
+the source's exclusive lock, so close your handle first.
+
+```javascript
+import { recover } from "plugmem";
+
+const report = await recover("memory.plugmem", "repaired.plugmem");
+// { kept: 1, droppedText: 0, droppedVector: 0, droppedMetadata: 0 }
+```
+
+**The source is never written.** It stays exactly as it was, as evidence; this
+produces a repaired copy beside it, and swapping them is your decision. `dst`
+must therefore be a different path — passing the same one throws.
+
+The three `dropped` counts are the damage: each is a fact the source could not
+produce intact. All zero means the image was content-clean and this was a
+compaction. Memory stays proportional to the record count rather than the file,
+so a database far larger than RAM can be recovered.
+
+It handles **content** damage — the kind `verify()` reports. A snapshot whose
+container will not parse at all is not salvageable here; that is what a backup
+is for.
+
+### The one thing the Rust library has and this does not
+
+`import`. The JSONL dump format is defined by
+[`plugmem-cli`](https://docs.rs/plugmem-cli/latest), not by the engine — there is
+no `import` verb to mirror. A Node program holding records already has
+`rememberMany` and `link`, which is what an importer is made of.
 
 ## Errors
 
@@ -294,6 +393,10 @@ path = "/path/to/memory.plugmem"
 [engine]
 dim = 768                     # embedding size (0 = vectors off)
 
+[recall]                      # optional — every key has a tuned default
+w_vec = 2.0                   # trust meaning over keywords in this memory
+half_life_days = 30           # and treat anything older than a month as stale
+
 [embedder]                    # optional — omit for lexical/tag/graph/time only
 kind  = "ollama"              # or openai / lmstudio / vllm / llamacpp
 url   = "http://localhost:11434/v1/embeddings"
@@ -303,6 +406,24 @@ model = "nomic-embed-text"
 fsync = "each_op"             # or "on_snapshot": faster, loses the journal tail on an OS crash
 ```
 
+`[engine]` is what a database is *built* with; changing one of those on an
+existing file is refused. `[recall]` and `[index]` are the opposite — reopening
+with different weights is how you change the ranking, so tune them freely. All
+of them are in the [full settings reference](https://github.com/m62624/plugmem/blob/main/crates/plugmem-host/SETTINGS.md).
+
+### When a key is misspelled
+
+Unknown keys and sections do not stop anything, but they are not swallowed
+either — a misspelled `w_vec` changes no behaviour, and silence would leave you
+believing you had tuned something. **Read them once after opening**, because a
+native addon has nowhere sensible to print:
+
+```javascript
+const db = await Plugmem.open("agent.plugmem", { config: "./plugmem.toml" });
+for (const warning of db.configWarnings()) console.warn(warning);
+// unknown setting [recall].w_vector — did you mean `w_vec`?
+```
+
 With an `[embedder]`, a text-only `remember`/`recall` embeds automatically, and
 the provider's HTTP call happens outside the engine lock. The `dim` open option
 sets the embedding size when there is no config; if the config built an
@@ -393,7 +514,7 @@ processes, sharing the OS page cache.
 
 ## Many memories in one directory
 
-**Default: one memory, one file.** `Plugmem.open(path)` and nothing here
+**Default: one logical memory backed by a local database layout.** `Plugmem.open(path)` and nothing here
 applies.
 
 The problem this solves: a process serving many conversations, tenants or
@@ -432,7 +553,7 @@ const byOwner: DbEntry[] = await ws.find("ann");            // → the same memo
 ```
 
 A name is `[a-z0-9][a-z0-9_-]*` and **cannot express a path**, so it resolves to
-exactly one file inside the directory — traversal is not filtered out, it is
+exactly one named database inside the directory — traversal is not filtered out, it is
 unconstructible. `ws.open(name, false)` refuses a name that does not exist yet,
 which is what a read should do so a typo is diagnosed rather than answered with
 an empty result.
@@ -477,8 +598,8 @@ data.
 
 ## What it is not for
 
-plugmem is for a local agent's memory: one process, one file, no service to
-operate. Its design centre is around 100 000 active facts on one machine, and
+plugmem is for local-first application and agent memory: one process, one local database,
+no service to operate. Its design centre is around 100 000 active facts on one machine, and
 the benchmarks track 1M-operation profiles to show how the same engine behaves
 under heavier local load.
 

package/index.d.ts

@@ -98,6 +98,16 @@ export interface RecallArgs {
    * the engine is still in the flat regime, below `flat_to_hnsw`.
    */
   ef?: number
+  /**
+   * How many edges the graph source may follow from an anchor entity
+   * (default: the configured `graph_depth`). `0` asks for the anchors' own
+   * facts and no neighbours.
+   *
+   * Per call for the same reason `k` and `tokenBudget` are: how wide a net
+   * to cast belongs to the question. "What is known around this person"
+   * wants more hops than "what is this person's stated preference".
+   */
+  graphDepth?: number
   /**
    * A precomputed embedding. Its length must equal the configured `dim`.
    *
@@ -123,6 +133,53 @@ export interface LinkArgs {
    */
   provenance?: number
 }
+/** Options for [`recover`]. */
+export interface RecoverOptions {
+  /**
+   * Embedding dimension, as `OpenOptions.dim`. It must match what the source
+   * database stores, so a memory with vectors needs this (or a `config`
+   * naming the embedder) or the salvage cannot open it.
+   */
+  dim?: number
+  /** Path to a `config.toml`; the standard discovery applies when omitted. */
+  config?: string
+}
+/**
+ * Salvages a content-corrupt memory: reads `src`, drops the facts that fail
+ * the content checks, and writes a clean, compacted image to `dst`.
+ *
+ * **`src` is never written.** The damaged file stays exactly as it was, as
+ * evidence — this produces a repaired copy beside it, and swapping them is
+ * your decision, not this function's. `dst` must therefore be a different
+ * path.
+ *
+ * **It handles *content* damage**, the kind `verify` reports: text that is not
+ * valid UTF-8, a vector slot that disagrees with its fact, metadata that will
+ * not decode. *Structural* damage — a snapshot whose container will not parse
+ * at all — is not salvageable here and rejects; that is what a backup is for.
+ *
+ * Memory stays proportional to the record count rather than the image: the
+ * two large pools are streamed through temp files, so a database far bigger
+ * than RAM can be recovered as long as its graph fits.
+ *
+ * **Async**: it opens, sweeps and rewrites a whole database.
+ *
+ * @throws `PLUGMEM_LOCKED` if either path is open elsewhere; `PLUGMEM_ENGINE`
+ * if `dst` resolves to the same file as `src`, or the source will not parse;
+ * `PLUGMEM_OPEN` for an IO failure on either path.
+ */
+export declare function recover(src: string, dst: string, options?: RecoverOptions | undefined | null): Promise<RecoverReport>
+/** Options for `Plugmem#scrub`. */
+export interface ScrubOptions {
+  /**
+   * Bytes to hash per step, at most. Default: the engine's own (1 MiB).
+   *
+   * This is the knob trading responsiveness for throughput: smaller steps
+   * return to JavaScript more often, larger ones spend less time crossing
+   * back and forth. It changes no answer, only the grain.
+   */
+  budget?: number
+}
 /** One similar / potentially-conflicting live fact surfaced by `remember`. */
 export interface Similar {
   /** The existing fact's id. */
@@ -263,6 +320,52 @@ export interface ExportedFact {
   /** Validity start (unix ms; preserved on import). */
   validFrom: number
 }
+/**
+ * One exported edge — the shape `exportEdges` streams, and the same fields the
+ * CLI's JSONL dump writes for an edge.
+ *
+ * Edges are not part of a fact's dump: a fact names its tags and metadata, but
+ * an edge is a statement *between* two entities and outlives any single fact.
+ * That is why a complete backup is the two streams together.
+ */
+export interface ExportedEdge {
+  /** Source entity name. */
+  src: string
+  /** The relation, verbatim. */
+  rel: string
+  /** Destination entity name. */
+  dst: string
+  /**
+   * The fact this edge follows from, if it was recorded with one. Absent
+   * rather than a sentinel, so "no provenance" cannot be mistaken for fact 0.
+   */
+  provenance?: number
+}
+/**
+ * What a `recover` salvaged, and what it had to leave behind.
+ *
+ * The three `dropped` counts are the damage: each is a fact the source could
+ * not produce intact, so a non-zero total means the recovered copy is smaller
+ * than the original claimed to be. Zeroes across the board mean the image was
+ * content-clean and this was a compaction.
+ */
+export interface RecoverReport {
+  /** Facts written to the destination — the survivors. */
+  kept: number
+  /** Dropped: the stored text was not valid UTF-8. */
+  droppedText: number
+  /** Dropped: the vector slot was out of range or disagreed with the fact. */
+  droppedVector: number
+  /** Dropped: the metadata blob did not decode to a well-formed map. */
+  droppedMetadata: number
+}
+/** How far a `Scrub` has got. */
+export interface ScrubProgress {
+  /** Bytes checksummed so far; equals `totalBytes` on the last step. */
+  doneBytes: number
+  /** The generation file's length — the total to checksum. */
+  totalBytes: number
+}
 /** One bounded page returned by `exportPage`. */
 export interface ExportPage {
   /** Open facts in fact-id order; never longer than the native page bound. */
@@ -498,6 +601,18 @@ export declare class Plugmem {
    * snapshot (`PLUGMEM_NEEDS_CHECKPOINT`), or on an IO error.
    */
   static open(path?: string | undefined | null, options?: OpenOptions | undefined | null): Promise<Plugmem>
+  /**
+   * What `config.toml` said that nothing claimed — a misspelled key, a
+   * misspelled section — one human-readable line each, empty when the file
+   * was clean.
+   *
+   * It is a value rather than a printed warning because a native addon has
+   * nowhere sensible to print: stderr belongs to the host application, which
+   * may be a server that logs as JSON. **Read it once after opening and log
+   * it your own way** — ignoring it puts you back where a typo silently
+   * changes nothing.
+   */
+  configWarnings(): Array<string>
   /**
    * The file this memory is open on.
    *
@@ -594,6 +709,50 @@ export declare class Plugmem {
    * mutate it during a snapshot-style backup; a read-only handle is stable.
    */
   exportPage(cursor?: number | undefined | null): Promise<ExportPage>
+  /**
+   * Streams every current edge to `onBatch`, in batches, and resolves with
+   * the number streamed.
+   *
+   * **The other half of a backup.** `export`/`exportPage` dump facts; an edge
+   * is a statement between two entities and belongs to no single fact, so a
+   * dump of facts alone silently loses the graph.
+   *
+   * **Async and bounded.** The walk runs on a libuv worker; batches reach
+   * `onBatch` on the JS thread through a threadsafe function with a queue of
+   * four. When the callback is slower than the walk, the *worker* waits — the
+   * event loop never does, and peak memory is four batches whatever the
+   * database's size.
+   *
+   * **When the promise resolves, every batch has been delivered.** Worth
+   * saying because a threadsafe-function call only queues work for the JS
+   * thread, so the natural implementation resolves early and hands you a
+   * half-filled accumulator on some runs and a full one on others. Each
+   * batch is acknowledged from the JS thread and the worker waits for the
+   * receipts, so reading your results straight after the `await` is correct.
+   *
+   * `onBatch` throwing is not caught here: it surfaces as an uncaught error,
+   * as it would from any callback Node invokes on your behalf.
+   */
+  exportEdges(onBatch: (edges: ExportedEdge[]) => void): Promise<number>
+  /**
+   * Starts a resumable byte-level check of the current published snapshot,
+   * and resolves with the [`Scrub`](crate::scrub::Scrub) that paces it.
+   *
+   * The counterpart to `verify`, not a replacement: `verify` asks whether
+   * the *content* agrees with itself, this asks whether the *bytes* are the
+   * ones that were written. Only the second catches a flipped bit that the
+   * structure accepts.
+   *
+   * Available on a writer as well as a read-only handle: a scrub reads the
+   * published generation from disk and has no opinion about which handle
+   * asked. It does need something published — checkpoint once first.
+   *
+   * **The returned object holds a shared lock** on the generation it scans
+   * for as long as it lives, so run it to completion or `close()` it.
+   *
+   * @throws `PLUGMEM_NEEDS_CHECKPOINT` when nothing has been published yet.
+   */
+  scrub(options?: ScrubOptions | undefined | null): Promise<Scrub>
   /** One fact's tags, or an empty array for an unknown or tombstoned id. */
   tagsOf(id: number): Array<string>
   /**
@@ -638,6 +797,46 @@ export declare class Plugmem {
    */
   close(): void
 }
+/**
+ * A resumable byte-level check of one snapshot generation.
+ *
+ * **Holding this object holds a lock.** It pins the generation it is scanning
+ * with a shared lock for its whole life, so the writer's garbage collection
+ * cannot reclaim that generation under it. Finish the scan, or `close()` it —
+ * do not park one indefinitely.
+ *
+ * One-shot: once it has returned `null`, or thrown, it is done. Ask the
+ * database for another to scan again.
+ */
+export declare class Scrub {
+  /**
+   * Hashes the next slice and resolves with the progress so far, or `null`
+   * when the scan is complete.
+   *
+   * **Async**: see the module note — a step is disk I/O, not arithmetic.
+   *
+   * Concurrent calls are serialized rather than refused: the cursor has one
+   * position, so two overlapping steps would be two different slices of the
+   * same scan, which is what a caller pacing it from two places asked for.
+   *
+   * @throws on the first mismatch, naming what failed its checksum. The
+   * scrub is finished at that point; a further call resolves `null`.
+   */
+  next(): Promise<ScrubProgress | null>
+  /**
+   * Releases the pinned generation now, abandoning an unfinished scan.
+   *
+   * Idempotent, and the same bargain as `Plugmem#close`: waiting for the
+   * garbage collector to do it means the writer cannot reclaim that
+   * generation until it happens.
+   */
+  close(): void
+  /**
+   * Whether this scrub still has work — `false` once it finished, failed or
+   * was closed. Cheap: it inspects the handle, not the file.
+   */
+  active(): boolean
+}
 /**
  * A directory of named memories — the napi mirror of
  * [`plugmem_host::Workspace`].

package/index.js

@@ -310,9 +310,11 @@ if (!nativeBinding) {
   throw new Error(`Failed to load native binding`)
 }
 
-const { Plugmem, MaintainMode, Workspace, version, about, settingsHelp, skill, skillFull, skillVersion } = nativeBinding
+const { Plugmem, recover, Scrub, MaintainMode, Workspace, version, about, settingsHelp, skill, skillFull, skillVersion } = nativeBinding
 
 module.exports.Plugmem = Plugmem
+module.exports.recover = recover
+module.exports.Scrub = Scrub
 module.exports.MaintainMode = MaintainMode
 module.exports.Workspace = Workspace
 module.exports.version = version

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "plugmem",
-  "version": "0.5.0",
-  "description": "Native Node.js addon for plugmem: an embedded long-term memory engine for LLM agents (remember / recall / revise / forget over one local file).",
+  "version": "0.6.0",
+  "description": "Native Node.js addon for plugmem: an embedded bitemporal memory and retrieval engine for local-first applications and agents (remember / recall / revise / forget over one local database).",
   "repository": {
     "type": "git",
     "url": "git+https://github.com/m62624/plugmem.git"
@@ -41,11 +41,11 @@
     "typecheck": "tsc -p tsconfig.json"
   },
   "optionalDependencies": {
-    "plugmem-linux-x64-gnu": "0.5.0",
-    "plugmem-linux-arm64-gnu": "0.5.0",
-    "plugmem-darwin-x64": "0.5.0",
-    "plugmem-darwin-arm64": "0.5.0",
-    "plugmem-win32-x64-msvc": "0.5.0",
-    "plugmem-win32-arm64-msvc": "0.5.0"
+    "plugmem-linux-x64-gnu": "0.6.0",
+    "plugmem-linux-arm64-gnu": "0.6.0",
+    "plugmem-darwin-x64": "0.6.0",
+    "plugmem-darwin-arm64": "0.6.0",
+    "plugmem-win32-x64-msvc": "0.6.0",
+    "plugmem-win32-arm64-msvc": "0.6.0"
   }
 }