@equationalapplications/core-llm-wiki 7.7.7 → 7.9.0

package/README.md

@@ -67,6 +67,82 @@ For programmatic pipelines — importing pre-classified data, building a GraphRA
 
 This is the GraphRAG seed path: load a corpus, walk it.
 
+## Temporal facts
+
+Facts carry an optional validity window. Nothing changes until you use it: facts without validity data are always current.
+
+```typescript
+// The user tells you they moved; the move happened on March 1.
+await wiki.supersede('user-123', seattleFactId, { title: 'Lives in San Francisco', body: 'Moved in March.' }, {
+  validFrom: Date.parse('2026-03-01'),
+});
+
+await wiki.read('user-123', 'where do I live');                                  // current: San Francisco
+await wiki.read('user-123', 'where do I live', { asOf: Date.parse('2026-02-01') }); // then: Seattle
+await wiki.history('user-123', seattleFactId);                                    // [Seattle, San Francisco]
+
+// Record when an event really happened (defaults to ingestion time):
+await wiki.write('user-123', { event_type: 'observation', summary: 'Moved to SF', occurred_at: Date.parse('2026-03-01') });
+```
+
+- Validity is half-open: `[valid_from, valid_to)`. At the switch instant only the new fact matches.
+- `asOf` answers "what do we now know was true at T". Learning late about a past change still answers correctly for the past.
+- `supersede()` refuses `immutable_document` facts (re-ingest the document instead), cycles, facts owned by another entity, and facts already superseded. It throws `WikiSupersedeError` with a `reason`.
+- `traverseGraph({ asOf })` only walks edges whose two facts are both live at that instant.
+- History is kept. Set `config.pruneSupersededAfter` (days) to let `runPrune` retire expired facts.
+- `getMemoryBundle()` and `exportDump()` return every non-deleted fact, including history.
+
+## Librarian strategies
+
+`config.librarian.strategy` selects how events become facts. The default, `'legacy'`, is unchanged from 7.x.
+
+```typescript
+new WikiMemory(db, {
+  llmProvider,
+  config: { librarian: { strategy: 'ops', gate: { k: 5, dupThreshold: 0.89, novelThreshold: 0.30 } } },
+});
+```
+
+`'ops'` works in three steps:
+
+1. **Extract.** Candidate facts come from the events since the last pass. Existing facts are not sent.
+2. **Gate, with no LLM call.** Each candidate is compared with its closest current facts. An identical fact is a no-op, and a candidate with no close match is added. Only candidates that are similar but different continue to step 3. Similar is not the same as identical: "lives in Seattle" and "moved to San Francisco" score as similar, so they go to the model.
+3. **Resolve, in one batched call.** The model returns `ADD`, `UPDATE`, `SUPERSEDE` or `NOOP` for each remaining candidate. `SUPERSEDE` uses `supersede()`, so history is kept and `read({ asOf })` still sees the old fact.
+
+- The thresholds depend on your embedding model. The defaults are provisional.
+- Without `embed`, the gate uses keyword search. Every keyword match then goes to the resolve step.
+- Under `'ops'`, `runLibrarian`'s `promptOverride` and `prompts.librarianSystemPrompt` are ignored: the ops prompts use a different schema.
+- A model answer that can't be used never loses a fact. It is added, or stored as a draft, and a diagnostic is emitted.
+
+## Deferred maintenance
+
+By default, `write()` starts the librarian and heal in the background when thresholds are crossed. On serverless or request-scoped hosts those background promises can be frozen or killed. Set `maintenance: 'deferred'` to decide when maintenance runs:
+
+```typescript
+const wiki = new WikiMemory(db, {
+  llmProvider,
+  config: { maintenance: 'deferred', librarian: { strategy: 'ops' } },
+});
+
+await wiki.write(entityId, { event_type: 'observation', summary, occurred_at });
+
+const report = await wiki.runPendingMaintenance({ tokenBudget: 20_000, deadlineMs: 8_000 });
+// report.stoppedReason: complete | deadline | budget_exhausted | budget_too_small
+// Nothing is still running here, so it is safe to persist the database.
+```
+
+- `getPendingMaintenance()` shows the backlog per entity without calling a model.
+- The budget covers text-generation tokens only; embedding calls are not counted. Figures are provider-reported when `generateTextWithUsage` exists, otherwise estimated at characters ÷ 4.
+- Entities share the budget fairly: each round runs one batch per entity.
+- `budget_too_small` names the first job and entity whose smallest step needed more than the whole budget; other entities may have hit the same limit in that run.
+- A crash mid-batch loses nothing. The librarian's progress marker only advances after a batch commits.
+- In `'auto'` mode, call `drain()` before the app is suspended or closed. `autoLibrarianTokenThreshold` also triggers the librarian on the size of the pending text.
+- Use `'deferred'` with the ops strategy. The legacy strategy reads only the newest 50 events per pass and marks them processed, so a backlog larger than that is skipped.
+- Under `'deferred'` or the ops strategy, `runPrune` never deletes events the librarian has not processed.
+- `JobManager` locks only work within one process. Guarantee a single writer per database file across processes.
+
+See the [SynapseTree integration guide](../../docs/synapsetree-integration.md) for the end-to-end pattern on tenant-aware hosts (S3-backed SQLite, request lifecycle, `entity_id` scheme, MCP surface, costs). Benchmarks live at [docs/benchmarks.md](../../docs/benchmarks.md).
+
 ## Installation
 
 ```bash
@@ -124,6 +200,15 @@ const wikiMemory = new WikiMemory(db, {
 // read() returns MiniSearch results, onRetrievalFallback called (embed threw)
 ```
 
+### Token budgets
+
+```typescript
+const bundle = await wiki.read('user-123', 'what should I cook?', { tokenBudget: 800 });
+const prompt = formatContext(bundle, { compact: true });
+```
+
+`tokenBudget` keeps the most relevant facts per token that fit within about 800 tokens. Tokens are estimated as characters ÷ 4. The top fact is always included, truncated if it alone is too big. Tasks and events are unaffected. `traverseGraph` accepts the same option and keeps nodes in breadth-first order. A `read_budget` diagnostic reports how many facts were packed.
+
 ## Configuration
 
 All `WikiConfig` fields are optional:
@@ -1169,6 +1254,15 @@ const { nodesWritten, edgesWritten, superseded } = await wikiMemory.upsertGraph(
 
 `upsertGraph` is "the tail of `ingestDocument` with the middle (LLM extraction) step removed" — it accepts caller-supplied nodes (`{ id, type, title, body? }`) and edges (`{ type, sourceId, targetId, id? }`) and writes them under the same `(sourceRef, sourceHash)` semantics. If a *different* live `sourceRef` already holds the same `sourceHash`, it throws `WikiSourceRefHashCollision`; re-writing the identical `(sourceRef, sourceHash)` is a no-op returning zero counts. The adapter parameter is required so writes participate in the caller's transaction.
 
+Core never sees your transaction commit, so it can't index the new nodes itself. Until you tell it, they won't show up in keyword or hybrid `read()` results on the same instance. Call `syncSearchIndex` **after** the commit, not inside the transaction:
+
+```typescript
+await db.withTransactionAsync((tx) => wikiMemory.upsertGraph('entity-123', params, tx));
+await wikiMemory.syncSearchIndex('entity-123'); // or syncSearchIndex() for every entity written
+```
+
+It is cheap on an entity that is already current, runs in order with core's own index syncs, and never rejects for a valid `entityId` (an empty or non-string id throws `TypeError`). It does not compute embeddings; those still come from the maintenance sweep.
+
 Entry IDs occupy one globally shared namespace within a database. An ID belongs
 to its existing entity while its row exists, including after soft deletion.
 Same-entity updates and resurrection retain their existing behavior; physically