@equationalapplications/core-llm-wiki 7.15.1 → 7.15.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -10,1614 +10,43 @@ Platform-agnostic TypeScript engine for hybrid LLM memory. Features episodic fac
10
10
 
11
11
  > Inspired by [Andrej Karpathy's LLM Wiki memory spec](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
12
12
 
13
+ **📖 Documentation: [equationalapplications.github.io/expo-llm-wiki/packages/core](https://equationalapplications.github.io/expo-llm-wiki/packages/core/)**. Start with the [Node.js quickstart](https://equationalapplications.github.io/expo-llm-wiki/start/quickstart-node/), or go straight to the [API reference](https://equationalapplications.github.io/expo-llm-wiki/reference/wiki-memory/).
13
14
 
14
15
  ## Features
15
16
 
16
- - **Platform-agnostic** — Zero runtime dependencies; works with any SQLite driver via the `SQLiteAdapter` interface
17
- - **Semantic search** — Vector embeddings via your LLM's `embed` function, ranked by cosine similarity
18
- - **Keyword fallback** — SQLite FTS5 index stored in the database file, or an in-memory [MiniSearch](https://github.com/lucaong/minisearch) index where FTS5 is unavailable (see `indexStrategy`), for offline/degraded scenarios when embeddings unavailable
19
- - **Retrieval tuning** — Per-call overrides for `maxResults`, `preFilterLimit`, `hybridWeight`, `tierWeights`, `tierFloors`, and `includeZeroWeightEntities`
20
- - **Multi-entity reads** — Search across multiple `entity_id` namespaces in one pass with per-entity score multipliers (`tierWeights`); `tierFloors` reserves each namespace's top-N matching results; optional `factScores` and `metadata` for explainability
21
- - **Immutable vs mutable facts** — Use `WikiFact.source_type` to distinguish document-sourced facts (`immutable_document`) from derived or user-provided facts (`librarian_inferred`, `user_stated`, `user_confirmed`). Immutable document facts are not rewritten by `runLibrarian()` or `runHeal()` and can only be removed by `forget()` or re-ingesting.
22
- - **Full-featured memory** — Facts, tasks, events, maintenance jobs (librarian, heal, reembed, prune)
23
- - **Type-safe** — Built with TypeScript, full type exports
24
- - **Interoperability:** Supports [Open Knowledge Format (OKF)](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf) v0.1 + v0.2 import and export via the [llm-wiki OKF profiles](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/okf-profile.md) (default `llm-wiki/2`, back-compat `llm-wiki/1`).
25
- - **Per-entity seeded ontology** — Optional Strict, Emergent, or Off modes govern LLM graph extraction; seed taxonomies per entity and persist typed facts with inline edges.
26
- - **Diagnostics** — Optional `onDiagnostic` hook with typed, content-free reports of dropped chunks, facts, edges, embedding failures and background-job failures ([Diagnostics](#diagnostics))
27
- - **Draft review** — `excludeDrafts` on reads and traversal, plus `listDrafts` / `promoteDraft` ([Draft Review](#draft-review))
28
- - **Evidence grounding** — Opt-in `grounding` check; facts that don't quote their source are stored as drafts ([Grounding](#grounding))
29
- - **Optional classifier** — `LLMProvider.classify` types facts during ontology backfill when you opt in with `ontology.backfillClassifier: 'auto'` ([Ontology backfill](#ontology-backfill))
30
-
31
- ## GraphRAG & Multi-Modal Retrieval
32
-
33
- `@equationalapplications/core-llm-wiki` exposes three complementary retrieval modes, each addressing a different shape of query:
34
-
35
- | Mode | API | Best for |
36
- |---|---|---|
37
- | **Semantic** (vector cosine) | `wiki.read(entityId, query)` with `embed` configured | Open-ended natural-language questions; "what do I know about X" |
38
- | **Keyword** (FTS5 or MiniSearch) | `wiki.read(entityId, query)` with `embed` absent or offline | Exact terms, identifiers, names; offline fallback |
39
- | **GraphRAG** (recursive CTE) | `wiki.traverseGraph(entityId, options)` + `formatGraphContext(result)` | Structural questions; "what connects to X", "everything two hops from this fact", "summarise the people, places, and projects linked to Alice" |
40
-
41
- The GraphRAG path is structurally distinct: it doesn't rank by relevance to a query string, it walks `llm_wiki_edges` from a known anchor fact. The result is dense and connected — subgraphs, not loose top-K hits.
42
-
43
- ### Graph traversal APIs
44
-
45
- ```typescript
46
- import { WikiMemory, formatGraphContext } from '@equationalapplications/core-llm-wiki';
47
-
48
- const graph = await wikiMemory.traverseGraph('user-123', {
49
- sourceId: '<anchor-fact-id>',
50
- maxDepth: 2,
51
- direction: 'both', // 'inbound' | 'outbound' | 'both'
52
- edgeTypes: ['reports_to'], // optional filter
53
- excludeSourceTypes: ['immutable_document'],
54
- minTraversalConfidence: 'inferred',
55
- maxTraversalNodes: 20,
56
- });
57
-
58
- const promptContext = formatGraphContext(graph);
59
- // → dense text block ready for prompt injection
60
- ```
61
-
62
- `traverseGraph` runs as a single recursive CTE in SQLite (see the root README's ["The SQL: how traversal works in one query"](../../README.md#the-sql-how-traversal-works-in-one-query) for the query shape). No external graph database.
63
-
64
- ### Deterministic graph seeding (no LLM)
65
-
66
- For programmatic pipelines — importing pre-classified data, building a GraphRAG corpus from a CSV, or seed-loading from a JSON file — use `upsertGraph()`. It writes nodes and edges directly under the same `(sourceRef, sourceHash)` ownership semantics as `ingestDocument()`, but skips the LLM extraction step. See [Direct Graph Write](#direct-graph-write) for the canonical signature, including the required `SQLiteAdapter` argument and transactional semantics.
67
-
68
- This is the GraphRAG seed path: load a corpus, walk it.
69
-
70
- ## Direct fact writes
71
-
72
- `remember()` stores one mutable fact in one transaction, with no text-generation (LLM) call, and returns the stored row. When `llmProvider.embed` is configured it does await one embedding call before resolving (failures surface as `embedding_failed`, not a throw). Use it when the host already knows the fact (a user preference, an agent's observation the user approved). To replace a fact, use `supersede()` so history is kept.
73
-
74
- ```typescript
75
- const fact = await wiki.remember('user-123', {
76
- title: 'Prefers merge commits',
77
- body: 'Kurt wants PRs merged with merge commits, never squashed.',
78
- tags: ['git'], // default []
79
- confidence: 'certain', // default
80
- source_type: 'user_stated', // or 'user_confirmed'; default 'user_stated'
81
- okf_type: 'Preference', // optional; resolved like an upsertGraph node type
82
- valid_from: Date.parse('2026-03-01'), // optional epoch ms
83
- });
84
- fact.id; // 'fact_…' — the row as stored: timestamps, defaults, isStale, trustTier
85
- ```
86
-
87
- - Validation throws `TypeError` before anything is written: `title` non-empty, `body` a string, `tags` an array of strings, known `confidence` and `source_type` (explicit `null` is rejected; omit the field to take the default), `okf_type` a non-empty string or null, finite and >= 0 `valid_from`. Nothing is clipped; length limits are the host's.
88
- - `okf_type` follows the `upsertGraph` rule: the manifest's canonical slug when known; `WikiStrictOntologyViolation` under strict mode otherwise; `null` otherwise.
89
- - Never starts the librarian or heal and writes no event, in `'auto'` and `'deferred'` mode alike. Embeds through `llmProvider.embed` when present (failures surface as `embedding_failed` with `operation: 'remember'`); keyword-only otherwise, and `getPendingMaintenance().reembedPending` picks it up once an `embed` exists.
90
- - Not idempotent: two identical calls store two facts. Hosts that expose retries own the idempotency key.
91
- - Opens its own transaction. Do not call it inside `withTransactionAsync`.
92
-
93
- ## Temporal facts
94
-
95
- Facts carry an optional validity window. Nothing changes until you use it: facts without validity data are always current.
96
-
97
- ```typescript
98
- // The user tells you they moved; the move happened on March 1.
99
- await wiki.supersede('user-123', seattleFactId, { title: 'Lives in San Francisco', body: 'Moved in March.' }, {
100
- validFrom: Date.parse('2026-03-01'),
101
- });
102
-
103
- await wiki.read('user-123', 'where do I live'); // current: San Francisco
104
- await wiki.read('user-123', 'where do I live', { asOf: Date.parse('2026-02-01') }); // then: Seattle
105
- await wiki.history('user-123', seattleFactId); // [Seattle, San Francisco]
106
-
107
- // Record when an event really happened (defaults to ingestion time):
108
- await wiki.write('user-123', { event_type: 'observation', summary: 'Moved to SF', occurred_at: Date.parse('2026-03-01') });
109
- ```
110
-
111
- - Validity is half-open: `[valid_from, valid_to)`. At the switch instant only the new fact matches.
112
- - `asOf` answers "what do we now know was true at T". Learning late about a past change still answers correctly for the past.
113
- - `supersede()` refuses `immutable_document` facts (re-ingest the document instead), cycles, facts owned by another entity, and facts already superseded. It throws `WikiSupersedeError` with a `reason`.
114
- - `traverseGraph({ asOf })` only walks edges whose two facts are both live at that instant.
115
- - History is kept. Set `config.pruneSupersededAfter` (days) to let `runPrune` retire expired facts.
116
- - `getMemoryBundle()` and `exportDump()` return every non-deleted fact, including history.
117
-
118
- ## Librarian strategies
119
-
120
- `config.librarian.strategy` selects how events become facts. The default, `'legacy'`, is unchanged from 7.x.
121
-
122
- ```typescript
123
- new WikiMemory(db, {
124
- llmProvider,
125
- config: { librarian: { strategy: 'ops', gate: { k: 5, dupThreshold: 0.89, novelThreshold: 0.30 } } },
126
- });
127
- ```
128
-
129
- `'ops'` works in three steps:
130
-
131
- 1. **Extract.** Candidate facts come from the events since the last pass. Existing facts are not sent.
132
- 2. **Gate, with no LLM call.** Each candidate is compared with its closest current facts. An identical fact is a no-op, and a candidate with no close match is added. Only candidates that are similar but different continue to step 3. Similar is not the same as identical: "lives in Seattle" and "moved to San Francisco" score as similar, so they go to the model.
133
- 3. **Resolve, in one batched call.** The model returns `ADD`, `UPDATE`, `SUPERSEDE` or `NOOP` for each remaining candidate. `SUPERSEDE` uses `supersede()`, so history is kept and `read({ asOf })` still sees the old fact.
134
-
135
- - The thresholds depend on your embedding model. The defaults are provisional.
136
- - Without `embed`, the gate uses keyword search. Every keyword match then goes to the resolve step.
137
- - Under `'ops'`, `runLibrarian`'s `promptOverride` and `prompts.librarianSystemPrompt` are ignored: the ops prompts use a different schema.
138
- - A model answer that can't be used never loses a fact. It is added, or stored as a draft, and a diagnostic is emitted.
139
-
140
- ## Deferred maintenance
141
-
142
- By default, `write()` starts the librarian and heal in the background when thresholds are crossed. On serverless or request-scoped hosts those background promises can be frozen or killed. Set `maintenance: 'deferred'` to decide when maintenance runs:
143
-
144
- ```typescript
145
- const wiki = new WikiMemory(db, {
146
- llmProvider,
147
- config: { maintenance: 'deferred', librarian: { strategy: 'ops' } },
148
- });
149
-
150
- await wiki.write(entityId, { event_type: 'observation', summary, occurred_at });
151
-
152
- const report = await wiki.runPendingMaintenance({ tokenBudget: 20_000, deadlineMs: 8_000 });
153
- // report.stoppedReason: complete | deadline | budget_exhausted | budget_too_small
154
- // Nothing is still running here, so it is safe to persist the database.
155
- ```
156
-
157
- - `getPendingMaintenance()` shows the backlog per entity without calling a model.
158
- - `remember()` and `supersede()` never schedule maintenance in either mode; only `write()` does.
159
- - The budget covers text-generation tokens only; embedding calls are not counted. Figures are provider-reported when `generateTextWithUsage` exists, otherwise estimated at characters ÷ 4.
160
- - Entities share the budget fairly: each round runs one batch per entity.
161
- - `budget_too_small` names the first job and entity whose smallest step needed more than the whole budget; other entities may have hit the same limit in that run.
162
- - A crash mid-batch loses nothing. The librarian's progress marker only advances after a batch commits.
163
- - In `'auto'` mode, call `drain()` before the app is suspended or closed. `autoLibrarianTokenThreshold` also triggers the librarian on the size of the pending text.
164
- - Use `'deferred'` with the ops strategy. The legacy strategy reads only the newest 50 events per pass and marks them processed, so a backlog larger than that is skipped.
165
- - Under `'deferred'` or the ops strategy, `runPrune` never deletes events the librarian has not processed.
166
- - `JobManager` locks only work within one process. Guarantee a single writer per database file across processes.
167
-
168
- See the [SynapseTree integration guide](../../docs/synapsetree-integration.md) for the end-to-end pattern on tenant-aware hosts (S3-backed SQLite, request lifecycle, `entity_id` scheme, MCP surface, costs). Benchmarks live at [docs/benchmarks.md](../../docs/benchmarks.md).
17
+ - **Any SQLite driver.** Pure TypeScript with no native code. You pass a small [`SQLiteAdapter`](https://equationalapplications.github.io/expo-llm-wiki/guides/sqlite-adapter/) for `better-sqlite3`, `sql.js`, `expo-sqlite` or anything else.
18
+ - **Bring your own inference.** One `generateText` function, plus an optional `embed` for semantic search. The engine owns prompts, JSON parsing and database writes.
19
+ - **Hybrid retrieval.** Vector cosine ranking with a keyword fallback (SQLite FTS5 or in-memory MiniSearch), [tunable](https://equationalapplications.github.io/expo-llm-wiki/guides/tune-retrieval/) per call.
20
+ - **Multi-entity reads.** Search several namespaces in one pass with `tierWeights` and `tierFloors`.
21
+ - **GraphRAG.** Per-entity [seeded ontologies](https://equationalapplications.github.io/expo-llm-wiki/guides/ontologies/) (Strict, Emergent or Off) and SQL-only [graph traversal](https://equationalapplications.github.io/expo-llm-wiki/guides/graphrag/).
22
+ - **Background maintenance.** Librarian, heal, reembed and prune jobs, run inline or [deferred](https://equationalapplications.github.io/expo-llm-wiki/guides/maintenance/).
23
+ - **Temporal facts, drafts and grounding** for [trust and provenance](https://equationalapplications.github.io/expo-llm-wiki/concepts/trust-and-provenance/).
24
+ - **OKF v0.1 and v0.2** [import and export](https://equationalapplications.github.io/expo-llm-wiki/guides/okf/).
25
+ - **Diagnostics.** An optional `onDiagnostic` hook with typed, content-free reports.
169
26
 
170
27
  ## Installation
171
28
 
172
29
  ```bash
173
30
  npm install @equationalapplications/core-llm-wiki
31
+ npm install better-sqlite3 # Node.js, or sql.js for the browser
174
32
  ```
175
33
 
176
- ## Semantic Search with Embeddings
177
-
178
- Provide an `embed` function in `llmProvider` to enable vector-based retrieval:
179
-
180
- ```typescript
181
- import { WikiMemory } from '@equationalapplications/core-llm-wiki';
182
-
183
- const wikiMemory = new WikiMemory(db, {
184
- llmProvider: {
185
- generateText: async ({ systemPrompt, userPrompt }) => {
186
- // Your LLM call for extracting facts, tasks
187
- return 'Model output';
188
- },
189
- maxOutputTokens: 4096, // optional — lets runHeal/runOntologyBackfill size their first LLM call correctly instead of discovering the ceiling via a truncated response and retrying smaller
190
- embed: async (text: string) => {
191
- // Your embedding service (e.g., OpenAI, Cohere, local)
192
- const response = await fetch('https://your-app.example.com/api/embed', {
193
- method: 'POST',
194
- body: JSON.stringify({ text }),
195
- });
196
- const { embedding } = await response.json();
197
- return embedding; // number[]
198
- },
199
- },
200
- });
201
-
202
- await wikiMemory.setup();
203
-
204
- // Query with semantic matching
205
- const memory = await wikiMemory.read('user-123', 'What should I do this weekend?');
206
- // Returns facts semantically similar to the query, not lexical matches
207
- // E.g., fact "Saturday hiking trip" ranks high even though no lexical overlap
208
- ```
209
-
210
- When `embed` is unavailable, `read()` silently falls back to keyword search. If an embedding attempt throws, `read()` falls back and calls `onRetrievalFallback` if provided:
211
-
212
- ```typescript
213
- const wikiMemory = new WikiMemory(db, {
214
- llmProvider: {
215
- generateText: async () => { /* ... */ },
216
- embed: undefined, // or throws on network error
217
- },
218
- onRetrievalFallback: (error) => {
219
- console.warn('Embedding retrieval unavailable, using keyword search:', error);
220
- },
221
- });
222
-
223
- // read() returns keyword results, onRetrievalFallback not called (embed absent is expected)
224
- // read() returns keyword results, onRetrievalFallback called (embed threw)
225
- ```
226
-
227
- ### Token budgets
228
-
229
- ```typescript
230
- const bundle = await wiki.read('user-123', 'what should I cook?', { tokenBudget: 800 });
231
- const prompt = formatContext(bundle, { compact: true });
232
- ```
233
-
234
- `tokenBudget` keeps the most relevant facts per token that fit within about 800 tokens. Tokens are estimated as characters ÷ 4. The top fact is always included, truncated if it alone is too big. Tasks and events are unaffected. `traverseGraph` accepts the same option and keeps nodes in breadth-first order. A `read_budget` diagnostic reports how many facts were packed.
235
-
236
- ## Configuration
237
-
238
- All `WikiConfig` fields are optional:
239
-
240
- ```typescript
241
- const wikiMemory = new WikiMemory(db, {
242
- llmProvider: { /* ... */ },
243
- config: {
244
- tablePrefix: 'llm_wiki_', // default: 'llm_wiki_'
245
- maxResults: 10, // default: 10
246
- autoLibrarianThreshold: 20, // default: 20 — events before librarian auto-runs
247
- autoHealThreshold: 100, // default: 100 — events before heal auto-runs
248
- maxChunkLength: 12000, // default: 12000 (char count per ingestDocument chunk; exported as DEFAULT_MAX_CHUNK_LENGTH)
249
- chunkOverlap: 400, // default: 400 (overlap between chunks in characters; exported as DEFAULT_CHUNK_OVERLAP)
250
- chunkConcurrency: 1, // default: 1 (parallel LLM calls per ingestDocument)
251
- maxEmbedChars: 6000, // default: 6000 (chars of title+body+tags sent to embed(); hard ceiling 16000; exported as DEFAULT_MAX_EMBED_CHARS)
252
- pruneRetainSoftDeletedFor: 7, // default: 7 (days before hard-deleting soft-deleted facts)
253
- pruneEventsAfter: 30, // default: 30 (days before hard-deleting old events)
254
- orphanAfterDays: 30, // default: 30 (days before runHeal flags sourceless facts; null to disable)
255
- staleInferredAfterDays: 60, // default: 60 (days before runHeal downgrades inferred facts; null to disable)
256
- preFilterLimit: 50, // default: undefined — keyword pre-filter before cosine scan; recommended for >500 facts
257
- hybridWeight: 0.7, // default: undefined — blend semantic (1.0) ↔ keyword (0.0); pure semantic when unset
258
- // Keyword-index backend (#257): 'fts5' stores the index inside the database file, fed by triggers on the
259
- // entries table, so memory stays flat at any size; setup() throws if the adapter's SQLite lacks FTS5 or json1.
260
- // 'minisearch' keeps the in-RAM MiniSearch index, whose memory grows with total stored text; 'auto' (default)
261
- // probes at setup() and uses FTS5 when present, else MiniSearch (e.g. sql.js, which ships without FTS5).
262
- // The first setup() that attaches FTS5 to an existing database rebuilds the index in one transaction, holding
263
- // the write lock for its whole duration. Keyword rankings differ between the two backends; set 'minisearch'
264
- // to keep the pre-7.11 behavior.
265
- // Under 'auto', a failed FTS5 init is retried once, then setup() completes with keyword search unavailable
266
- // (queries throw WikiSearchUnavailableError); it never silently demotes to MiniSearch. While unavailable,
267
- // runHeal returns searchUnavailable: true, runPendingMaintenance lists skipped jobs in perEntity[].searchUnavailable,
268
- // and a no-arg syncSearchIndex() retries the init (after a failure, backing off from 5 s up to 5 min). MiniSearch holds all
269
- // entry text in the JS heap — the measured ceiling is ~1M entries on a default Node heap (--max-old-space-size).
270
- indexStrategy: 'auto', // default: 'auto' — 'fts5' | 'minisearch' | 'auto'
271
- enableOutbox: false, // default: false — when true, entry/task mutations write to an internal SQLite outbox table for external sync (e.g. via @equationalapplications/prisma-outbox)
272
- excludeDrafts: false, // default: false — engine default for read()/traverseGraph() excludeDrafts; see Draft Review
273
- grounding: { mode: 'off' }, // default: off — 'draft' checks evidence quotes; see Grounding
274
- ontology: {
275
- // mode, seedManifests: see Per-Entity Seeded Ontology
276
- backfillClassifier: 'llm', // default: 'llm' — 'auto' uses llmProvider.classify when present; see Ontology backfill
277
- classifyMinConfidence: 0.5, // default: 0.5 — classifier answers below this are left untyped
278
- },
279
-
280
- // Global prompt overrides — librarianSystemPrompt and healSystemPrompt apply to write() auto-runs;
281
- // ingestSystemPrompt applies only to explicit ingestDocument() calls.
282
- // ⚠ Overrides replace the entire default prompt, including the JSON output contract.
283
- // See "JSON Output Contracts" in the Prompt Management & Overrides section below.
284
- prompts: {
285
- ingestSystemPrompt: `Extract core facts from this document: {{documentChunk}}\n\nReturn ONLY valid JSON: { "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }] }. No markdown.`,
286
- librarianSystemPrompt: `You are an expert curator. Synthesize these thoughts:\n{{events}}\n\nCurrent Facts:\n{{currentFacts}}\n\nReturn ONLY valid JSON: { "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }], "tasks": [{ "description": "string", "priority": 0 }] }. No markdown.`,
287
- healSystemPrompt: `Fix the memory graph based on these candidates: {{healCandidates}}\n\nReturn ONLY valid JSON: { "downgraded": ["factId"], "deleted": ["factId"], "newFacts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }] }. No markdown.`,
288
- },
289
- },
290
- // Host callbacks sit beside llmProvider, not inside config:
291
- // onDiagnostic: (d) => { ... }, // see Diagnostics
292
- });
293
- ```
294
-
295
- ### Opening inside a writer lease
296
-
297
- One-time upgrade work (schema migrations, the `source_ref` normalize) can take minutes on a large store. If you open the store inside a time-limited writer lease, refuse that work instead of running it:
298
-
299
- ```typescript
300
- try {
301
- await wiki.setup({ migrations: 'refuse' }); // also settable as config.migrations
302
- } catch (err) {
303
- if (!(err instanceof WikiMigrationRequiredError)) throw err;
304
- // err.pendingWork says what is pending. Release the lease, then:
305
- await wiki.setup(); // runs the work out of lease (resumable if interrupted)
306
- // ...re-acquire the lease and open again with 'refuse'.
307
- }
308
- ```
309
-
310
- `'allow'` (default) runs everything; `'auto'` throws on a pending schema migration but runs a pending normalize. Await `setup()` before any other call on the instance.
311
-
312
- ## Prompt Management & Overrides
313
-
314
- Core maintenance tasks (`ingestDocument`, `runLibrarian`, `runHeal`) use system prompts to instruct the LLM. You can customize these prompts using `{{mustache}}` style variables to inject context dynamically.
315
-
316
- > **JSON Output Contracts:** Prompt overrides replace the *entire* default system prompt, including the JSON response schema the parser depends on. Your override **must** instruct the LLM to return raw JSON — no markdown. Required shapes:
317
- >
318
- > | Operation | Required JSON shape |
319
- > |-----------|-------------------|
320
- > | `ingestDocument` | `{ "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain\|inferred\|tentative" }] }` |
321
- > | `runLibrarian` | `{ "facts": [...], "tasks": [{ "description": "string", "priority": 5 }] }` — `priority` is an integer 0–10 |
322
- > | `runHeal` | `{ "downgraded": ["factId"], "deleted": ["factId"], "newFacts": [...] }` |
323
- >
324
- > **Grounding:** when [`config.grounding`](#grounding) is on, the evidence instruction is appended after your override (and after ontology context) for every writer in `grounding.writers`. Your override does not need to ask for `evidence` itself.
34
+ Requires Node.js 20 or later. Using Expo or React? Install [`expo-llm-wiki`](https://equationalapplications.github.io/expo-llm-wiki/packages/expo/) or [`react-llm-wiki`](https://equationalapplications.github.io/expo-llm-wiki/packages/react/) instead. Both re-export core.
325
35
 
326
- ### Global Overrides (Auto-Runs)
327
-
328
- If your application relies on `write()` to automatically maintain the memory graph in the background (via `autoLibrarianThreshold` and `autoHealThreshold`), configure custom prompts globally at instantiation. This ensures the internal `WriteService` uses your domain-specific instructions when it triggers an auto-run.
329
-
330
- ```typescript
331
- const wikiMemory = new WikiMemory(db, {
332
- llmProvider,
333
- config: {
334
- prompts: {
335
- // Override must include the JSON output contract — it replaces the entire default prompt.
336
- librarianSystemPrompt: `You are an expert curator. Synthesize these thoughts:\n{{events}}\n\nCurrent Facts:\n{{currentFacts}}\n\nReturn ONLY a valid JSON object: { "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }], "tasks": [{ "description": "string", "priority": 0 }] }. No markdown.`,
337
- },
338
- },
339
- });
340
-
341
- // WriteService uses the global prompt whenever autoLibrarianThreshold is hit
342
- await wikiMemory.write('user-123', { event_type: 'observation', summary: '...' });
343
- ```
344
-
345
- Available `{{variables}}` per prompt type:
346
-
347
- | Prompt | Variables |
348
- |--------|-----------|
349
- | `ingestSystemPrompt` | `{{documentChunk}}` |
350
- | `librarianSystemPrompt` | `{{events}}`, `{{currentFacts}}` |
351
- | `healSystemPrompt` | `{{healCandidates}}`, `{{documentAnchors}}`, `{{allTasks}}`, `{{recentEvents}}` |
352
-
353
- When a template contains `{{variable}}` tags, the matching data is hydrated directly into `systemPrompt` and a short fixed string is used as `userPrompt`. When a template has no `{{}}` tags, the raw data is appended as `userPrompt` — backward compatible with plain-string overrides.
354
-
355
- ### Runtime Overrides (Manual Execution)
356
-
357
- Pass `promptOverride` per-call for one-off instructions. **Runtime overrides apply only to that single call — they do not persist for future auto-runs triggered by `write()`.**
358
-
359
- ```typescript
360
- // Override the base default AND global config for this single execution.
361
- // Each override must include the JSON output contract (replaces the entire default prompt).
362
- await wikiMemory.runLibrarian('user-123', {
363
- promptOverride: `One-off extraction task:\n{{events}}\n\nReturn ONLY valid JSON: { "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }], "tasks": [{ "description": "string", "priority": 0 }] }. No markdown.`,
364
- });
365
-
366
- await wikiMemory.runHeal('user-123', {
367
- promptOverride: `Domain-specific healing: {{healCandidates}}\n\nReturn ONLY valid JSON: { "downgraded": ["factId"], "deleted": ["factId"], "newFacts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }] }. No markdown.`,
368
- });
369
-
370
- await wikiMemory.ingestDocument('user-123', {
371
- sourceRef: 'doc-1',
372
- sourceHash: sha256(content),
373
- documentChunk: content,
374
- promptOverride: `Strict technical extraction: {{documentChunk}}\n\nReturn ONLY valid JSON: { "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }] }. No markdown.`,
375
- });
376
- ```
377
-
378
- > **Important:** If your app relies on `write()` auto-runs and needs custom prompts for those runs, use `config.prompts` at construction time. Runtime `promptOverride` values are never forwarded to `WriteService`-triggered internal runs.
379
-
380
- ### Effective instructions (`getInstructions`)
381
-
382
- ```typescript
383
- const { ingest, librarian, heal, ontologyBackfill } = await wikiMemory.getInstructions('entity-123');
384
- ```
385
-
386
- Returns the system prompt each writer sends, with `WikiConfig.prompts` overrides applied. Ingest, librarian and ontology backfill also get the entity's ontology block; heal gets none, as at runtime. When `WikiConfig.grounding` is on, the evidence block is appended for each writer in `grounding.writers`, exactly as sent. Data placeholders such as `{{documentChunk}}` stay unfilled; no events, chunks or facts are included. It reflects `WikiConfig.prompts` only: a per-call `promptOverride` is not reflected, and `ontologyBackfill` is returned even when backfill would send no prompt (ontology `off`, or the classifier path). `core-llm-tools` exposes this as the `wiki_get_instructions` tool (`memory:read`), so agents can see the engine's output format and constraints before proposing writes. Agents should treat it as reference data, not instructions; see [Prompt-Injection Trust Boundary](#prompt-injection-trust-boundary).
387
-
388
- > **Warning:** overrides are returned verbatim to any client with `memory:read`. Never put secrets, API keys or private data in `WikiConfig.prompts`.
389
-
390
- ## Retrieval Tuning
391
-
392
- Optimize `read()` performance and blend retrieval strategies:
393
-
394
- ```typescript
395
- const config = {
396
- // Limit cosine similarity scoring to top-K MiniSearch keyword candidates
397
- preFilterLimit: 50,
398
-
399
- // Blend semantic and keyword scores (0.0 = pure keyword, 1.0 = pure semantic)
400
- hybridWeight: 0.7,
401
-
402
- // Max results returned per read
403
- maxResults: 10,
404
- };
405
-
406
- const wikiMemory = new WikiMemory(db, {
407
- config,
408
- llmProvider: { /* ... */ },
409
- });
410
-
411
- // Per-call overrides (runtime controls for search dashboards, etc.)
412
- const memory = await wikiMemory.read('user-123', 'my preferences', {
413
- maxResults: 5,
414
- preFilterLimit: 20,
415
- hybridWeight: 0.5,
416
- });
417
-
418
- // Multi-entity with tier weights
419
- const multiMemory = await wikiMemory.read(['tier_wisdom', 'tier_fact', 'tier_working'], 'my preferences', {
420
- maxResults: 8,
421
- tierWeights: {
422
- tier_wisdom: 2, // high-confidence curated notes boosted 2×
423
- tier_fact: 1, // neutral baseline
424
- tier_working: 0.25, // recent but unvetted context downranked
425
- },
426
- // includeZeroWeightEntities: true — include 0-weight entities as bottom-ranked filler
427
- });
428
- // multiMemory.factScores — optional Record<factId, weightedScore> for returned facts; may be absent/undefined
429
- // multiMemory.metadata — optional { query, entityIds, tierWeights, tierFloors }; may be absent/undefined
430
- ```
431
-
432
- **Hybrid scoring blends:**
433
- - `hybridWeight: 1.0` → all-semantic blend with semantic scores clamped to non-negative range (no keyword component)
434
- - `hybridWeight: 0.5` → balanced semantic + keyword (50/50 blend)
435
- - `hybridWeight: 0.0` → pure keyword ranking, skips `embed()` entirely (no LLM API cost)
436
-
437
- True cosine-range pure semantic ranking (including negative cosine values) is used when `hybridWeight` is left `undefined`.
438
-
439
- **Tier weights:**
440
- - `tierWeights` applies a per-entity multiplier after semantic/keyword scoring: `finalScore = retrievalScore × weight`
441
- - Missing weights default to `1.0`. Negative weights clamp to `0`. Non-finite weights default to `1.0`.
442
- - `tierWeights[entity] = 0` skips that entity's scored retrieval branch (no compute cost).
443
- - `includeZeroWeightEntities: true` includes zero-weight entities as bottom-ranked filler instead of skipping them.
444
- - `tierFloors: Record<string, number>` reserves that entity's top-N scored results before the global `maxResults` cut — a floor on entity `X` means at least `N` of the returned facts come from `X`. Floors apply *after* scoring (including `tierWeights`) and *after* `preFilterLimit`'s candidate selection, so a row excluded by `preFilterLimit` cannot be resurrected by a floor. An entity with fewer matching facts than its floor contributes what it has — this is not an error. Only meaningful when `entityId` is an array; ignored for single-string calls and for empty-query ("recent facts") reads. Throws `WikiInvalidReadOptions` when a floor cannot be satisfied by construction: floors summing above `maxResults`, a floor keyed to an entity not in `entityId`, or a floor on an entity excluded by `tierWeights: 0` when `includeZeroWeightEntities` is not set. With an external `vectorRanker`, ranker-omitted rows (typically un-embedded facts) for a floored entity are pre-reserved before the global backfill budget is spent, so an unembedded floored entity still satisfies its floor.
445
- - `factScores` is present for array-shaped `entityId` calls only when the query is non-empty and at least one fact is scored; empty-query ("recent facts") reads leave it absent even when `entityId` is an array. Plain string calls never expose it. `metadata` is present for all array-shaped calls regardless of query.
446
- - `maxResults` applies globally across all requested entities.
447
- - Tasks are capped at `min(20 × entityCount, 200)`; events at `min(10 × entityCount, 100)` for multi-entity reads.
448
-
449
- **Pre-filtering optimization:**
450
- When `preFilterLimit: 50` is set with 1000 facts, cosine similarity is computed only for the top 50 keyword matches, reducing O(N) scoring to O(50).
451
-
452
- ## Draft Review
453
-
454
- Facts can carry `lifecycle_status: 'draft'`, for example when a host marks model output as unreviewed. Drafts stay visible by default. To keep them out of results:
36
+ ## Quick look
455
37
 
456
38
  ```ts
457
- await wiki.read('user-1', 'deploy process', { excludeDrafts: true });
458
- await wiki.traverseGraph('user-1', { sourceId, excludeDrafts: true });
459
- // or engine-wide:
460
- createWiki(db, { llmProvider, config: { excludeDrafts: true } });
461
- ```
39
+ import { createWiki, formatContext } from '@equationalapplications/core-llm-wiki';
462
40
 
463
- - On every `read()` path, drafts are removed **before** `maxResults`, `tierFloors`, and pre-filter cuts, so they never take slots from reviewed facts.
464
- - In traversal, drafts are dead ends. The starting fact is always returned.
465
- - Status is read from SQLite on every call. A promotion is visible immediately, with no re-indexing.
41
+ const wiki = createWiki(adapter, { llmProvider }); // your SQLiteAdapter and LLMProvider
42
+ await wiki.setup();
466
43
 
467
- Review API:
468
-
469
- ```ts
470
- const { facts, nextCursor } = await wiki.listDrafts('user-1', { limit: 50 });
471
- await wiki.promoteDraft(facts[0].id, 'user-1', { by: 'human:alice' }); // → stable, trustTier 'human-reviewed'
472
- // Reject with setLifecycleStatus(id, entityId, 'deprecated') or forget().
44
+ await wiki.write('user-123', { event_type: 'observation', summary: 'Loves hiking on weekends.' });
45
+ await wiki.runLibrarian('user-123'); // extract facts from events
46
+ const context = formatContext(await wiki.read('user-123', 'weekend plans'));
473
47
  ```
474
48
 
475
- `promoteDraft` throws `WikiDraftNotFound` when no live draft with that id exists for the entity. The error is contextless by design. Promotion does not change `updated_at`, so a promoted fact keeps its recency position.
476
-
477
- ## Grounding
478
-
479
- Opt-in, deterministic evidence check for LLM-authored facts. When on, writers you choose must quote the source they were shown. A fact whose quotes are missing or not found is stored as a `draft` (see [Draft Review](#draft-review)), not rejected. A fact whose quotes all check out is stored `stable` with a `process:grounding-check` verifier, so its `trustTier` is `machine-confirmed`. Default off: 7.x write behavior is unchanged.
480
-
481
- ```ts
482
- new WikiMemory(db, {
483
- llmProvider,
484
- config: {
485
- grounding: {
486
- mode: 'draft', // 'off' (default) | 'draft'
487
- writers: ['ingest'], // default ['ingest']; also 'librarian', 'heal'
488
- minEvidenceChars: 20, // shorter quotes count as absent
489
- maxEvidence: 3, // quotes asked for per fact
490
- maxEvidenceChars: 300,
491
- },
492
- },
493
- });
494
- ```
495
-
496
- - **What counts as source.**
497
- - Ingest: the chunk text.
498
- - Librarian: the `summary` of each event in the prompt.
499
- - Heal: the `summary` of each recent event in the prompt, plus the bodies of non-draft document anchors. When heal is a writer, anchors are shown with their body clipped to 800 characters.
500
- - Instructions, the ontology manifest, existing facts and identifiers never count, so a model cannot ground a claim by quoting them.
501
- - **The check.** Both sides are normalized with NFKC, whitespace runs collapse to one space, and matching is case-sensitive. A fact with more than 10 quotes, or any quote not found, fails.
502
- - **Diagnostics.** `grounding_missing` (reasons `no_evidence`, `evidence_too_short`) and `grounding_failed` (reasons `quote_not_found`, `too_many_quotes`), one per fact, with the new fact's `factId`. Quotes are never included.
503
- - **Duplicate titles in one ingest.** When chunks yield facts with the same title, ingest keeps one: a grounded fact beats one with missing or failed evidence, and on a tie the first one wins. The others are reported as `fact_deduplicated`. A fact already stored for the same `sourceRef` is not replaced by a later partial ingest, and that skip is reported the same way. Either way the diagnostic carries `{ sourceRef, chunkIndex, itemIndex }` for the skipped fact, whether or not grounding is enabled.
504
- - **`upsertGraph`** nodes are host-supplied and never grounded.
505
- - **Librarian and heal** synthesize across events, so their pass rates are unknown. Measure them on your own event log before opting them in.
506
- - Evidence quotes are not stored.
507
-
508
- ## Pluggable Vector Retrieval
509
-
510
- When your entity corpus grows, in-process cosine similarity scoring becomes a bottleneck. The optional **`VectorRanker`** interface lets you delegate semantic ranking to [**sqlite-vec**](https://github.com/asg017/sqlite-vec), [**sqlite-vss**](https://github.com/asg017/sqlite-vss), or an external vector database while `WikiMemory` handles embedding validation, hybrid scoring, and tier-2 row hydration.
511
-
512
- ### `VectorRanker` purpose
513
-
514
- `VectorRanker` provides an optional injection point for approximate nearest-neighbor (ANN) ranking:
515
-
516
- ```typescript
517
- export interface VectorRanker {
518
- /**
519
- * Return semantic scores for facts in scope, sorted by similarity.
520
- * - `entityId`: restricts results to one entity
521
- * - `queryVec`: the embedded query (Float32Array or number[])
522
- * - `candidateIds` (optional): when set, rank only within this set (MiniSearch pre-filter mode)
523
- * - `limit`: requested top-K count
524
- */
525
- rankBySimilarity(args: VectorRankerRankArgs): Promise<VectorRankerSemanticResult[]>;
526
-
527
- /**
528
- * Optional hook called after embedding persistence (upsert, reembed, delete).
529
- * Implementations use this to keep external indexes (sqlite-vec, remote ANN) in sync.
530
- */
531
- onEmbeddingPersisted?(event: {
532
- entityId: string;
533
- factId: string;
534
- vector: Float32Array | null; // null = embedding removed
535
- }): void | Promise<void>;
536
- }
537
- ```
538
-
539
- **When no ranker is configured**, `WikiMemory` uses built-in JS cosine similarity — the same behavior as today. When a ranker is supplied and embeddings preconditions are met (`embed` available, dimensions match, no mismatches), `WikiMemory` delegates scoring to the ranker and blends results with keyword scores.
540
-
541
- ### Example: sqlite-vec adapter
542
-
543
- ```typescript
544
- import { WikiMemory } from '@equationalapplications/core-llm-wiki';
545
- import type { VectorRanker, VectorRankerRankArgs, VectorRankerSemanticResult } from '@equationalapplications/core-llm-wiki';
546
-
547
- // Minimal sqlite-vec adapter (pseudo-code)
548
- const sqliteVecRanker: VectorRanker = {
549
- async rankBySimilarity(args: VectorRankerRankArgs): Promise<VectorRankerSemanticResult[]> {
550
- const { entityId, queryVec, candidateIds, limit } = args;
551
-
552
- // Build KNN query using sqlite-vec's distance functions.
553
- // sqlite-vec returns cosine distance (0 = identical, 2 = opposite) ascending.
554
- // Invert to semanticScore: higher = more similar, matching VectorRanker contract.
555
- let sql = `SELECT id, (1.0 - distance) AS semanticScore FROM vec_facts
556
- WHERE entity_id = ? AND deleted_at IS NULL`;
557
- const params: any[] = [entityId];
558
-
559
- // Apply pre-filter if provided
560
- if (candidateIds) {
561
- sql += ` AND id IN (${candidateIds.map(() => '?').join(',')})`;
562
- params.push(...candidateIds);
563
- }
564
-
565
- // KNN search (example syntax; adjust for your sqlite-vec version)
566
- sql += ` ORDER BY vec MATCH vec_neighbor(?) LIMIT ?`;
567
- params.push(queryVec, limit);
568
-
569
- const rows = await db.getAllAsync<{ id: string; semanticScore: number }>(sql, params);
570
- return rows; // sorted descending by semanticScore (closest distance → highest similarity)
571
- },
572
-
573
- async onEmbeddingPersisted(event) {
574
- const { entityId, factId, vector } = event;
575
- if (vector) {
576
- // Upsert into sqlite-vec table
577
- await db.runAsync(
578
- `INSERT OR REPLACE INTO vec_facts (id, entity_id, vec) VALUES (?, ?, ?)`,
579
- [factId, entityId, vector]
580
- );
581
- } else {
582
- // Delete when embedding is removed
583
- await db.runAsync(`DELETE FROM vec_facts WHERE id = ?`, [factId]);
584
- }
585
- },
586
- };
587
-
588
- const wikiMemory = new WikiMemory(db, {
589
- llmProvider: { /* ... */ },
590
- vectorRanker: sqliteVecRanker,
591
- });
592
-
593
- // read() now uses sqlite-vec for scoring instead of JS cosine
594
- const memory = await wikiMemory.read('user-123', 'my preferences');
595
- ```
596
-
597
- ### Fallback policies
598
-
599
- When `rankBySimilarity` rejects (e.g., ANN service outage, misconfiguration), `WikiMemory` applies a recovery policy:
600
-
601
- ```typescript
602
- export type VectorRankerFallback =
603
- | 'js-cosine' // (default) Score candidates in-process with JS cosine — same as no ranker
604
- | 'keyword' // Skip semantic ranking; return keyword-only results
605
- | 'empty' // Semantic facts list empty for this read; tasks/events still included
606
- | 'throw'; // Reject read() with the ranker error
607
-
608
- const wikiMemory = new WikiMemory(db, {
609
- llmProvider: { /* ... */ },
610
- vectorRanker: sqliteVecRanker,
611
- vectorRankerFallback: 'js-cosine', // default
612
- onVectorRankerFallback: (info) => {
613
- console.warn(
614
- `Ranker failed (policy: ${info.policy}); error:`,
615
- info.error
616
- );
617
- },
618
- });
619
- ```
620
-
621
- - **`'js-cosine'` (default):** Seamless degradation; same behavior as if no ranker was configured.
622
- - **`'keyword'`:** Useful when semantic ranking is optional; keyword search proceeds normally.
623
- - **`'empty'`:** Return no facts for this query (but tasks/events still load); useful for strict consistency.
624
- - **`'throw'`:** Propagate the error and fail the read.
625
-
626
- ### `onEmbeddingPersisted` eventual consistency
627
-
628
- If `vectorRanker.onEmbeddingPersisted` returns a pending Promise, the hook **may resolve asynchronously**. This supports ANN indexes that rebuild on a schedule (e.g., sqlite-vec triggers on transaction commit) or external services with eventual consistency.
629
-
630
- **Best practice:**
631
- - If your adapter has **synchronous guarantees** (in-process sqlite-vec, same transaction), await the promise.
632
- - If your adapter is **eventually consistent** (remote ANN, async rebuild), document the lag and document that queries may miss recently-added facts until the index refreshes.
633
- - The **SQLite blob remains the source of truth**; `WikiMemory` always writes embeddings to `embedding_blob` first before calling the hook.
634
-
635
- ### Hybrid scoring with ranker
636
-
637
- When both `vectorRanker` and `hybridWeight` are configured, `WikiMemory` still applies hybrid blending after the ranker returns scores:
638
-
639
- ```typescript
640
- const wikiMemory = new WikiMemory(db, {
641
- config: {
642
- hybridWeight: 0.7, // 70% semantic, 30% keyword
643
- },
644
- vectorRanker: sqliteVecRanker,
645
- });
646
-
647
- // ranker returns semanticScore; WikiMemory blends with MiniSearch keyword score
648
- const memory = await wikiMemory.read('user-123', 'my preferences', {
649
- hybridWeight: 0.5, // per-call override to 50/50 blend
650
- });
651
- ```
652
-
653
- Note on semantics:
654
- - Leave `hybridWeight` undefined for true pure-semantic cosine-range scoring.
655
- - Set `hybridWeight: 1` for an all-semantic variant that clamps negative semantic scores to 0.
656
-
657
- For details on hybrid scoring formulas and trade-offs, see [Retrieval Tuning](#retrieval-tuning) above.
658
-
659
- ### Spec and issue reference
660
-
661
- - **Full spec:** [`docs/superpowers/specs/2026-05-07-pluggable-vector-retrieval.md`](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/superpowers/specs/2026-05-07-pluggable-vector-retrieval.md)
662
- - **GitHub issue:** [#15](https://github.com/equationalapplications/expo-llm-wiki/issues/15)
663
-
664
- ## Vector Cache
665
-
666
- Parsed embedding vectors from full-scan `read()` calls are cached in memory, keyed by entity ID (max 16 entities, max 500 vectors per entity). This avoids redundant `Float32Array` parsing on repeated queries for the same entity. When the 16-entity limit is reached, the oldest-inserted entity is evicted to make room; if an entity exceeds 500 facts, its vectors are not cached at all for that read.
667
-
668
- After heavy read workloads or on memory-constrained runtimes, you can release the entire cache explicitly:
669
-
670
- ```typescript
671
- // Release all cached embedding vectors
672
- wikiMemory.clearVectorCache();
673
- ```
674
-
675
- The cache is also automatically invalidated on any mutation (`runLibrarian`, `runHeal`, `runPrune`, `runReembed`, `ingestDocument`, `importDump`, `forget`).
676
-
677
- ## Re-Embedding & Retry Behavior
678
-
679
- `runReembed(entityId?, opts?)` returns `{ embedded, skipped, failed, deferred, permanentlyFailed }`.
680
-
681
- - `failed` — attempted this sweep and failed.
682
- - `deferred` — previously failed and still inside its exponential backoff window (60s doubling, capped at 24h). Not an error; a later sweep will retry.
683
- - `permanentlyFailed` — excluded for good: a `float32_overflow` failure, or 5 failed attempts. Pass `{ force: true }` to retry these anyway.
684
-
685
- Convergence loops should test `failed`, never `deferred` — deferred rows clear themselves once their backoff elapses.
686
-
687
- Two caveats on that guidance:
688
-
689
- - A `storage_error` failure counts in `failed` but is never marked for backoff — a failure marker is itself a DB write, which is exactly what is broken. While DB writes fail, every sweep re-attempts every row, so a loop testing `failed` can spin without converging. Treat repeated `storage_error` failures as an infrastructure signal to fix, not a retry window to wait out.
690
- - `{ force: true }` bypasses more than `permanentlyFailed`: it also skips the backoff window, so rows that would otherwise be `deferred` are re-attempted immediately.
691
-
692
- ## Entity Status
693
-
694
- `WikiMemory` exposes the in-flight job state for a single entity through two complementary APIs.
695
-
696
- ### `getEntityStatus(entityId)`
697
-
698
- Synchronous point-in-time snapshot:
699
-
700
- ```typescript
701
- const status = wikiMemory.getEntityStatus('user-42');
702
- // { ingesting: boolean, librarian: boolean, heal: boolean }
703
- ```
704
-
705
- Use this when you only need the current value (e.g. inside a request handler).
706
-
707
- ### `subscribeEntityStatus(entityId, callback)`
708
-
709
- Push-based change notification — the callback fires synchronously once with the current status, then again on every transition where any of the three booleans flips. There is no polling and no duplicate snapshots.
710
-
711
- ```typescript
712
- const unsubscribe = wikiMemory.subscribeEntityStatus('user-42', (status) => {
713
- console.log(status); // { ingesting, librarian, heal }
714
- });
715
-
716
- // Later:
717
- unsubscribe(); // idempotent — safe to call more than once
718
- ```
719
-
720
- Notes:
721
-
722
- - The first invocation happens **before** `subscribeEntityStatus` returns. Treat it as the initial render value.
723
- - Each emission may be a fresh object literal. Do not rely on referential equality between callbacks; equality of the three booleans is the contract.
724
- - A throwing callback is caught (logged via `console.error`) and does not block other subscribers or the underlying job.
725
- - Subscriptions are scoped to a single `entityId`. There is no wildcard or "all entities" form.
726
-
727
- ## Diagnostics
728
-
729
- Pass `onDiagnostic` to receive typed, content-free reports of events core used to drop silently or only log: failed chunks, rejected facts and tasks, dedupe drops, dropped edges, embedding and host-hook failures, heal skips, and background-job failures.
730
-
731
- ```ts
732
- const wiki = createWiki(db, {
733
- llmProvider,
734
- onDiagnostic: (d) => {
735
- // d.code, d.severity ('info' | 'warn' | 'error'), d.operation, d.trigger ('call' | 'auto'),
736
- // d.entityId, d.at, d.message, d.detail?: { factId, sourceRef, chunkIndex, itemIndex,
737
- // edgeType, sourceNodeType, targetNodeType, reason, count, chunkIndexes }
738
- telemetry.record(d);
739
- },
740
- });
741
- ```
742
-
743
- - **Content-free.** Diagnostics carry IDs, indexes, counts, ontology slugs and reason slugs only. They never carry titles, bodies, LLM output, provider error messages, or hashes of content.
744
- - **Isolated.** The hook is called synchronously. A throwing or rejecting hook never affects the operation, and its failure is logged with `console.warn`.
745
- - **After commit.** Transactional diagnostics are delivered after the operation's transaction commits. An operation that throws delivers none; the exception is the signal. `upsertGraph` runs in your transaction, so its diagnostics are delivered when it resolves. Disregard them if you roll back.
746
- - **`trigger`.** `'auto'` marks work started by `autoLibrarianThreshold` / `autoHealThreshold`. A failed background job is reported as `background_job_failed` with `operation` set to the job.
747
- - **`operation`** names the entry point: `ingest`, `upsertGraph`, `librarian`, `heal`, `ontologyBackfill`, `reembed`, `importDump`, `write`, `supersede`, `remember`, `read`, `prune`. New values may be added in minor releases.
748
- - **Forward-compatible.** New codes may be added in minor releases; ignore codes you don't recognize.
749
- - **Console output is unchanged** when a hook is set and succeeds: every existing core line is identical. A throwing or rejecting hook adds only its own `console.warn`, and a non-function `onDiagnostic` warns once.
750
-
751
- ## Per-Entity Seeded Ontology
752
-
753
- Control how librarian and ingest passes classify facts and extract graph relationships. The system defaults to **`off`** so existing deployments behave unchanged.
754
-
755
- ### The Three Modes
756
-
757
- | Mode | Behavior |
758
- |------|----------|
759
- | **`off`** (default) | No ontology guidance. LLM output and persistence match pre-ontology behavior: `okf_type` stays `null` on LLM-created facts; maintenance passes do not create edges. OKF import still populates `okf_type` and edges independently. |
760
- | **`strict`** | The LLM must use only `node_types` and `edge_types` from the entity manifest. Invalid `okf_type` falls back to an untyped fact with no edges; invalid individual edges are dropped while a valid `okf_type` and matching edges are kept. |
761
- | **`emergent`** | Same validation as Strict, plus the LLM may return `ontology_updates` with new node/edge types. Updates are append-only (deduped by `type` string) and take effect before facts from the same response are validated. |
762
-
763
- Mode resolution per entity: persisted DB row `mode` (when present) → `seedManifests[entityId].mode` (when no row but a seed exists) → `WikiConfig.ontology.mode` → `'off'`.
764
-
765
- ### WikiConfig
766
-
767
- Set a global default mode and bootstrap manifests for known entities at construction time:
768
-
769
- ```typescript
770
- const wikiMemory = new WikiMemory(db, {
771
- llmProvider,
772
- config: {
773
- ontology: {
774
- mode: 'strict', // global default when an entity has no per-entity override
775
- seedManifests: {
776
- 'team-alpha': {
777
- mode: 'emergent', // optional per-entity override
778
- manifest: {
779
- node_types: [
780
- { type: 'person', description: 'An individual or user.' },
781
- { type: 'project', description: 'An ongoing initiative.' },
782
- ],
783
- edge_types: [
784
- {
785
- type: 'contributes_to',
786
- source_type: 'person',
787
- target_type: 'project',
788
- description: 'Person working on a project.',
789
- },
790
- ],
791
- },
792
- },
793
- },
794
- },
795
- },
796
- });
797
- ```
798
-
799
- `seedManifests` entries are written to SQLite the first time an entity's ontology is
800
- resolved *inside a transaction* (ingest, heal) and no row exists for it. A read outside a
801
- transaction — including `getOntologyManifest` — resolves the seed without persisting it, so
802
- a configured entity can report a manifest while still having no row in `entity_manifests`.
803
- That distinction matters for `ifAbsent` below.
804
-
805
- ### Public API
806
-
807
- Read or seed an entity's ontology at runtime:
808
-
809
- ```typescript
810
- // Read effective mode + manifest (DB row, then seedManifests fallback)
811
- const ontology = await wikiMemory.getOntologyManifest('team-alpha');
812
- // { mode: 'emergent', manifest: { node_types: [...], edge_types: [...] } }
813
- // null when no row and no seed entry
814
-
815
- // Seed or replace manifest; optional per-entity mode override
816
- await wikiMemory.setOntologyManifest('team-alpha', {
817
- node_types: [{ type: 'person', description: 'An individual.' }],
818
- edge_types: [{
819
- type: 'reports_to',
820
- source_type: 'person',
821
- target_type: 'person',
822
- description: 'Reporting hierarchy.',
823
- }],
824
- }, { mode: 'strict' });
825
- ```
826
-
827
- Seed several entities atomically — all manifests are written in one transaction,
828
- so a failure partway through leaves none of them behind:
829
-
830
- ```typescript
831
- const { written, skipped } = await wikiMemory.setOntologyManifests(
832
- [
833
- { entityId: 'tier_fact', manifest, mode: 'strict' },
834
- { entityId: 'tier_wisdom', manifest, mode: 'strict' },
835
- ],
836
- { ifAbsent: true },
837
- );
838
- // written: entities whose manifest this call wrote
839
- // skipped: entities that already had a persisted manifest (only under `ifAbsent`)
840
- ```
841
-
842
- `ifAbsent` makes each write create-if-absent rather than an upsert, so a
843
- concurrent initializer loses the race by writing nothing instead of overwriting
844
- a manifest it never read. Omit it for replace-on-conflict, which is what
845
- `setOntologyManifest` does.
846
-
847
- **`ifAbsent` tests for a persisted row, not for an effective manifest.** An entity
848
- whose manifest comes from `WikiConfig.ontology.seedManifests` has no row until an
849
- ingest materializes one, so `ifAbsent` writes over the configured seed and reports
850
- the entity in `written`; after an ingest has run for that entity the same call
851
- reports it in `skipped`. Don't mix the two seeding routes for one entity: seed it
852
- through the config, or through this method, not both.
853
-
854
- The method takes data, never a transaction handle: `WikiMemory` serializes
855
- transactions on the adapter it is given, so a transaction opened on the adapter
856
- you passed to `createWiki` would not participate in that serialization.
857
-
858
- ### Fact Shape Extensions
859
-
860
- In **Strict** and **Emergent** modes, librarian and ingest JSON may include typed facts with inline edges:
861
-
862
- ```json
863
- {
864
- "facts": [{
865
- "title": "Jane reports to Bob",
866
- "body": "Jane reports to Bob Smith.",
867
- "tags": [],
868
- "confidence": "certain",
869
- "okf_type": "person",
870
- "edges": [{ "edge_type": "reports_to", "target_title": "Bob Smith" }]
871
- }]
872
- }
873
- ```
874
-
875
- - `okf_type` maps to a `node_types[].type` entry (case-insensitive lookup; canonical manifest casing is persisted).
876
- - `edges` are resolved by `target_title` within the same maintenance transaction and persisted via `EdgeRepository`.
877
- - Invalid `okf_type` falls back to `null` with no edges for that fact. Invalid individual edges are dropped; valid `okf_type` and matching edges are still persisted.
878
-
879
- See the design spec: [`docs/superpowers/specs/2026-06-23-per-entity-seeded-ontology-design.md`](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/superpowers/specs/2026-06-23-per-entity-seeded-ontology-design.md).
880
-
881
- ### Ontology type inheritance
882
-
883
- `OntologyNodeType` accepts an optional `parent_type` field for **one-level type inheritance** — the building block for polymorphic queries like "all CreativeWorks" without deep, multi-level hierarchies.
884
-
885
- ```ts
886
- // packages/core/src/types.ts
887
- export interface OntologyNodeType {
888
- type: string;
889
- description: string;
890
- /** Optional parent type slug. One level only — the parent must exist in the
891
- * same manifest and must not itself declare a `parent_type`. */
892
- parent_type?: string;
893
- }
894
- ```
895
-
896
- A concrete type declares `parent_type: '<parent-slug>'`; the parent must be a top-level node in the same manifest. A `design_spec` node with `parent_type: 'creativework'` is treated as both itself *and* a `creativework` during edge matching.
897
-
898
- #### Inheritance rules (strictly enforced)
899
-
900
- `validateManifest` enforces the **one-level invariant** at every persist and read (`entity_manifests.manifest_json` is `JSON.parse`d on every read, so validation runs on the read path too). Violations throw:
901
-
902
- | Condition | Error |
903
- |-----------|-------|
904
- | `parent_type` references an unknown slug | `Parent type not found: <slug>` |
905
- | `parent_type` equals the node's own slug | `Self-parent: <type>` |
906
- | The referenced parent also declares a `parent_type` | `Parent chain too deep: <a> → <b> → <grandparent>` |
907
- | `parent_type` is present but blank (`''` / whitespace) or non-string (`number`, `null`, `object`) | `Ontology parent_type must be a non-empty string when present: <type>` |
908
-
909
- Two consequences worth knowing:
910
-
911
- - The depth check rejects **2-cycles** (`a → b → a`) which a self-parent check alone would miss.
912
- - A *present* `parent_type` key is required to be a usable string; an *absent* key (`undefined`) is fine and means "no parent". This is the same rule core applies to blank slugs elsewhere (`Ontology node type slug must be non-empty`).
913
-
914
- Parent types are **instantiable** — a fact may be classified as bare `creativework`, and `resolveNodeType('creativework')` returns `'creativework'` normally. There is no `abstract` flag; if you want a parent to stay abstract, write the description to steer classification toward the concrete children.
915
-
916
- #### Edge matching (symmetric, exact-first)
917
-
918
- Edge matching is parent-aware on **both** sides and routes through a single primitive, `typeSatisfies(declaredType, concreteType, manifest)`:
919
-
920
- - `declaredType === concreteType` (case-insensitive) → **exact match**, short-circuits before the node lookup. Manifests with no `parent_type` behave bit-for-bit as before.
921
- - Otherwise: looks up `concreteType`'s definition in the manifest and returns `true` if its `parent_type` equals `declaredType`. One hop only — `typeSatisfies` never recurses.
922
-
923
- `typeSatisfies` is used at four gates — `validateInlineEdges` (source), `OntologyService.resolveEdges` (source and target, via two-pass), `IngestionService.upsertGraph` (source) — so a parent-satisfied edge that validates will also persist.
924
-
925
- **Targets resolve exact-first.** When resolving an edge against a target fact, `OntologyService.resolveEdges` runs two passes over candidate defs:
926
-
927
- ```ts
928
- // Packages/core/src/services/OntologyService.ts (abridged)
929
- const targetType = (target.okf_type ?? '').trim().toLowerCase();
930
- const def = candidates.find(d => d.target_type.trim().toLowerCase() === targetType)
931
- ?? candidates.find(d => typeSatisfies(d.target_type, targetType, manifest));
932
- ```
933
-
934
- The exact pass runs first, so a `design_spec` target with both `about creativework → creativework` and `about creativework → design_spec` declared resolves to the **narrower** row — array order never decides which pass wins. Ties within a single pass are immaterial: only `def.type` is read off the winning def, and `validateManifest` already rejects a manifest whose triples spell one edge name with different casing, so every candidate within a pass yields a byte-identical edge row.
935
-
936
- > **Accepted cost:** declaring `→ creativework` now admits every child of `creativework`, and there is no way to say "the parent type only." A type that needs an exact-only target must not be given children.
937
-
938
- #### Emergent prompt schema
939
-
940
- In `emergent` mode the LLM may propose new types via `ontology_updates`. The prompt schema (`EMERGENT_EXTRA` in `packages/core/src/prompts/ontology.ts`) advertises the optional `parent_type` so emergent proposals can suggest children of an existing type:
941
-
942
- ```json
943
- "ontology_updates": {
944
- "node_types": [{ "type": "slug", "description": "...", "parent_type": "optional existing slug" }],
945
- "edge_types": [{ "type": "slug", "source_type": "...", "target_type": "...", "description": "..." }]
946
- }
947
- ```
948
-
949
- The rest of the manifest reaches the LLM unchanged — `buildPromptContext` does `JSON.stringify(manifest, null, 2)`, so an established manifest's `parent_type` fields appear verbatim alongside `type` and `description`.
950
-
951
- Emergent proposals are **untrusted input**: `mergeOntologyUpdates` drops any `parent_type` (rather than throwing) when it is a non-string, blank, unresolvable, self-referential, or whose referenced parent already declares its own parent. The lenient merge contract keeps malformed LLM proposals from aborting an ingest transaction. Changing an established type's parent is a `setOntologyManifest` operation.
952
-
953
- #### Backwards compatibility
954
-
955
- `parent_type` is optional and manifests persist as a whole JSON blob in `entity_manifests.manifest_json`. Existing manifests without the field validate identically; no SQLite migration is needed. `setOntologyManifest` rejects a two-level chain at the public API boundary, so callers cannot accidentally introduce an unenforced chain through a typo.
956
-
957
- `typeSatisfies` is intentionally **not** re-exported from the package's public surface (`packages/core/src/index.ts`). No host needs the primitive to author or validate a manifest; publishing it would freeze an internal matching rule into the package's public surface.
958
-
959
- See the design spec: [`docs/superpowers/specs/2026-08-28-ontology-parent-field-spec.md`](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/superpowers/specs/2026-08-28-ontology-parent-field-spec.md).
960
-
961
- ### Ontology backfill
962
-
963
- Facts that enter the store without passing through the librarian (synced-down
964
- remote facts via `importDump`, or facts created before the ontology feature)
965
- have `okf_type = NULL` and no edges, so `traverseGraph` cannot reach them.
966
- `runOntologyBackfill` types them in place:
967
-
968
- ```ts
969
- const result = await wiki.runOntologyBackfill(entityId);
970
- // { scanned, typed, failedValidation, edgesAdded, remaining, deferred, skipped }
971
- ```
972
-
973
- - **When to call it:** you own the trigger — the library provides the operation,
974
- not a scheduler (same as `runLibrarian`/`runHeal`). A good default is after
975
- each sync/import completes. `WikiBusyError` under concurrency is safe to
976
- swallow; the next trigger retries.
977
- - **Cost model:** one or more LLM calls only when eligible untyped facts exist;
978
- otherwise one SELECT. Each run scans at most 25 facts (override via
979
- `options.batchSize`), oldest first. Facts are sent to the LLM in bounded
980
- sub-batches — sized from `llmProvider.maxOutputTokens` when supplied,
981
- otherwise a conservative default — with the full serialized prompt
982
- (system + user) capped at 40k chars. A sub-batch whose response is truncated
983
- or fails to parse is halved and retried automatically; a single fact that
984
- still fails alone is counted in `skipped` rather than aborting the run.
985
- - **Convergence:** loop `while (result.remaining > 0)` to drain a backlog.
986
- `remaining` counts only *eligible* untyped facts, so the loop terminates even
987
- when unclassifiable facts exist; those are cooldown-stamped and retried after
988
- 7 days (`deferred` reports them). Because of that weekly retry, host
989
- dashboards may see small recurring bumps in `scanned`/`failedValidation` as
990
- the unclassifiable backlog is re-tested — expected, not a regression.
991
- - **Strictly additive:** never creates, deletes, or rewrites facts, and never
992
- overwrites an existing `okf_type` (guarded in SQL, not just convention).
993
- `updated_at` is never touched, so sync merge resolution is unaffected.
994
- - **Prompt customization:** per-call `options.promptOverride`, or global
995
- `config.prompts.ontologyBackfillSystemPrompt` (template may use `{{facts}}`
996
- and the ontology placeholders).
997
-
998
- #### Classifier mode (optional)
999
-
1000
- A System-One classifier (for example TypeSafe's Jev, an OpenJev-style open model, or a local ONNX classifier) can type facts during backfill without generating text. Add `classify` to your provider and opt in:
1001
-
1002
- ```ts
1003
- const wiki = createWiki(db, {
1004
- llmProvider: { generateText, classify },
1005
- config: { ontology: { backfillClassifier: 'auto', classifyMinConfidence: 0.5 } },
1006
- });
1007
- await wiki.runOntologyBackfill('user-1'); // uses classify
1008
- await wiki.runOntologyBackfill('user-1', { classifier: 'llm' }); // force the generative path
1009
- ```
1010
-
1011
- - Providing `classify` changes nothing by itself. The default is `'llm'`.
1012
- - Each untyped fact gets one `choice` question over the entity manifest's node types. Answers below `classifyMinConfidence` (default 0.5) are left untyped and retried after the cooldown.
1013
- - **No edges are proposed in classifier mode** (`edgesAdded: 0`): a classifier cannot extract edge targets. Run with `classifier: 'llm'` when you want edges.
1014
- - Manifests with more than 255 node types, or providers without `classify`, use the generative path.
1015
- - Answers are validated as untrusted. Off-list choices and out-of-range probabilities count toward `failedValidation`. A thrown `classify` counts toward `skipped` and is retried on the next pass.
1016
-
1017
- Example adapter for Cloudflare Workers AI's `typesafe/jev`. This is illustrative, not a supported package; check the provider's current API reference before use.
1018
-
1019
- ```ts
1020
- const classify: LLMProvider['classify'] = async ({ state, questions }) => {
1021
- const jevQuestions = Object.fromEntries(Object.entries(questions).map(([key, q]) => [key,
1022
- q.kind === 'choice' ? { type: 'choice', instructions: q.instructions, criteria: Object.fromEntries(q.options.map((o) => [o, o])) }
1023
- : q.kind === 'score' ? { type: 'score', instructions: q.instructions, criteria: q.levels }
1024
- : { type: 'noul', instructions: q.instructions },
1025
- ]));
1026
- const res = await env.AI.run('typesafe/jev', { state, questions: jevQuestions });
1027
- const answers = Object.fromEntries(Object.entries(res.answers).map(([key, a]: [string, any]) => [key,
1028
- a.type === 'choice' ? { kind: 'choice', choice: a.choice, confidence: a.confidence, probabilities: a.probabilities }
1029
- : a.type === 'score' ? {
1030
- kind: 'score', score: a.score, confidence: a.confidence,
1031
- probabilities: Object.keys(a.probabilities).sort((x, y) => Number(x) - Number(y)).map((k) => a.probabilities[k]),
1032
- }
1033
- : { kind: 'binary', probability: a.noul },
1034
- ]));
1035
- return { answers };
1036
- };
1037
- ```
1038
-
1039
- ## OKF Import/Export
1040
-
1041
- The core package integrates with `@equationalapplications/core-okf` to seamlessly adapt wiki data dumps to and from Open Knowledge Format (OKF) bundles (v0.1 and v0.2; `formatOkfBundle` defaults to the v0.2 / `llm-wiki/2` profile).
1042
-
1043
- ### Exporting an OKF Bundle
1044
-
1045
- Convert an existing wiki dump into a flat array of OKF files, ready to be written to disk or zipped:
1046
-
1047
- ```typescript
1048
- import { formatOkfBundle } from '@equationalapplications/core-llm-wiki';
1049
-
1050
- const dump = await wiki.exportDump(['entity-123']);
1051
- const { files } = formatOkfBundle(dump);
1052
-
1053
- // files: Array<{ path: string; content: string }>
1054
- // e.g., [{ path: 'entities/entity-123/facts/fact_abc.md', content: '---\n...' }]
1055
- ```
1056
-
1057
- ### Importing an OKF Bundle
1058
-
1059
- Parse raw OKF files back into a `MemoryDump` that the wiki can ingest:
1060
-
1061
- ```typescript
1062
- import { parseOkfBundle } from '@equationalapplications/core-llm-wiki';
1063
-
1064
- // Assuming you read OKF files for this entity (e.g. under `entities/entity-123/`) from disk/zip into OkfFile[] shape
1065
- const dump = parseOkfBundle('entity-123', files, {
1066
- defaultSchema: 'fact',
1067
- typeMapping: {
1068
- 'custom_type': 'fact',
1069
- 'archived': 'ignore', // Skips these concepts
1070
- },
1071
- });
1072
-
1073
- await wiki.importDump(dump, { merge: true });
1074
- ```
1075
-
1076
- **Routing Precedence:** Concepts are routed into either the `entries` (facts) or `tasks` tables based on a three-step fallback:
1077
-
1078
- 1. `OkfImportOptions.typeMapping` explicitly mapping an OKF `type` to `'fact'`, `'task'`, or `'ignore'`.
1079
- 2. Directory convention (e.g., files in `/facts/` become facts, `/tasks/` become tasks).
1080
- 3. The `OkfImportOptions.defaultSchema` (defaults to `'fact'`).
1081
-
1082
- ### WikiEdge and the `## Related` Section
1083
-
1084
- A `WikiEdge` represents a directed link between two concepts (`source_id`, `target_id`, `edge_type`). In profile llm-wiki/1, edges are serialized in a trailing `## Related` section on each concept file. `parseOkfBundle()` extracts them into `WikiEdge` rows and strips the section from stored bodies; `formatOkfBundle()` emits the section from the `edges` array in the dump.
1085
-
1086
- See [`docs/okf-profile.md`](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/okf-profile.md) for the full normative spec and [`packages/okf/fixtures/`](https://github.com/equationalapplications/expo-llm-wiki/tree/main/packages/okf/fixtures) for conformance bundles.
1087
-
1088
- `importDump` persists an imported entity summary (profile ≥ 1 bundles) and
1089
- `exportDump` re-emits it, so the profile §4 round-trip holds across storage.
1090
- Read it directly with `wiki.getEntitySummary(entityId)`. The value lives in the
1091
- `{prefix}meta` table under `entity_summary:{entity_id}` and is removed by
1092
- `forget(entityId, { clearAll: true })`. See the
1093
- [design spec](../../docs/superpowers/specs/2026-07-05-okf-summary-persistence-design.md)
1094
- for import/merge semantics and wipe-path cleanup.
1095
-
1096
- ### The `okf_type` Field
1097
-
1098
- Facts and tasks include a nullable `okf_type` column. This preserves the literal OKF `type` string from an imported bundle frontmatter, independent of whether the item was routed to the `entries` or `tasks` table. When `formatOkfBundle` runs, it restores this specific string, falling back to `'fact'` or `'task'` if the field is null (ensuring non-imported rows export cleanly).
1099
-
1100
- ## Security
1101
-
1102
- `@equationalapplications/core-llm-wiki` enforces multiple security layers:
1103
-
1104
- ### VectorRanker Adapter Security
1105
-
1106
- If implementing a custom `VectorRanker`:
1107
-
1108
- - **SQL Injection**: ALWAYS use parameterized queries for `entityId`, `factId`, `candidateIds`. Never concatenate into SQL strings.
1109
- - **Entity Isolation**: Filter by `entityId` in all queries to prevent cross-tenant data leaks.
1110
- - **Credential Scrubbing**: Strip API keys, tokens, connection strings from thrown errors before surfacing to host.
1111
- - **Resource Limits**: Cap `limit` and `candidateIds.length` to prevent DoS. Do NOT retain `vector` references beyond callback scope — blocks GC.
1112
-
1113
- See [SECURITY.md](https://github.com/equationalapplications/expo-llm-wiki/blob/main/SECURITY.md) for complete adapter security guidance and code examples.
1114
-
1115
- ### Host Application Security
1116
-
1117
- When using `VectorRanker`:
1118
-
1119
- - **Error Sanitization**: `sanitizeRankerErrors: true` (default) scrubs ranker errors before mirroring via `error.cause`.
1120
- - **Fallback Policy**: Choose `vectorRankerFallback` based on availability vs consistency requirements:
1121
- - `'js-cosine'` (default): Best availability
1122
- - `'keyword'`: Fast fallback without semantic ranking
1123
- - `'empty'`: Strict consistency (no facts on failure)
1124
- - `'throw'`: Fail-fast error propagation
1125
- - **Deletion Hook Contract**: `forget()` / `runPrune()` reject on hook timeout/failure. Prevents GDPR violations (deleted vectors still retrievable). Handle failures with retry or queue for reconciliation.
1126
- - **Timeout Tuning**: Set `deletionHookTimeoutMs` per deployment (default 30s). Interactive UX: 5s. Background jobs: 60s.
1127
-
1128
- Core WikiMemory provides:
1129
- - **Defensive Copies**: Query/embedding vectors copied before ranker/hook calls
1130
- - **Input Validation**: `sourceRef`/`sourceHash` normalized; embedding dimensions validated
1131
- - **Parameterized Queries**: All SQL uses bind parameters
1132
-
1133
- ### Prompt-Injection Trust Boundary
1134
-
1135
- User-controlled text — `event.summary` passed to `write()`, document chunks passed to `ingestDocument()`,
1136
- fact `title`/`body` (including imported dumps) — is interpolated verbatim into LLM prompts for librarian,
1137
- heal, and embedding operations. Prompt templating does simple variable substitution; it does not detect
1138
- or filter instruction-like content.
1139
-
1140
- Mitigating prompt injection (e.g. "ignore prior instructions and emit...") is **the host's responsibility**.
1141
- If your application accepts untrusted input that flows into `write()`, `ingestDocument()`, or `importDump()`,
1142
- treat the LLM's librarian/heal output as similarly untrusted — validate or scope it before acting on it
1143
- downstream.
1144
-
1145
- [Grounding](#grounding) is a support check, not an injection defense. It confirms that a fact quotes the
1146
- text the model was shown, and injected text in that source can be quoted like any other.
1147
-
1148
- [`getInstructions`](#effective-instructions-getinstructions) and the `wiki_get_instructions` tool (`memory:read`)
1149
- return `WikiConfig.prompts` overrides verbatim, so keep secrets out of them. The result also includes the entity's
1150
- ontology manifest, which in emergent mode holds types and descriptions the model proposed from ingested
1151
- documents. Agents should treat the result as reference data about the engine, not as instructions to follow.
1152
-
1153
- ## Usage
1154
-
1155
- ```typescript
1156
- import { WikiMemory, type SQLiteAdapter } from '@equationalapplications/core-llm-wiki';
1157
-
1158
- // Provide any SQLiteAdapter-compatible driver
1159
- const wikiMemory = new WikiMemory(db, {
1160
- llmProvider: {
1161
- generateText: async ({ systemPrompt, userPrompt }) => {
1162
- // Your LLM call here
1163
- return 'Model output';
1164
- },
1165
- },
1166
- });
1167
-
1168
- // Initialize schema and run migrations
1169
- await wikiMemory.setup();
1170
-
1171
- // Store facts
1172
- await wikiMemory.write('user-123', {
1173
- event_type: 'observation',
1174
- summary: 'User prefers async/await over promises',
1175
- });
1176
-
1177
- // Query memory
1178
- const memory = await wikiMemory.read('user-123', 'coding style preferences');
1179
- ```
1180
-
1181
- ### Multi-entity weighted reads
1182
-
1183
- `read()` accepts either one entity id or an array of entity ids. Facts are always merged globally before `maxResults` is applied. For single-entity reads, tasks are uncapped and events are capped at 10. For multi-entity reads, tasks are capped at `min(20 × entity count, 200)` and events at `min(10 × entity count, 100)` — per-entity representation in the returned bundle is not guaranteed.
1184
-
1185
- ```ts
1186
- const memory = await wikiMemory.read(['tier_wisdom', 'tier_fact', 'tier_working'], 'Which source should I trust?', {
1187
- maxResults: 8,
1188
- tierWeights: {
1189
- tier_wisdom: 2,
1190
- tier_fact: 1,
1191
- tier_working: 0.25,
1192
- },
1193
- });
1194
-
1195
- console.log(memory.metadata);
1196
- console.log(memory.factScores);
1197
- ```
1198
-
1199
- ### Librarian prompt override contract
1200
-
1201
- Core exports prompt utilities for weighted retrieval-based synthesis. Use `mapLibrarianOptionsToReadOptions()` to map `entityWeights` to `tierWeights`, then hydrate a prompt with `query`, `context`, and `tasks`.
1202
-
1203
- ```ts
1204
- import {
1205
- DEFAULT_LIBRARIAN_SYNTHESIS_PROMPT,
1206
- formatContext,
1207
- hydrateLibrarianPrompt,
1208
- mapLibrarianOptionsToReadOptions,
1209
- validateLibrarianPromptTemplate,
1210
- } from '@equationalapplications/core-llm-wiki';
1211
-
1212
- const options = {
1213
- entityWeights: { tier_wisdom: 2, tier_fact: 1, tier_working: 0.25 },
1214
- systemPrompt: `You are a strict fact checker.
1215
- Question:
1216
- {{query}}
1217
-
1218
- Retrieved context:
1219
- {{context}}
1220
-
1221
- {{tasks}}`,
1222
- };
1223
-
1224
- const query = 'Which source should I trust for recent project decisions?';
1225
-
1226
- const memory = await wikiMemory.read(['tier_wisdom', 'tier_fact', 'tier_working'], query, {
1227
- ...mapLibrarianOptionsToReadOptions(options),
1228
- maxResults: 8,
1229
- });
1230
-
1231
- const template = options.systemPrompt ?? DEFAULT_LIBRARIAN_SYNTHESIS_PROMPT;
1232
- const warnings = validateLibrarianPromptTemplate(template, {
1233
- custom: options.systemPrompt != null,
1234
- taskCount: memory.tasks.length,
1235
- });
1236
-
1237
- for (const warning of warnings) console.warn(warning);
1238
-
1239
- const finalPrompt = hydrateLibrarianPrompt(template, {
1240
- query,
1241
- context: formatContext(memory, { includeEntityIds: true, includeFactScores: true }),
1242
- tasks: formatContext({ facts: [], tasks: memory.tasks, events: [] }, { format: 'plain' }),
1243
- });
1244
- ```
1245
-
1246
- ## Platform Random Source
1247
-
1248
- Wiki record IDs must be cryptographically random. The core engine resolves a random source in this order:
1249
-
1250
- 1. `crypto.randomUUID()` (Web / Node 19+)
1251
- 2. `crypto.getRandomValues()` (Web / Node / polyfilled global)
1252
- 3. A source injected via `configureRandomSource()` (e.g. `expo-crypto` on Hermes/React Native)
1253
-
1254
- Web and Node are unchanged — global `crypto` wins when present. React Native / Hermes typically has no `crypto` global; use a platform package or inject your own implementation:
1255
-
1256
- ```typescript
1257
- import { configureRandomSource } from '@equationalapplications/core-llm-wiki';
1258
- import { getRandomValues } from 'expo-crypto';
1259
-
1260
- // Call once at module load, before any wiki writes
1261
- configureRandomSource(getRandomValues);
1262
- ```
1263
-
1264
- `@equationalapplications/expo-llm-wiki` does this automatically on import (main entry and `/factory` subpath). If you use `@equationalapplications/core-llm-wiki` directly on React Native without the expo package, you must call `configureRandomSource()` yourself or polyfill `globalThis.crypto.getRandomValues`.
1265
-
1266
- ## Entity Enumeration
1267
-
1268
- List all entities that have stored data in the wiki:
1269
-
1270
- ```typescript
1271
- const entityIds = await wikiMemory.listEntityIds();
1272
- // Returns all entity_ids with at least one row (including soft-deleted-only entities)
1273
- // Optional prefix filter: await wikiMemory.listEntityIds({ prefix: 'tier_' });
1274
- ```
1275
-
1276
- Use this for maintenance scheduling, multi-entity operations, or discovering which namespaces exist. Includes entities with only soft-deleted rows so `runPrune()` can reclaim orphaned storage.
1277
-
1278
- ## Source Reference Enumeration
1279
-
1280
- List all documents currently stored for an entity:
1281
-
1282
- ```typescript
1283
- const sourceRefs = await wikiMemory.listSourceRefs('user-123');
1284
- // One row per live sourceRef (soft-deleted rows are excluded):
1285
- // Array<{ sourceRef: string; sourceHash: string | null; factCount: number; lastIngestedAt: number }>
1286
- // factCount — number of live facts under that sourceRef
1287
- // lastIngestedAt — Unix timestamp in ms from the most recently updated live entry
1288
- ```
1289
-
1290
- Use this to audit stored documents, validate external sync state, or preview the blast radius before `forget()` operations.
1291
-
1292
- ## Direct Graph Write
1293
-
1294
- Write structured graph data directly without LLM extraction — useful for programmatic fact ingestion, parsers, and deterministic pipelines:
1295
-
1296
- ```typescript
1297
- const { nodesWritten, edgesWritten, superseded } = await wikiMemory.upsertGraph('entity-123', {
1298
- sourceRef: 'codebase_main.ts',
1299
- sourceHash: sha256(sourceCode),
1300
- nodes: [
1301
- { id: 'fn_processData', type: 'function', title: 'processData', body: 'Processes user data' },
1302
- { id: 'class_UserService', type: 'class', title: 'UserService', body: 'User management service' },
1303
- ],
1304
- edges: [
1305
- { type: 'calls', sourceId: 'fn_processData', targetId: 'class_UserService' },
1306
- ],
1307
- }, adapter); // SQLiteAdapter from your platform driver — writes join the caller's transaction
1308
- ```
1309
-
1310
- `upsertGraph` is "the tail of `ingestDocument` with the middle (LLM extraction) step removed" — it accepts caller-supplied nodes (`{ id, type, title, body? }`) and edges (`{ type, sourceId, targetId, id? }`) and writes them under the same `(sourceRef, sourceHash)` semantics. If a *different* live `sourceRef` already holds the same `sourceHash`, it throws `WikiSourceRefHashCollision`; re-writing the identical `(sourceRef, sourceHash)` is a no-op returning zero counts. The adapter parameter is required so writes participate in the caller's transaction.
1311
-
1312
- Core never sees your transaction commit, so it can't index the new nodes itself. Until you tell it, they won't show up in keyword or hybrid `read()` results on the same instance. Call `syncSearchIndex` **after** the commit, not inside the transaction:
1313
-
1314
- ```typescript
1315
- await db.withTransactionAsync((tx) => wikiMemory.upsertGraph('entity-123', params, tx));
1316
- await wikiMemory.syncSearchIndex('entity-123'); // or syncSearchIndex() for every entity written
1317
- ```
1318
-
1319
- It is cheap on an entity that is already current, runs in order with core's own index syncs, and never rejects for a valid `entityId` (an empty or non-string id throws `TypeError`). It does not compute embeddings; those still come from the maintenance sweep.
1320
-
1321
- Entry IDs occupy one globally shared namespace within a database. An ID belongs
1322
- to its existing entity while its row exists, including after soft deletion.
1323
- Same-entity updates and resurrection retain their existing behavior; physically
1324
- absent IDs may be inserted. Hosts remain responsible for authenticating callers
1325
- and selecting the authorized entity ID. Entity-qualified deterministic IDs can
1326
- avoid accidental collisions, but are not required by the API.
1327
-
1328
- A foreign node ID rejects the entire non-no-op graph write with
1329
- `WikiGraphNodeOwnershipConflict` (code `WIKI_GRAPH_NODE_OWNERSHIP_CONFLICT`).
1330
- The error does not identify the owner, conflicting node, or stored details.
1331
- A collision visible to node preflight performs no persistent writes, including
1332
- seed-manifest writes. The live source-hash probe still runs first: an identical
1333
- mapping is a zero-count no-op; a different live reference retains its hash
1334
- collision error. Deleted source-reference mappings do not trigger that probe.
1335
-
1336
- Call `upsertGraph` inside the host transaction and pass that callback's `tx`.
1337
- It neither starts nor ends a transaction. Propagate later failures so the host
1338
- transaction rolls back partial writes; catching a late error and committing is
1339
- not atomic. Foreign edge-ID collisions retain their existing late bare `Error`
1340
- and require host rollback, even though the foreign edge is never overwritten.
1341
-
1342
- `ingestDocument` shares ownership enforcement on both its full and partial paths
1343
- and may also throw `WikiGraphNodeOwnershipConflict`. It normally generates new
1344
- IDs, so collisions are unexpected. Unlike direct graph preflight, ingestion may
1345
- already have persisted ontology or other state when rejection occurs; its
1346
- transaction rollback, not a zero-write preflight guarantee, protects atomicity.
1347
-
1348
- ## Duplicate Hash Detection
1349
-
1350
- Control behavior when a different live `sourceRef` already holds the same `sourceHash`. The option is passed as a third argument to `ingestDocument`, not inside the params object:
1351
-
1352
- ```typescript
1353
- await wikiMemory.ingestDocument(
1354
- 'entity-123',
1355
- {
1356
- sourceRef: 'doc.md',
1357
- sourceHash: sha256(content),
1358
- documentChunk: content,
1359
- },
1360
- { onDuplicateHash: 'ingest' } // 'ingest' (default) | 'skip' | 'throw'
1361
- );
1362
- ```
1363
-
1364
- - `'ingest'` (default): No duplicate pre-check; extraction proceeds as before this option existed. If a different live `sourceRef` is found holding the same hash at commit time (a concurrent-writer race caught by the source-ref unique index), the call still throws `WikiDuplicateHashError`.
1365
- - `'skip'`: Pre-check before any LLM call; if a different live `sourceRef` already holds the hash, return a zero-chunk result without writing.
1366
- - `'throw'`: Pre-check before any LLM call; throw `WikiDuplicateHashError` (carries the canonical `sourceRef`).
1367
- - The guard only considers **live** references — soft-deleted refs do not trigger it in any mode.
1368
-
1369
- When the pre-check passes but a concurrent writer claims the hash before the write commits, the transaction rolls back after the LLM extraction has already run. You still receive the diagnostics that describe that extraction — `ingest_chunk_failed`, `fact_rejected` and `fact_deduplicated` — in every mode, and also in the rarer case where the winning writer has itself rolled back by the time core looks, so the call surfaces the raw database error instead of `WikiDuplicateHashError`. Diagnostics tied to the rolled-back write (`grounding_*`, `edge_dropped`) are dropped, because the `factId`s they carry name rows that were never committed.
1370
-
1371
- ## Batch Change Detection
1372
-
1373
- Check multiple documents for changes in one call:
1374
-
1375
- ```typescript
1376
- const batch = [
1377
- { sourceRef: 'doc1.md', sourceHash: sha256(content1) },
1378
- { sourceRef: 'doc2.md', sourceHash: sha256(content2) },
1379
- { sourceRef: 'doc3.md', sourceHash: sha256(content3) },
1380
- ];
1381
- const changes = await wikiMemory.hasChanged('entity-123', batch);
1382
- // changes: Array<{ sourceRef: string; changed: boolean; duplicateOf?: string }>
1383
- // duplicateOf — when present, the canonical stored different sourceRef holding
1384
- // the same hash (DB-normalized spelling; sourceRef echoes the raw caller value).
1385
- // Per-document change detection; internally batched across queries
1386
- ```
1387
-
1388
- `pendingSources` returns one status per input, in order, and adds the partial-ingest state:
1389
-
1390
- ```typescript
1391
- const statuses = await wikiMemory.pendingSources('entity-123', batch);
1392
- // Array<{ sourceRef: string; status: 'new' | 'changed' | 'partial' | 'current' }>
1393
- ```
1394
-
1395
- - `current` means exactly what `hasChanged` returning `false` means.
1396
- - `partial` means live facts exist for the ref, but none has a stored hash: for example a first ingest where a chunk failed, or imported rows that carry no hash. Re-ingest to retry. A failed re-ingest of a ref that already has hashed rows reports `changed`, not `partial`.
1397
-
1398
- ## Lint
1399
-
1400
- Read-only health report for one entity. It reports problems and never repairs them.
1401
-
1402
- ```typescript
1403
- const report = await wikiMemory.lint('entity-123');
1404
- // {
1405
- // danglingEdges, // source or target missing, soft-deleted, or another entity's
1406
- // manifestViolations, // (source type, edge type, target type) not in the effective manifest
1407
- // untypedFacts, // okf_type is null
1408
- // drafts, // lifecycle_status = 'draft' (see Draft Review)
1409
- // unverifiedInferred, // librarian_inferred facts with no okf_verified entry
1410
- // sample: { danglingEdgeIds, manifestViolationEdgeIds }, // up to 20 each
1411
- // }
1412
- ```
1413
-
1414
- - Manifest violations are 0 when ontology is off or the manifest is empty.
1415
- - An edge with an untyped endpoint counts as a violation.
1416
- - Partial-ingest rows are not reported here; use `pendingSources`.
1417
-
1418
- ## Dry-Run Deletion
1419
-
1420
- Preview deletion impact without writing:
1421
-
1422
- ```typescript
1423
- const preview = await wikiMemory.forget('entity-123', { sourceRef: 'doc.md' }, { dryRun: true });
1424
- // preview: { deleted: { entries: number; tasks: number } }
1425
- // No database writes performed; safe for blast-radius validation
1426
- ```
1427
-
1428
- ## Chunking Utilities
1429
-
1430
- `ingestDocument()` splits a document into chunks before extraction. That same chunking is exported as a pure function, so a consumer can reproduce ingest-time chunk boundaries exactly — useful for recovering the passage a fact was extracted from, by re-chunking the source and ranking chunks against the fact's stored embedding.
1431
-
1432
- ```typescript
1433
- import {
1434
- chunkText,
1435
- safeSlice,
1436
- DEFAULT_MAX_CHUNK_LENGTH, // 12000
1437
- DEFAULT_CHUNK_OVERLAP, // 400
1438
- } from '@equationalapplications/core-llm-wiki';
1439
-
1440
- const { chunks, truncated } = chunkText(
1441
- sourceDocument,
1442
- DEFAULT_MAX_CHUNK_LENGTH,
1443
- DEFAULT_CHUNK_OVERLAP
1444
- );
1445
- ```
1446
-
1447
- `chunkText(input, maxChunkLength, overlap)` returns `{ chunks, truncated }`. It prefers to split on a paragraph break, then a sentence terminator, then whitespace, falling back to a hard cut only when none is found within the window — `truncated` is `true` if any split needed that fallback. Consecutive chunks repeat up to `overlap` characters so context isn't lost at a boundary (less than `overlap` when the previous chunk was shorter than that). Empty/whitespace-only input returns `{ chunks: [], truncated: false }` without validating `maxChunkLength`/`overlap`; otherwise throws if `maxChunkLength` is not an integer >= 2, or if `overlap` is not a non-negative integer < `maxChunkLength`.
1448
-
1449
- `safeSlice(value, start, end?)` slices like `String.prototype.slice` but clamps out-of-range indices, swaps a start-after-end range, and never splits a UTF-16 surrogate pair.
1450
-
1451
- `DEFAULT_MAX_CHUNK_LENGTH` and `DEFAULT_CHUNK_OVERLAP` are the values `ingestDocument()` uses when neither the call nor `WikiConfig` overrides them. Pass the same values your host app configured (`config.maxChunkLength` / `config.chunkOverlap`) to match a non-default ingest.
1452
-
1453
- > **Note:** `ingestDocument()` clamps the resolved overlap — whether it came from `chunkOverlap`, `WikiConfig`, or `DEFAULT_CHUNK_OVERLAP` — to `maxChunkLength - 1`. The clamp is evaluated on every call and is a no-op at the shipped defaults (`400 < 12000 - 1`), but it also bites when a custom `maxChunkLength` alone leaves the resolved overlap too large (e.g. `maxChunkLength: 100` with the default overlap `400` ingests at an effective overlap of `99`). That clamp is internal: passing the unclamped pair straight to `chunkText` throws, since it requires `overlap < maxChunkLength`. Apply the same `Math.min(overlap, maxChunkLength - 1)` yourself when re-chunking under a custom config.
1454
-
1455
- ## Schema Migrations
1456
-
1457
- `wikiMemory.setup()` runs pending schema migrations in order and records the applied version. Migrations are additive and safe to re-run.
1458
-
1459
- **Migration v9** (`add_live_hash_unique_index`) adds a partial `UNIQUE` index on `(entity_id, source_hash)` scoped to live (`deleted_at IS NULL`), non-null-hash rows. This closes a TOCTOU race where two concurrent `ingestDocument()` calls for different `sourceRef`s could both pass the duplicate-hash pre-check before either write committed, leaving two live rows with the same content hash.
1460
-
1461
- If your existing database already has live rows that violate this invariant (created before the app-level race fix shipped), `setup()` throws:
1462
-
1463
- ```text
1464
- Migration v9 (add_live_hash_unique_index) failed: existing live rows violate the new UNIQUE index. …
1465
- ```
1466
-
1467
- The error lists the offending `(entity_id, source_hash)` groups. `setup()` performs no destructive cleanup on this path — the index is not created and the schema version is not advanced, so it's safe to retry after remediation. To remediate, either soft-delete (or re-ingest with a corrected `sourceRef`) all but one live row per listed group, then re-run `setup()`.
1468
-
1469
- ## Adapter Interface
1470
-
1471
- Implement `SQLiteAdapter` to use your platform's SQLite driver:
1472
-
1473
- ```typescript
1474
- export interface SQLiteAdapter {
1475
- execAsync(sql: string): Promise<void>;
1476
- runAsync(sql: string, params?: unknown[]): Promise<{ changes: number; lastInsertRowId: number }>;
1477
- getAllAsync<T>(sql: string, params?: unknown[]): Promise<T[]>;
1478
- getFirstAsync<T>(sql: string, params?: unknown[]): Promise<T | null>;
1479
- withTransactionAsync<T>(fn: (tx: SQLiteAdapter) => Promise<T>): Promise<T>;
1480
- closeAsync(): Promise<void>;
1481
- }
1482
- ```
1483
-
1484
- `runAsync().changes` must accurately count written rows: a conflict-suppressed
1485
- write reports 0, while a permitted value-identical update still counts its row.
1486
- Host adapters are responsible for this contract; see the full affected-row
1487
- requirements on [`SQLiteAdapter.runAsync`](src/types.ts). Ownership checks,
1488
- edge counts, metadata compare-and-set, and maintenance depend on accurate counts.
1489
-
1490
- `@equationalapplications/expo-llm-wiki` provides a pre-built adapter for Expo/React Native. For web and Node.js, implement the interface yourself — examples below.
1491
-
1492
- **Browser ([sql.js](https://github.com/sql-js/sql.js)):**
1493
-
1494
- ```typescript
1495
- import initSqlJs from 'sql.js';
1496
- import type { SQLiteAdapter } from '@equationalapplications/core-llm-wiki';
1497
-
1498
- const SQL = await initSqlJs({ locateFile: (f) => `/wasm/${f}` });
1499
- const sqlDb = new SQL.Database();
1500
-
1501
- const adapter: SQLiteAdapter = {
1502
- async execAsync(sql) { sqlDb.run(sql); },
1503
- async runAsync(sql, params = []) {
1504
- sqlDb.run(sql, params as any[]);
1505
- // sql.js doesn't expose lastInsertRowId; hardcode 0 since WikiMemory uses internal ID generation
1506
- return { changes: sqlDb.getRowsModified(), lastInsertRowId: 0 };
1507
- },
1508
- async getAllAsync<T>(sql, params = []) {
1509
- const stmt = sqlDb.prepare(sql);
1510
- stmt.bind(params as any[]);
1511
- const rows: T[] = [];
1512
- while (stmt.step()) rows.push(stmt.getAsObject() as T);
1513
- stmt.free();
1514
- return rows;
1515
- },
1516
- async getFirstAsync<T>(sql, params = []) {
1517
- const stmt = sqlDb.prepare(sql);
1518
- stmt.bind(params as any[]);
1519
- const row = stmt.step() ? stmt.getAsObject() as T : null;
1520
- stmt.free();
1521
- return row;
1522
- },
1523
- async withTransactionAsync(fn) {
1524
- sqlDb.run('BEGIN');
1525
- try { const r = await fn(this); sqlDb.run('COMMIT'); return r; }
1526
- catch (e) { sqlDb.run('ROLLBACK'); throw e; }
1527
- },
1528
- async closeAsync() { sqlDb.close(); },
1529
- };
1530
- ```
1531
-
1532
- **Node.js ([better-sqlite3](https://github.com/WiseLibs/better-sqlite3)):**
1533
-
1534
- ```typescript
1535
- import Database from 'better-sqlite3';
1536
- import type { SQLiteAdapter } from '@equationalapplications/core-llm-wiki';
1537
-
1538
- const db = new Database('wiki.db');
1539
-
1540
- const adapter: SQLiteAdapter = {
1541
- async execAsync(sql) { db.exec(sql); },
1542
- async runAsync(sql, params = []) {
1543
- const info = db.prepare(sql).run(...(params as any[]));
1544
- return { changes: info.changes, lastInsertRowId: Number(info.lastInsertRowid) };
1545
- },
1546
- async getAllAsync<T>(sql, params = []) {
1547
- return db.prepare(sql).all(...(params as any[])) as T[];
1548
- },
1549
- async getFirstAsync<T>(sql, params = []) {
1550
- return (db.prepare(sql).get(...(params as any[])) ?? null) as T | null;
1551
- },
1552
- async withTransactionAsync(fn) {
1553
- db.exec('BEGIN');
1554
- try { const r = await fn(this); db.exec('COMMIT'); return r; }
1555
- catch (e) { db.exec('ROLLBACK'); throw e; }
1556
- },
1557
- async closeAsync() { db.close(); },
1558
- };
1559
- ```
1560
-
1561
- ## Concurrency
1562
-
1563
- All write APIs are safe to call from concurrent async contexts. Transactions are
1564
- serialized internally on the single database connection — you never need to know that
1565
- SQLite forbids nested `BEGIN`, and you never need to throttle callers yourself.
1566
-
1567
- - One connection per database file per process is the supported topology.
1568
- - Non-transactional reads are **not** serialized, so read latency is unaffected.
1569
- - Inside a transaction callback, use only the provided `tx` handle — never the outer
1570
- database handle. Using the outer handle deadlocks; opening a nested transaction throws.
1571
-
1572
- ### `WikiTransactionError`
1573
-
1574
- Thrown when a SQLite driver error escapes a transaction (nested `BEGIN`,
1575
- `SQLITE_BUSY`, constraint violation). Stable `instanceof` target for observability:
1576
-
1577
- - `error.cause` — the original driver error (the chain bottoms out here, not at a
1578
- rollback red herring).
1579
- - `error.sqliteErrorCode` — best-effort SQLite code (e.g. `'SQLITE_BUSY'`), present
1580
- when it can be parsed. Group on this in Sentry/Datadog instead of inspecting
1581
- `error.cause` recursively.
1582
-
1583
- Domain errors thrown from your own callback logic (validation, `WikiBusyError`)
1584
- pass through unwrapped with their original type intact.
1585
-
1586
- ## How It Works
1587
-
1588
- ```mermaid
1589
- flowchart TD
1590
- A["read(entityId | entityId[], query, options?)"] --> B{hybridWeight = 0?}
1591
- B -->|Yes| C["MiniSearch only<br/>(skip embed)"]
1592
- B -->|No| D{embed available?}
1593
- D -->|No| C
1594
- D -->|Yes| F["Embed query"]
1595
- F -->|throws| E["onRetrievalFallback<br/>callback"]
1596
- E --> C
1597
- F -->|succeeds| G{preFilterLimit<br/>active?}
1598
- G -->|Yes| H["MiniSearch pre-filter<br/>top K candidates"]
1599
- H --> I["Phase 1: Cosine score<br/>top K candidates"]
1600
- G -->|No| J["Phase 1: Cosine score<br/>all facts"]
1601
- J --> K["Cache vectors<br/>in-memory<br/>(full scan only)"]
1602
- K --> L{hybridWeight = 1?}
1603
- I --> L
1604
- L -->|Yes| M["Pure semantic<br/>ranking"]
1605
- L -->|No| N["Hybrid blend:<br/>semantic + keyword<br/>via MiniSearch"]
1606
- M --> O["Phase 2: Fetch full rows<br/>top maxResults"]
1607
- N --> O
1608
- C --> P["MiniSearch ranking"]
1609
- P --> O
1610
- O --> R["Track access"]
1611
- R --> Q["Return MemoryBundle"]
1612
- ```
1613
-
1614
- The flowchart shows:
1615
- 1. **Fast-path** when `hybridWeight = 0` (pure keyword, no embed cost)
1616
- 2. **Fallback chain** when embed unavailable (MiniSearch silently) or throws (`onRetrievalFallback` callback, then MiniSearch)
1617
- 3. **Pre-filtering** to limit cosine scoring to top-K keyword matches (O(N) → O(K))
1618
- 4. **Two-phase SELECT**: phase 1 scores all/filtered facts with minimal columns, phase 2 fetches full rows for winners
1619
- 5. **Hybrid scoring** to blend semantic and keyword rankings
1620
- 6. **Vector caching** on full scans only; reads with `preFilterLimit` active skip cache population
49
+ The [Node.js quickstart](https://equationalapplications.github.io/expo-llm-wiki/start/quickstart-node/) shows a complete `adapter` and `llmProvider`.
1621
50
 
1622
51
  ## Monorepo Ecosystem
1623
52
 
@@ -1633,30 +62,6 @@ The flowchart shows:
1633
62
  | [@equationalapplications/schema-software-org](https://github.com/equationalapplications/expo-llm-wiki/blob/main/packages/schema-software-org/README.md) | Software-organization executive ontology manifest — 17 node types, 40 edges, warm-agent superset, data-only |
1634
63
  | [@equationalapplications/mcp-server](https://github.com/equationalapplications/expo-llm-wiki/blob/main/packages/mcp-server/README.md) | MCP server exposing SynapseTree memory tools to coding agents |
1635
64
 
1636
- ## OKF v0.2 conformance (llm-wiki/2)
1637
-
1638
- `formatOkfBundle` defaults to `okf_version: 0.2` + `profile: llm-wiki/2`. To force a v0.1 export, pass `{ profile: 'llm-wiki/1' }`:
1639
-
1640
- ```ts
1641
- formatOkfBundle(dump, { profile: 'llm-wiki/1' });
1642
- ```
1643
-
1644
- `parseOkfBundle` auto-detects the profile from the root index frontmatter:
1645
-
1646
- - `profile === 'llm-wiki/2'` → v0.2 path (new fields populated).
1647
- - `profile === 'llm-wiki/1'` → v0.1 path (unchanged behavior).
1648
- - `profile === undefined && okf_version === '0.1'` → v0.1 legacy path.
1649
- - `profile === undefined && okf_version === undefined` → profile-0 (legacy) path.
1650
-
1651
- New public methods on `WikiMemory` for OKF v0.2 trust/provenance writes (each DAO method omits `updated_at` from its SQL):
1652
-
1653
- - `writeOkfTrust(entryId, entityId, verified)`
1654
- - `writeOkfSources(entryId, entityId, sources)`
1655
- - `setLifecycleStatus(entryId, entityId, status)`
1656
- - `setStaleAfter(entryId, entityId, date | null)`
1657
- - `setGeneratedBy(entryId, entityId, actor)`
1658
- - (and `…Task` variants of all five)
1659
-
1660
65
  ## License
1661
66
 
1662
67
  MIT