@equationalapplications/core-llm-wiki 7.15.1 → 7.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +20 -1615
- package/dist/{chunk-BC3QHSF3.mjs → chunk-G7F5QPVR.mjs} +32 -4
- package/dist/chunk-G7F5QPVR.mjs.map +1 -0
- package/dist/index.d.mts +2 -2
- package/dist/index.d.ts +2 -2
- package/dist/index.js +30 -2
- package/dist/index.js.map +1 -1
- package/dist/index.mjs +2 -2
- package/dist/{testing-BlGhilQw.d.mts → testing-Des97-_i.d.mts} +20 -0
- package/dist/{testing-BlGhilQw.d.ts → testing-Des97-_i.d.ts} +20 -0
- package/dist/testing.d.mts +1 -1
- package/dist/testing.d.ts +1 -1
- package/dist/testing.js +30 -2
- package/dist/testing.js.map +1 -1
- package/dist/testing.mjs +1 -1
- package/package.json +3 -3
- package/dist/chunk-BC3QHSF3.mjs.map +0 -1
package/README.md
CHANGED
|
@@ -10,1614 +10,43 @@ Platform-agnostic TypeScript engine for hybrid LLM memory. Features episodic fac
|
|
|
10
10
|
|
|
11
11
|
> Inspired by [Andrej Karpathy's LLM Wiki memory spec](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
|
|
12
12
|
|
|
13
|
+
**📖 Documentation: [equationalapplications.github.io/expo-llm-wiki/packages/core](https://equationalapplications.github.io/expo-llm-wiki/packages/core/)**. Start with the [Node.js quickstart](https://equationalapplications.github.io/expo-llm-wiki/start/quickstart-node/), or go straight to the [API reference](https://equationalapplications.github.io/expo-llm-wiki/reference/wiki-memory/).
|
|
13
14
|
|
|
14
15
|
## Features
|
|
15
16
|
|
|
16
|
-
- **
|
|
17
|
-
- **
|
|
18
|
-
- **
|
|
19
|
-
- **
|
|
20
|
-
- **
|
|
21
|
-
- **
|
|
22
|
-
- **
|
|
23
|
-
- **
|
|
24
|
-
- **
|
|
25
|
-
- **Per-entity seeded ontology** — Optional Strict, Emergent, or Off modes govern LLM graph extraction; seed taxonomies per entity and persist typed facts with inline edges.
|
|
26
|
-
- **Diagnostics** — Optional `onDiagnostic` hook with typed, content-free reports of dropped chunks, facts, edges, embedding failures and background-job failures ([Diagnostics](#diagnostics))
|
|
27
|
-
- **Draft review** — `excludeDrafts` on reads and traversal, plus `listDrafts` / `promoteDraft` ([Draft Review](#draft-review))
|
|
28
|
-
- **Evidence grounding** — Opt-in `grounding` check; facts that don't quote their source are stored as drafts ([Grounding](#grounding))
|
|
29
|
-
- **Optional classifier** — `LLMProvider.classify` types facts during ontology backfill when you opt in with `ontology.backfillClassifier: 'auto'` ([Ontology backfill](#ontology-backfill))
|
|
30
|
-
|
|
31
|
-
## GraphRAG & Multi-Modal Retrieval
|
|
32
|
-
|
|
33
|
-
`@equationalapplications/core-llm-wiki` exposes three complementary retrieval modes, each addressing a different shape of query:
|
|
34
|
-
|
|
35
|
-
| Mode | API | Best for |
|
|
36
|
-
|---|---|---|
|
|
37
|
-
| **Semantic** (vector cosine) | `wiki.read(entityId, query)` with `embed` configured | Open-ended natural-language questions; "what do I know about X" |
|
|
38
|
-
| **Keyword** (FTS5 or MiniSearch) | `wiki.read(entityId, query)` with `embed` absent or offline | Exact terms, identifiers, names; offline fallback |
|
|
39
|
-
| **GraphRAG** (recursive CTE) | `wiki.traverseGraph(entityId, options)` + `formatGraphContext(result)` | Structural questions; "what connects to X", "everything two hops from this fact", "summarise the people, places, and projects linked to Alice" |
|
|
40
|
-
|
|
41
|
-
The GraphRAG path is structurally distinct: it doesn't rank by relevance to a query string, it walks `llm_wiki_edges` from a known anchor fact. The result is dense and connected — subgraphs, not loose top-K hits.
|
|
42
|
-
|
|
43
|
-
### Graph traversal APIs
|
|
44
|
-
|
|
45
|
-
```typescript
|
|
46
|
-
import { WikiMemory, formatGraphContext } from '@equationalapplications/core-llm-wiki';
|
|
47
|
-
|
|
48
|
-
const graph = await wikiMemory.traverseGraph('user-123', {
|
|
49
|
-
sourceId: '<anchor-fact-id>',
|
|
50
|
-
maxDepth: 2,
|
|
51
|
-
direction: 'both', // 'inbound' | 'outbound' | 'both'
|
|
52
|
-
edgeTypes: ['reports_to'], // optional filter
|
|
53
|
-
excludeSourceTypes: ['immutable_document'],
|
|
54
|
-
minTraversalConfidence: 'inferred',
|
|
55
|
-
maxTraversalNodes: 20,
|
|
56
|
-
});
|
|
57
|
-
|
|
58
|
-
const promptContext = formatGraphContext(graph);
|
|
59
|
-
// → dense text block ready for prompt injection
|
|
60
|
-
```
|
|
61
|
-
|
|
62
|
-
`traverseGraph` runs as a single recursive CTE in SQLite (see the root README's ["The SQL: how traversal works in one query"](../../README.md#the-sql-how-traversal-works-in-one-query) for the query shape). No external graph database.
|
|
63
|
-
|
|
64
|
-
### Deterministic graph seeding (no LLM)
|
|
65
|
-
|
|
66
|
-
For programmatic pipelines — importing pre-classified data, building a GraphRAG corpus from a CSV, or seed-loading from a JSON file — use `upsertGraph()`. It writes nodes and edges directly under the same `(sourceRef, sourceHash)` ownership semantics as `ingestDocument()`, but skips the LLM extraction step. See [Direct Graph Write](#direct-graph-write) for the canonical signature, including the required `SQLiteAdapter` argument and transactional semantics.
|
|
67
|
-
|
|
68
|
-
This is the GraphRAG seed path: load a corpus, walk it.
|
|
69
|
-
|
|
70
|
-
## Direct fact writes
|
|
71
|
-
|
|
72
|
-
`remember()` stores one mutable fact in one transaction, with no text-generation (LLM) call, and returns the stored row. When `llmProvider.embed` is configured it does await one embedding call before resolving (failures surface as `embedding_failed`, not a throw). Use it when the host already knows the fact (a user preference, an agent's observation the user approved). To replace a fact, use `supersede()` so history is kept.
|
|
73
|
-
|
|
74
|
-
```typescript
|
|
75
|
-
const fact = await wiki.remember('user-123', {
|
|
76
|
-
title: 'Prefers merge commits',
|
|
77
|
-
body: 'Kurt wants PRs merged with merge commits, never squashed.',
|
|
78
|
-
tags: ['git'], // default []
|
|
79
|
-
confidence: 'certain', // default
|
|
80
|
-
source_type: 'user_stated', // or 'user_confirmed'; default 'user_stated'
|
|
81
|
-
okf_type: 'Preference', // optional; resolved like an upsertGraph node type
|
|
82
|
-
valid_from: Date.parse('2026-03-01'), // optional epoch ms
|
|
83
|
-
});
|
|
84
|
-
fact.id; // 'fact_…' — the row as stored: timestamps, defaults, isStale, trustTier
|
|
85
|
-
```
|
|
86
|
-
|
|
87
|
-
- Validation throws `TypeError` before anything is written: `title` non-empty, `body` a string, `tags` an array of strings, known `confidence` and `source_type` (explicit `null` is rejected; omit the field to take the default), `okf_type` a non-empty string or null, finite and >= 0 `valid_from`. Nothing is clipped; length limits are the host's.
|
|
88
|
-
- `okf_type` follows the `upsertGraph` rule: the manifest's canonical slug when known; `WikiStrictOntologyViolation` under strict mode otherwise; `null` otherwise.
|
|
89
|
-
- Never starts the librarian or heal and writes no event, in `'auto'` and `'deferred'` mode alike. Embeds through `llmProvider.embed` when present (failures surface as `embedding_failed` with `operation: 'remember'`); keyword-only otherwise, and `getPendingMaintenance().reembedPending` picks it up once an `embed` exists.
|
|
90
|
-
- Not idempotent: two identical calls store two facts. Hosts that expose retries own the idempotency key.
|
|
91
|
-
- Opens its own transaction. Do not call it inside `withTransactionAsync`.
|
|
92
|
-
|
|
93
|
-
## Temporal facts
|
|
94
|
-
|
|
95
|
-
Facts carry an optional validity window. Nothing changes until you use it: facts without validity data are always current.
|
|
96
|
-
|
|
97
|
-
```typescript
|
|
98
|
-
// The user tells you they moved; the move happened on March 1.
|
|
99
|
-
await wiki.supersede('user-123', seattleFactId, { title: 'Lives in San Francisco', body: 'Moved in March.' }, {
|
|
100
|
-
validFrom: Date.parse('2026-03-01'),
|
|
101
|
-
});
|
|
102
|
-
|
|
103
|
-
await wiki.read('user-123', 'where do I live'); // current: San Francisco
|
|
104
|
-
await wiki.read('user-123', 'where do I live', { asOf: Date.parse('2026-02-01') }); // then: Seattle
|
|
105
|
-
await wiki.history('user-123', seattleFactId); // [Seattle, San Francisco]
|
|
106
|
-
|
|
107
|
-
// Record when an event really happened (defaults to ingestion time):
|
|
108
|
-
await wiki.write('user-123', { event_type: 'observation', summary: 'Moved to SF', occurred_at: Date.parse('2026-03-01') });
|
|
109
|
-
```
|
|
110
|
-
|
|
111
|
-
- Validity is half-open: `[valid_from, valid_to)`. At the switch instant only the new fact matches.
|
|
112
|
-
- `asOf` answers "what do we now know was true at T". Learning late about a past change still answers correctly for the past.
|
|
113
|
-
- `supersede()` refuses `immutable_document` facts (re-ingest the document instead), cycles, facts owned by another entity, and facts already superseded. It throws `WikiSupersedeError` with a `reason`.
|
|
114
|
-
- `traverseGraph({ asOf })` only walks edges whose two facts are both live at that instant.
|
|
115
|
-
- History is kept. Set `config.pruneSupersededAfter` (days) to let `runPrune` retire expired facts.
|
|
116
|
-
- `getMemoryBundle()` and `exportDump()` return every non-deleted fact, including history.
|
|
117
|
-
|
|
118
|
-
## Librarian strategies
|
|
119
|
-
|
|
120
|
-
`config.librarian.strategy` selects how events become facts. The default, `'legacy'`, is unchanged from 7.x.
|
|
121
|
-
|
|
122
|
-
```typescript
|
|
123
|
-
new WikiMemory(db, {
|
|
124
|
-
llmProvider,
|
|
125
|
-
config: { librarian: { strategy: 'ops', gate: { k: 5, dupThreshold: 0.89, novelThreshold: 0.30 } } },
|
|
126
|
-
});
|
|
127
|
-
```
|
|
128
|
-
|
|
129
|
-
`'ops'` works in three steps:
|
|
130
|
-
|
|
131
|
-
1. **Extract.** Candidate facts come from the events since the last pass. Existing facts are not sent.
|
|
132
|
-
2. **Gate, with no LLM call.** Each candidate is compared with its closest current facts. An identical fact is a no-op, and a candidate with no close match is added. Only candidates that are similar but different continue to step 3. Similar is not the same as identical: "lives in Seattle" and "moved to San Francisco" score as similar, so they go to the model.
|
|
133
|
-
3. **Resolve, in one batched call.** The model returns `ADD`, `UPDATE`, `SUPERSEDE` or `NOOP` for each remaining candidate. `SUPERSEDE` uses `supersede()`, so history is kept and `read({ asOf })` still sees the old fact.
|
|
134
|
-
|
|
135
|
-
- The thresholds depend on your embedding model. The defaults are provisional.
|
|
136
|
-
- Without `embed`, the gate uses keyword search. Every keyword match then goes to the resolve step.
|
|
137
|
-
- Under `'ops'`, `runLibrarian`'s `promptOverride` and `prompts.librarianSystemPrompt` are ignored: the ops prompts use a different schema.
|
|
138
|
-
- A model answer that can't be used never loses a fact. It is added, or stored as a draft, and a diagnostic is emitted.
|
|
139
|
-
|
|
140
|
-
## Deferred maintenance
|
|
141
|
-
|
|
142
|
-
By default, `write()` starts the librarian and heal in the background when thresholds are crossed. On serverless or request-scoped hosts those background promises can be frozen or killed. Set `maintenance: 'deferred'` to decide when maintenance runs:
|
|
143
|
-
|
|
144
|
-
```typescript
|
|
145
|
-
const wiki = new WikiMemory(db, {
|
|
146
|
-
llmProvider,
|
|
147
|
-
config: { maintenance: 'deferred', librarian: { strategy: 'ops' } },
|
|
148
|
-
});
|
|
149
|
-
|
|
150
|
-
await wiki.write(entityId, { event_type: 'observation', summary, occurred_at });
|
|
151
|
-
|
|
152
|
-
const report = await wiki.runPendingMaintenance({ tokenBudget: 20_000, deadlineMs: 8_000 });
|
|
153
|
-
// report.stoppedReason: complete | deadline | budget_exhausted | budget_too_small
|
|
154
|
-
// Nothing is still running here, so it is safe to persist the database.
|
|
155
|
-
```
|
|
156
|
-
|
|
157
|
-
- `getPendingMaintenance()` shows the backlog per entity without calling a model.
|
|
158
|
-
- `remember()` and `supersede()` never schedule maintenance in either mode; only `write()` does.
|
|
159
|
-
- The budget covers text-generation tokens only; embedding calls are not counted. Figures are provider-reported when `generateTextWithUsage` exists, otherwise estimated at characters ÷ 4.
|
|
160
|
-
- Entities share the budget fairly: each round runs one batch per entity.
|
|
161
|
-
- `budget_too_small` names the first job and entity whose smallest step needed more than the whole budget; other entities may have hit the same limit in that run.
|
|
162
|
-
- A crash mid-batch loses nothing. The librarian's progress marker only advances after a batch commits.
|
|
163
|
-
- In `'auto'` mode, call `drain()` before the app is suspended or closed. `autoLibrarianTokenThreshold` also triggers the librarian on the size of the pending text.
|
|
164
|
-
- Use `'deferred'` with the ops strategy. The legacy strategy reads only the newest 50 events per pass and marks them processed, so a backlog larger than that is skipped.
|
|
165
|
-
- Under `'deferred'` or the ops strategy, `runPrune` never deletes events the librarian has not processed.
|
|
166
|
-
- `JobManager` locks only work within one process. Guarantee a single writer per database file across processes.
|
|
167
|
-
|
|
168
|
-
See the [SynapseTree integration guide](../../docs/synapsetree-integration.md) for the end-to-end pattern on tenant-aware hosts (S3-backed SQLite, request lifecycle, `entity_id` scheme, MCP surface, costs). Benchmarks live at [docs/benchmarks.md](../../docs/benchmarks.md).
|
|
17
|
+
- **Any SQLite driver.** Pure TypeScript with no native code. You pass a small [`SQLiteAdapter`](https://equationalapplications.github.io/expo-llm-wiki/guides/sqlite-adapter/) for `better-sqlite3`, `sql.js`, `expo-sqlite` or anything else.
|
|
18
|
+
- **Bring your own inference.** One `generateText` function, plus an optional `embed` for semantic search. The engine owns prompts, JSON parsing and database writes.
|
|
19
|
+
- **Hybrid retrieval.** Vector cosine ranking with a keyword fallback (SQLite FTS5 or in-memory MiniSearch), [tunable](https://equationalapplications.github.io/expo-llm-wiki/guides/tune-retrieval/) per call.
|
|
20
|
+
- **Multi-entity reads.** Search several namespaces in one pass with `tierWeights` and `tierFloors`.
|
|
21
|
+
- **GraphRAG.** Per-entity [seeded ontologies](https://equationalapplications.github.io/expo-llm-wiki/guides/ontologies/) (Strict, Emergent or Off) and SQL-only [graph traversal](https://equationalapplications.github.io/expo-llm-wiki/guides/graphrag/).
|
|
22
|
+
- **Background maintenance.** Librarian, heal, reembed and prune jobs, run inline or [deferred](https://equationalapplications.github.io/expo-llm-wiki/guides/maintenance/).
|
|
23
|
+
- **Temporal facts, drafts and grounding** for [trust and provenance](https://equationalapplications.github.io/expo-llm-wiki/concepts/trust-and-provenance/).
|
|
24
|
+
- **OKF v0.1 and v0.2** [import and export](https://equationalapplications.github.io/expo-llm-wiki/guides/okf/).
|
|
25
|
+
- **Diagnostics.** An optional `onDiagnostic` hook with typed, content-free reports.
|
|
169
26
|
|
|
170
27
|
## Installation
|
|
171
28
|
|
|
172
29
|
```bash
|
|
173
30
|
npm install @equationalapplications/core-llm-wiki
|
|
31
|
+
npm install better-sqlite3 # Node.js, or sql.js for the browser
|
|
174
32
|
```
|
|
175
33
|
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
Provide an `embed` function in `llmProvider` to enable vector-based retrieval:
|
|
179
|
-
|
|
180
|
-
```typescript
|
|
181
|
-
import { WikiMemory } from '@equationalapplications/core-llm-wiki';
|
|
182
|
-
|
|
183
|
-
const wikiMemory = new WikiMemory(db, {
|
|
184
|
-
llmProvider: {
|
|
185
|
-
generateText: async ({ systemPrompt, userPrompt }) => {
|
|
186
|
-
// Your LLM call for extracting facts, tasks
|
|
187
|
-
return 'Model output';
|
|
188
|
-
},
|
|
189
|
-
maxOutputTokens: 4096, // optional — lets runHeal/runOntologyBackfill size their first LLM call correctly instead of discovering the ceiling via a truncated response and retrying smaller
|
|
190
|
-
embed: async (text: string) => {
|
|
191
|
-
// Your embedding service (e.g., OpenAI, Cohere, local)
|
|
192
|
-
const response = await fetch('https://your-app.example.com/api/embed', {
|
|
193
|
-
method: 'POST',
|
|
194
|
-
body: JSON.stringify({ text }),
|
|
195
|
-
});
|
|
196
|
-
const { embedding } = await response.json();
|
|
197
|
-
return embedding; // number[]
|
|
198
|
-
},
|
|
199
|
-
},
|
|
200
|
-
});
|
|
201
|
-
|
|
202
|
-
await wikiMemory.setup();
|
|
203
|
-
|
|
204
|
-
// Query with semantic matching
|
|
205
|
-
const memory = await wikiMemory.read('user-123', 'What should I do this weekend?');
|
|
206
|
-
// Returns facts semantically similar to the query, not lexical matches
|
|
207
|
-
// E.g., fact "Saturday hiking trip" ranks high even though no lexical overlap
|
|
208
|
-
```
|
|
209
|
-
|
|
210
|
-
When `embed` is unavailable, `read()` silently falls back to keyword search. If an embedding attempt throws, `read()` falls back and calls `onRetrievalFallback` if provided:
|
|
211
|
-
|
|
212
|
-
```typescript
|
|
213
|
-
const wikiMemory = new WikiMemory(db, {
|
|
214
|
-
llmProvider: {
|
|
215
|
-
generateText: async () => { /* ... */ },
|
|
216
|
-
embed: undefined, // or throws on network error
|
|
217
|
-
},
|
|
218
|
-
onRetrievalFallback: (error) => {
|
|
219
|
-
console.warn('Embedding retrieval unavailable, using keyword search:', error);
|
|
220
|
-
},
|
|
221
|
-
});
|
|
222
|
-
|
|
223
|
-
// read() returns keyword results, onRetrievalFallback not called (embed absent is expected)
|
|
224
|
-
// read() returns keyword results, onRetrievalFallback called (embed threw)
|
|
225
|
-
```
|
|
226
|
-
|
|
227
|
-
### Token budgets
|
|
228
|
-
|
|
229
|
-
```typescript
|
|
230
|
-
const bundle = await wiki.read('user-123', 'what should I cook?', { tokenBudget: 800 });
|
|
231
|
-
const prompt = formatContext(bundle, { compact: true });
|
|
232
|
-
```
|
|
233
|
-
|
|
234
|
-
`tokenBudget` keeps the most relevant facts per token that fit within about 800 tokens. Tokens are estimated as characters ÷ 4. The top fact is always included, truncated if it alone is too big. Tasks and events are unaffected. `traverseGraph` accepts the same option and keeps nodes in breadth-first order. A `read_budget` diagnostic reports how many facts were packed.
|
|
235
|
-
|
|
236
|
-
## Configuration
|
|
237
|
-
|
|
238
|
-
All `WikiConfig` fields are optional:
|
|
239
|
-
|
|
240
|
-
```typescript
|
|
241
|
-
const wikiMemory = new WikiMemory(db, {
|
|
242
|
-
llmProvider: { /* ... */ },
|
|
243
|
-
config: {
|
|
244
|
-
tablePrefix: 'llm_wiki_', // default: 'llm_wiki_'
|
|
245
|
-
maxResults: 10, // default: 10
|
|
246
|
-
autoLibrarianThreshold: 20, // default: 20 — events before librarian auto-runs
|
|
247
|
-
autoHealThreshold: 100, // default: 100 — events before heal auto-runs
|
|
248
|
-
maxChunkLength: 12000, // default: 12000 (char count per ingestDocument chunk; exported as DEFAULT_MAX_CHUNK_LENGTH)
|
|
249
|
-
chunkOverlap: 400, // default: 400 (overlap between chunks in characters; exported as DEFAULT_CHUNK_OVERLAP)
|
|
250
|
-
chunkConcurrency: 1, // default: 1 (parallel LLM calls per ingestDocument)
|
|
251
|
-
maxEmbedChars: 6000, // default: 6000 (chars of title+body+tags sent to embed(); hard ceiling 16000; exported as DEFAULT_MAX_EMBED_CHARS)
|
|
252
|
-
pruneRetainSoftDeletedFor: 7, // default: 7 (days before hard-deleting soft-deleted facts)
|
|
253
|
-
pruneEventsAfter: 30, // default: 30 (days before hard-deleting old events)
|
|
254
|
-
orphanAfterDays: 30, // default: 30 (days before runHeal flags sourceless facts; null to disable)
|
|
255
|
-
staleInferredAfterDays: 60, // default: 60 (days before runHeal downgrades inferred facts; null to disable)
|
|
256
|
-
preFilterLimit: 50, // default: undefined — keyword pre-filter before cosine scan; recommended for >500 facts
|
|
257
|
-
hybridWeight: 0.7, // default: undefined — blend semantic (1.0) ↔ keyword (0.0); pure semantic when unset
|
|
258
|
-
// Keyword-index backend (#257): 'fts5' stores the index inside the database file, fed by triggers on the
|
|
259
|
-
// entries table, so memory stays flat at any size; setup() throws if the adapter's SQLite lacks FTS5 or json1.
|
|
260
|
-
// 'minisearch' keeps the in-RAM MiniSearch index, whose memory grows with total stored text; 'auto' (default)
|
|
261
|
-
// probes at setup() and uses FTS5 when present, else MiniSearch (e.g. sql.js, which ships without FTS5).
|
|
262
|
-
// The first setup() that attaches FTS5 to an existing database rebuilds the index in one transaction, holding
|
|
263
|
-
// the write lock for its whole duration. Keyword rankings differ between the two backends; set 'minisearch'
|
|
264
|
-
// to keep the pre-7.11 behavior.
|
|
265
|
-
// Under 'auto', a failed FTS5 init is retried once, then setup() completes with keyword search unavailable
|
|
266
|
-
// (queries throw WikiSearchUnavailableError); it never silently demotes to MiniSearch. While unavailable,
|
|
267
|
-
// runHeal returns searchUnavailable: true, runPendingMaintenance lists skipped jobs in perEntity[].searchUnavailable,
|
|
268
|
-
// and a no-arg syncSearchIndex() retries the init (after a failure, backing off from 5 s up to 5 min). MiniSearch holds all
|
|
269
|
-
// entry text in the JS heap — the measured ceiling is ~1M entries on a default Node heap (--max-old-space-size).
|
|
270
|
-
indexStrategy: 'auto', // default: 'auto' — 'fts5' | 'minisearch' | 'auto'
|
|
271
|
-
enableOutbox: false, // default: false — when true, entry/task mutations write to an internal SQLite outbox table for external sync (e.g. via @equationalapplications/prisma-outbox)
|
|
272
|
-
excludeDrafts: false, // default: false — engine default for read()/traverseGraph() excludeDrafts; see Draft Review
|
|
273
|
-
grounding: { mode: 'off' }, // default: off — 'draft' checks evidence quotes; see Grounding
|
|
274
|
-
ontology: {
|
|
275
|
-
// mode, seedManifests: see Per-Entity Seeded Ontology
|
|
276
|
-
backfillClassifier: 'llm', // default: 'llm' — 'auto' uses llmProvider.classify when present; see Ontology backfill
|
|
277
|
-
classifyMinConfidence: 0.5, // default: 0.5 — classifier answers below this are left untyped
|
|
278
|
-
},
|
|
279
|
-
|
|
280
|
-
// Global prompt overrides — librarianSystemPrompt and healSystemPrompt apply to write() auto-runs;
|
|
281
|
-
// ingestSystemPrompt applies only to explicit ingestDocument() calls.
|
|
282
|
-
// ⚠ Overrides replace the entire default prompt, including the JSON output contract.
|
|
283
|
-
// See "JSON Output Contracts" in the Prompt Management & Overrides section below.
|
|
284
|
-
prompts: {
|
|
285
|
-
ingestSystemPrompt: `Extract core facts from this document: {{documentChunk}}\n\nReturn ONLY valid JSON: { "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }] }. No markdown.`,
|
|
286
|
-
librarianSystemPrompt: `You are an expert curator. Synthesize these thoughts:\n{{events}}\n\nCurrent Facts:\n{{currentFacts}}\n\nReturn ONLY valid JSON: { "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }], "tasks": [{ "description": "string", "priority": 0 }] }. No markdown.`,
|
|
287
|
-
healSystemPrompt: `Fix the memory graph based on these candidates: {{healCandidates}}\n\nReturn ONLY valid JSON: { "downgraded": ["factId"], "deleted": ["factId"], "newFacts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }] }. No markdown.`,
|
|
288
|
-
},
|
|
289
|
-
},
|
|
290
|
-
// Host callbacks sit beside llmProvider, not inside config:
|
|
291
|
-
// onDiagnostic: (d) => { ... }, // see Diagnostics
|
|
292
|
-
});
|
|
293
|
-
```
|
|
294
|
-
|
|
295
|
-
### Opening inside a writer lease
|
|
296
|
-
|
|
297
|
-
One-time upgrade work (schema migrations, the `source_ref` normalize) can take minutes on a large store. If you open the store inside a time-limited writer lease, refuse that work instead of running it:
|
|
298
|
-
|
|
299
|
-
```typescript
|
|
300
|
-
try {
|
|
301
|
-
await wiki.setup({ migrations: 'refuse' }); // also settable as config.migrations
|
|
302
|
-
} catch (err) {
|
|
303
|
-
if (!(err instanceof WikiMigrationRequiredError)) throw err;
|
|
304
|
-
// err.pendingWork says what is pending. Release the lease, then:
|
|
305
|
-
await wiki.setup(); // runs the work out of lease (resumable if interrupted)
|
|
306
|
-
// ...re-acquire the lease and open again with 'refuse'.
|
|
307
|
-
}
|
|
308
|
-
```
|
|
309
|
-
|
|
310
|
-
`'allow'` (default) runs everything; `'auto'` throws on a pending schema migration but runs a pending normalize. Await `setup()` before any other call on the instance.
|
|
311
|
-
|
|
312
|
-
## Prompt Management & Overrides
|
|
313
|
-
|
|
314
|
-
Core maintenance tasks (`ingestDocument`, `runLibrarian`, `runHeal`) use system prompts to instruct the LLM. You can customize these prompts using `{{mustache}}` style variables to inject context dynamically.
|
|
315
|
-
|
|
316
|
-
> **JSON Output Contracts:** Prompt overrides replace the *entire* default system prompt, including the JSON response schema the parser depends on. Your override **must** instruct the LLM to return raw JSON — no markdown. Required shapes:
|
|
317
|
-
>
|
|
318
|
-
> | Operation | Required JSON shape |
|
|
319
|
-
> |-----------|-------------------|
|
|
320
|
-
> | `ingestDocument` | `{ "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain\|inferred\|tentative" }] }` |
|
|
321
|
-
> | `runLibrarian` | `{ "facts": [...], "tasks": [{ "description": "string", "priority": 5 }] }` — `priority` is an integer 0–10 |
|
|
322
|
-
> | `runHeal` | `{ "downgraded": ["factId"], "deleted": ["factId"], "newFacts": [...] }` |
|
|
323
|
-
>
|
|
324
|
-
> **Grounding:** when [`config.grounding`](#grounding) is on, the evidence instruction is appended after your override (and after ontology context) for every writer in `grounding.writers`. Your override does not need to ask for `evidence` itself.
|
|
34
|
+
Requires Node.js 20 or later. Using Expo or React? Install [`expo-llm-wiki`](https://equationalapplications.github.io/expo-llm-wiki/packages/expo/) or [`react-llm-wiki`](https://equationalapplications.github.io/expo-llm-wiki/packages/react/) instead. Both re-export core.
|
|
325
35
|
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
If your application relies on `write()` to automatically maintain the memory graph in the background (via `autoLibrarianThreshold` and `autoHealThreshold`), configure custom prompts globally at instantiation. This ensures the internal `WriteService` uses your domain-specific instructions when it triggers an auto-run.
|
|
329
|
-
|
|
330
|
-
```typescript
|
|
331
|
-
const wikiMemory = new WikiMemory(db, {
|
|
332
|
-
llmProvider,
|
|
333
|
-
config: {
|
|
334
|
-
prompts: {
|
|
335
|
-
// Override must include the JSON output contract — it replaces the entire default prompt.
|
|
336
|
-
librarianSystemPrompt: `You are an expert curator. Synthesize these thoughts:\n{{events}}\n\nCurrent Facts:\n{{currentFacts}}\n\nReturn ONLY a valid JSON object: { "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }], "tasks": [{ "description": "string", "priority": 0 }] }. No markdown.`,
|
|
337
|
-
},
|
|
338
|
-
},
|
|
339
|
-
});
|
|
340
|
-
|
|
341
|
-
// WriteService uses the global prompt whenever autoLibrarianThreshold is hit
|
|
342
|
-
await wikiMemory.write('user-123', { event_type: 'observation', summary: '...' });
|
|
343
|
-
```
|
|
344
|
-
|
|
345
|
-
Available `{{variables}}` per prompt type:
|
|
346
|
-
|
|
347
|
-
| Prompt | Variables |
|
|
348
|
-
|--------|-----------|
|
|
349
|
-
| `ingestSystemPrompt` | `{{documentChunk}}` |
|
|
350
|
-
| `librarianSystemPrompt` | `{{events}}`, `{{currentFacts}}` |
|
|
351
|
-
| `healSystemPrompt` | `{{healCandidates}}`, `{{documentAnchors}}`, `{{allTasks}}`, `{{recentEvents}}` |
|
|
352
|
-
|
|
353
|
-
When a template contains `{{variable}}` tags, the matching data is hydrated directly into `systemPrompt` and a short fixed string is used as `userPrompt`. When a template has no `{{}}` tags, the raw data is appended as `userPrompt` — backward compatible with plain-string overrides.
|
|
354
|
-
|
|
355
|
-
### Runtime Overrides (Manual Execution)
|
|
356
|
-
|
|
357
|
-
Pass `promptOverride` per-call for one-off instructions. **Runtime overrides apply only to that single call — they do not persist for future auto-runs triggered by `write()`.**
|
|
358
|
-
|
|
359
|
-
```typescript
|
|
360
|
-
// Override the base default AND global config for this single execution.
|
|
361
|
-
// Each override must include the JSON output contract (replaces the entire default prompt).
|
|
362
|
-
await wikiMemory.runLibrarian('user-123', {
|
|
363
|
-
promptOverride: `One-off extraction task:\n{{events}}\n\nReturn ONLY valid JSON: { "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }], "tasks": [{ "description": "string", "priority": 0 }] }. No markdown.`,
|
|
364
|
-
});
|
|
365
|
-
|
|
366
|
-
await wikiMemory.runHeal('user-123', {
|
|
367
|
-
promptOverride: `Domain-specific healing: {{healCandidates}}\n\nReturn ONLY valid JSON: { "downgraded": ["factId"], "deleted": ["factId"], "newFacts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }] }. No markdown.`,
|
|
368
|
-
});
|
|
369
|
-
|
|
370
|
-
await wikiMemory.ingestDocument('user-123', {
|
|
371
|
-
sourceRef: 'doc-1',
|
|
372
|
-
sourceHash: sha256(content),
|
|
373
|
-
documentChunk: content,
|
|
374
|
-
promptOverride: `Strict technical extraction: {{documentChunk}}\n\nReturn ONLY valid JSON: { "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }] }. No markdown.`,
|
|
375
|
-
});
|
|
376
|
-
```
|
|
377
|
-
|
|
378
|
-
> **Important:** If your app relies on `write()` auto-runs and needs custom prompts for those runs, use `config.prompts` at construction time. Runtime `promptOverride` values are never forwarded to `WriteService`-triggered internal runs.
|
|
379
|
-
|
|
380
|
-
### Effective instructions (`getInstructions`)
|
|
381
|
-
|
|
382
|
-
```typescript
|
|
383
|
-
const { ingest, librarian, heal, ontologyBackfill } = await wikiMemory.getInstructions('entity-123');
|
|
384
|
-
```
|
|
385
|
-
|
|
386
|
-
Returns the system prompt each writer sends, with `WikiConfig.prompts` overrides applied. Ingest, librarian and ontology backfill also get the entity's ontology block; heal gets none, as at runtime. When `WikiConfig.grounding` is on, the evidence block is appended for each writer in `grounding.writers`, exactly as sent. Data placeholders such as `{{documentChunk}}` stay unfilled; no events, chunks or facts are included. It reflects `WikiConfig.prompts` only: a per-call `promptOverride` is not reflected, and `ontologyBackfill` is returned even when backfill would send no prompt (ontology `off`, or the classifier path). `core-llm-tools` exposes this as the `wiki_get_instructions` tool (`memory:read`), so agents can see the engine's output format and constraints before proposing writes. Agents should treat it as reference data, not instructions; see [Prompt-Injection Trust Boundary](#prompt-injection-trust-boundary).
|
|
387
|
-
|
|
388
|
-
> **Warning:** overrides are returned verbatim to any client with `memory:read`. Never put secrets, API keys or private data in `WikiConfig.prompts`.
|
|
389
|
-
|
|
390
|
-
## Retrieval Tuning
|
|
391
|
-
|
|
392
|
-
Optimize `read()` performance and blend retrieval strategies:
|
|
393
|
-
|
|
394
|
-
```typescript
|
|
395
|
-
const config = {
|
|
396
|
-
// Limit cosine similarity scoring to top-K MiniSearch keyword candidates
|
|
397
|
-
preFilterLimit: 50,
|
|
398
|
-
|
|
399
|
-
// Blend semantic and keyword scores (0.0 = pure keyword, 1.0 = pure semantic)
|
|
400
|
-
hybridWeight: 0.7,
|
|
401
|
-
|
|
402
|
-
// Max results returned per read
|
|
403
|
-
maxResults: 10,
|
|
404
|
-
};
|
|
405
|
-
|
|
406
|
-
const wikiMemory = new WikiMemory(db, {
|
|
407
|
-
config,
|
|
408
|
-
llmProvider: { /* ... */ },
|
|
409
|
-
});
|
|
410
|
-
|
|
411
|
-
// Per-call overrides (runtime controls for search dashboards, etc.)
|
|
412
|
-
const memory = await wikiMemory.read('user-123', 'my preferences', {
|
|
413
|
-
maxResults: 5,
|
|
414
|
-
preFilterLimit: 20,
|
|
415
|
-
hybridWeight: 0.5,
|
|
416
|
-
});
|
|
417
|
-
|
|
418
|
-
// Multi-entity with tier weights
|
|
419
|
-
const multiMemory = await wikiMemory.read(['tier_wisdom', 'tier_fact', 'tier_working'], 'my preferences', {
|
|
420
|
-
maxResults: 8,
|
|
421
|
-
tierWeights: {
|
|
422
|
-
tier_wisdom: 2, // high-confidence curated notes boosted 2×
|
|
423
|
-
tier_fact: 1, // neutral baseline
|
|
424
|
-
tier_working: 0.25, // recent but unvetted context downranked
|
|
425
|
-
},
|
|
426
|
-
// includeZeroWeightEntities: true — include 0-weight entities as bottom-ranked filler
|
|
427
|
-
});
|
|
428
|
-
// multiMemory.factScores — optional Record<factId, weightedScore> for returned facts; may be absent/undefined
|
|
429
|
-
// multiMemory.metadata — optional { query, entityIds, tierWeights, tierFloors }; may be absent/undefined
|
|
430
|
-
```
|
|
431
|
-
|
|
432
|
-
**Hybrid scoring blends:**
|
|
433
|
-
- `hybridWeight: 1.0` → all-semantic blend with semantic scores clamped to non-negative range (no keyword component)
|
|
434
|
-
- `hybridWeight: 0.5` → balanced semantic + keyword (50/50 blend)
|
|
435
|
-
- `hybridWeight: 0.0` → pure keyword ranking, skips `embed()` entirely (no LLM API cost)
|
|
436
|
-
|
|
437
|
-
True cosine-range pure semantic ranking (including negative cosine values) is used when `hybridWeight` is left `undefined`.
|
|
438
|
-
|
|
439
|
-
**Tier weights:**
|
|
440
|
-
- `tierWeights` applies a per-entity multiplier after semantic/keyword scoring: `finalScore = retrievalScore × weight`
|
|
441
|
-
- Missing weights default to `1.0`. Negative weights clamp to `0`. Non-finite weights default to `1.0`.
|
|
442
|
-
- `tierWeights[entity] = 0` skips that entity's scored retrieval branch (no compute cost).
|
|
443
|
-
- `includeZeroWeightEntities: true` includes zero-weight entities as bottom-ranked filler instead of skipping them.
|
|
444
|
-
- `tierFloors: Record<string, number>` reserves that entity's top-N scored results before the global `maxResults` cut — a floor on entity `X` means at least `N` of the returned facts come from `X`. Floors apply *after* scoring (including `tierWeights`) and *after* `preFilterLimit`'s candidate selection, so a row excluded by `preFilterLimit` cannot be resurrected by a floor. An entity with fewer matching facts than its floor contributes what it has — this is not an error. Only meaningful when `entityId` is an array; ignored for single-string calls and for empty-query ("recent facts") reads. Throws `WikiInvalidReadOptions` when a floor cannot be satisfied by construction: floors summing above `maxResults`, a floor keyed to an entity not in `entityId`, or a floor on an entity excluded by `tierWeights: 0` when `includeZeroWeightEntities` is not set. With an external `vectorRanker`, ranker-omitted rows (typically un-embedded facts) for a floored entity are pre-reserved before the global backfill budget is spent, so an unembedded floored entity still satisfies its floor.
|
|
445
|
-
- `factScores` is present for array-shaped `entityId` calls only when the query is non-empty and at least one fact is scored; empty-query ("recent facts") reads leave it absent even when `entityId` is an array. Plain string calls never expose it. `metadata` is present for all array-shaped calls regardless of query.
|
|
446
|
-
- `maxResults` applies globally across all requested entities.
|
|
447
|
-
- Tasks are capped at `min(20 × entityCount, 200)`; events at `min(10 × entityCount, 100)` for multi-entity reads.
|
|
448
|
-
|
|
449
|
-
**Pre-filtering optimization:**
|
|
450
|
-
When `preFilterLimit: 50` is set with 1000 facts, cosine similarity is computed only for the top 50 keyword matches, reducing O(N) scoring to O(50).
|
|
451
|
-
|
|
452
|
-
## Draft Review
|
|
453
|
-
|
|
454
|
-
Facts can carry `lifecycle_status: 'draft'`, for example when a host marks model output as unreviewed. Drafts stay visible by default. To keep them out of results:
|
|
36
|
+
## Quick look
|
|
455
37
|
|
|
456
38
|
```ts
|
|
457
|
-
|
|
458
|
-
await wiki.traverseGraph('user-1', { sourceId, excludeDrafts: true });
|
|
459
|
-
// or engine-wide:
|
|
460
|
-
createWiki(db, { llmProvider, config: { excludeDrafts: true } });
|
|
461
|
-
```
|
|
39
|
+
import { createWiki, formatContext } from '@equationalapplications/core-llm-wiki';
|
|
462
40
|
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
- Status is read from SQLite on every call. A promotion is visible immediately, with no re-indexing.
|
|
41
|
+
const wiki = createWiki(adapter, { llmProvider }); // your SQLiteAdapter and LLMProvider
|
|
42
|
+
await wiki.setup();
|
|
466
43
|
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
const { facts, nextCursor } = await wiki.listDrafts('user-1', { limit: 50 });
|
|
471
|
-
await wiki.promoteDraft(facts[0].id, 'user-1', { by: 'human:alice' }); // → stable, trustTier 'human-reviewed'
|
|
472
|
-
// Reject with setLifecycleStatus(id, entityId, 'deprecated') or forget().
|
|
44
|
+
await wiki.write('user-123', { event_type: 'observation', summary: 'Loves hiking on weekends.' });
|
|
45
|
+
await wiki.runLibrarian('user-123'); // extract facts from events
|
|
46
|
+
const context = formatContext(await wiki.read('user-123', 'weekend plans'));
|
|
473
47
|
```
|
|
474
48
|
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
## Grounding
|
|
478
|
-
|
|
479
|
-
Opt-in, deterministic evidence check for LLM-authored facts. When on, writers you choose must quote the source they were shown. A fact whose quotes are missing or not found is stored as a `draft` (see [Draft Review](#draft-review)), not rejected. A fact whose quotes all check out is stored `stable` with a `process:grounding-check` verifier, so its `trustTier` is `machine-confirmed`. Default off: 7.x write behavior is unchanged.
|
|
480
|
-
|
|
481
|
-
```ts
|
|
482
|
-
new WikiMemory(db, {
|
|
483
|
-
llmProvider,
|
|
484
|
-
config: {
|
|
485
|
-
grounding: {
|
|
486
|
-
mode: 'draft', // 'off' (default) | 'draft'
|
|
487
|
-
writers: ['ingest'], // default ['ingest']; also 'librarian', 'heal'
|
|
488
|
-
minEvidenceChars: 20, // shorter quotes count as absent
|
|
489
|
-
maxEvidence: 3, // quotes asked for per fact
|
|
490
|
-
maxEvidenceChars: 300,
|
|
491
|
-
},
|
|
492
|
-
},
|
|
493
|
-
});
|
|
494
|
-
```
|
|
495
|
-
|
|
496
|
-
- **What counts as source.**
|
|
497
|
-
- Ingest: the chunk text.
|
|
498
|
-
- Librarian: the `summary` of each event in the prompt.
|
|
499
|
-
- Heal: the `summary` of each recent event in the prompt, plus the bodies of non-draft document anchors. When heal is a writer, anchors are shown with their body clipped to 800 characters.
|
|
500
|
-
- Instructions, the ontology manifest, existing facts and identifiers never count, so a model cannot ground a claim by quoting them.
|
|
501
|
-
- **The check.** Both sides are normalized with NFKC, whitespace runs collapse to one space, and matching is case-sensitive. A fact with more than 10 quotes, or any quote not found, fails.
|
|
502
|
-
- **Diagnostics.** `grounding_missing` (reasons `no_evidence`, `evidence_too_short`) and `grounding_failed` (reasons `quote_not_found`, `too_many_quotes`), one per fact, with the new fact's `factId`. Quotes are never included.
|
|
503
|
-
- **Duplicate titles in one ingest.** When chunks yield facts with the same title, ingest keeps one: a grounded fact beats one with missing or failed evidence, and on a tie the first one wins. The others are reported as `fact_deduplicated`. A fact already stored for the same `sourceRef` is not replaced by a later partial ingest, and that skip is reported the same way. Either way the diagnostic carries `{ sourceRef, chunkIndex, itemIndex }` for the skipped fact, whether or not grounding is enabled.
|
|
504
|
-
- **`upsertGraph`** nodes are host-supplied and never grounded.
|
|
505
|
-
- **Librarian and heal** synthesize across events, so their pass rates are unknown. Measure them on your own event log before opting them in.
|
|
506
|
-
- Evidence quotes are not stored.
|
|
507
|
-
|
|
508
|
-
## Pluggable Vector Retrieval
|
|
509
|
-
|
|
510
|
-
When your entity corpus grows, in-process cosine similarity scoring becomes a bottleneck. The optional **`VectorRanker`** interface lets you delegate semantic ranking to [**sqlite-vec**](https://github.com/asg017/sqlite-vec), [**sqlite-vss**](https://github.com/asg017/sqlite-vss), or an external vector database while `WikiMemory` handles embedding validation, hybrid scoring, and tier-2 row hydration.
|
|
511
|
-
|
|
512
|
-
### `VectorRanker` purpose
|
|
513
|
-
|
|
514
|
-
`VectorRanker` provides an optional injection point for approximate nearest-neighbor (ANN) ranking:
|
|
515
|
-
|
|
516
|
-
```typescript
|
|
517
|
-
export interface VectorRanker {
|
|
518
|
-
/**
|
|
519
|
-
* Return semantic scores for facts in scope, sorted by similarity.
|
|
520
|
-
* - `entityId`: restricts results to one entity
|
|
521
|
-
* - `queryVec`: the embedded query (Float32Array or number[])
|
|
522
|
-
* - `candidateIds` (optional): when set, rank only within this set (MiniSearch pre-filter mode)
|
|
523
|
-
* - `limit`: requested top-K count
|
|
524
|
-
*/
|
|
525
|
-
rankBySimilarity(args: VectorRankerRankArgs): Promise<VectorRankerSemanticResult[]>;
|
|
526
|
-
|
|
527
|
-
/**
|
|
528
|
-
* Optional hook called after embedding persistence (upsert, reembed, delete).
|
|
529
|
-
* Implementations use this to keep external indexes (sqlite-vec, remote ANN) in sync.
|
|
530
|
-
*/
|
|
531
|
-
onEmbeddingPersisted?(event: {
|
|
532
|
-
entityId: string;
|
|
533
|
-
factId: string;
|
|
534
|
-
vector: Float32Array | null; // null = embedding removed
|
|
535
|
-
}): void | Promise<void>;
|
|
536
|
-
}
|
|
537
|
-
```
|
|
538
|
-
|
|
539
|
-
**When no ranker is configured**, `WikiMemory` uses built-in JS cosine similarity — the same behavior as today. When a ranker is supplied and embeddings preconditions are met (`embed` available, dimensions match, no mismatches), `WikiMemory` delegates scoring to the ranker and blends results with keyword scores.
|
|
540
|
-
|
|
541
|
-
### Example: sqlite-vec adapter
|
|
542
|
-
|
|
543
|
-
```typescript
|
|
544
|
-
import { WikiMemory } from '@equationalapplications/core-llm-wiki';
|
|
545
|
-
import type { VectorRanker, VectorRankerRankArgs, VectorRankerSemanticResult } from '@equationalapplications/core-llm-wiki';
|
|
546
|
-
|
|
547
|
-
// Minimal sqlite-vec adapter (pseudo-code)
|
|
548
|
-
const sqliteVecRanker: VectorRanker = {
|
|
549
|
-
async rankBySimilarity(args: VectorRankerRankArgs): Promise<VectorRankerSemanticResult[]> {
|
|
550
|
-
const { entityId, queryVec, candidateIds, limit } = args;
|
|
551
|
-
|
|
552
|
-
// Build KNN query using sqlite-vec's distance functions.
|
|
553
|
-
// sqlite-vec returns cosine distance (0 = identical, 2 = opposite) ascending.
|
|
554
|
-
// Invert to semanticScore: higher = more similar, matching VectorRanker contract.
|
|
555
|
-
let sql = `SELECT id, (1.0 - distance) AS semanticScore FROM vec_facts
|
|
556
|
-
WHERE entity_id = ? AND deleted_at IS NULL`;
|
|
557
|
-
const params: any[] = [entityId];
|
|
558
|
-
|
|
559
|
-
// Apply pre-filter if provided
|
|
560
|
-
if (candidateIds) {
|
|
561
|
-
sql += ` AND id IN (${candidateIds.map(() => '?').join(',')})`;
|
|
562
|
-
params.push(...candidateIds);
|
|
563
|
-
}
|
|
564
|
-
|
|
565
|
-
// KNN search (example syntax; adjust for your sqlite-vec version)
|
|
566
|
-
sql += ` ORDER BY vec MATCH vec_neighbor(?) LIMIT ?`;
|
|
567
|
-
params.push(queryVec, limit);
|
|
568
|
-
|
|
569
|
-
const rows = await db.getAllAsync<{ id: string; semanticScore: number }>(sql, params);
|
|
570
|
-
return rows; // sorted descending by semanticScore (closest distance → highest similarity)
|
|
571
|
-
},
|
|
572
|
-
|
|
573
|
-
async onEmbeddingPersisted(event) {
|
|
574
|
-
const { entityId, factId, vector } = event;
|
|
575
|
-
if (vector) {
|
|
576
|
-
// Upsert into sqlite-vec table
|
|
577
|
-
await db.runAsync(
|
|
578
|
-
`INSERT OR REPLACE INTO vec_facts (id, entity_id, vec) VALUES (?, ?, ?)`,
|
|
579
|
-
[factId, entityId, vector]
|
|
580
|
-
);
|
|
581
|
-
} else {
|
|
582
|
-
// Delete when embedding is removed
|
|
583
|
-
await db.runAsync(`DELETE FROM vec_facts WHERE id = ?`, [factId]);
|
|
584
|
-
}
|
|
585
|
-
},
|
|
586
|
-
};
|
|
587
|
-
|
|
588
|
-
const wikiMemory = new WikiMemory(db, {
|
|
589
|
-
llmProvider: { /* ... */ },
|
|
590
|
-
vectorRanker: sqliteVecRanker,
|
|
591
|
-
});
|
|
592
|
-
|
|
593
|
-
// read() now uses sqlite-vec for scoring instead of JS cosine
|
|
594
|
-
const memory = await wikiMemory.read('user-123', 'my preferences');
|
|
595
|
-
```
|
|
596
|
-
|
|
597
|
-
### Fallback policies
|
|
598
|
-
|
|
599
|
-
When `rankBySimilarity` rejects (e.g., ANN service outage, misconfiguration), `WikiMemory` applies a recovery policy:
|
|
600
|
-
|
|
601
|
-
```typescript
|
|
602
|
-
export type VectorRankerFallback =
|
|
603
|
-
| 'js-cosine' // (default) Score candidates in-process with JS cosine — same as no ranker
|
|
604
|
-
| 'keyword' // Skip semantic ranking; return keyword-only results
|
|
605
|
-
| 'empty' // Semantic facts list empty for this read; tasks/events still included
|
|
606
|
-
| 'throw'; // Reject read() with the ranker error
|
|
607
|
-
|
|
608
|
-
const wikiMemory = new WikiMemory(db, {
|
|
609
|
-
llmProvider: { /* ... */ },
|
|
610
|
-
vectorRanker: sqliteVecRanker,
|
|
611
|
-
vectorRankerFallback: 'js-cosine', // default
|
|
612
|
-
onVectorRankerFallback: (info) => {
|
|
613
|
-
console.warn(
|
|
614
|
-
`Ranker failed (policy: ${info.policy}); error:`,
|
|
615
|
-
info.error
|
|
616
|
-
);
|
|
617
|
-
},
|
|
618
|
-
});
|
|
619
|
-
```
|
|
620
|
-
|
|
621
|
-
- **`'js-cosine'` (default):** Seamless degradation; same behavior as if no ranker was configured.
|
|
622
|
-
- **`'keyword'`:** Useful when semantic ranking is optional; keyword search proceeds normally.
|
|
623
|
-
- **`'empty'`:** Return no facts for this query (but tasks/events still load); useful for strict consistency.
|
|
624
|
-
- **`'throw'`:** Propagate the error and fail the read.
|
|
625
|
-
|
|
626
|
-
### `onEmbeddingPersisted` eventual consistency
|
|
627
|
-
|
|
628
|
-
If `vectorRanker.onEmbeddingPersisted` returns a pending Promise, the hook **may resolve asynchronously**. This supports ANN indexes that rebuild on a schedule (e.g., sqlite-vec triggers on transaction commit) or external services with eventual consistency.
|
|
629
|
-
|
|
630
|
-
**Best practice:**
|
|
631
|
-
- If your adapter has **synchronous guarantees** (in-process sqlite-vec, same transaction), await the promise.
|
|
632
|
-
- If your adapter is **eventually consistent** (remote ANN, async rebuild), document the lag and document that queries may miss recently-added facts until the index refreshes.
|
|
633
|
-
- The **SQLite blob remains the source of truth**; `WikiMemory` always writes embeddings to `embedding_blob` first before calling the hook.
|
|
634
|
-
|
|
635
|
-
### Hybrid scoring with ranker
|
|
636
|
-
|
|
637
|
-
When both `vectorRanker` and `hybridWeight` are configured, `WikiMemory` still applies hybrid blending after the ranker returns scores:
|
|
638
|
-
|
|
639
|
-
```typescript
|
|
640
|
-
const wikiMemory = new WikiMemory(db, {
|
|
641
|
-
config: {
|
|
642
|
-
hybridWeight: 0.7, // 70% semantic, 30% keyword
|
|
643
|
-
},
|
|
644
|
-
vectorRanker: sqliteVecRanker,
|
|
645
|
-
});
|
|
646
|
-
|
|
647
|
-
// ranker returns semanticScore; WikiMemory blends with MiniSearch keyword score
|
|
648
|
-
const memory = await wikiMemory.read('user-123', 'my preferences', {
|
|
649
|
-
hybridWeight: 0.5, // per-call override to 50/50 blend
|
|
650
|
-
});
|
|
651
|
-
```
|
|
652
|
-
|
|
653
|
-
Note on semantics:
|
|
654
|
-
- Leave `hybridWeight` undefined for true pure-semantic cosine-range scoring.
|
|
655
|
-
- Set `hybridWeight: 1` for an all-semantic variant that clamps negative semantic scores to 0.
|
|
656
|
-
|
|
657
|
-
For details on hybrid scoring formulas and trade-offs, see [Retrieval Tuning](#retrieval-tuning) above.
|
|
658
|
-
|
|
659
|
-
### Spec and issue reference
|
|
660
|
-
|
|
661
|
-
- **Full spec:** [`docs/superpowers/specs/2026-05-07-pluggable-vector-retrieval.md`](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/superpowers/specs/2026-05-07-pluggable-vector-retrieval.md)
|
|
662
|
-
- **GitHub issue:** [#15](https://github.com/equationalapplications/expo-llm-wiki/issues/15)
|
|
663
|
-
|
|
664
|
-
## Vector Cache
|
|
665
|
-
|
|
666
|
-
Parsed embedding vectors from full-scan `read()` calls are cached in memory, keyed by entity ID (max 16 entities, max 500 vectors per entity). This avoids redundant `Float32Array` parsing on repeated queries for the same entity. When the 16-entity limit is reached, the oldest-inserted entity is evicted to make room; if an entity exceeds 500 facts, its vectors are not cached at all for that read.
|
|
667
|
-
|
|
668
|
-
After heavy read workloads or on memory-constrained runtimes, you can release the entire cache explicitly:
|
|
669
|
-
|
|
670
|
-
```typescript
|
|
671
|
-
// Release all cached embedding vectors
|
|
672
|
-
wikiMemory.clearVectorCache();
|
|
673
|
-
```
|
|
674
|
-
|
|
675
|
-
The cache is also automatically invalidated on any mutation (`runLibrarian`, `runHeal`, `runPrune`, `runReembed`, `ingestDocument`, `importDump`, `forget`).
|
|
676
|
-
|
|
677
|
-
## Re-Embedding & Retry Behavior
|
|
678
|
-
|
|
679
|
-
`runReembed(entityId?, opts?)` returns `{ embedded, skipped, failed, deferred, permanentlyFailed }`.
|
|
680
|
-
|
|
681
|
-
- `failed` — attempted this sweep and failed.
|
|
682
|
-
- `deferred` — previously failed and still inside its exponential backoff window (60s doubling, capped at 24h). Not an error; a later sweep will retry.
|
|
683
|
-
- `permanentlyFailed` — excluded for good: a `float32_overflow` failure, or 5 failed attempts. Pass `{ force: true }` to retry these anyway.
|
|
684
|
-
|
|
685
|
-
Convergence loops should test `failed`, never `deferred` — deferred rows clear themselves once their backoff elapses.
|
|
686
|
-
|
|
687
|
-
Two caveats on that guidance:
|
|
688
|
-
|
|
689
|
-
- A `storage_error` failure counts in `failed` but is never marked for backoff — a failure marker is itself a DB write, which is exactly what is broken. While DB writes fail, every sweep re-attempts every row, so a loop testing `failed` can spin without converging. Treat repeated `storage_error` failures as an infrastructure signal to fix, not a retry window to wait out.
|
|
690
|
-
- `{ force: true }` bypasses more than `permanentlyFailed`: it also skips the backoff window, so rows that would otherwise be `deferred` are re-attempted immediately.
|
|
691
|
-
|
|
692
|
-
## Entity Status
|
|
693
|
-
|
|
694
|
-
`WikiMemory` exposes the in-flight job state for a single entity through two complementary APIs.
|
|
695
|
-
|
|
696
|
-
### `getEntityStatus(entityId)`
|
|
697
|
-
|
|
698
|
-
Synchronous point-in-time snapshot:
|
|
699
|
-
|
|
700
|
-
```typescript
|
|
701
|
-
const status = wikiMemory.getEntityStatus('user-42');
|
|
702
|
-
// { ingesting: boolean, librarian: boolean, heal: boolean }
|
|
703
|
-
```
|
|
704
|
-
|
|
705
|
-
Use this when you only need the current value (e.g. inside a request handler).
|
|
706
|
-
|
|
707
|
-
### `subscribeEntityStatus(entityId, callback)`
|
|
708
|
-
|
|
709
|
-
Push-based change notification — the callback fires synchronously once with the current status, then again on every transition where any of the three booleans flips. There is no polling and no duplicate snapshots.
|
|
710
|
-
|
|
711
|
-
```typescript
|
|
712
|
-
const unsubscribe = wikiMemory.subscribeEntityStatus('user-42', (status) => {
|
|
713
|
-
console.log(status); // { ingesting, librarian, heal }
|
|
714
|
-
});
|
|
715
|
-
|
|
716
|
-
// Later:
|
|
717
|
-
unsubscribe(); // idempotent — safe to call more than once
|
|
718
|
-
```
|
|
719
|
-
|
|
720
|
-
Notes:
|
|
721
|
-
|
|
722
|
-
- The first invocation happens **before** `subscribeEntityStatus` returns. Treat it as the initial render value.
|
|
723
|
-
- Each emission may be a fresh object literal. Do not rely on referential equality between callbacks; equality of the three booleans is the contract.
|
|
724
|
-
- A throwing callback is caught (logged via `console.error`) and does not block other subscribers or the underlying job.
|
|
725
|
-
- Subscriptions are scoped to a single `entityId`. There is no wildcard or "all entities" form.
|
|
726
|
-
|
|
727
|
-
## Diagnostics
|
|
728
|
-
|
|
729
|
-
Pass `onDiagnostic` to receive typed, content-free reports of events core used to drop silently or only log: failed chunks, rejected facts and tasks, dedupe drops, dropped edges, embedding and host-hook failures, heal skips, and background-job failures.
|
|
730
|
-
|
|
731
|
-
```ts
|
|
732
|
-
const wiki = createWiki(db, {
|
|
733
|
-
llmProvider,
|
|
734
|
-
onDiagnostic: (d) => {
|
|
735
|
-
// d.code, d.severity ('info' | 'warn' | 'error'), d.operation, d.trigger ('call' | 'auto'),
|
|
736
|
-
// d.entityId, d.at, d.message, d.detail?: { factId, sourceRef, chunkIndex, itemIndex,
|
|
737
|
-
// edgeType, sourceNodeType, targetNodeType, reason, count, chunkIndexes }
|
|
738
|
-
telemetry.record(d);
|
|
739
|
-
},
|
|
740
|
-
});
|
|
741
|
-
```
|
|
742
|
-
|
|
743
|
-
- **Content-free.** Diagnostics carry IDs, indexes, counts, ontology slugs and reason slugs only. They never carry titles, bodies, LLM output, provider error messages, or hashes of content.
|
|
744
|
-
- **Isolated.** The hook is called synchronously. A throwing or rejecting hook never affects the operation, and its failure is logged with `console.warn`.
|
|
745
|
-
- **After commit.** Transactional diagnostics are delivered after the operation's transaction commits. An operation that throws delivers none; the exception is the signal. `upsertGraph` runs in your transaction, so its diagnostics are delivered when it resolves. Disregard them if you roll back.
|
|
746
|
-
- **`trigger`.** `'auto'` marks work started by `autoLibrarianThreshold` / `autoHealThreshold`. A failed background job is reported as `background_job_failed` with `operation` set to the job.
|
|
747
|
-
- **`operation`** names the entry point: `ingest`, `upsertGraph`, `librarian`, `heal`, `ontologyBackfill`, `reembed`, `importDump`, `write`, `supersede`, `remember`, `read`, `prune`. New values may be added in minor releases.
|
|
748
|
-
- **Forward-compatible.** New codes may be added in minor releases; ignore codes you don't recognize.
|
|
749
|
-
- **Console output is unchanged** when a hook is set and succeeds: every existing core line is identical. A throwing or rejecting hook adds only its own `console.warn`, and a non-function `onDiagnostic` warns once.
|
|
750
|
-
|
|
751
|
-
## Per-Entity Seeded Ontology
|
|
752
|
-
|
|
753
|
-
Control how librarian and ingest passes classify facts and extract graph relationships. The system defaults to **`off`** so existing deployments behave unchanged.
|
|
754
|
-
|
|
755
|
-
### The Three Modes
|
|
756
|
-
|
|
757
|
-
| Mode | Behavior |
|
|
758
|
-
|------|----------|
|
|
759
|
-
| **`off`** (default) | No ontology guidance. LLM output and persistence match pre-ontology behavior: `okf_type` stays `null` on LLM-created facts; maintenance passes do not create edges. OKF import still populates `okf_type` and edges independently. |
|
|
760
|
-
| **`strict`** | The LLM must use only `node_types` and `edge_types` from the entity manifest. Invalid `okf_type` falls back to an untyped fact with no edges; invalid individual edges are dropped while a valid `okf_type` and matching edges are kept. |
|
|
761
|
-
| **`emergent`** | Same validation as Strict, plus the LLM may return `ontology_updates` with new node/edge types. Updates are append-only (deduped by `type` string) and take effect before facts from the same response are validated. |
|
|
762
|
-
|
|
763
|
-
Mode resolution per entity: persisted DB row `mode` (when present) → `seedManifests[entityId].mode` (when no row but a seed exists) → `WikiConfig.ontology.mode` → `'off'`.
|
|
764
|
-
|
|
765
|
-
### WikiConfig
|
|
766
|
-
|
|
767
|
-
Set a global default mode and bootstrap manifests for known entities at construction time:
|
|
768
|
-
|
|
769
|
-
```typescript
|
|
770
|
-
const wikiMemory = new WikiMemory(db, {
|
|
771
|
-
llmProvider,
|
|
772
|
-
config: {
|
|
773
|
-
ontology: {
|
|
774
|
-
mode: 'strict', // global default when an entity has no per-entity override
|
|
775
|
-
seedManifests: {
|
|
776
|
-
'team-alpha': {
|
|
777
|
-
mode: 'emergent', // optional per-entity override
|
|
778
|
-
manifest: {
|
|
779
|
-
node_types: [
|
|
780
|
-
{ type: 'person', description: 'An individual or user.' },
|
|
781
|
-
{ type: 'project', description: 'An ongoing initiative.' },
|
|
782
|
-
],
|
|
783
|
-
edge_types: [
|
|
784
|
-
{
|
|
785
|
-
type: 'contributes_to',
|
|
786
|
-
source_type: 'person',
|
|
787
|
-
target_type: 'project',
|
|
788
|
-
description: 'Person working on a project.',
|
|
789
|
-
},
|
|
790
|
-
],
|
|
791
|
-
},
|
|
792
|
-
},
|
|
793
|
-
},
|
|
794
|
-
},
|
|
795
|
-
},
|
|
796
|
-
});
|
|
797
|
-
```
|
|
798
|
-
|
|
799
|
-
`seedManifests` entries are written to SQLite the first time an entity's ontology is
|
|
800
|
-
resolved *inside a transaction* (ingest, heal) and no row exists for it. A read outside a
|
|
801
|
-
transaction — including `getOntologyManifest` — resolves the seed without persisting it, so
|
|
802
|
-
a configured entity can report a manifest while still having no row in `entity_manifests`.
|
|
803
|
-
That distinction matters for `ifAbsent` below.
|
|
804
|
-
|
|
805
|
-
### Public API
|
|
806
|
-
|
|
807
|
-
Read or seed an entity's ontology at runtime:
|
|
808
|
-
|
|
809
|
-
```typescript
|
|
810
|
-
// Read effective mode + manifest (DB row, then seedManifests fallback)
|
|
811
|
-
const ontology = await wikiMemory.getOntologyManifest('team-alpha');
|
|
812
|
-
// { mode: 'emergent', manifest: { node_types: [...], edge_types: [...] } }
|
|
813
|
-
// null when no row and no seed entry
|
|
814
|
-
|
|
815
|
-
// Seed or replace manifest; optional per-entity mode override
|
|
816
|
-
await wikiMemory.setOntologyManifest('team-alpha', {
|
|
817
|
-
node_types: [{ type: 'person', description: 'An individual.' }],
|
|
818
|
-
edge_types: [{
|
|
819
|
-
type: 'reports_to',
|
|
820
|
-
source_type: 'person',
|
|
821
|
-
target_type: 'person',
|
|
822
|
-
description: 'Reporting hierarchy.',
|
|
823
|
-
}],
|
|
824
|
-
}, { mode: 'strict' });
|
|
825
|
-
```
|
|
826
|
-
|
|
827
|
-
Seed several entities atomically — all manifests are written in one transaction,
|
|
828
|
-
so a failure partway through leaves none of them behind:
|
|
829
|
-
|
|
830
|
-
```typescript
|
|
831
|
-
const { written, skipped } = await wikiMemory.setOntologyManifests(
|
|
832
|
-
[
|
|
833
|
-
{ entityId: 'tier_fact', manifest, mode: 'strict' },
|
|
834
|
-
{ entityId: 'tier_wisdom', manifest, mode: 'strict' },
|
|
835
|
-
],
|
|
836
|
-
{ ifAbsent: true },
|
|
837
|
-
);
|
|
838
|
-
// written: entities whose manifest this call wrote
|
|
839
|
-
// skipped: entities that already had a persisted manifest (only under `ifAbsent`)
|
|
840
|
-
```
|
|
841
|
-
|
|
842
|
-
`ifAbsent` makes each write create-if-absent rather than an upsert, so a
|
|
843
|
-
concurrent initializer loses the race by writing nothing instead of overwriting
|
|
844
|
-
a manifest it never read. Omit it for replace-on-conflict, which is what
|
|
845
|
-
`setOntologyManifest` does.
|
|
846
|
-
|
|
847
|
-
**`ifAbsent` tests for a persisted row, not for an effective manifest.** An entity
|
|
848
|
-
whose manifest comes from `WikiConfig.ontology.seedManifests` has no row until an
|
|
849
|
-
ingest materializes one, so `ifAbsent` writes over the configured seed and reports
|
|
850
|
-
the entity in `written`; after an ingest has run for that entity the same call
|
|
851
|
-
reports it in `skipped`. Don't mix the two seeding routes for one entity: seed it
|
|
852
|
-
through the config, or through this method, not both.
|
|
853
|
-
|
|
854
|
-
The method takes data, never a transaction handle: `WikiMemory` serializes
|
|
855
|
-
transactions on the adapter it is given, so a transaction opened on the adapter
|
|
856
|
-
you passed to `createWiki` would not participate in that serialization.
|
|
857
|
-
|
|
858
|
-
### Fact Shape Extensions
|
|
859
|
-
|
|
860
|
-
In **Strict** and **Emergent** modes, librarian and ingest JSON may include typed facts with inline edges:
|
|
861
|
-
|
|
862
|
-
```json
|
|
863
|
-
{
|
|
864
|
-
"facts": [{
|
|
865
|
-
"title": "Jane reports to Bob",
|
|
866
|
-
"body": "Jane reports to Bob Smith.",
|
|
867
|
-
"tags": [],
|
|
868
|
-
"confidence": "certain",
|
|
869
|
-
"okf_type": "person",
|
|
870
|
-
"edges": [{ "edge_type": "reports_to", "target_title": "Bob Smith" }]
|
|
871
|
-
}]
|
|
872
|
-
}
|
|
873
|
-
```
|
|
874
|
-
|
|
875
|
-
- `okf_type` maps to a `node_types[].type` entry (case-insensitive lookup; canonical manifest casing is persisted).
|
|
876
|
-
- `edges` are resolved by `target_title` within the same maintenance transaction and persisted via `EdgeRepository`.
|
|
877
|
-
- Invalid `okf_type` falls back to `null` with no edges for that fact. Invalid individual edges are dropped; valid `okf_type` and matching edges are still persisted.
|
|
878
|
-
|
|
879
|
-
See the design spec: [`docs/superpowers/specs/2026-06-23-per-entity-seeded-ontology-design.md`](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/superpowers/specs/2026-06-23-per-entity-seeded-ontology-design.md).
|
|
880
|
-
|
|
881
|
-
### Ontology type inheritance
|
|
882
|
-
|
|
883
|
-
`OntologyNodeType` accepts an optional `parent_type` field for **one-level type inheritance** — the building block for polymorphic queries like "all CreativeWorks" without deep, multi-level hierarchies.
|
|
884
|
-
|
|
885
|
-
```ts
|
|
886
|
-
// packages/core/src/types.ts
|
|
887
|
-
export interface OntologyNodeType {
|
|
888
|
-
type: string;
|
|
889
|
-
description: string;
|
|
890
|
-
/** Optional parent type slug. One level only — the parent must exist in the
|
|
891
|
-
* same manifest and must not itself declare a `parent_type`. */
|
|
892
|
-
parent_type?: string;
|
|
893
|
-
}
|
|
894
|
-
```
|
|
895
|
-
|
|
896
|
-
A concrete type declares `parent_type: '<parent-slug>'`; the parent must be a top-level node in the same manifest. A `design_spec` node with `parent_type: 'creativework'` is treated as both itself *and* a `creativework` during edge matching.
|
|
897
|
-
|
|
898
|
-
#### Inheritance rules (strictly enforced)
|
|
899
|
-
|
|
900
|
-
`validateManifest` enforces the **one-level invariant** at every persist and read (`entity_manifests.manifest_json` is `JSON.parse`d on every read, so validation runs on the read path too). Violations throw:
|
|
901
|
-
|
|
902
|
-
| Condition | Error |
|
|
903
|
-
|-----------|-------|
|
|
904
|
-
| `parent_type` references an unknown slug | `Parent type not found: <slug>` |
|
|
905
|
-
| `parent_type` equals the node's own slug | `Self-parent: <type>` |
|
|
906
|
-
| The referenced parent also declares a `parent_type` | `Parent chain too deep: <a> → <b> → <grandparent>` |
|
|
907
|
-
| `parent_type` is present but blank (`''` / whitespace) or non-string (`number`, `null`, `object`) | `Ontology parent_type must be a non-empty string when present: <type>` |
|
|
908
|
-
|
|
909
|
-
Two consequences worth knowing:
|
|
910
|
-
|
|
911
|
-
- The depth check rejects **2-cycles** (`a → b → a`) which a self-parent check alone would miss.
|
|
912
|
-
- A *present* `parent_type` key is required to be a usable string; an *absent* key (`undefined`) is fine and means "no parent". This is the same rule core applies to blank slugs elsewhere (`Ontology node type slug must be non-empty`).
|
|
913
|
-
|
|
914
|
-
Parent types are **instantiable** — a fact may be classified as bare `creativework`, and `resolveNodeType('creativework')` returns `'creativework'` normally. There is no `abstract` flag; if you want a parent to stay abstract, write the description to steer classification toward the concrete children.
|
|
915
|
-
|
|
916
|
-
#### Edge matching (symmetric, exact-first)
|
|
917
|
-
|
|
918
|
-
Edge matching is parent-aware on **both** sides and routes through a single primitive, `typeSatisfies(declaredType, concreteType, manifest)`:
|
|
919
|
-
|
|
920
|
-
- `declaredType === concreteType` (case-insensitive) → **exact match**, short-circuits before the node lookup. Manifests with no `parent_type` behave bit-for-bit as before.
|
|
921
|
-
- Otherwise: looks up `concreteType`'s definition in the manifest and returns `true` if its `parent_type` equals `declaredType`. One hop only — `typeSatisfies` never recurses.
|
|
922
|
-
|
|
923
|
-
`typeSatisfies` is used at four gates — `validateInlineEdges` (source), `OntologyService.resolveEdges` (source and target, via two-pass), `IngestionService.upsertGraph` (source) — so a parent-satisfied edge that validates will also persist.
|
|
924
|
-
|
|
925
|
-
**Targets resolve exact-first.** When resolving an edge against a target fact, `OntologyService.resolveEdges` runs two passes over candidate defs:
|
|
926
|
-
|
|
927
|
-
```ts
|
|
928
|
-
// Packages/core/src/services/OntologyService.ts (abridged)
|
|
929
|
-
const targetType = (target.okf_type ?? '').trim().toLowerCase();
|
|
930
|
-
const def = candidates.find(d => d.target_type.trim().toLowerCase() === targetType)
|
|
931
|
-
?? candidates.find(d => typeSatisfies(d.target_type, targetType, manifest));
|
|
932
|
-
```
|
|
933
|
-
|
|
934
|
-
The exact pass runs first, so a `design_spec` target with both `about creativework → creativework` and `about creativework → design_spec` declared resolves to the **narrower** row — array order never decides which pass wins. Ties within a single pass are immaterial: only `def.type` is read off the winning def, and `validateManifest` already rejects a manifest whose triples spell one edge name with different casing, so every candidate within a pass yields a byte-identical edge row.
|
|
935
|
-
|
|
936
|
-
> **Accepted cost:** declaring `→ creativework` now admits every child of `creativework`, and there is no way to say "the parent type only." A type that needs an exact-only target must not be given children.
|
|
937
|
-
|
|
938
|
-
#### Emergent prompt schema
|
|
939
|
-
|
|
940
|
-
In `emergent` mode the LLM may propose new types via `ontology_updates`. The prompt schema (`EMERGENT_EXTRA` in `packages/core/src/prompts/ontology.ts`) advertises the optional `parent_type` so emergent proposals can suggest children of an existing type:
|
|
941
|
-
|
|
942
|
-
```json
|
|
943
|
-
"ontology_updates": {
|
|
944
|
-
"node_types": [{ "type": "slug", "description": "...", "parent_type": "optional existing slug" }],
|
|
945
|
-
"edge_types": [{ "type": "slug", "source_type": "...", "target_type": "...", "description": "..." }]
|
|
946
|
-
}
|
|
947
|
-
```
|
|
948
|
-
|
|
949
|
-
The rest of the manifest reaches the LLM unchanged — `buildPromptContext` does `JSON.stringify(manifest, null, 2)`, so an established manifest's `parent_type` fields appear verbatim alongside `type` and `description`.
|
|
950
|
-
|
|
951
|
-
Emergent proposals are **untrusted input**: `mergeOntologyUpdates` drops any `parent_type` (rather than throwing) when it is a non-string, blank, unresolvable, self-referential, or whose referenced parent already declares its own parent. The lenient merge contract keeps malformed LLM proposals from aborting an ingest transaction. Changing an established type's parent is a `setOntologyManifest` operation.
|
|
952
|
-
|
|
953
|
-
#### Backwards compatibility
|
|
954
|
-
|
|
955
|
-
`parent_type` is optional and manifests persist as a whole JSON blob in `entity_manifests.manifest_json`. Existing manifests without the field validate identically; no SQLite migration is needed. `setOntologyManifest` rejects a two-level chain at the public API boundary, so callers cannot accidentally introduce an unenforced chain through a typo.
|
|
956
|
-
|
|
957
|
-
`typeSatisfies` is intentionally **not** re-exported from the package's public surface (`packages/core/src/index.ts`). No host needs the primitive to author or validate a manifest; publishing it would freeze an internal matching rule into the package's public surface.
|
|
958
|
-
|
|
959
|
-
See the design spec: [`docs/superpowers/specs/2026-08-28-ontology-parent-field-spec.md`](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/superpowers/specs/2026-08-28-ontology-parent-field-spec.md).
|
|
960
|
-
|
|
961
|
-
### Ontology backfill
|
|
962
|
-
|
|
963
|
-
Facts that enter the store without passing through the librarian (synced-down
|
|
964
|
-
remote facts via `importDump`, or facts created before the ontology feature)
|
|
965
|
-
have `okf_type = NULL` and no edges, so `traverseGraph` cannot reach them.
|
|
966
|
-
`runOntologyBackfill` types them in place:
|
|
967
|
-
|
|
968
|
-
```ts
|
|
969
|
-
const result = await wiki.runOntologyBackfill(entityId);
|
|
970
|
-
// { scanned, typed, failedValidation, edgesAdded, remaining, deferred, skipped }
|
|
971
|
-
```
|
|
972
|
-
|
|
973
|
-
- **When to call it:** you own the trigger — the library provides the operation,
|
|
974
|
-
not a scheduler (same as `runLibrarian`/`runHeal`). A good default is after
|
|
975
|
-
each sync/import completes. `WikiBusyError` under concurrency is safe to
|
|
976
|
-
swallow; the next trigger retries.
|
|
977
|
-
- **Cost model:** one or more LLM calls only when eligible untyped facts exist;
|
|
978
|
-
otherwise one SELECT. Each run scans at most 25 facts (override via
|
|
979
|
-
`options.batchSize`), oldest first. Facts are sent to the LLM in bounded
|
|
980
|
-
sub-batches — sized from `llmProvider.maxOutputTokens` when supplied,
|
|
981
|
-
otherwise a conservative default — with the full serialized prompt
|
|
982
|
-
(system + user) capped at 40k chars. A sub-batch whose response is truncated
|
|
983
|
-
or fails to parse is halved and retried automatically; a single fact that
|
|
984
|
-
still fails alone is counted in `skipped` rather than aborting the run.
|
|
985
|
-
- **Convergence:** loop `while (result.remaining > 0)` to drain a backlog.
|
|
986
|
-
`remaining` counts only *eligible* untyped facts, so the loop terminates even
|
|
987
|
-
when unclassifiable facts exist; those are cooldown-stamped and retried after
|
|
988
|
-
7 days (`deferred` reports them). Because of that weekly retry, host
|
|
989
|
-
dashboards may see small recurring bumps in `scanned`/`failedValidation` as
|
|
990
|
-
the unclassifiable backlog is re-tested — expected, not a regression.
|
|
991
|
-
- **Strictly additive:** never creates, deletes, or rewrites facts, and never
|
|
992
|
-
overwrites an existing `okf_type` (guarded in SQL, not just convention).
|
|
993
|
-
`updated_at` is never touched, so sync merge resolution is unaffected.
|
|
994
|
-
- **Prompt customization:** per-call `options.promptOverride`, or global
|
|
995
|
-
`config.prompts.ontologyBackfillSystemPrompt` (template may use `{{facts}}`
|
|
996
|
-
and the ontology placeholders).
|
|
997
|
-
|
|
998
|
-
#### Classifier mode (optional)
|
|
999
|
-
|
|
1000
|
-
A System-One classifier (for example TypeSafe's Jev, an OpenJev-style open model, or a local ONNX classifier) can type facts during backfill without generating text. Add `classify` to your provider and opt in:
|
|
1001
|
-
|
|
1002
|
-
```ts
|
|
1003
|
-
const wiki = createWiki(db, {
|
|
1004
|
-
llmProvider: { generateText, classify },
|
|
1005
|
-
config: { ontology: { backfillClassifier: 'auto', classifyMinConfidence: 0.5 } },
|
|
1006
|
-
});
|
|
1007
|
-
await wiki.runOntologyBackfill('user-1'); // uses classify
|
|
1008
|
-
await wiki.runOntologyBackfill('user-1', { classifier: 'llm' }); // force the generative path
|
|
1009
|
-
```
|
|
1010
|
-
|
|
1011
|
-
- Providing `classify` changes nothing by itself. The default is `'llm'`.
|
|
1012
|
-
- Each untyped fact gets one `choice` question over the entity manifest's node types. Answers below `classifyMinConfidence` (default 0.5) are left untyped and retried after the cooldown.
|
|
1013
|
-
- **No edges are proposed in classifier mode** (`edgesAdded: 0`): a classifier cannot extract edge targets. Run with `classifier: 'llm'` when you want edges.
|
|
1014
|
-
- Manifests with more than 255 node types, or providers without `classify`, use the generative path.
|
|
1015
|
-
- Answers are validated as untrusted. Off-list choices and out-of-range probabilities count toward `failedValidation`. A thrown `classify` counts toward `skipped` and is retried on the next pass.
|
|
1016
|
-
|
|
1017
|
-
Example adapter for Cloudflare Workers AI's `typesafe/jev`. This is illustrative, not a supported package; check the provider's current API reference before use.
|
|
1018
|
-
|
|
1019
|
-
```ts
|
|
1020
|
-
const classify: LLMProvider['classify'] = async ({ state, questions }) => {
|
|
1021
|
-
const jevQuestions = Object.fromEntries(Object.entries(questions).map(([key, q]) => [key,
|
|
1022
|
-
q.kind === 'choice' ? { type: 'choice', instructions: q.instructions, criteria: Object.fromEntries(q.options.map((o) => [o, o])) }
|
|
1023
|
-
: q.kind === 'score' ? { type: 'score', instructions: q.instructions, criteria: q.levels }
|
|
1024
|
-
: { type: 'noul', instructions: q.instructions },
|
|
1025
|
-
]));
|
|
1026
|
-
const res = await env.AI.run('typesafe/jev', { state, questions: jevQuestions });
|
|
1027
|
-
const answers = Object.fromEntries(Object.entries(res.answers).map(([key, a]: [string, any]) => [key,
|
|
1028
|
-
a.type === 'choice' ? { kind: 'choice', choice: a.choice, confidence: a.confidence, probabilities: a.probabilities }
|
|
1029
|
-
: a.type === 'score' ? {
|
|
1030
|
-
kind: 'score', score: a.score, confidence: a.confidence,
|
|
1031
|
-
probabilities: Object.keys(a.probabilities).sort((x, y) => Number(x) - Number(y)).map((k) => a.probabilities[k]),
|
|
1032
|
-
}
|
|
1033
|
-
: { kind: 'binary', probability: a.noul },
|
|
1034
|
-
]));
|
|
1035
|
-
return { answers };
|
|
1036
|
-
};
|
|
1037
|
-
```
|
|
1038
|
-
|
|
1039
|
-
## OKF Import/Export
|
|
1040
|
-
|
|
1041
|
-
The core package integrates with `@equationalapplications/core-okf` to seamlessly adapt wiki data dumps to and from Open Knowledge Format (OKF) bundles (v0.1 and v0.2; `formatOkfBundle` defaults to the v0.2 / `llm-wiki/2` profile).
|
|
1042
|
-
|
|
1043
|
-
### Exporting an OKF Bundle
|
|
1044
|
-
|
|
1045
|
-
Convert an existing wiki dump into a flat array of OKF files, ready to be written to disk or zipped:
|
|
1046
|
-
|
|
1047
|
-
```typescript
|
|
1048
|
-
import { formatOkfBundle } from '@equationalapplications/core-llm-wiki';
|
|
1049
|
-
|
|
1050
|
-
const dump = await wiki.exportDump(['entity-123']);
|
|
1051
|
-
const { files } = formatOkfBundle(dump);
|
|
1052
|
-
|
|
1053
|
-
// files: Array<{ path: string; content: string }>
|
|
1054
|
-
// e.g., [{ path: 'entities/entity-123/facts/fact_abc.md', content: '---\n...' }]
|
|
1055
|
-
```
|
|
1056
|
-
|
|
1057
|
-
### Importing an OKF Bundle
|
|
1058
|
-
|
|
1059
|
-
Parse raw OKF files back into a `MemoryDump` that the wiki can ingest:
|
|
1060
|
-
|
|
1061
|
-
```typescript
|
|
1062
|
-
import { parseOkfBundle } from '@equationalapplications/core-llm-wiki';
|
|
1063
|
-
|
|
1064
|
-
// Assuming you read OKF files for this entity (e.g. under `entities/entity-123/`) from disk/zip into OkfFile[] shape
|
|
1065
|
-
const dump = parseOkfBundle('entity-123', files, {
|
|
1066
|
-
defaultSchema: 'fact',
|
|
1067
|
-
typeMapping: {
|
|
1068
|
-
'custom_type': 'fact',
|
|
1069
|
-
'archived': 'ignore', // Skips these concepts
|
|
1070
|
-
},
|
|
1071
|
-
});
|
|
1072
|
-
|
|
1073
|
-
await wiki.importDump(dump, { merge: true });
|
|
1074
|
-
```
|
|
1075
|
-
|
|
1076
|
-
**Routing Precedence:** Concepts are routed into either the `entries` (facts) or `tasks` tables based on a three-step fallback:
|
|
1077
|
-
|
|
1078
|
-
1. `OkfImportOptions.typeMapping` explicitly mapping an OKF `type` to `'fact'`, `'task'`, or `'ignore'`.
|
|
1079
|
-
2. Directory convention (e.g., files in `/facts/` become facts, `/tasks/` become tasks).
|
|
1080
|
-
3. The `OkfImportOptions.defaultSchema` (defaults to `'fact'`).
|
|
1081
|
-
|
|
1082
|
-
### WikiEdge and the `## Related` Section
|
|
1083
|
-
|
|
1084
|
-
A `WikiEdge` represents a directed link between two concepts (`source_id`, `target_id`, `edge_type`). In profile llm-wiki/1, edges are serialized in a trailing `## Related` section on each concept file. `parseOkfBundle()` extracts them into `WikiEdge` rows and strips the section from stored bodies; `formatOkfBundle()` emits the section from the `edges` array in the dump.
|
|
1085
|
-
|
|
1086
|
-
See [`docs/okf-profile.md`](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/okf-profile.md) for the full normative spec and [`packages/okf/fixtures/`](https://github.com/equationalapplications/expo-llm-wiki/tree/main/packages/okf/fixtures) for conformance bundles.
|
|
1087
|
-
|
|
1088
|
-
`importDump` persists an imported entity summary (profile ≥ 1 bundles) and
|
|
1089
|
-
`exportDump` re-emits it, so the profile §4 round-trip holds across storage.
|
|
1090
|
-
Read it directly with `wiki.getEntitySummary(entityId)`. The value lives in the
|
|
1091
|
-
`{prefix}meta` table under `entity_summary:{entity_id}` and is removed by
|
|
1092
|
-
`forget(entityId, { clearAll: true })`. See the
|
|
1093
|
-
[design spec](../../docs/superpowers/specs/2026-07-05-okf-summary-persistence-design.md)
|
|
1094
|
-
for import/merge semantics and wipe-path cleanup.
|
|
1095
|
-
|
|
1096
|
-
### The `okf_type` Field
|
|
1097
|
-
|
|
1098
|
-
Facts and tasks include a nullable `okf_type` column. This preserves the literal OKF `type` string from an imported bundle frontmatter, independent of whether the item was routed to the `entries` or `tasks` table. When `formatOkfBundle` runs, it restores this specific string, falling back to `'fact'` or `'task'` if the field is null (ensuring non-imported rows export cleanly).
|
|
1099
|
-
|
|
1100
|
-
## Security
|
|
1101
|
-
|
|
1102
|
-
`@equationalapplications/core-llm-wiki` enforces multiple security layers:
|
|
1103
|
-
|
|
1104
|
-
### VectorRanker Adapter Security
|
|
1105
|
-
|
|
1106
|
-
If implementing a custom `VectorRanker`:
|
|
1107
|
-
|
|
1108
|
-
- **SQL Injection**: ALWAYS use parameterized queries for `entityId`, `factId`, `candidateIds`. Never concatenate into SQL strings.
|
|
1109
|
-
- **Entity Isolation**: Filter by `entityId` in all queries to prevent cross-tenant data leaks.
|
|
1110
|
-
- **Credential Scrubbing**: Strip API keys, tokens, connection strings from thrown errors before surfacing to host.
|
|
1111
|
-
- **Resource Limits**: Cap `limit` and `candidateIds.length` to prevent DoS. Do NOT retain `vector` references beyond callback scope — blocks GC.
|
|
1112
|
-
|
|
1113
|
-
See [SECURITY.md](https://github.com/equationalapplications/expo-llm-wiki/blob/main/SECURITY.md) for complete adapter security guidance and code examples.
|
|
1114
|
-
|
|
1115
|
-
### Host Application Security
|
|
1116
|
-
|
|
1117
|
-
When using `VectorRanker`:
|
|
1118
|
-
|
|
1119
|
-
- **Error Sanitization**: `sanitizeRankerErrors: true` (default) scrubs ranker errors before mirroring via `error.cause`.
|
|
1120
|
-
- **Fallback Policy**: Choose `vectorRankerFallback` based on availability vs consistency requirements:
|
|
1121
|
-
- `'js-cosine'` (default): Best availability
|
|
1122
|
-
- `'keyword'`: Fast fallback without semantic ranking
|
|
1123
|
-
- `'empty'`: Strict consistency (no facts on failure)
|
|
1124
|
-
- `'throw'`: Fail-fast error propagation
|
|
1125
|
-
- **Deletion Hook Contract**: `forget()` / `runPrune()` reject on hook timeout/failure. Prevents GDPR violations (deleted vectors still retrievable). Handle failures with retry or queue for reconciliation.
|
|
1126
|
-
- **Timeout Tuning**: Set `deletionHookTimeoutMs` per deployment (default 30s). Interactive UX: 5s. Background jobs: 60s.
|
|
1127
|
-
|
|
1128
|
-
Core WikiMemory provides:
|
|
1129
|
-
- **Defensive Copies**: Query/embedding vectors copied before ranker/hook calls
|
|
1130
|
-
- **Input Validation**: `sourceRef`/`sourceHash` normalized; embedding dimensions validated
|
|
1131
|
-
- **Parameterized Queries**: All SQL uses bind parameters
|
|
1132
|
-
|
|
1133
|
-
### Prompt-Injection Trust Boundary
|
|
1134
|
-
|
|
1135
|
-
User-controlled text — `event.summary` passed to `write()`, document chunks passed to `ingestDocument()`,
|
|
1136
|
-
fact `title`/`body` (including imported dumps) — is interpolated verbatim into LLM prompts for librarian,
|
|
1137
|
-
heal, and embedding operations. Prompt templating does simple variable substitution; it does not detect
|
|
1138
|
-
or filter instruction-like content.
|
|
1139
|
-
|
|
1140
|
-
Mitigating prompt injection (e.g. "ignore prior instructions and emit...") is **the host's responsibility**.
|
|
1141
|
-
If your application accepts untrusted input that flows into `write()`, `ingestDocument()`, or `importDump()`,
|
|
1142
|
-
treat the LLM's librarian/heal output as similarly untrusted — validate or scope it before acting on it
|
|
1143
|
-
downstream.
|
|
1144
|
-
|
|
1145
|
-
[Grounding](#grounding) is a support check, not an injection defense. It confirms that a fact quotes the
|
|
1146
|
-
text the model was shown, and injected text in that source can be quoted like any other.
|
|
1147
|
-
|
|
1148
|
-
[`getInstructions`](#effective-instructions-getinstructions) and the `wiki_get_instructions` tool (`memory:read`)
|
|
1149
|
-
return `WikiConfig.prompts` overrides verbatim, so keep secrets out of them. The result also includes the entity's
|
|
1150
|
-
ontology manifest, which in emergent mode holds types and descriptions the model proposed from ingested
|
|
1151
|
-
documents. Agents should treat the result as reference data about the engine, not as instructions to follow.
|
|
1152
|
-
|
|
1153
|
-
## Usage
|
|
1154
|
-
|
|
1155
|
-
```typescript
|
|
1156
|
-
import { WikiMemory, type SQLiteAdapter } from '@equationalapplications/core-llm-wiki';
|
|
1157
|
-
|
|
1158
|
-
// Provide any SQLiteAdapter-compatible driver
|
|
1159
|
-
const wikiMemory = new WikiMemory(db, {
|
|
1160
|
-
llmProvider: {
|
|
1161
|
-
generateText: async ({ systemPrompt, userPrompt }) => {
|
|
1162
|
-
// Your LLM call here
|
|
1163
|
-
return 'Model output';
|
|
1164
|
-
},
|
|
1165
|
-
},
|
|
1166
|
-
});
|
|
1167
|
-
|
|
1168
|
-
// Initialize schema and run migrations
|
|
1169
|
-
await wikiMemory.setup();
|
|
1170
|
-
|
|
1171
|
-
// Store facts
|
|
1172
|
-
await wikiMemory.write('user-123', {
|
|
1173
|
-
event_type: 'observation',
|
|
1174
|
-
summary: 'User prefers async/await over promises',
|
|
1175
|
-
});
|
|
1176
|
-
|
|
1177
|
-
// Query memory
|
|
1178
|
-
const memory = await wikiMemory.read('user-123', 'coding style preferences');
|
|
1179
|
-
```
|
|
1180
|
-
|
|
1181
|
-
### Multi-entity weighted reads
|
|
1182
|
-
|
|
1183
|
-
`read()` accepts either one entity id or an array of entity ids. Facts are always merged globally before `maxResults` is applied. For single-entity reads, tasks are uncapped and events are capped at 10. For multi-entity reads, tasks are capped at `min(20 × entity count, 200)` and events at `min(10 × entity count, 100)` — per-entity representation in the returned bundle is not guaranteed.
|
|
1184
|
-
|
|
1185
|
-
```ts
|
|
1186
|
-
const memory = await wikiMemory.read(['tier_wisdom', 'tier_fact', 'tier_working'], 'Which source should I trust?', {
|
|
1187
|
-
maxResults: 8,
|
|
1188
|
-
tierWeights: {
|
|
1189
|
-
tier_wisdom: 2,
|
|
1190
|
-
tier_fact: 1,
|
|
1191
|
-
tier_working: 0.25,
|
|
1192
|
-
},
|
|
1193
|
-
});
|
|
1194
|
-
|
|
1195
|
-
console.log(memory.metadata);
|
|
1196
|
-
console.log(memory.factScores);
|
|
1197
|
-
```
|
|
1198
|
-
|
|
1199
|
-
### Librarian prompt override contract
|
|
1200
|
-
|
|
1201
|
-
Core exports prompt utilities for weighted retrieval-based synthesis. Use `mapLibrarianOptionsToReadOptions()` to map `entityWeights` to `tierWeights`, then hydrate a prompt with `query`, `context`, and `tasks`.
|
|
1202
|
-
|
|
1203
|
-
```ts
|
|
1204
|
-
import {
|
|
1205
|
-
DEFAULT_LIBRARIAN_SYNTHESIS_PROMPT,
|
|
1206
|
-
formatContext,
|
|
1207
|
-
hydrateLibrarianPrompt,
|
|
1208
|
-
mapLibrarianOptionsToReadOptions,
|
|
1209
|
-
validateLibrarianPromptTemplate,
|
|
1210
|
-
} from '@equationalapplications/core-llm-wiki';
|
|
1211
|
-
|
|
1212
|
-
const options = {
|
|
1213
|
-
entityWeights: { tier_wisdom: 2, tier_fact: 1, tier_working: 0.25 },
|
|
1214
|
-
systemPrompt: `You are a strict fact checker.
|
|
1215
|
-
Question:
|
|
1216
|
-
{{query}}
|
|
1217
|
-
|
|
1218
|
-
Retrieved context:
|
|
1219
|
-
{{context}}
|
|
1220
|
-
|
|
1221
|
-
{{tasks}}`,
|
|
1222
|
-
};
|
|
1223
|
-
|
|
1224
|
-
const query = 'Which source should I trust for recent project decisions?';
|
|
1225
|
-
|
|
1226
|
-
const memory = await wikiMemory.read(['tier_wisdom', 'tier_fact', 'tier_working'], query, {
|
|
1227
|
-
...mapLibrarianOptionsToReadOptions(options),
|
|
1228
|
-
maxResults: 8,
|
|
1229
|
-
});
|
|
1230
|
-
|
|
1231
|
-
const template = options.systemPrompt ?? DEFAULT_LIBRARIAN_SYNTHESIS_PROMPT;
|
|
1232
|
-
const warnings = validateLibrarianPromptTemplate(template, {
|
|
1233
|
-
custom: options.systemPrompt != null,
|
|
1234
|
-
taskCount: memory.tasks.length,
|
|
1235
|
-
});
|
|
1236
|
-
|
|
1237
|
-
for (const warning of warnings) console.warn(warning);
|
|
1238
|
-
|
|
1239
|
-
const finalPrompt = hydrateLibrarianPrompt(template, {
|
|
1240
|
-
query,
|
|
1241
|
-
context: formatContext(memory, { includeEntityIds: true, includeFactScores: true }),
|
|
1242
|
-
tasks: formatContext({ facts: [], tasks: memory.tasks, events: [] }, { format: 'plain' }),
|
|
1243
|
-
});
|
|
1244
|
-
```
|
|
1245
|
-
|
|
1246
|
-
## Platform Random Source
|
|
1247
|
-
|
|
1248
|
-
Wiki record IDs must be cryptographically random. The core engine resolves a random source in this order:
|
|
1249
|
-
|
|
1250
|
-
1. `crypto.randomUUID()` (Web / Node 19+)
|
|
1251
|
-
2. `crypto.getRandomValues()` (Web / Node / polyfilled global)
|
|
1252
|
-
3. A source injected via `configureRandomSource()` (e.g. `expo-crypto` on Hermes/React Native)
|
|
1253
|
-
|
|
1254
|
-
Web and Node are unchanged — global `crypto` wins when present. React Native / Hermes typically has no `crypto` global; use a platform package or inject your own implementation:
|
|
1255
|
-
|
|
1256
|
-
```typescript
|
|
1257
|
-
import { configureRandomSource } from '@equationalapplications/core-llm-wiki';
|
|
1258
|
-
import { getRandomValues } from 'expo-crypto';
|
|
1259
|
-
|
|
1260
|
-
// Call once at module load, before any wiki writes
|
|
1261
|
-
configureRandomSource(getRandomValues);
|
|
1262
|
-
```
|
|
1263
|
-
|
|
1264
|
-
`@equationalapplications/expo-llm-wiki` does this automatically on import (main entry and `/factory` subpath). If you use `@equationalapplications/core-llm-wiki` directly on React Native without the expo package, you must call `configureRandomSource()` yourself or polyfill `globalThis.crypto.getRandomValues`.
|
|
1265
|
-
|
|
1266
|
-
## Entity Enumeration
|
|
1267
|
-
|
|
1268
|
-
List all entities that have stored data in the wiki:
|
|
1269
|
-
|
|
1270
|
-
```typescript
|
|
1271
|
-
const entityIds = await wikiMemory.listEntityIds();
|
|
1272
|
-
// Returns all entity_ids with at least one row (including soft-deleted-only entities)
|
|
1273
|
-
// Optional prefix filter: await wikiMemory.listEntityIds({ prefix: 'tier_' });
|
|
1274
|
-
```
|
|
1275
|
-
|
|
1276
|
-
Use this for maintenance scheduling, multi-entity operations, or discovering which namespaces exist. Includes entities with only soft-deleted rows so `runPrune()` can reclaim orphaned storage.
|
|
1277
|
-
|
|
1278
|
-
## Source Reference Enumeration
|
|
1279
|
-
|
|
1280
|
-
List all documents currently stored for an entity:
|
|
1281
|
-
|
|
1282
|
-
```typescript
|
|
1283
|
-
const sourceRefs = await wikiMemory.listSourceRefs('user-123');
|
|
1284
|
-
// One row per live sourceRef (soft-deleted rows are excluded):
|
|
1285
|
-
// Array<{ sourceRef: string; sourceHash: string | null; factCount: number; lastIngestedAt: number }>
|
|
1286
|
-
// factCount — number of live facts under that sourceRef
|
|
1287
|
-
// lastIngestedAt — Unix timestamp in ms from the most recently updated live entry
|
|
1288
|
-
```
|
|
1289
|
-
|
|
1290
|
-
Use this to audit stored documents, validate external sync state, or preview the blast radius before `forget()` operations.
|
|
1291
|
-
|
|
1292
|
-
## Direct Graph Write
|
|
1293
|
-
|
|
1294
|
-
Write structured graph data directly without LLM extraction — useful for programmatic fact ingestion, parsers, and deterministic pipelines:
|
|
1295
|
-
|
|
1296
|
-
```typescript
|
|
1297
|
-
const { nodesWritten, edgesWritten, superseded } = await wikiMemory.upsertGraph('entity-123', {
|
|
1298
|
-
sourceRef: 'codebase_main.ts',
|
|
1299
|
-
sourceHash: sha256(sourceCode),
|
|
1300
|
-
nodes: [
|
|
1301
|
-
{ id: 'fn_processData', type: 'function', title: 'processData', body: 'Processes user data' },
|
|
1302
|
-
{ id: 'class_UserService', type: 'class', title: 'UserService', body: 'User management service' },
|
|
1303
|
-
],
|
|
1304
|
-
edges: [
|
|
1305
|
-
{ type: 'calls', sourceId: 'fn_processData', targetId: 'class_UserService' },
|
|
1306
|
-
],
|
|
1307
|
-
}, adapter); // SQLiteAdapter from your platform driver — writes join the caller's transaction
|
|
1308
|
-
```
|
|
1309
|
-
|
|
1310
|
-
`upsertGraph` is "the tail of `ingestDocument` with the middle (LLM extraction) step removed" — it accepts caller-supplied nodes (`{ id, type, title, body? }`) and edges (`{ type, sourceId, targetId, id? }`) and writes them under the same `(sourceRef, sourceHash)` semantics. If a *different* live `sourceRef` already holds the same `sourceHash`, it throws `WikiSourceRefHashCollision`; re-writing the identical `(sourceRef, sourceHash)` is a no-op returning zero counts. The adapter parameter is required so writes participate in the caller's transaction.
|
|
1311
|
-
|
|
1312
|
-
Core never sees your transaction commit, so it can't index the new nodes itself. Until you tell it, they won't show up in keyword or hybrid `read()` results on the same instance. Call `syncSearchIndex` **after** the commit, not inside the transaction:
|
|
1313
|
-
|
|
1314
|
-
```typescript
|
|
1315
|
-
await db.withTransactionAsync((tx) => wikiMemory.upsertGraph('entity-123', params, tx));
|
|
1316
|
-
await wikiMemory.syncSearchIndex('entity-123'); // or syncSearchIndex() for every entity written
|
|
1317
|
-
```
|
|
1318
|
-
|
|
1319
|
-
It is cheap on an entity that is already current, runs in order with core's own index syncs, and never rejects for a valid `entityId` (an empty or non-string id throws `TypeError`). It does not compute embeddings; those still come from the maintenance sweep.
|
|
1320
|
-
|
|
1321
|
-
Entry IDs occupy one globally shared namespace within a database. An ID belongs
|
|
1322
|
-
to its existing entity while its row exists, including after soft deletion.
|
|
1323
|
-
Same-entity updates and resurrection retain their existing behavior; physically
|
|
1324
|
-
absent IDs may be inserted. Hosts remain responsible for authenticating callers
|
|
1325
|
-
and selecting the authorized entity ID. Entity-qualified deterministic IDs can
|
|
1326
|
-
avoid accidental collisions, but are not required by the API.
|
|
1327
|
-
|
|
1328
|
-
A foreign node ID rejects the entire non-no-op graph write with
|
|
1329
|
-
`WikiGraphNodeOwnershipConflict` (code `WIKI_GRAPH_NODE_OWNERSHIP_CONFLICT`).
|
|
1330
|
-
The error does not identify the owner, conflicting node, or stored details.
|
|
1331
|
-
A collision visible to node preflight performs no persistent writes, including
|
|
1332
|
-
seed-manifest writes. The live source-hash probe still runs first: an identical
|
|
1333
|
-
mapping is a zero-count no-op; a different live reference retains its hash
|
|
1334
|
-
collision error. Deleted source-reference mappings do not trigger that probe.
|
|
1335
|
-
|
|
1336
|
-
Call `upsertGraph` inside the host transaction and pass that callback's `tx`.
|
|
1337
|
-
It neither starts nor ends a transaction. Propagate later failures so the host
|
|
1338
|
-
transaction rolls back partial writes; catching a late error and committing is
|
|
1339
|
-
not atomic. Foreign edge-ID collisions retain their existing late bare `Error`
|
|
1340
|
-
and require host rollback, even though the foreign edge is never overwritten.
|
|
1341
|
-
|
|
1342
|
-
`ingestDocument` shares ownership enforcement on both its full and partial paths
|
|
1343
|
-
and may also throw `WikiGraphNodeOwnershipConflict`. It normally generates new
|
|
1344
|
-
IDs, so collisions are unexpected. Unlike direct graph preflight, ingestion may
|
|
1345
|
-
already have persisted ontology or other state when rejection occurs; its
|
|
1346
|
-
transaction rollback, not a zero-write preflight guarantee, protects atomicity.
|
|
1347
|
-
|
|
1348
|
-
## Duplicate Hash Detection
|
|
1349
|
-
|
|
1350
|
-
Control behavior when a different live `sourceRef` already holds the same `sourceHash`. The option is passed as a third argument to `ingestDocument`, not inside the params object:
|
|
1351
|
-
|
|
1352
|
-
```typescript
|
|
1353
|
-
await wikiMemory.ingestDocument(
|
|
1354
|
-
'entity-123',
|
|
1355
|
-
{
|
|
1356
|
-
sourceRef: 'doc.md',
|
|
1357
|
-
sourceHash: sha256(content),
|
|
1358
|
-
documentChunk: content,
|
|
1359
|
-
},
|
|
1360
|
-
{ onDuplicateHash: 'ingest' } // 'ingest' (default) | 'skip' | 'throw'
|
|
1361
|
-
);
|
|
1362
|
-
```
|
|
1363
|
-
|
|
1364
|
-
- `'ingest'` (default): No duplicate pre-check; extraction proceeds as before this option existed. If a different live `sourceRef` is found holding the same hash at commit time (a concurrent-writer race caught by the source-ref unique index), the call still throws `WikiDuplicateHashError`.
|
|
1365
|
-
- `'skip'`: Pre-check before any LLM call; if a different live `sourceRef` already holds the hash, return a zero-chunk result without writing.
|
|
1366
|
-
- `'throw'`: Pre-check before any LLM call; throw `WikiDuplicateHashError` (carries the canonical `sourceRef`).
|
|
1367
|
-
- The guard only considers **live** references — soft-deleted refs do not trigger it in any mode.
|
|
1368
|
-
|
|
1369
|
-
When the pre-check passes but a concurrent writer claims the hash before the write commits, the transaction rolls back after the LLM extraction has already run. You still receive the diagnostics that describe that extraction — `ingest_chunk_failed`, `fact_rejected` and `fact_deduplicated` — in every mode, and also in the rarer case where the winning writer has itself rolled back by the time core looks, so the call surfaces the raw database error instead of `WikiDuplicateHashError`. Diagnostics tied to the rolled-back write (`grounding_*`, `edge_dropped`) are dropped, because the `factId`s they carry name rows that were never committed.
|
|
1370
|
-
|
|
1371
|
-
## Batch Change Detection
|
|
1372
|
-
|
|
1373
|
-
Check multiple documents for changes in one call:
|
|
1374
|
-
|
|
1375
|
-
```typescript
|
|
1376
|
-
const batch = [
|
|
1377
|
-
{ sourceRef: 'doc1.md', sourceHash: sha256(content1) },
|
|
1378
|
-
{ sourceRef: 'doc2.md', sourceHash: sha256(content2) },
|
|
1379
|
-
{ sourceRef: 'doc3.md', sourceHash: sha256(content3) },
|
|
1380
|
-
];
|
|
1381
|
-
const changes = await wikiMemory.hasChanged('entity-123', batch);
|
|
1382
|
-
// changes: Array<{ sourceRef: string; changed: boolean; duplicateOf?: string }>
|
|
1383
|
-
// duplicateOf — when present, the canonical stored different sourceRef holding
|
|
1384
|
-
// the same hash (DB-normalized spelling; sourceRef echoes the raw caller value).
|
|
1385
|
-
// Per-document change detection; internally batched across queries
|
|
1386
|
-
```
|
|
1387
|
-
|
|
1388
|
-
`pendingSources` returns one status per input, in order, and adds the partial-ingest state:
|
|
1389
|
-
|
|
1390
|
-
```typescript
|
|
1391
|
-
const statuses = await wikiMemory.pendingSources('entity-123', batch);
|
|
1392
|
-
// Array<{ sourceRef: string; status: 'new' | 'changed' | 'partial' | 'current' }>
|
|
1393
|
-
```
|
|
1394
|
-
|
|
1395
|
-
- `current` means exactly what `hasChanged` returning `false` means.
|
|
1396
|
-
- `partial` means live facts exist for the ref, but none has a stored hash: for example a first ingest where a chunk failed, or imported rows that carry no hash. Re-ingest to retry. A failed re-ingest of a ref that already has hashed rows reports `changed`, not `partial`.
|
|
1397
|
-
|
|
1398
|
-
## Lint
|
|
1399
|
-
|
|
1400
|
-
Read-only health report for one entity. It reports problems and never repairs them.
|
|
1401
|
-
|
|
1402
|
-
```typescript
|
|
1403
|
-
const report = await wikiMemory.lint('entity-123');
|
|
1404
|
-
// {
|
|
1405
|
-
// danglingEdges, // source or target missing, soft-deleted, or another entity's
|
|
1406
|
-
// manifestViolations, // (source type, edge type, target type) not in the effective manifest
|
|
1407
|
-
// untypedFacts, // okf_type is null
|
|
1408
|
-
// drafts, // lifecycle_status = 'draft' (see Draft Review)
|
|
1409
|
-
// unverifiedInferred, // librarian_inferred facts with no okf_verified entry
|
|
1410
|
-
// sample: { danglingEdgeIds, manifestViolationEdgeIds }, // up to 20 each
|
|
1411
|
-
// }
|
|
1412
|
-
```
|
|
1413
|
-
|
|
1414
|
-
- Manifest violations are 0 when ontology is off or the manifest is empty.
|
|
1415
|
-
- An edge with an untyped endpoint counts as a violation.
|
|
1416
|
-
- Partial-ingest rows are not reported here; use `pendingSources`.
|
|
1417
|
-
|
|
1418
|
-
## Dry-Run Deletion
|
|
1419
|
-
|
|
1420
|
-
Preview deletion impact without writing:
|
|
1421
|
-
|
|
1422
|
-
```typescript
|
|
1423
|
-
const preview = await wikiMemory.forget('entity-123', { sourceRef: 'doc.md' }, { dryRun: true });
|
|
1424
|
-
// preview: { deleted: { entries: number; tasks: number } }
|
|
1425
|
-
// No database writes performed; safe for blast-radius validation
|
|
1426
|
-
```
|
|
1427
|
-
|
|
1428
|
-
## Chunking Utilities
|
|
1429
|
-
|
|
1430
|
-
`ingestDocument()` splits a document into chunks before extraction. That same chunking is exported as a pure function, so a consumer can reproduce ingest-time chunk boundaries exactly — useful for recovering the passage a fact was extracted from, by re-chunking the source and ranking chunks against the fact's stored embedding.
|
|
1431
|
-
|
|
1432
|
-
```typescript
|
|
1433
|
-
import {
|
|
1434
|
-
chunkText,
|
|
1435
|
-
safeSlice,
|
|
1436
|
-
DEFAULT_MAX_CHUNK_LENGTH, // 12000
|
|
1437
|
-
DEFAULT_CHUNK_OVERLAP, // 400
|
|
1438
|
-
} from '@equationalapplications/core-llm-wiki';
|
|
1439
|
-
|
|
1440
|
-
const { chunks, truncated } = chunkText(
|
|
1441
|
-
sourceDocument,
|
|
1442
|
-
DEFAULT_MAX_CHUNK_LENGTH,
|
|
1443
|
-
DEFAULT_CHUNK_OVERLAP
|
|
1444
|
-
);
|
|
1445
|
-
```
|
|
1446
|
-
|
|
1447
|
-
`chunkText(input, maxChunkLength, overlap)` returns `{ chunks, truncated }`. It prefers to split on a paragraph break, then a sentence terminator, then whitespace, falling back to a hard cut only when none is found within the window — `truncated` is `true` if any split needed that fallback. Consecutive chunks repeat up to `overlap` characters so context isn't lost at a boundary (less than `overlap` when the previous chunk was shorter than that). Empty/whitespace-only input returns `{ chunks: [], truncated: false }` without validating `maxChunkLength`/`overlap`; otherwise throws if `maxChunkLength` is not an integer >= 2, or if `overlap` is not a non-negative integer < `maxChunkLength`.
|
|
1448
|
-
|
|
1449
|
-
`safeSlice(value, start, end?)` slices like `String.prototype.slice` but clamps out-of-range indices, swaps a start-after-end range, and never splits a UTF-16 surrogate pair.
|
|
1450
|
-
|
|
1451
|
-
`DEFAULT_MAX_CHUNK_LENGTH` and `DEFAULT_CHUNK_OVERLAP` are the values `ingestDocument()` uses when neither the call nor `WikiConfig` overrides them. Pass the same values your host app configured (`config.maxChunkLength` / `config.chunkOverlap`) to match a non-default ingest.
|
|
1452
|
-
|
|
1453
|
-
> **Note:** `ingestDocument()` clamps the resolved overlap — whether it came from `chunkOverlap`, `WikiConfig`, or `DEFAULT_CHUNK_OVERLAP` — to `maxChunkLength - 1`. The clamp is evaluated on every call and is a no-op at the shipped defaults (`400 < 12000 - 1`), but it also bites when a custom `maxChunkLength` alone leaves the resolved overlap too large (e.g. `maxChunkLength: 100` with the default overlap `400` ingests at an effective overlap of `99`). That clamp is internal: passing the unclamped pair straight to `chunkText` throws, since it requires `overlap < maxChunkLength`. Apply the same `Math.min(overlap, maxChunkLength - 1)` yourself when re-chunking under a custom config.
|
|
1454
|
-
|
|
1455
|
-
## Schema Migrations
|
|
1456
|
-
|
|
1457
|
-
`wikiMemory.setup()` runs pending schema migrations in order and records the applied version. Migrations are additive and safe to re-run.
|
|
1458
|
-
|
|
1459
|
-
**Migration v9** (`add_live_hash_unique_index`) adds a partial `UNIQUE` index on `(entity_id, source_hash)` scoped to live (`deleted_at IS NULL`), non-null-hash rows. This closes a TOCTOU race where two concurrent `ingestDocument()` calls for different `sourceRef`s could both pass the duplicate-hash pre-check before either write committed, leaving two live rows with the same content hash.
|
|
1460
|
-
|
|
1461
|
-
If your existing database already has live rows that violate this invariant (created before the app-level race fix shipped), `setup()` throws:
|
|
1462
|
-
|
|
1463
|
-
```text
|
|
1464
|
-
Migration v9 (add_live_hash_unique_index) failed: existing live rows violate the new UNIQUE index. …
|
|
1465
|
-
```
|
|
1466
|
-
|
|
1467
|
-
The error lists the offending `(entity_id, source_hash)` groups. `setup()` performs no destructive cleanup on this path — the index is not created and the schema version is not advanced, so it's safe to retry after remediation. To remediate, either soft-delete (or re-ingest with a corrected `sourceRef`) all but one live row per listed group, then re-run `setup()`.
|
|
1468
|
-
|
|
1469
|
-
## Adapter Interface
|
|
1470
|
-
|
|
1471
|
-
Implement `SQLiteAdapter` to use your platform's SQLite driver:
|
|
1472
|
-
|
|
1473
|
-
```typescript
|
|
1474
|
-
export interface SQLiteAdapter {
|
|
1475
|
-
execAsync(sql: string): Promise<void>;
|
|
1476
|
-
runAsync(sql: string, params?: unknown[]): Promise<{ changes: number; lastInsertRowId: number }>;
|
|
1477
|
-
getAllAsync<T>(sql: string, params?: unknown[]): Promise<T[]>;
|
|
1478
|
-
getFirstAsync<T>(sql: string, params?: unknown[]): Promise<T | null>;
|
|
1479
|
-
withTransactionAsync<T>(fn: (tx: SQLiteAdapter) => Promise<T>): Promise<T>;
|
|
1480
|
-
closeAsync(): Promise<void>;
|
|
1481
|
-
}
|
|
1482
|
-
```
|
|
1483
|
-
|
|
1484
|
-
`runAsync().changes` must accurately count written rows: a conflict-suppressed
|
|
1485
|
-
write reports 0, while a permitted value-identical update still counts its row.
|
|
1486
|
-
Host adapters are responsible for this contract; see the full affected-row
|
|
1487
|
-
requirements on [`SQLiteAdapter.runAsync`](src/types.ts). Ownership checks,
|
|
1488
|
-
edge counts, metadata compare-and-set, and maintenance depend on accurate counts.
|
|
1489
|
-
|
|
1490
|
-
`@equationalapplications/expo-llm-wiki` provides a pre-built adapter for Expo/React Native. For web and Node.js, implement the interface yourself — examples below.
|
|
1491
|
-
|
|
1492
|
-
**Browser ([sql.js](https://github.com/sql-js/sql.js)):**
|
|
1493
|
-
|
|
1494
|
-
```typescript
|
|
1495
|
-
import initSqlJs from 'sql.js';
|
|
1496
|
-
import type { SQLiteAdapter } from '@equationalapplications/core-llm-wiki';
|
|
1497
|
-
|
|
1498
|
-
const SQL = await initSqlJs({ locateFile: (f) => `/wasm/${f}` });
|
|
1499
|
-
const sqlDb = new SQL.Database();
|
|
1500
|
-
|
|
1501
|
-
const adapter: SQLiteAdapter = {
|
|
1502
|
-
async execAsync(sql) { sqlDb.run(sql); },
|
|
1503
|
-
async runAsync(sql, params = []) {
|
|
1504
|
-
sqlDb.run(sql, params as any[]);
|
|
1505
|
-
// sql.js doesn't expose lastInsertRowId; hardcode 0 since WikiMemory uses internal ID generation
|
|
1506
|
-
return { changes: sqlDb.getRowsModified(), lastInsertRowId: 0 };
|
|
1507
|
-
},
|
|
1508
|
-
async getAllAsync<T>(sql, params = []) {
|
|
1509
|
-
const stmt = sqlDb.prepare(sql);
|
|
1510
|
-
stmt.bind(params as any[]);
|
|
1511
|
-
const rows: T[] = [];
|
|
1512
|
-
while (stmt.step()) rows.push(stmt.getAsObject() as T);
|
|
1513
|
-
stmt.free();
|
|
1514
|
-
return rows;
|
|
1515
|
-
},
|
|
1516
|
-
async getFirstAsync<T>(sql, params = []) {
|
|
1517
|
-
const stmt = sqlDb.prepare(sql);
|
|
1518
|
-
stmt.bind(params as any[]);
|
|
1519
|
-
const row = stmt.step() ? stmt.getAsObject() as T : null;
|
|
1520
|
-
stmt.free();
|
|
1521
|
-
return row;
|
|
1522
|
-
},
|
|
1523
|
-
async withTransactionAsync(fn) {
|
|
1524
|
-
sqlDb.run('BEGIN');
|
|
1525
|
-
try { const r = await fn(this); sqlDb.run('COMMIT'); return r; }
|
|
1526
|
-
catch (e) { sqlDb.run('ROLLBACK'); throw e; }
|
|
1527
|
-
},
|
|
1528
|
-
async closeAsync() { sqlDb.close(); },
|
|
1529
|
-
};
|
|
1530
|
-
```
|
|
1531
|
-
|
|
1532
|
-
**Node.js ([better-sqlite3](https://github.com/WiseLibs/better-sqlite3)):**
|
|
1533
|
-
|
|
1534
|
-
```typescript
|
|
1535
|
-
import Database from 'better-sqlite3';
|
|
1536
|
-
import type { SQLiteAdapter } from '@equationalapplications/core-llm-wiki';
|
|
1537
|
-
|
|
1538
|
-
const db = new Database('wiki.db');
|
|
1539
|
-
|
|
1540
|
-
const adapter: SQLiteAdapter = {
|
|
1541
|
-
async execAsync(sql) { db.exec(sql); },
|
|
1542
|
-
async runAsync(sql, params = []) {
|
|
1543
|
-
const info = db.prepare(sql).run(...(params as any[]));
|
|
1544
|
-
return { changes: info.changes, lastInsertRowId: Number(info.lastInsertRowid) };
|
|
1545
|
-
},
|
|
1546
|
-
async getAllAsync<T>(sql, params = []) {
|
|
1547
|
-
return db.prepare(sql).all(...(params as any[])) as T[];
|
|
1548
|
-
},
|
|
1549
|
-
async getFirstAsync<T>(sql, params = []) {
|
|
1550
|
-
return (db.prepare(sql).get(...(params as any[])) ?? null) as T | null;
|
|
1551
|
-
},
|
|
1552
|
-
async withTransactionAsync(fn) {
|
|
1553
|
-
db.exec('BEGIN');
|
|
1554
|
-
try { const r = await fn(this); db.exec('COMMIT'); return r; }
|
|
1555
|
-
catch (e) { db.exec('ROLLBACK'); throw e; }
|
|
1556
|
-
},
|
|
1557
|
-
async closeAsync() { db.close(); },
|
|
1558
|
-
};
|
|
1559
|
-
```
|
|
1560
|
-
|
|
1561
|
-
## Concurrency
|
|
1562
|
-
|
|
1563
|
-
All write APIs are safe to call from concurrent async contexts. Transactions are
|
|
1564
|
-
serialized internally on the single database connection — you never need to know that
|
|
1565
|
-
SQLite forbids nested `BEGIN`, and you never need to throttle callers yourself.
|
|
1566
|
-
|
|
1567
|
-
- One connection per database file per process is the supported topology.
|
|
1568
|
-
- Non-transactional reads are **not** serialized, so read latency is unaffected.
|
|
1569
|
-
- Inside a transaction callback, use only the provided `tx` handle — never the outer
|
|
1570
|
-
database handle. Using the outer handle deadlocks; opening a nested transaction throws.
|
|
1571
|
-
|
|
1572
|
-
### `WikiTransactionError`
|
|
1573
|
-
|
|
1574
|
-
Thrown when a SQLite driver error escapes a transaction (nested `BEGIN`,
|
|
1575
|
-
`SQLITE_BUSY`, constraint violation). Stable `instanceof` target for observability:
|
|
1576
|
-
|
|
1577
|
-
- `error.cause` — the original driver error (the chain bottoms out here, not at a
|
|
1578
|
-
rollback red herring).
|
|
1579
|
-
- `error.sqliteErrorCode` — best-effort SQLite code (e.g. `'SQLITE_BUSY'`), present
|
|
1580
|
-
when it can be parsed. Group on this in Sentry/Datadog instead of inspecting
|
|
1581
|
-
`error.cause` recursively.
|
|
1582
|
-
|
|
1583
|
-
Domain errors thrown from your own callback logic (validation, `WikiBusyError`)
|
|
1584
|
-
pass through unwrapped with their original type intact.
|
|
1585
|
-
|
|
1586
|
-
## How It Works
|
|
1587
|
-
|
|
1588
|
-
```mermaid
|
|
1589
|
-
flowchart TD
|
|
1590
|
-
A["read(entityId | entityId[], query, options?)"] --> B{hybridWeight = 0?}
|
|
1591
|
-
B -->|Yes| C["MiniSearch only<br/>(skip embed)"]
|
|
1592
|
-
B -->|No| D{embed available?}
|
|
1593
|
-
D -->|No| C
|
|
1594
|
-
D -->|Yes| F["Embed query"]
|
|
1595
|
-
F -->|throws| E["onRetrievalFallback<br/>callback"]
|
|
1596
|
-
E --> C
|
|
1597
|
-
F -->|succeeds| G{preFilterLimit<br/>active?}
|
|
1598
|
-
G -->|Yes| H["MiniSearch pre-filter<br/>top K candidates"]
|
|
1599
|
-
H --> I["Phase 1: Cosine score<br/>top K candidates"]
|
|
1600
|
-
G -->|No| J["Phase 1: Cosine score<br/>all facts"]
|
|
1601
|
-
J --> K["Cache vectors<br/>in-memory<br/>(full scan only)"]
|
|
1602
|
-
K --> L{hybridWeight = 1?}
|
|
1603
|
-
I --> L
|
|
1604
|
-
L -->|Yes| M["Pure semantic<br/>ranking"]
|
|
1605
|
-
L -->|No| N["Hybrid blend:<br/>semantic + keyword<br/>via MiniSearch"]
|
|
1606
|
-
M --> O["Phase 2: Fetch full rows<br/>top maxResults"]
|
|
1607
|
-
N --> O
|
|
1608
|
-
C --> P["MiniSearch ranking"]
|
|
1609
|
-
P --> O
|
|
1610
|
-
O --> R["Track access"]
|
|
1611
|
-
R --> Q["Return MemoryBundle"]
|
|
1612
|
-
```
|
|
1613
|
-
|
|
1614
|
-
The flowchart shows:
|
|
1615
|
-
1. **Fast-path** when `hybridWeight = 0` (pure keyword, no embed cost)
|
|
1616
|
-
2. **Fallback chain** when embed unavailable (MiniSearch silently) or throws (`onRetrievalFallback` callback, then MiniSearch)
|
|
1617
|
-
3. **Pre-filtering** to limit cosine scoring to top-K keyword matches (O(N) → O(K))
|
|
1618
|
-
4. **Two-phase SELECT**: phase 1 scores all/filtered facts with minimal columns, phase 2 fetches full rows for winners
|
|
1619
|
-
5. **Hybrid scoring** to blend semantic and keyword rankings
|
|
1620
|
-
6. **Vector caching** on full scans only; reads with `preFilterLimit` active skip cache population
|
|
49
|
+
The [Node.js quickstart](https://equationalapplications.github.io/expo-llm-wiki/start/quickstart-node/) shows a complete `adapter` and `llmProvider`.
|
|
1621
50
|
|
|
1622
51
|
## Monorepo Ecosystem
|
|
1623
52
|
|
|
@@ -1633,30 +62,6 @@ The flowchart shows:
|
|
|
1633
62
|
| [@equationalapplications/schema-software-org](https://github.com/equationalapplications/expo-llm-wiki/blob/main/packages/schema-software-org/README.md) | Software-organization executive ontology manifest — 17 node types, 40 edges, warm-agent superset, data-only |
|
|
1634
63
|
| [@equationalapplications/mcp-server](https://github.com/equationalapplications/expo-llm-wiki/blob/main/packages/mcp-server/README.md) | MCP server exposing SynapseTree memory tools to coding agents |
|
|
1635
64
|
|
|
1636
|
-
## OKF v0.2 conformance (llm-wiki/2)
|
|
1637
|
-
|
|
1638
|
-
`formatOkfBundle` defaults to `okf_version: 0.2` + `profile: llm-wiki/2`. To force a v0.1 export, pass `{ profile: 'llm-wiki/1' }`:
|
|
1639
|
-
|
|
1640
|
-
```ts
|
|
1641
|
-
formatOkfBundle(dump, { profile: 'llm-wiki/1' });
|
|
1642
|
-
```
|
|
1643
|
-
|
|
1644
|
-
`parseOkfBundle` auto-detects the profile from the root index frontmatter:
|
|
1645
|
-
|
|
1646
|
-
- `profile === 'llm-wiki/2'` → v0.2 path (new fields populated).
|
|
1647
|
-
- `profile === 'llm-wiki/1'` → v0.1 path (unchanged behavior).
|
|
1648
|
-
- `profile === undefined && okf_version === '0.1'` → v0.1 legacy path.
|
|
1649
|
-
- `profile === undefined && okf_version === undefined` → profile-0 (legacy) path.
|
|
1650
|
-
|
|
1651
|
-
New public methods on `WikiMemory` for OKF v0.2 trust/provenance writes (each DAO method omits `updated_at` from its SQL):
|
|
1652
|
-
|
|
1653
|
-
- `writeOkfTrust(entryId, entityId, verified)`
|
|
1654
|
-
- `writeOkfSources(entryId, entityId, sources)`
|
|
1655
|
-
- `setLifecycleStatus(entryId, entityId, status)`
|
|
1656
|
-
- `setStaleAfter(entryId, entityId, date | null)`
|
|
1657
|
-
- `setGeneratedBy(entryId, entityId, actor)`
|
|
1658
|
-
- (and `…Task` variants of all five)
|
|
1659
|
-
|
|
1660
65
|
## License
|
|
1661
66
|
|
|
1662
67
|
MIT
|