@equationalapplications/core-llm-wiki 5.5.0 → 6.0.0

package/README.md

@@ -10,34 +10,6 @@ Platform-agnostic TypeScript engine for hybrid LLM memory. Features episodic fac
 
 > Inspired by [Andrej Karpathy's LLM Wiki memory spec](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
 
-## Recent changes
-
-### 5.4.1 — Ingest parse resilience (issue #92)
-
-- `parseJsonResponse` now tolerates bare `"` characters in LLM output via a
-  container-aware repair pass. The function's signature is unchanged
-  (`parseJsonResponse<T>(text: string): T`); the runtime contract widens —
-  the new `WikiParseError` type carries `{ tier, position, slice }` instead
-  of an opaque `Error`, and `tier: 'repair'` is now reachable for balanced
-  invalid payloads (e.g. `{"facts":}`) where the walker found a candidate
-  span that `JSON.parse` ultimately rejected.
-- `IngestionService.ingestDocument` no longer rejects the whole call when one
-  chunk fails. Sibling chunks commit; the result's `parseFailures[]` records
-  per-chunk failures. A new typed error `WikiIngestEmptyError` is thrown when
-  every chunk failed. New typed error `WikiParseError` carries `{tier,
-  position, slice}`. New result fields: `ingestedChunks`, `failedChunks`,
-  `parseFailures?`.
-- Partial-commit semantics: when some chunks fail, the document's
-  `(entity, sourceHash) → sourceRef` ownership is **not** recorded and the
-  partial rows are stored with `source_hash = NULL`. Subsequent runs with
-  the same hash see `hasChanged` return `true` and the failed chunks retry
-  on the next pass; the retry's `appendPartialFacts` dedupes against the
-  prior partial's surviving rows so titles are never duplicated. On full
-  success the next run supersedes everything atomically.
-- `INGEST_SYSTEM_PROMPT` and `ONTOLOGY_BACKFILL_SYSTEM_PROMPT` were tightened
-  to call out JSON-escape discipline explicitly.
-
-**Hosts must update (host-facing migration)**: a host that today treats `ingestDocument` throwing as the only failure signal will, after 5.4.1, see a successful return with `parseFailures[]` set when a subset of chunks failed. Inspect `result.failedChunks` and surface `result.parseFailures[]` for observability — do not rely on the throw for partial failures. Only `WikiIngestEmptyError` (every chunk failed) still throws. Catch `WikiParseError` explicitly if you previously caught `Error` to log parse failures — the new `tier` field replaces the opaque `message`.
 
 ## Features
 
@@ -49,9 +21,48 @@ Platform-agnostic TypeScript engine for hybrid LLM memory. Features episodic fac
 - **Immutable vs mutable facts** — Use `WikiFact.source_type` to distinguish document-sourced facts (`immutable_document`) from derived or user-provided facts (`librarian_inferred`, `user_stated`, `user_confirmed`). Immutable document facts are not rewritten by `runLibrarian()` or `runHeal()` and can only be removed by `forget()` or re-ingesting.
 - **Full-featured memory** — Facts, tasks, events, maintenance jobs (librarian, heal, reembed, prune)
 - **Type-safe** — Built with TypeScript, full type exports
-- **Interoperability:** Supports [Open Knowledge Format (OKF) v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf) import and export via the [llm-wiki OKF profile (llm-wiki/1)](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/okf-profile.md).
+- **Interoperability:** Supports [Open Knowledge Format (OKF)](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf) v0.1 + v0.2 import and export via the [llm-wiki OKF profiles](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/okf-profile.md) (default `llm-wiki/2`, back-compat `llm-wiki/1`).
 - **Per-entity seeded ontology** — Optional Strict, Emergent, or Off modes govern LLM graph extraction; seed taxonomies per entity and persist typed facts with inline edges.
 
+## GraphRAG & Multi-Modal Retrieval
+
+`@equationalapplications/core-llm-wiki` exposes three complementary retrieval modes, each addressing a different shape of query:
+
+| Mode | API | Best for |
+|---|---|---|
+| **Semantic** (vector cosine) | `wiki.read(entityId, query)` with `embed` configured | Open-ended natural-language questions; "what do I know about X" |
+| **Keyword** (MiniSearch) | `wiki.read(entityId, query)` with `embed` absent or offline | Exact terms, identifiers, names; offline fallback |
+| **GraphRAG** (recursive CTE) | `wiki.traverseGraph(entityId, options)` + `formatGraphContext(result)` | Structural questions; "what connects to X", "everything two hops from this fact", "summarise the people, places, and projects linked to Alice" |
+
+The GraphRAG path is structurally distinct: it doesn't rank by relevance to a query string, it walks `llm_wiki_edges` from a known anchor fact. The result is dense and connected — subgraphs, not loose top-K hits.
+
+### Graph traversal APIs
+
+```typescript
+import { WikiMemory, formatGraphContext } from '@equationalapplications/core-llm-wiki';
+
+const graph = await wikiMemory.traverseGraph('user-123', {
+  sourceId: '<anchor-fact-id>',
+  maxDepth: 2,
+  direction: 'both',          // 'inbound' | 'outbound' | 'both'
+  edgeTypes: ['reports_to'],   // optional filter
+  excludeSourceTypes: ['immutable_document'],
+  minTraversalConfidence: 'inferred',
+  maxTraversalNodes: 20,
+});
+
+const promptContext = formatGraphContext(graph);
+// → dense text block ready for prompt injection
+```
+
+`traverseGraph` runs as a single recursive CTE in SQLite (see the root README's ["The SQL: how traversal works in one query"](../../README.md#the-sql-how-traversal-works-in-one-query) for the query shape). No external graph database.
+
+### Deterministic graph seeding (no LLM)
+
+For programmatic pipelines — importing pre-classified data, building a GraphRAG corpus from a CSV, or seed-loading from a JSON file — use `upsertGraph()`. It writes nodes and edges directly under the same `(sourceRef, sourceHash)` ownership semantics as `ingestDocument()`, but skips the LLM extraction step. See [Direct Graph Write](#direct-graph-write) for the canonical signature, including the required `SQLiteAdapter` argument and transactional semantics.
+
+This is the GraphRAG seed path: load a corpus, walk it.
+
 ## Installation
 
 ```bash
@@ -610,7 +621,7 @@ const result = await wiki.runOntologyBackfill(entityId);
 
 ## OKF Import/Export
 
-The core package integrates with `@equationalapplications/core-okf` to seamlessly adapt wiki data dumps to and from Open Knowledge Format (OKF) v0.1 bundles.
+The core package integrates with `@equationalapplications/core-okf` to seamlessly adapt wiki data dumps to and from Open Knowledge Format (OKF) bundles (v0.1 and v0.2; `formatOkfBundle` defaults to the v0.2 / `llm-wiki/2` profile).
 
 ### Exporting an OKF Bundle
 
@@ -827,6 +838,100 @@ configureRandomSource(getRandomValues);
 
 `@equationalapplications/expo-llm-wiki` does this automatically on import (main entry and `/factory` subpath). If you use `@equationalapplications/core-llm-wiki` directly on React Native without the expo package, you must call `configureRandomSource()` yourself or polyfill `globalThis.crypto.getRandomValues`.
 
+## Entity Enumeration
+
+List all entities that have stored data in the wiki:
+
+```typescript
+const entityIds = await wikiMemory.listEntityIds();
+// Returns all entity_ids with at least one row (including soft-deleted-only entities)
+// Optional prefix filter: await wikiMemory.listEntityIds({ prefix: 'tier_' });
+```
+
+Use this for maintenance scheduling, multi-entity operations, or discovering which namespaces exist. Includes entities with only soft-deleted rows so `runPrune()` can reclaim orphaned storage.
+
+## Source Reference Enumeration
+
+List all documents currently stored for an entity:
+
+```typescript
+const sourceRefs = await wikiMemory.listSourceRefs('user-123');
+// One row per live sourceRef (soft-deleted rows are excluded):
+// Array<{ sourceRef: string; sourceHash: string | null; factCount: number; lastIngestedAt: number }>
+// factCount — number of live facts under that sourceRef
+// lastIngestedAt — Unix timestamp in ms from the most recently updated live entry
+```
+
+Use this to audit stored documents, validate external sync state, or preview the blast radius before `forget()` operations.
+
+## Direct Graph Write
+
+Write structured graph data directly without LLM extraction — useful for programmatic fact ingestion, parsers, and deterministic pipelines:
+
+```typescript
+const { nodesWritten, edgesWritten, superseded } = await wikiMemory.upsertGraph('entity-123', {
+  sourceRef: 'codebase_main.ts',
+  sourceHash: sha256(sourceCode),
+  nodes: [
+    { id: 'fn_processData', type: 'function', title: 'processData', body: 'Processes user data' },
+    { id: 'class_UserService', type: 'class', title: 'UserService', body: 'User management service' },
+  ],
+  edges: [
+    { type: 'calls', sourceId: 'fn_processData', targetId: 'class_UserService' },
+  ],
+}, adapter); // SQLiteAdapter from your platform driver — writes join the caller's transaction
+```
+
+`upsertGraph` is "the tail of `ingestDocument` with the middle (LLM extraction) step removed" — it accepts caller-supplied nodes (`{ id, type, title, body? }`) and edges (`{ type, sourceId, targetId, id? }`) and writes them under the same `(sourceRef, sourceHash)` semantics. If a *different* live `sourceRef` already holds the same `sourceHash`, it throws `WikiSourceRefHashCollision`; re-writing the identical `(sourceRef, sourceHash)` is a no-op returning zero counts. The adapter parameter is required so writes participate in the caller's transaction.
+
+## Duplicate Hash Detection
+
+Control behavior when a different live `sourceRef` already holds the same `sourceHash`. The option is passed as a third argument to `ingestDocument`, not inside the params object:
+
+```typescript
+await wikiMemory.ingestDocument(
+  'entity-123',
+  {
+    sourceRef: 'doc.md',
+    sourceHash: sha256(content),
+    documentChunk: content,
+  },
+  { onDuplicateHash: 'ingest' }  // 'ingest' (default) | 'skip' | 'throw'
+);
+```
+
+- `'ingest'` (default): No duplicate pre-check; extraction proceeds as before this option existed. If a different live `sourceRef` is found holding the same hash at commit time (a concurrent-writer race caught by the source-ref unique index), the call still throws `WikiDuplicateHashError`.
+- `'skip'`: Pre-check before any LLM call; if a different live `sourceRef` already holds the hash, return a zero-chunk result without writing.
+- `'throw'`: Pre-check before any LLM call; throw `WikiDuplicateHashError` (carries the canonical `sourceRef`).
+- The guard only considers **live** references — soft-deleted refs do not trigger it in any mode.
+
+## Batch Change Detection
+
+Check multiple documents for changes in one call:
+
+```typescript
+const batch = [
+  { sourceRef: 'doc1.md', sourceHash: sha256(content1) },
+  { sourceRef: 'doc2.md', sourceHash: sha256(content2) },
+  { sourceRef: 'doc3.md', sourceHash: sha256(content3) },
+];
+const changes = await wikiMemory.hasChanged('entity-123', batch);
+// changes: Array<{ sourceRef: string; changed: boolean; duplicateOf?: string }>
+// duplicateOf — when present, the canonical stored different sourceRef holding
+// the same hash (DB-normalized spelling; sourceRef echoes the raw caller value).
+// Per-document change detection; internally batched across queries
+```
+
+## Dry-Run Deletion
+
+Preview deletion impact without writing:
+
+```typescript
+const preview = await wikiMemory.forget('entity-123', { sourceRef: 'doc.md' }, { dryRun: true });
+// preview: { deleted: { entries: number; tasks: number } }
+// No database writes performed; safe for blast-radius validation
+```
+
 ## Chunking Utilities
 
 `ingestDocument()` splits a document into chunks before extraction. That same chunking is exported as a pure function, so a consumer can reproduce ingest-time chunk boundaries exactly — useful for recovering the passage a fact was extracted from, by re-chunking the source and ranking chunks against the fact's stored embedding.
@@ -1024,7 +1129,7 @@ The flowchart shows:
 | [@equationalapplications/react-llm-wiki](https://github.com/equationalapplications/expo-llm-wiki/blob/main/packages/react/README.md) | Persistent episodic memory for Web |
 | [@equationalapplications/prisma-outbox](https://github.com/equationalapplications/expo-llm-wiki/blob/main/packages/prisma-outbox/README.md) | Sync SQLite outbox events to Prisma |
 | [@equationalapplications/core-llm-tools](https://github.com/equationalapplications/expo-llm-wiki/blob/main/packages/core-llm-tools/README.md) | Gemini tool schemas and capability injector |
-| [@equationalapplications/core-okf](https://github.com/equationalapplications/expo-llm-wiki/blob/main/packages/okf/README.md) | Zero-dependency Open Knowledge Format (OKF) v0.1 primitives — parse and produce interoperable knowledge bundles. |
+| [@equationalapplications/core-okf](https://github.com/equationalapplications/expo-llm-wiki/blob/main/packages/okf/README.md) | Zero-dependency Open Knowledge Format (OKF) v0.1 + v0.2 primitives — parse and produce interoperable knowledge bundles. |
 | [@equationalapplications/schema-org-llm-wiki](https://github.com/equationalapplications/expo-llm-wiki/blob/main/packages/schema-org/README.md) | Curated schema.org warm-agent ontology manifest |
 
 ## OKF v0.2 conformance (llm-wiki/2)