@equationalapplications/core-llm-wiki 5.4.0 → 5.5.1

package/README.md

@@ -10,6 +10,35 @@ Platform-agnostic TypeScript engine for hybrid LLM memory. Features episodic fac
 
 > Inspired by [Andrej Karpathy's LLM Wiki memory spec](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
 
+## Recent changes
+
+### 5.4.1 — Ingest parse resilience (issue #92)
+
+- `parseJsonResponse` now tolerates bare `"` characters in LLM output via a
+  container-aware repair pass. The function's signature is unchanged
+  (`parseJsonResponse<T>(text: string): T`); the runtime contract widens —
+  the new `WikiParseError` type carries `{ tier, position, slice }` instead
+  of an opaque `Error`, and `tier: 'repair'` is now reachable for balanced
+  invalid payloads (e.g. `{"facts":}`) where the walker found a candidate
+  span that `JSON.parse` ultimately rejected.
+- `IngestionService.ingestDocument` no longer rejects the whole call when one
+  chunk fails. Sibling chunks commit; the result's `parseFailures[]` records
+  per-chunk failures. A new typed error `WikiIngestEmptyError` is thrown when
+  every chunk failed. New typed error `WikiParseError` carries `{tier,
+  position, slice}`. New result fields: `ingestedChunks`, `failedChunks`,
+  `parseFailures?`.
+- Partial-commit semantics: when some chunks fail, the document's
+  `(entity, sourceHash) → sourceRef` ownership is **not** recorded and the
+  partial rows are stored with `source_hash = NULL`. Subsequent runs with
+  the same hash see `hasChanged` return `true` and the failed chunks retry
+  on the next pass; the retry's `appendPartialFacts` dedupes against the
+  prior partial's surviving rows so titles are never duplicated. On full
+  success the next run supersedes everything atomically.
+- `INGEST_SYSTEM_PROMPT` and `ONTOLOGY_BACKFILL_SYSTEM_PROMPT` were tightened
+  to call out JSON-escape discipline explicitly.
+
+**Hosts must update (host-facing migration)**: a host that today treats `ingestDocument` throwing as the only failure signal will, after 5.4.1, see a successful return with `parseFailures[]` set when a subset of chunks failed. Inspect `result.failedChunks` and surface `result.parseFailures[]` for observability — do not rely on the throw for partial failures. Only `WikiIngestEmptyError` (every chunk failed) still throws. Catch `WikiParseError` explicitly if you previously caught `Error` to log parse failures — the new `tier` field replaces the opaque `message`.
+
 ## Features
 
 - **Platform-agnostic** — Zero runtime dependencies; works with any SQLite driver via the `SQLiteAdapter` interface
@@ -23,6 +52,45 @@ Platform-agnostic TypeScript engine for hybrid LLM memory. Features episodic fac
 - **Interoperability:** Supports [Open Knowledge Format (OKF) v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf) import and export via the [llm-wiki OKF profile (llm-wiki/1)](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/okf-profile.md).
 - **Per-entity seeded ontology** — Optional Strict, Emergent, or Off modes govern LLM graph extraction; seed taxonomies per entity and persist typed facts with inline edges.
 
+## GraphRAG & Multi-Modal Retrieval
+
+`@equationalapplications/core-llm-wiki` exposes three complementary retrieval modes, each addressing a different shape of query:
+
+| Mode | API | Best for |
+|---|---|---|
+| **Semantic** (vector cosine) | `wiki.read(entityId, query)` with `embed` configured | Open-ended natural-language questions; "what do I know about X" |
+| **Keyword** (MiniSearch) | `wiki.read(entityId, query)` with `embed` absent or offline | Exact terms, identifiers, names; offline fallback |
+| **GraphRAG** (recursive CTE) | `wiki.traverseGraph(entityId, options)` + `formatGraphContext(result)` | Structural questions; "what connects to X", "everything two hops from this fact", "summarise the people, places, and projects linked to Alice" |
+
+The GraphRAG path is structurally distinct: it doesn't rank by relevance to a query string, it walks `llm_wiki_edges` from a known anchor fact. The result is dense and connected — subgraphs, not loose top-K hits.
+
+### Graph traversal APIs
+
+```typescript
+import { WikiMemory, formatGraphContext } from '@equationalapplications/core-llm-wiki';
+
+const graph = await wikiMemory.traverseGraph('user-123', {
+  sourceId: '<anchor-fact-id>',
+  maxDepth: 2,
+  direction: 'both',          // 'inbound' | 'outbound' | 'both'
+  edgeTypes: ['reports_to'],   // optional filter
+  excludeSourceTypes: ['immutable_document'],
+  minTraversalConfidence: 'inferred',
+  maxTraversalNodes: 20,
+});
+
+const promptContext = formatGraphContext(graph);
+// → dense text block ready for prompt injection
+```
+
+`traverseGraph` runs as a single recursive CTE in SQLite (see the root README's ["The SQL: how traversal works in one query"](../../README.md#the-sql-how-traversal-works-in-one-query) for the query shape). No external graph database.
+
+### Deterministic graph seeding (no LLM)
+
+For programmatic pipelines — importing pre-classified data, building a GraphRAG corpus from a CSV, or seed-loading from a JSON file — use `upsertGraph()`. It writes nodes and edges directly under the same `(sourceRef, sourceHash)` ownership semantics as `ingestDocument()`, but skips the LLM extraction step. See [Direct Graph Write](#direct-graph-write) for the canonical signature, including the required `SQLiteAdapter` argument and transactional semantics.
+
+This is the GraphRAG seed path: load a corpus, walk it.
+
 ## Installation
 
 ```bash
@@ -798,6 +866,100 @@ configureRandomSource(getRandomValues);
 
 `@equationalapplications/expo-llm-wiki` does this automatically on import (main entry and `/factory` subpath). If you use `@equationalapplications/core-llm-wiki` directly on React Native without the expo package, you must call `configureRandomSource()` yourself or polyfill `globalThis.crypto.getRandomValues`.
 
+## Entity Enumeration
+
+List all entities that have stored data in the wiki:
+
+```typescript
+const entityIds = await wikiMemory.listEntityIds();
+// Returns all entity_ids with at least one row (including soft-deleted-only entities)
+// Optional prefix filter: await wikiMemory.listEntityIds({ prefix: 'tier_' });
+```
+
+Use this for maintenance scheduling, multi-entity operations, or discovering which namespaces exist. Includes entities with only soft-deleted rows so `runPrune()` can reclaim orphaned storage.
+
+## Source Reference Enumeration
+
+List all documents currently stored for an entity:
+
+```typescript
+const sourceRefs = await wikiMemory.listSourceRefs('user-123');
+// One row per live sourceRef (soft-deleted rows are excluded):
+// Array<{ sourceRef: string; sourceHash: string | null; factCount: number; lastIngestedAt: number }>
+// factCount — number of live facts under that sourceRef
+// lastIngestedAt — Unix timestamp in ms from the most recently updated live entry
+```
+
+Use this to audit stored documents, validate external sync state, or preview the blast radius before `forget()` operations.
+
+## Direct Graph Write
+
+Write structured graph data directly without LLM extraction — useful for programmatic fact ingestion, parsers, and deterministic pipelines:
+
+```typescript
+const { nodesWritten, edgesWritten, superseded } = await wikiMemory.upsertGraph('entity-123', {
+  sourceRef: 'codebase_main.ts',
+  sourceHash: sha256(sourceCode),
+  nodes: [
+    { id: 'fn_processData', type: 'function', title: 'processData', body: 'Processes user data' },
+    { id: 'class_UserService', type: 'class', title: 'UserService', body: 'User management service' },
+  ],
+  edges: [
+    { type: 'calls', sourceId: 'fn_processData', targetId: 'class_UserService' },
+  ],
+}, adapter); // SQLiteAdapter from your platform driver — writes join the caller's transaction
+```
+
+`upsertGraph` is "the tail of `ingestDocument` with the middle (LLM extraction) step removed" — it accepts caller-supplied nodes (`{ id, type, title, body? }`) and edges (`{ type, sourceId, targetId, id? }`) and writes them under the same `(sourceRef, sourceHash)` semantics. If a *different* live `sourceRef` already holds the same `sourceHash`, it throws `WikiSourceRefHashCollision`; re-writing the identical `(sourceRef, sourceHash)` is a no-op returning zero counts. The adapter parameter is required so writes participate in the caller's transaction.
+
+## Duplicate Hash Detection
+
+Control behavior when a different live `sourceRef` already holds the same `sourceHash`. The option is passed as a third argument to `ingestDocument`, not inside the params object:
+
+```typescript
+await wikiMemory.ingestDocument(
+  'entity-123',
+  {
+    sourceRef: 'doc.md',
+    sourceHash: sha256(content),
+    documentChunk: content,
+  },
+  { onDuplicateHash: 'ingest' }  // 'ingest' (default) | 'skip' | 'throw'
+);
+```
+
+- `'ingest'` (default): No duplicate pre-check; extraction proceeds as before this option existed. If a different live `sourceRef` is found holding the same hash at commit time (a concurrent-writer race caught by the source-ref unique index), the call still throws `WikiDuplicateHashError`.
+- `'skip'`: Pre-check before any LLM call; if a different live `sourceRef` already holds the hash, return a zero-chunk result without writing.
+- `'throw'`: Pre-check before any LLM call; throw `WikiDuplicateHashError` (carries the canonical `sourceRef`).
+- The guard only considers **live** references — soft-deleted refs do not trigger it in any mode.
+
+## Batch Change Detection
+
+Check multiple documents for changes in one call:
+
+```typescript
+const batch = [
+  { sourceRef: 'doc1.md', sourceHash: sha256(content1) },
+  { sourceRef: 'doc2.md', sourceHash: sha256(content2) },
+  { sourceRef: 'doc3.md', sourceHash: sha256(content3) },
+];
+const changes = await wikiMemory.hasChanged('entity-123', batch);
+// changes: Array<{ sourceRef: string; changed: boolean; duplicateOf?: string }>
+// duplicateOf — when present, the canonical stored different sourceRef holding
+// the same hash (DB-normalized spelling; sourceRef echoes the raw caller value).
+// Per-document change detection; internally batched across queries
+```
+
+## Dry-Run Deletion
+
+Preview deletion impact without writing:
+
+```typescript
+const preview = await wikiMemory.forget('entity-123', { sourceRef: 'doc.md' }, { dryRun: true });
+// preview: { deleted: { entries: number; tasks: number } }
+// No database writes performed; safe for blast-radius validation
+```
+
 ## Chunking Utilities
 
 `ingestDocument()` splits a document into chunks before extraction. That same chunking is exported as a pure function, so a consumer can reproduce ingest-time chunk boundaries exactly — useful for recovering the passage a fact was extracted from, by re-chunking the source and ranking chunks against the fact's stored embedding.