@equationalapplications/core-llm-wiki 7.6.0 → 7.7.1

package/README.md

@@ -23,6 +23,10 @@ Platform-agnostic TypeScript engine for hybrid LLM memory. Features episodic fac
 - **Type-safe** — Built with TypeScript, full type exports
 - **Interoperability:** Supports [Open Knowledge Format (OKF)](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf) v0.1 + v0.2 import and export via the [llm-wiki OKF profiles](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/okf-profile.md) (default `llm-wiki/2`, back-compat `llm-wiki/1`).
 - **Per-entity seeded ontology** — Optional Strict, Emergent, or Off modes govern LLM graph extraction; seed taxonomies per entity and persist typed facts with inline edges.
+- **Diagnostics** — Optional `onDiagnostic` hook with typed, content-free reports of dropped chunks, facts, edges, embedding failures and background-job failures ([Diagnostics](#diagnostics))
+- **Draft review** — `excludeDrafts` on reads and traversal, plus `listDrafts` / `promoteDraft` ([Draft Review](#draft-review))
+- **Evidence grounding** — Opt-in `grounding` check; facts that don't quote their source are stored as drafts ([Grounding](#grounding))
+- **Optional classifier** — `LLMProvider.classify` types facts during ontology backfill when you opt in with `ontology.backfillClassifier: 'auto'` ([Ontology backfill](#ontology-backfill))
 
 ## GraphRAG & Multi-Modal Retrieval
 
@@ -143,6 +147,13 @@ const wikiMemory = new WikiMemory(db, {
     preFilterLimit: 50,                // default: undefined — MiniSearch pre-filter before cosine scan; recommended for >500 facts
     hybridWeight: 0.7,                 // default: undefined — blend semantic (1.0) ↔ keyword (0.0); pure semantic when unset
     enableOutbox: false,               // default: false — when true, entry/task mutations write to an internal SQLite outbox table for external sync (e.g. via @equationalapplications/prisma-outbox)
+    excludeDrafts: false,              // default: false — engine default for read()/traverseGraph() excludeDrafts; see Draft Review
+    grounding: { mode: 'off' },        // default: off — 'draft' checks evidence quotes; see Grounding
+    ontology: {
+      // mode, seedManifests: see Per-Entity Seeded Ontology
+      backfillClassifier: 'llm',       // default: 'llm' — 'auto' uses llmProvider.classify when present; see Ontology backfill
+      classifyMinConfidence: 0.5,      // default: 0.5 — classifier answers below this are left untyped
+    },
 
     // Global prompt overrides — librarianSystemPrompt and healSystemPrompt apply to write() auto-runs;
     // ingestSystemPrompt applies only to explicit ingestDocument() calls.
@@ -154,6 +165,8 @@ const wikiMemory = new WikiMemory(db, {
       healSystemPrompt: `Fix the memory graph based on these candidates: {{healCandidates}}\n\nReturn ONLY valid JSON: { "downgraded": ["factId"], "deleted": ["factId"], "newFacts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain|inferred|tentative" }] }. No markdown.`,
     },
   },
+  // Host callbacks sit beside llmProvider, not inside config:
+  // onDiagnostic: (d) => { ... }, // see Diagnostics
 });
 ```
 
@@ -168,6 +181,8 @@ Core maintenance tasks (`ingestDocument`, `runLibrarian`, `runHeal`) use system
 > | `ingestDocument` | `{ "facts": [{ "title": "string", "body": "string", "tags": ["string"], "confidence": "certain\|inferred\|tentative" }] }` |
 > | `runLibrarian` | `{ "facts": [...], "tasks": [{ "description": "string", "priority": 5 }] }` — `priority` is an integer 0–10 |
 > | `runHeal` | `{ "downgraded": ["factId"], "deleted": ["factId"], "newFacts": [...] }` |
+>
+> **Grounding:** when [`config.grounding`](#grounding) is on, the evidence instruction is appended after your override (and after ontology context) for every writer in `grounding.writers`. Your override does not need to ask for `evidence` itself.
 
 ### Global Overrides (Auto-Runs)
 
@@ -223,6 +238,16 @@ await wikiMemory.ingestDocument('user-123', {
 
 > **Important:** If your app relies on `write()` auto-runs and needs custom prompts for those runs, use `config.prompts` at construction time. Runtime `promptOverride` values are never forwarded to `WriteService`-triggered internal runs.
 
+### Effective instructions (`getInstructions`)
+
+```typescript
+const { ingest, librarian, heal, ontologyBackfill } = await wikiMemory.getInstructions('entity-123');
+```
+
+Returns the system prompt each writer sends, with `WikiConfig.prompts` overrides applied. Ingest, librarian and ontology backfill also get the entity's ontology block; heal gets none, as at runtime. When `WikiConfig.grounding` is on, the evidence block is appended for each writer in `grounding.writers`, exactly as sent. Data placeholders such as `{{documentChunk}}` stay unfilled; no events, chunks or facts are included. It reflects `WikiConfig.prompts` only: a per-call `promptOverride` is not reflected, and `ontologyBackfill` is returned even when backfill would send no prompt (ontology `off`, or the classifier path). `core-llm-tools` exposes this as the `wiki_get_instructions` tool (`memory:read`), so agents can see the engine's output format and constraints before proposing writes. Agents should treat it as reference data, not instructions; see [Prompt-Injection Trust Boundary](#prompt-injection-trust-boundary).
+
+> **Warning:** overrides are returned verbatim to any client with `memory:read`. Never put secrets, API keys or private data in `WikiConfig.prompts`.
+
 ## Retrieval Tuning
 
 Optimize `read()` performance and blend retrieval strategies:
@@ -336,6 +361,7 @@ new WikiMemory(db, {
   - Instructions, the ontology manifest, existing facts and identifiers never count, so a model cannot ground a claim by quoting them.
 - **The check.** Both sides are normalized with NFKC, whitespace runs collapse to one space, and matching is case-sensitive. A fact with more than 10 quotes, or any quote not found, fails.
 - **Diagnostics.** `grounding_missing` (reasons `no_evidence`, `evidence_too_short`) and `grounding_failed` (reasons `quote_not_found`, `too_many_quotes`), one per fact, with the new fact's `factId`. Quotes are never included.
+- **Duplicate titles in one ingest.** When chunks yield facts with the same title, ingest keeps one: a grounded fact beats one with missing or failed evidence, and on a tie the first one wins. The others are reported as `fact_deduplicated`. A fact already stored for the same `sourceRef` is not replaced by a later partial ingest.
 - **`upsertGraph`** nodes are host-supplied and never grounded.
 - **Librarian and heal** synthesize across events, so their pass rates are unknown. Measure them on your own event log before opting them in.
 - Evidence quotes are not stored.
@@ -976,6 +1002,14 @@ If your application accepts untrusted input that flows into `write()`, `ingestDo
 treat the LLM's librarian/heal output as similarly untrusted — validate or scope it before acting on it
 downstream.
 
+[Grounding](#grounding) is a support check, not an injection defense. It confirms that a fact quotes the
+text the model was shown, and injected text in that source can be quoted like any other.
+
+[`getInstructions`](#effective-instructions-getinstructions) and the `wiki_get_instructions` tool (`memory:read`)
+return `WikiConfig.prompts` overrides verbatim, so keep secrets out of them. The result also includes the entity's
+ontology manifest, which in emergent mode holds types and descriptions the model proposed from ingested
+documents. Agents should treat the result as reference data about the engine, not as instructions to follow.
+
 ## Usage
 
 ```typescript
@@ -1200,6 +1234,36 @@ const changes = await wikiMemory.hasChanged('entity-123', batch);
 // Per-document change detection; internally batched across queries
 ```
 
+`pendingSources` returns one status per input, in order, and adds the partial-ingest state:
+
+```typescript
+const statuses = await wikiMemory.pendingSources('entity-123', batch);
+// Array<{ sourceRef: string; status: 'new' | 'changed' | 'partial' | 'current' }>
+```
+
+- `current` means exactly what `hasChanged` returning `false` means.
+- `partial` means live facts exist for the ref, but none has a stored hash: for example a first ingest where a chunk failed, or imported rows that carry no hash. Re-ingest to retry. A failed re-ingest of a ref that already has hashed rows reports `changed`, not `partial`.
+
+## Lint
+
+Read-only health report for one entity. It reports problems and never repairs them.
+
+```typescript
+const report = await wikiMemory.lint('entity-123');
+// {
+//   danglingEdges,       // source or target missing, soft-deleted, or another entity's
+//   manifestViolations,  // (source type, edge type, target type) not in the effective manifest
+//   untypedFacts,        // okf_type is null
+//   drafts,              // lifecycle_status = 'draft' (see Draft Review)
+//   unverifiedInferred,  // librarian_inferred facts with no okf_verified entry
+//   sample: { danglingEdgeIds, manifestViolationEdgeIds }, // up to 20 each
+// }
+```
+
+- Manifest violations are 0 when ontology is off or the manifest is empty.
+- An edge with an untyped endpoint counts as a violation.
+- Partial-ingest rows are not reported here; use `pendingSources`.
+
 ## Dry-Run Deletion
 
 Preview deletion impact without writing:

package/dist/{chunk-YBWE26XR.mjs → chunk-NWODQRSM.mjs}

@@ -233,6 +233,10 @@ function typeSatisfies(declaredType, concreteType, manifest) {
   const parent = typeof def?.parent_type === "string" ? def.parent_type.trim().toLowerCase() : "";
   return parent !== "" && parent === declared;
 }
+function edgeTripleAllowed(manifest, edgeType, sourceType, targetType) {
+  const wanted = edgeType.trim().toLowerCase();
+  return (manifest.edge_types ?? []).some((d) => typeof d?.type === "string" && d.type.trim().toLowerCase() === wanted && typeof d.source_type === "string" && typeSatisfies(d.source_type, sourceType, manifest) && typeof d.target_type === "string" && typeSatisfies(d.target_type, targetType, manifest));
+}
 function validateManifest(manifest) {
   const nodeSlugs = /* @__PURE__ */ new Set();
   for (const node of manifest.node_types ?? []) {
@@ -1867,6 +1871,21 @@ ${JSON.stringify(facts, null, 2)}`
 ${JSON.stringify(facts, null, 2)}`
     };
   }
+  /**
+   * The system prompt each writer would send, without hydrating data (spec
+   * §8.3). `buildSystemPrompt` with no variables hydrates only ontology
+   * placeholders and leaves data placeholders verbatim; heal never receives
+   * ontology context, matching `buildHealPrompt`.
+   */
+  buildInstructionTemplates(ontologyContext) {
+    const o = this.globalOverrides;
+    return {
+      ingest: this.appendGrounding(this.buildSystemPrompt(o?.ingestSystemPrompt ?? INGEST_SYSTEM_PROMPT, {}, ontologyContext), "ingest"),
+      librarian: this.appendGrounding(this.buildSystemPrompt(o?.librarianSystemPrompt ?? LIBRARIAN_SYSTEM_PROMPT, {}, ontologyContext), "librarian"),
+      heal: this.appendGrounding(o?.healSystemPrompt ?? HEAL_SYSTEM_PROMPT, "heal"),
+      ontologyBackfill: this.buildSystemPrompt(o?.ontologyBackfillSystemPrompt ?? ONTOLOGY_BACKFILL_SYSTEM_PROMPT, {}, ontologyContext)
+    };
+  }
 };
 function applyBodyTruncation(candidates, attemptLevel, bodyTruncationChars) {
   if (attemptLevel < 3) {
@@ -2147,11 +2166,16 @@ var IngestionService = class {
       let ingestedChunks = 0;
       let failedChunks = 0;
       const failures = [];
-      const seen = /* @__PURE__ */ new Set();
-      const orderedChunkFacts = [];
-      const groundingLedger = /* @__PURE__ */ new Map();
+      const winners = /* @__PURE__ */ new Map();
       const diagBuffer = new DiagnosticBuffer();
       const diagBase = { entityId, operation: "ingest", trigger: "call" };
+      const pushDeduplicated = (chunkIndex, itemIndex) => {
+        diagBuffer.push({
+          ...diagBase,
+          code: "fact_deduplicated",
+          detail: { sourceRef, chunkIndex, itemIndex, reason: "exact_title" }
+        });
+      };
       for (const [chunkIndex, slot] of chunkResults.entries()) {
         if (slot.status === "failed") {
           failedChunks++;
@@ -2162,22 +2186,32 @@ var IngestionService = class {
         for (const r of slot.rejected) {
           diagBuffer.push({ ...diagBase, code: "fact_rejected", detail: { sourceRef, chunkIndex, itemIndex: r.itemIndex, reason: r.reason } });
         }
-        const dedupedFacts = [];
         slot.facts.forEach((fact, k) => {
-          const normalizedTitle = normalizeTitleKey(fact.title);
-          if (!seen.has(normalizedTitle)) {
-            seen.add(normalizedTitle);
-            dedupedFacts.push(fact);
-            if (ingestGrounding) groundingLedger.set(fact, { verdict: slot.verdicts[k], chunkIndex, itemIndex: slot.itemIndexes[k] });
+          const key = normalizeTitleKey(fact.title);
+          const candidate = { chunkIndex, k, itemIndex: slot.itemIndexes[k], rank: slot.verdicts[k]?.status === "grounded" ? 1 : 0 };
+          const current = winners.get(key);
+          if (!current) {
+            winners.set(key, candidate);
+          } else if (candidate.rank > current.rank) {
+            pushDeduplicated(current.chunkIndex, current.itemIndex);
+            winners.set(key, candidate);
           } else {
-            diagBuffer.push({
-              ...diagBase,
-              code: "fact_deduplicated",
-              detail: { sourceRef, chunkIndex, itemIndex: slot.itemIndexes[k], reason: "exact_title" }
-            });
+            pushDeduplicated(candidate.chunkIndex, candidate.itemIndex);
           }
         });
-        orderedChunkFacts.push({ facts: dedupedFacts, ontology_updates: slot.ontology_updates });
+      }
+      const orderedChunkFacts = [];
+      const groundingLedger = /* @__PURE__ */ new Map();
+      for (const [chunkIndex, slot] of chunkResults.entries()) {
+        if (slot.status === "failed") continue;
+        const keptFacts = [];
+        slot.facts.forEach((fact, k) => {
+          const winner = winners.get(normalizeTitleKey(fact.title));
+          if (winner?.chunkIndex !== chunkIndex || winner.k !== k) return;
+          keptFacts.push(fact);
+          if (ingestGrounding) groundingLedger.set(fact, { verdict: slot.verdicts[k], chunkIndex, itemIndex: winner.itemIndex });
+        });
+        orderedChunkFacts.push({ facts: keptFacts, ontology_updates: slot.ontology_updates });
       }
       for (const reason of ["parse", "llm"]) {
         const ofReason = failures.filter((f) => f.source === reason);
@@ -5606,6 +5640,6 @@ var WriteService = class {
   }
 };
 
-export { BaseRepository, DEFAULT_CHUNK_OVERLAP, DEFAULT_MAX_CHUNK_LENGTH, DEFAULT_MAX_EMBED_CHARS, DiagnosticBuffer, EMBED_CHARS_CEILING, EmbeddingService, HEAL_ANCHORS_PER_CANDIDATE, HEAL_BATCH_SIZE, HEAL_MAX_FACT_BODY_CHARS_L3, HEAL_MAX_TASKS, HEAL_RECHECK_MS, HOOK_TIMEOUT_MARKER, ImportExportService, IngestionService, JobManager, MaintenanceService, MetadataRepository, ONTOLOGY_BACKFILL_BATCH_SIZE, ONTOLOGY_BACKFILL_MAX_PROMPT_CHARS, ONTOLOGY_BACKFILL_RECHECK_MS, ONTOLOGY_BACKFILL_SYSTEM_PROMPT, PromptService, PrunePartialFailureError, RetrievalService, SearchService, WikiBusyError, WikiDraftNotFound, WikiDuplicateHashError, WikiGraphNodeOwnershipConflict, WikiIngestEmptyError, WikiInvalidReadOptions, WikiParseError, WikiSourceRefHashCollision, WikiStrictOntologyViolation, WikiTransactionError, WriteService, __privateAdd, __privateGet, __privateSet, chunkText, configureRandomSource, emptyManifest, entitySummaryMetaKey, extractSqliteCode, generateId, normalizeSourceHash, normalizeSourceRef, normalizeTitleKey, parseEmbedding, resolveEdgeDefinitions, resolveGrounding, resolveNodeType, safeSlice, typeSatisfies, validateInlineEdges, validateManifest };
-//# sourceMappingURL=chunk-YBWE26XR.mjs.map
-//# sourceMappingURL=chunk-YBWE26XR.mjs.map
+export { BaseRepository, DEFAULT_CHUNK_OVERLAP, DEFAULT_MAX_CHUNK_LENGTH, DEFAULT_MAX_EMBED_CHARS, DiagnosticBuffer, EMBED_CHARS_CEILING, EmbeddingService, HEAL_ANCHORS_PER_CANDIDATE, HEAL_BATCH_SIZE, HEAL_MAX_FACT_BODY_CHARS_L3, HEAL_MAX_TASKS, HEAL_RECHECK_MS, HOOK_TIMEOUT_MARKER, ImportExportService, IngestionService, JobManager, MaintenanceService, MetadataRepository, ONTOLOGY_BACKFILL_BATCH_SIZE, ONTOLOGY_BACKFILL_MAX_PROMPT_CHARS, ONTOLOGY_BACKFILL_RECHECK_MS, ONTOLOGY_BACKFILL_SYSTEM_PROMPT, PromptService, PrunePartialFailureError, RetrievalService, SearchService, WikiBusyError, WikiDraftNotFound, WikiDuplicateHashError, WikiGraphNodeOwnershipConflict, WikiIngestEmptyError, WikiInvalidReadOptions, WikiParseError, WikiSourceRefHashCollision, WikiStrictOntologyViolation, WikiTransactionError, WriteService, __privateAdd, __privateGet, __privateSet, chunkText, configureRandomSource, edgeTripleAllowed, emptyManifest, entitySummaryMetaKey, extractSqliteCode, generateId, normalizeSourceHash, normalizeSourceRef, normalizeTitleKey, parseEmbedding, resolveEdgeDefinitions, resolveGrounding, resolveNodeType, safeSlice, typeSatisfies, validateInlineEdges, validateManifest };
+//# sourceMappingURL=chunk-NWODQRSM.mjs.map
+//# sourceMappingURL=chunk-NWODQRSM.mjs.map