@equationalapplications/core-llm-wiki 4.15.3 → 4.16.0

package/README.md

@@ -21,6 +21,7 @@ Platform-agnostic TypeScript engine for hybrid LLM memory. Features episodic fac
 - **Full-featured memory** — Facts, tasks, events, maintenance jobs (librarian, heal, reembed, prune)
 - **Type-safe** — Built with TypeScript, full type exports
 - **Interoperability:** Supports [Open Knowledge Format (OKF) v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf) import and export.
+- **Per-entity seeded ontology** — Optional Strict, Emergent, or Off modes govern LLM graph extraction; seed taxonomies per entity and persist typed facts with inline edges.
 
 ## Installation
 
@@ -445,6 +446,101 @@ Notes:
 - A throwing callback is caught (logged via `console.error`) and does not block other subscribers or the underlying job.
 - Subscriptions are scoped to a single `entityId`. There is no wildcard or "all entities" form.
 
+## Per-Entity Seeded Ontology
+
+Control how librarian and ingest passes classify facts and extract graph relationships. The system defaults to **`off`** so existing deployments behave unchanged.
+
+### The Three Modes
+
+| Mode | Behavior |
+|------|----------|
+| **`off`** (default) | No ontology guidance. LLM output and persistence match pre-ontology behavior: `okf_type` stays `null` on LLM-created facts; maintenance passes do not create edges. OKF import still populates `okf_type` and edges independently. |
+| **`strict`** | The LLM must use only `node_types` and `edge_types` from the entity manifest. Invalid `okf_type` falls back to an untyped fact with no edges; invalid individual edges are dropped while a valid `okf_type` and matching edges are kept. |
+| **`emergent`** | Same validation as Strict, plus the LLM may return `ontology_updates` with new node/edge types. Updates are append-only (deduped by `type` string) and take effect before facts from the same response are validated. |
+
+Mode resolution per entity: persisted DB row `mode` (when present) → `seedManifests[entityId].mode` (when no row but a seed exists) → `WikiConfig.ontology.mode` → `'off'`.
+
+### WikiConfig
+
+Set a global default mode and bootstrap manifests for known entities at construction time:
+
+```typescript
+const wikiMemory = new WikiMemory(db, {
+  llmProvider,
+  config: {
+    ontology: {
+      mode: 'strict', // global default when an entity has no per-entity override
+      seedManifests: {
+        'team-alpha': {
+          mode: 'emergent', // optional per-entity override
+          manifest: {
+            node_types: [
+              { type: 'person', description: 'An individual or user.' },
+              { type: 'project', description: 'An ongoing initiative.' },
+            ],
+            edge_types: [
+              {
+                type: 'contributes_to',
+                source_type: 'person',
+                target_type: 'project',
+                description: 'Person working on a project.',
+              },
+            ],
+          },
+        },
+      },
+    },
+  },
+});
+```
+
+`seedManifests` entries are written to SQLite on first access when no row exists for that entity.
+
+### Public API
+
+Read or seed an entity's ontology at runtime:
+
+```typescript
+// Read effective mode + manifest (DB row, then seedManifests fallback)
+const ontology = await wikiMemory.getOntologyManifest('team-alpha');
+// { mode: 'emergent', manifest: { node_types: [...], edge_types: [...] } }
+// null when no row and no seed entry
+
+// Seed or replace manifest; optional per-entity mode override
+await wikiMemory.setOntologyManifest('team-alpha', {
+  node_types: [{ type: 'person', description: 'An individual.' }],
+  edge_types: [{
+    type: 'reports_to',
+    source_type: 'person',
+    target_type: 'person',
+    description: 'Reporting hierarchy.',
+  }],
+}, { mode: 'strict' });
+```
+
+### Fact Shape Extensions
+
+In **Strict** and **Emergent** modes, librarian and ingest JSON may include typed facts with inline edges:
+
+```json
+{
+  "facts": [{
+    "title": "Jane reports to Bob",
+    "body": "Jane reports to Bob Smith.",
+    "tags": [],
+    "confidence": "certain",
+    "okf_type": "person",
+    "edges": [{ "edge_type": "reports_to", "target_title": "Bob Smith" }]
+  }]
+}
+```
+
+- `okf_type` maps to a `node_types[].type` entry (case-insensitive lookup; canonical manifest casing is persisted).
+- `edges` are resolved by `target_title` within the same maintenance transaction and persisted via `EdgeRepository`.
+- Invalid `okf_type` falls back to `null` with no edges for that fact. Invalid individual edges are dropped; valid `okf_type` and matching edges are still persisted.
+
+See the design spec: [`docs/superpowers/specs/2026-06-23-per-entity-seeded-ontology-design.md`](https://github.com/equationalapplications/expo-llm-wiki/blob/main/docs/superpowers/specs/2026-06-23-per-entity-seeded-ontology-design.md).
+
 ## OKF Import/Export
 
 The core package integrates with `@equationalapplications/core-okf` to seamlessly adapt wiki data dumps to and from Open Knowledge Format (OKF) v0.1 bundles.

package/dist/{chunk-J4GBC6CP.mjs → chunk-2BGLPRT3.mjs}

@@ -606,30 +606,54 @@ var PromptService = class {
       return typeof value === "string" ? value : JSON.stringify(value, null, 2);
     });
   }
-  buildIngestPrompt(documentChunk, runtimeOverride) {
+  hasOntologyPlaceholders(template) {
+    return /\{\{\s*ontology(?:Manifest|ModeInstructions)\s*\}\}/.test(template);
+  }
+  buildSystemPrompt(template, variables, ontologyContext) {
+    const shouldHydrate = Object.keys(variables).some(
+      (key) => new RegExp(`\\{\\{\\s*${key}\\s*\\}\\}`).test(template)
+    ) || ontologyContext != null && this.hasOntologyPlaceholders(template);
+    const hydrated = shouldHydrate ? this.hydrate(template, { ...variables, ...ontologyContext ?? {} }) : template;
+    return this.hasOntologyPlaceholders(template) ? ontologyContext != null ? hydrated : hydrated.replace(/\{\{\s*ontology(?:Manifest|ModeInstructions)\s*\}\}/g, "") : this.appendOntology(hydrated, ontologyContext);
+  }
+  appendOntology(systemPrompt, ctx) {
+    if (!ctx) return systemPrompt;
+    return `${systemPrompt}
+
+${ctx.ontologyModeInstructions}`;
+  }
+  buildIngestPrompt(documentChunk, runtimeOverride, ontologyContext) {
     const template = runtimeOverride ?? this.globalOverrides?.ingestSystemPrompt ?? INGEST_SYSTEM_PROMPT;
-    if (/\{\{\s*documentChunk\s*\}\}/.test(template)) {
+    const hasDocumentChunk = /\{\{\s*documentChunk\s*\}\}/.test(template);
+    if (hasDocumentChunk || this.hasOntologyPlaceholders(template)) {
       return {
-        systemPrompt: this.hydrate(template, { documentChunk }),
-        userPrompt: "Please extract the facts."
+        systemPrompt: this.buildSystemPrompt(template, { documentChunk }, ontologyContext),
+        userPrompt: hasDocumentChunk ? "Please extract the facts." : `Document Chunk:
+${documentChunk}`
       };
     }
     return {
-      systemPrompt: template,
+      systemPrompt: this.appendOntology(template, ontologyContext),
       userPrompt: `Document Chunk:
 ${documentChunk}`
     };
   }
-  buildLibrarianPrompt(events, currentFacts, runtimeOverride) {
+  buildLibrarianPrompt(events, currentFacts, runtimeOverride, ontologyContext) {
     const template = runtimeOverride ?? this.globalOverrides?.librarianSystemPrompt ?? LIBRARIAN_SYSTEM_PROMPT;
-    if (/\{\{\s*events\s*\}\}/.test(template) || /\{\{\s*currentFacts\s*\}\}/.test(template)) {
+    const hasEvents = /\{\{\s*events\s*\}\}/.test(template);
+    const hasCurrentFacts = /\{\{\s*currentFacts\s*\}\}/.test(template);
+    if (hasEvents || hasCurrentFacts || this.hasOntologyPlaceholders(template)) {
       return {
-        systemPrompt: this.hydrate(template, { events, currentFacts }),
-        userPrompt: "Please synthesize the context."
+        systemPrompt: this.buildSystemPrompt(template, { events, currentFacts }, ontologyContext),
+        userPrompt: hasEvents || hasCurrentFacts ? "Please synthesize the context." : `Events:
+${JSON.stringify(events, null, 2)}
+
+Current Facts:
+${JSON.stringify(currentFacts, null, 2)}`
       };
     }
     return {
-      systemPrompt: template,
+      systemPrompt: this.appendOntology(template, ontologyContext),
       userPrompt: `Events:
 ${JSON.stringify(events, null, 2)}
 
@@ -875,6 +899,91 @@ function jaccardScore(a, b) {
   return intersection.size / union.size;
 }
 
+// src/utils/ontology.ts
+function emptyManifest() {
+  return { node_types: [], edge_types: [] };
+}
+function normalizeTitleKey(title) {
+  return title.trim().toLowerCase().replace(/\s+/g, " ");
+}
+function resolveNodeType(raw, manifest) {
+  const slug = raw.trim();
+  if (!slug) return null;
+  const hit = manifest.node_types.find((n) => n.type.toLowerCase() === slug.toLowerCase());
+  return hit?.type ?? null;
+}
+function resolveEdgeDefinition(rawEdgeType, manifest) {
+  const slug = rawEdgeType.trim();
+  if (!slug) return null;
+  return manifest.edge_types.find((e) => e.type.toLowerCase() === slug.toLowerCase()) ?? null;
+}
+function validateManifest(manifest) {
+  const nodeSlugs = /* @__PURE__ */ new Set();
+  for (const node of manifest.node_types ?? []) {
+    const type = node.type?.trim();
+    if (!type) throw new Error("Ontology node type slug must be non-empty");
+    const key = type.toLowerCase();
+    if (nodeSlugs.has(key)) throw new Error(`Duplicate node type: ${type}`);
+    nodeSlugs.add(key);
+  }
+  const edgeSlugs = /* @__PURE__ */ new Set();
+  for (const edge of manifest.edge_types ?? []) {
+    const edgeType = edge.type?.trim();
+    const sourceType = edge.source_type?.trim();
+    const targetType = edge.target_type?.trim();
+    if (!edgeType) throw new Error("Ontology edge type slug must be non-empty");
+    const edgeKey = edgeType.toLowerCase();
+    if (edgeSlugs.has(edgeKey)) throw new Error(`Duplicate edge type: ${edgeType}`);
+    edgeSlugs.add(edgeKey);
+    if (!sourceType || !targetType || !nodeSlugs.has(sourceType.toLowerCase()) || !nodeSlugs.has(targetType.toLowerCase())) {
+      throw new Error(`Edge type ${edgeType} references unknown node type`);
+    }
+  }
+}
+function mergeOntologyUpdates(current, updates) {
+  const node_types = [...current.node_types];
+  const edge_types = [...current.edge_types];
+  const nodeSlugs = new Set(node_types.map((n) => n.type.trim().toLowerCase()));
+  const edgeSlugs = new Set(edge_types.map((e) => e.type.trim().toLowerCase()));
+  for (const node of updates.node_types ?? []) {
+    const type = node?.type?.trim();
+    if (!type) continue;
+    const key = type.toLowerCase();
+    if (nodeSlugs.has(key)) continue;
+    node_types.push({ type, description: String(node.description ?? "") });
+    nodeSlugs.add(key);
+  }
+  for (const edge of updates.edge_types ?? []) {
+    const edgeType = edge?.type?.trim();
+    const sourceType = edge?.source_type?.trim();
+    const targetType = edge?.target_type?.trim();
+    if (!edgeType || !sourceType || !targetType) continue;
+    const edgeKey = edgeType.toLowerCase();
+    if (edgeSlugs.has(edgeKey)) continue;
+    if (!nodeSlugs.has(sourceType.toLowerCase()) || !nodeSlugs.has(targetType.toLowerCase())) continue;
+    edge_types.push({
+      type: edgeType,
+      source_type: sourceType,
+      target_type: targetType,
+      description: String(edge.description ?? "")
+    });
+    edgeSlugs.add(edgeKey);
+  }
+  return { node_types, edge_types };
+}
+function validateInlineEdges(sourceType, _targetType, edges, manifest) {
+  if (!Array.isArray(edges)) return [];
+  const valid = [];
+  for (const edge of edges) {
+    if (typeof edge?.edge_type !== "string" || typeof edge?.target_title !== "string") continue;
+    const def = resolveEdgeDefinition(edge.edge_type, manifest);
+    if (!def) continue;
+    if (def.source_type.toLowerCase() !== sourceType.toLowerCase()) continue;
+    valid.push({ edge_type: def.type, target_title: edge.target_title });
+  }
+  return valid;
+}
+
 // src/utils/ids.ts
 function generateId(prefix = "") {
   if (typeof crypto !== "undefined" && typeof crypto.randomUUID === "function") {
@@ -892,7 +1001,7 @@ function generateId(prefix = "") {
 
 // src/services/IngestionService.ts
 var IngestionService = class {
-  constructor(db, prefix, options, entryRepo, searchService, jobManager, embeddingService, promptService) {
+  constructor(db, prefix, options, entryRepo, searchService, jobManager, embeddingService, promptService, ontologyService) {
     this.db = db;
     this.prefix = prefix;
     this.options = options;
@@ -900,6 +1009,7 @@ var IngestionService = class {
     this.searchService = searchService;
     this.jobManager = jobManager;
     this.embeddingService = embeddingService;
+    this.ontologyService = ontologyService;
     this.promptService = promptService ?? new PromptService(this.options.config?.prompts);
   }
   async ingestDocument(entityId, params) {
@@ -924,23 +1034,33 @@ var IngestionService = class {
       if (chunks.length === 0) return { truncated: false, chunks: 0 };
       const chunkResults = await withConcurrency(
         chunks.map((chunk) => async () => {
-          const { systemPrompt, userPrompt } = this.promptService.buildIngestPrompt(chunk, params.promptOverride);
+          const ontologyContext = await this.ontologyService?.buildPromptContext(entityId) ?? null;
+          const { systemPrompt, userPrompt } = this.promptService.buildIngestPrompt(
+            chunk,
+            params.promptOverride,
+            ontologyContext
+          );
           const responseText = await this.options.llmProvider.generateText({ systemPrompt, userPrompt });
           const result = parseJsonResponse(responseText);
-          return (Array.isArray(result.facts) ? result.facts : []).map(validateFact).filter((f) => f !== null);
+          return {
+            facts: (Array.isArray(result.facts) ? result.facts : []).map(validateFact).filter((f) => f !== null),
+            ontology_updates: result.ontology_updates
+          };
         }),
         chunkConcurrency
       );
       const seen = /* @__PURE__ */ new Set();
-      const allValidFacts = [];
-      for (const facts of chunkResults) {
-        for (const fact of facts) {
-          const normalized = fact.title.trim().toLowerCase().replace(/\s+/g, " ");
-          if (!seen.has(normalized)) {
-            seen.add(normalized);
-            allValidFacts.push(fact);
+      const orderedChunkFacts = [];
+      for (const chunkResult of chunkResults) {
+        const dedupedFacts = [];
+        for (const fact of chunkResult.facts) {
+          const normalizedTitle = normalizeTitleKey(fact.title);
+          if (!seen.has(normalizedTitle)) {
+            seen.add(normalizedTitle);
+            dedupedFacts.push(fact);
           }
         }
+        orderedChunkFacts.push({ facts: dedupedFacts, ontology_updates: chunkResult.ontology_updates });
       }
       const now = Date.now();
       const insertedFacts = [];
@@ -948,26 +1068,63 @@ var IngestionService = class {
       await this.db.withTransactionAsync(async (tx) => {
         deletedSourceFactIds.push(...await this.entryRepo.findIdsBySource(entityId, sourceRef, null, tx, false));
         await this.entryRepo.softDeleteBySource(entityId, tx, sourceRef, null);
-        for (const fact of allValidFacts) {
-          const id = generateId("fact_");
-          const wikiFact = {
-            id,
-            entity_id: entityId,
-            title: fact.title,
-            body: fact.body,
-            tags: fact.tags,
-            confidence: fact.confidence,
-            source_type: "immutable_document",
-            source_hash: sourceHash,
-            source_ref: sourceRef,
-            created_at: now,
-            updated_at: now,
-            last_accessed_at: null,
-            access_count: 0,
-            deleted_at: null
-          };
-          await this.entryRepo.upsert(wikiFact, tx);
-          insertedFacts.push({ id, entity_id: entityId, title: fact.title, body: fact.body, tags: JSON.stringify(fact.tags) });
+        const titleIndex = /* @__PURE__ */ new Map();
+        const pendingEdges = [];
+        const existingFacts = await this.entryRepo.findRecentByEntityId(entityId, 500, tx);
+        for (const existing of existingFacts) {
+          titleIndex.set(normalizeTitleKey(existing.title), {
+            id: existing.id,
+            okf_type: existing.okf_type ?? null
+          });
+        }
+        let ontologyState = await this.ontologyService?.getEffectiveState(entityId, tx) ?? { mode: "off", manifest: { node_types: [], edge_types: [] } };
+        let { mode, manifest } = ontologyState;
+        for (const { facts, ontology_updates } of orderedChunkFacts) {
+          if (mode === "emergent" && ontology_updates && this.ontologyService) {
+            manifest = await this.ontologyService.mergeEmergentUpdates(entityId, ontology_updates, tx);
+            ontologyState = await this.ontologyService.getEffectiveState(entityId, tx);
+            mode = ontologyState.mode;
+          }
+          for (const fact of facts) {
+            const ontologyFact = fact;
+            const normalized = this.ontologyService?.validateAndNormalizeFact(ontologyFact, manifest) ?? { okf_type: null, edges: [] };
+            const id = generateId("fact_");
+            const wikiFact = {
+              id,
+              entity_id: entityId,
+              title: fact.title,
+              body: fact.body,
+              tags: fact.tags,
+              confidence: fact.confidence,
+              source_type: "immutable_document",
+              source_hash: sourceHash,
+              source_ref: sourceRef,
+              created_at: now,
+              updated_at: now,
+              last_accessed_at: null,
+              access_count: 0,
+              deleted_at: null,
+              okf_type: normalized.okf_type
+            };
+            await this.entryRepo.upsert(wikiFact, tx);
+            insertedFacts.push({ id, entity_id: entityId, title: fact.title, body: fact.body, tags: JSON.stringify(fact.tags) });
+            titleIndex.set(normalizeTitleKey(fact.title), { id, okf_type: normalized.okf_type });
+            if (normalized.edges.length > 0) {
+              pendingEdges.push({ sourceId: id, sourceType: normalized.okf_type, edges: normalized.edges });
+            }
+          }
+        }
+        for (const item of pendingEdges) {
+          await this.ontologyService?.resolveAndPersistEdges(
+            entityId,
+            item.sourceId,
+            item.sourceType,
+            item.edges ?? [],
+            manifest,
+            titleIndex,
+            tx,
+            now
+          );
         }
       });
       await this.searchService.sync(entityId);
@@ -994,7 +1151,7 @@ var IngestionService = class {
 var FUZZY_THRESHOLD = 0.5;
 var MIN_TOKENS_TO_QUALIFY = 3;
 var MaintenanceService = class {
-  constructor(db, prefix, options, entryRepo, taskRepo, eventRepo, metadataRepo, searchService, jobManager, embeddingService, promptService) {
+  constructor(db, prefix, options, entryRepo, taskRepo, eventRepo, metadataRepo, searchService, jobManager, embeddingService, promptService, ontologyService) {
     this.db = db;
     this.prefix = prefix;
     this.options = options;
@@ -1005,6 +1162,7 @@ var MaintenanceService = class {
     this.searchService = searchService;
     this.jobManager = jobManager;
     this.embeddingService = embeddingService;
+    this.ontologyService = ontologyService;
     this.promptService = promptService ?? new PromptService(this.options.config?.prompts);
   }
   async runPrune(entityId, options) {
@@ -1227,21 +1385,36 @@ var MaintenanceService = class {
         tags: typeof rest.tags === "string" ? JSON.parse(rest.tags) : rest.tags
       };
     });
+    const ontologyContext = await this.ontologyService?.buildPromptContext(entityId) ?? null;
     const { systemPrompt, userPrompt } = this.promptService.buildLibrarianPrompt(
       events.reverse(),
       currentFacts,
-      promptOverride
+      promptOverride,
+      ontologyContext
     );
     const responseText = await this.options.llmProvider.generateText({ systemPrompt, userPrompt });
     const result = parseJsonResponse(responseText);
     const facts = Array.isArray(result.facts) ? result.facts : [];
     const tasks = Array.isArray(result.tasks) ? result.tasks : [];
+    const ontologyUpdates = result.ontology_updates;
     const validFacts = facts.map(validateFact).filter((f) => f !== null);
     const validTasks = tasks.map(validateTask).filter((t) => t !== null);
     const now = Date.now();
     const insertedFacts = [];
     await this.db.withTransactionAsync(async (tx) => {
+      let { mode, manifest } = await this.ontologyService?.getEffectiveState(entityId, tx) ?? { mode: "off", manifest: { node_types: [], edge_types: [] } };
+      if (mode === "emergent" && ontologyUpdates && this.ontologyService) {
+        manifest = await this.ontologyService.mergeEmergentUpdates(entityId, ontologyUpdates, tx);
+      }
+      const titleIndex = /* @__PURE__ */ new Map();
+      for (const existing of currentFactsRows) {
+        titleIndex.set(normalizeTitleKey(existing.title), {
+          id: existing.id,
+          okf_type: existing.okf_type ?? null
+        });
+      }
       const factsForDedupe = await this.entryRepo.findRecentByEntityId(entityId, 100, tx);
+      const pendingEdges = [];
       for (const fact of validFacts) {
         const newTokens = titleTokens(fact.title);
         let skip = false;
@@ -1258,6 +1431,8 @@ var MaintenanceService = class {
           }
         }
         if (skip) continue;
+        const ontologyFact = fact;
+        const normalized = this.ontologyService?.validateAndNormalizeFact(ontologyFact, manifest) ?? { okf_type: null, edges: [] };
         const id = generateId("fact_");
         const factObj = {
           id,
@@ -1273,11 +1448,28 @@ var MaintenanceService = class {
           updated_at: now,
           last_accessed_at: null,
           access_count: 0,
-          deleted_at: null
+          deleted_at: null,
+          okf_type: normalized.okf_type
         };
         await this.entryRepo.upsert(factObj, tx);
         insertedFacts.push({ id, entity_id: entityId, title: fact.title, body: fact.body, tags: JSON.stringify(fact.tags) });
         factsForDedupe.push(factObj);
+        titleIndex.set(normalizeTitleKey(fact.title), { id, okf_type: normalized.okf_type });
+        if (normalized.edges.length > 0) {
+          pendingEdges.push({ sourceId: id, sourceType: normalized.okf_type, edges: normalized.edges });
+        }
+      }
+      for (const item of pendingEdges) {
+        await this.ontologyService?.resolveAndPersistEdges(
+          entityId,
+          item.sourceId,
+          item.sourceType,
+          item.edges ?? [],
+          manifest,
+          titleIndex,
+          tx,
+          now
+        );
       }
       for (const task of validTasks) {
         const id = generateId("task_");
@@ -2614,6 +2806,6 @@ var WriteService = class {
   }
 };
 
-export { EmbeddingService, HOOK_TIMEOUT_MARKER, ImportExportService, IngestionService, JobManager, MaintenanceService, PromptService, PrunePartialFailureError, RetrievalService, SearchService, WikiBusyError, WriteService, __privateAdd, __privateGet, __privateSet, generateId, normalizeSourceHash, normalizeSourceRef, parseEmbedding };
-//# sourceMappingURL=chunk-J4GBC6CP.mjs.map
-//# sourceMappingURL=chunk-J4GBC6CP.mjs.map
+export { EmbeddingService, HOOK_TIMEOUT_MARKER, ImportExportService, IngestionService, JobManager, MaintenanceService, PromptService, PrunePartialFailureError, RetrievalService, SearchService, WikiBusyError, WriteService, __privateAdd, __privateGet, __privateSet, emptyManifest, generateId, mergeOntologyUpdates, normalizeSourceHash, normalizeSourceRef, normalizeTitleKey, parseEmbedding, resolveEdgeDefinition, resolveNodeType, validateInlineEdges, validateManifest };
+//# sourceMappingURL=chunk-2BGLPRT3.mjs.map
+//# sourceMappingURL=chunk-2BGLPRT3.mjs.map